At the beginning of 2006, I decided to undertake a year-long case study of Google Analytics. Many people praise the service, but I've always reserved my judgement. My experience with third-party statistical services is that they under-report traffic. I haven't finished my case study, as I prefer to look at data over the span of a year, but the numbers I'm seeing from Google Analytics are disturbing. If I had to make a decision today, I'd yank all the Google Analytics code and say, "Not for me."
But it's only time to ask for insight on why the numbers may perform the way they are. I can rationalize some discrepancies between third-party statistics and what my own server logs show by allowing for people who disable Javascript in their browsers, assuming some traffic is driven by proxy servers that may not refetch/reactivate Javascript, assuming some traffic is actually robots (not identified in my robots filter), etc. But what I cannot rationalize is how Google Analytics thinks I've lost 50% of my traffic when my server logs indicate that traffic has held steady.
What is even more curious is that some pages continue to perform well according to both Google Analytics and the server logs, whereas other pages show a severe dropoff in traffic from Analytics but either no dropoff or significantly less in the server logs. For the month of January, one page about a popular television actress led the Google Analytics report with 6000 unique visitors and 8100 page views. According to my server log, that page received 10,000 views (disallowing for known robots).
For July, Google Analytics says the same page received only 700 visitors and 900 views -- but my server logs indicate over 8,000 views. I don't have the opportunity right now to look at the raw logs. Sometimes I find that my server software lumps data together that I don't want lumped together, but nothing like this kind of discrepancy has ever shown up.
Our white cheese dip page showed more than 9,000 views in January per server logs and more than 7,000 in July (NOTE: Summer dropoffs are typical for Xenite.Org, which has large appeal among students of all ages). Google Analytics shows about the same numbers for those months.
The discrepancies vary across the selection of pages. For Xenite's home page, we show over 12,000 views in January but Google shows 2200. In July, Google thinks we only received about 800 page views on the main page. Our server logs indicate 3,000. Maybe the robot filtering doesn't work the way I think it should on my server logs -- I have changed statistics programs enough through the years that I occasionally get a little confused about how to interpret the results. But these kinds of discrepancies just don't make sense.
What I find unusable in Google Analytics testimonials from other people is that all the gung ho recommendations fail to mention any sorts of comparisons. While my case study is intended to run through the end of 2006, the declining data patterns for Analytics make absolutely no sense whatsoever.
Another measure of traffic comes in the form of referrals from Google. In January, we recorded (as Big Daddy rolled out) about 28,000 referrals from Google.com/search (just one of many Google referring URLs). In July we recorded about 16,000 referrals from that same URL. We did lose some search placements in July for a while, which happens occasionally. And as I mentioned above, we normally see a decrease in traffic during the summer months (I've been tracking Xenite.Org's traffic for nine summers now -- this is a normal drop).
Our server logs indicate we had 97,000 visitors in January and 81,000 visitors in July. So while a greater proportion of our summer traffic comes from non-Google sources, we didn't lose almost 50,000 visitors -- which the Google Analytics data suggests we should have.
So, my question to the rest of you is: has anyone else seen similar disparities in statistics? What are your thoughts on why Google Analytics would diverge so much from raw server data in a consistently downward spiral from January through July (this was not a sudden dropoff in the Analytics data -- the overall numbers have declined each month).
Click Here to read the entire article...
