When Search Updates Collide

I don’t read WebmasterWorld. I cannot stand all the complaining and references to PR; and I find the vast majority of posts there are about as useful to Search Engine Optimization as a sore thumb is to a carpenter. Which is not to say everyone who posts at WMW is clueless. Quite the opposite: some of the best minds in SEO have participated in the forum discussions through the years. Nonetheless, I now leave it to Barry Schwartz (RustyBrick) to sift through the nonsense and find the occasional gem. But I still won’t read the discussion threads. The last time I tried that was earlier this year and after pouring through 15 pages of drivel I asked myself, “Why did I do this?”

In my book, WMW is good for one thing and one thing only: letting everyone else know when Google changes something. The volume of complaints increases at WebmasterWorld more than in any other forum I monitor. That makes WMW a great weather gauge for when something changes at a search engine, especially Google. So many people complain it’s hard to miss the upswell in notices.

I’ve been debating whether I should discuss how I look for updates in Google. Technically, I don’t know any better than anyone else whether an update has occurred. According to Matt Cutts, Google is continuously updating its index. But what many people in the SEO community call an “update” is more than just Google adding page listings or recalculating internal PageRank.

Matt wrote: “Usually, what registers with an update to the webmaster community is when we update an algorithm (or its data), change our scoring algorithms, or switch over to a new piece of infrastructure.”

Now, I’m not going to ask Matt or anyone else at Goole to confirm my call. It doesn’t really matter, as the sheer weight of all their little changes could actually be felt on occasion only after a certain amount of “momentum” has built up. Nonetheless, I say that Google went into some sort of update (or momentum reached a critical mass) the weekend of November 18 and 19.

Two years ago (late 2004) I first noticed soon after the Thanksgiving weekend that I could upload a page to Xenite.Org and see it indexed by Google in a week or less. If I got the page out there by Friday morning it was almost sure to appear in the index by Monday morning. With only occasional lapses, that pattern has mostly held true ever since. These weekend data pushes (or whatever they are) were radical changes from Google’s previous behavior. We were all used to the monthly “Google dance” and Google gradually or quietly replaced the dance with a weekly jiggle.

Now, extremely active, well-linked sites have gotten content into the index on a daily basis for as long as I can remember. I just felt like Google had tightened its crawl-to-index efficiencies enough that suddenly they could dump a ton of data on the weekends starting in late 2004.

I first wrote about this apparent change in Google’s behavior in On The Googleness of Being, a paper I uploaded to Spider-Food after being quiet in the SEO community for nearly a year. Googleness was my introduction (to the SEO community) of my take on Google’s increasing reliance on trust. I have since refined my Google Trust interpretations in several ways. But the paper (which is quite long and was followed by considerable discussion and elaboration) is still very close to what I think I see in Google today.

Since 2003 Google has unleashed something some people call “the Christmas Crush” on the unsuspecting Webmastering community each Fall. Every November they change the algorithm, or add filters, or do something that pretty much upsets tons of etailers whose Web sites lose rankings. The etailers have to switch to or increase their PPC campaigns. It’s now almost common practice for good SEOs to recommend a holiday PPC budget “just in case”. After all, they did it in 2003, 2004, and in 2005 — and many of us agree they did it again this year.

Barry first reported on the November 2006 Google Update on November 20 (Monday before Thanksgiving). By then I suspected something was up because all my Google Alerts changed. I stopped seeing new Web content appear in the alerts. Normally I see tons of new or newly reindexed Web pages appear in my alerts. But for several weeks after November 19 I saw hardly any Web pages. Plenty of blog and news pages continued to show up in the alerts.

Of course, one could say that Google might have broken the Web alerts feature — but that would indeed constitute some sort of update, would it not?

On the other hand, I have been uploading sites at a frenzied pace since about mid-Summer this year. Dozens upon dozens of domains with tons of content. I was tracking sites entering into the index and ranking for their targeted expressions within a two-month window from release to rank. It was almost like clockwork.

After November 19, it was more like, “New site? What site? Links to new site? What links to new site?”

I’ve been getting links from many different sources. Most of my trusted sources dried up. It didn’t matter how often the pages were updated. It didn’t matter if all the links were pointing to CNN, the officlal White House Web site, or SaveTheWhales. Google’s cache refused to budge on showing anything after November 18 for nearly all my sources.

How do you explain to customers who are used to seeing new sites rise in the rankings that, um, Google stopped updating the index? If the customers understood this stuff as well as I did, they probably wouldn’t be customers. Of course, having gone through this many times before, I knew all I had to do was be patient. So I slowed the pace of site creation, increased the pace of link creation, and experimented with some site submissions (they didn’t seem to work but they didn’t hurt).

By November 29 more people were complaining at WMW and other forums and Barry once again wrote about a November 2006 Google update. I gently reminded him that he had already announced this one. On December 8, for the third time, Barry announced a possible late 2006 Google update. By this time, Matt Cutts intervened and said, “No, it’s just a data refresh (this week)”. Matt said nothing about whether there was an actual update in the previous weeks.

But we do know of several changes in Google behavior coming out of this period. First, members of SEO Refugee noticed on December 6 that allinanchor: appears to be working again; second, Google added refinements to the site: command and SERoudtbale reported as much on December 8. On December 9 Matt announced the new Google UI Plus Box feature. Now, in an earlier post (too old for me to find quickly) Matt said that Google might push out as many as 30 tweaks a month across its various Web properties. These three changes may only seem like “tweaks” to Matt and they may have no direct connection to the change in SERPs and Alerts that started on November 19.

Still, we’ve got five indicators of a major update rolling out: numerous discussions across various fora about lost rankings; sudden drop in Web page content for Google Alerts; Matt’s confirmation of a data refresh in the main index the week of December 7 (they normally follow a major update with a data refresh); and at least three permanent interface changes rolled out about the same time.

Also, my Alerts began returning to normal after December 7-9 and by the next Monday I was starting to see cache updates, new listings, and apparent link love passing from sites. Christmas came early for me.

Now, because of all my seasoned experience, I tried to both console and reassure concerned Webmasters over at Highrankings where, in their November 2006 Google Update discussion I wrote: “…until the update or whatever it is finishes, there is no point in doing any analysis, as the rankings will readjust themselves once Google flips the switches back on that have been temporarily flipped off.”

I have said as much through every major Google update for at least the past year-and-a-half, though most people ignore me and try to analyze the update before it’s finished rolling out.

Still, despite all my seasoned experience, I could not help but shoot my mouth off at SEO Refugee, where I wrote in another of their update discussions: “It appears to me that the update implemented a new link filter”. Ooo! I know better than to analyze an update before it’s over.

Well, I’ve since come to feel that my guess was correct nonetheless. This week I’m seeing far more effect from all those links I deployed throughout November and early December in Yahoo! than in Google. And I’ve seen movement in Google. Of course, Google doesn’t trust every link (nor does Yahoo!, but I think they’re looking at trust in a different way). Sometimes I seed links on pages that I think will eventually develop trust (they link out to very good content and they attract links). Yahoo! may trust those seed links sooner than Google.

But I don’t think that explains everything. I believe Google has implemented a new link filter. And, in keeping with my long-term policy of not sharing my conclusions for at least a year (because I need more time to evaluate the anecdotal evidence), I am not going to say what I think that filter is looking at. They may, for all I know, deploy a new filter every week. But this update just smacks of a huge filtering project despite the other changes that have been well documented.

And it could simply be that their crawling methodology is acting as a natural filter, without Google having to implement any actual filtering. It’s impossible for me to know, sitting on the outside, looking in at this black box we call “Google”.

But their timing was just a little off this year because Yahoo! announced an update on December 12. For once, I’ve been able to compare changes in SERPs between two aggressive crawl-to-index search engines (only Windows Live is more aggresive) at approximately the same time. It’s a rare search engine eclipse and I only regret that I’ve been so busy this week I haven’t been able to dig into the search results as deeply as I would like.

I feel like a kid in a candy store.

Of course, I can only hope that the good people who were hurt by this Christmas Crush have recovered their rankings and traffic in time to reap some useful benefit. If not, those who survive in business to next fall, here is a word of advice for you: budget for a PPC campaign starting around November 2007.