Optimizing search results pages

Google doesn’t like indexing search results pages unless they add value to their users. Of course, one of the oldest SEO tricks is to point links to search results pages in the hope that they’ll be crawled. But do you understand why people used to do this and what the possible outcome of doing this today may be?

Many people in the SEO community continue to do “stuff that works” because it once worked (or everyone thought it once worked) and because everyone else is doing it. Some of us even do things out of pure laziness. It is entirely possible (probably extremely likely) that I still have somewhere on one or more of my personal sites that infamous “revisit-after” meta tag, if only because it takes so long to comb through 50,000 pages of content (and because I occasionally resurrect old pages and often simply create new pages by copying old pages and stripping out the indexable content).

The “revisit-after” meta tag was honored by one relatively small Canadian search engine some years ago. For a few years, a number of us who were struggling with AOL proxy servers used the tag in the hope that proxy servers were honoring it. As so often happens, people implemented the tag, thought they saw a change, and then assumed there was some sort of cause and effect. Me? I’m just lazy. It was easier to reuse my ugly old HTML code rather than to completely redesign new page stuff every time I wanted to put up new content.

Besides. It once worked (for that one search engine).

Search results pages used to be very useful for optimization. That was back in the day when Inktomi ruled the Web. Although many comparisons have been made between Classic Inktomi and present-day Google, there were significant distinctions. Classic Inktomi was the “search engine behind the search engines”. Inktomi crawled the Web, divided everything into a primary and secondary index, and then sold its primary index database to about 30 other search engines (none of which did their own crawling).

Each of those search engines had the option of “tweaking” the Inktomi database so that the relevance scores were calculated somewhat differently. Each search engine also had the option of combining the Inktomi search results with other search results. Some directories (like Yahoo! and Looksmart) supplemented their own directory search results with search engine results. Besides Inktomi, Altavista also resold its database. So did Google, in the old days.

Another “search engine behind search engines” was DirectHit (now owned by Ask and apparently somehow updated and incorporated into Ask’s algorithm, code-named “Edison”). DirectHit was never as popular as Inktomi (nor as useful and relevant as Inktomi results) but it was picked up by several search services.

At the height of the First Golden Age of Search Engines, there were several services providing databases: Google, Inktomi, Altavista, and DMOZ. The various clients (at one point there may have been 40-50 altogether) accepted submissions, might do some of their own crawling, accepted database updates on different schedules, and otherwise ensured that their search results were unique enough to be different. And there were indeed some great search services back then. NBCi was one of the best. Unfortunately, they just could not monetize their services enough to keep going.

But one of the great challenges of the First Golden Age, for optimizers, was getting and keeping your content in all the indexes. Few people were able to do it consistently. Most people didn’t know how to do it or why it happened for them. But one of the reasons some sites stayed in longer was that they appeared in enough search results that there were links to search results pages that listed those sites.

Search results pages are indeed legitimate content. In fact, there was a time when it was very common for people on news groups (aka as Usenet and the Altnet) to provide links to search results to help other people find content, back up points in arguments, and otherwise share information. Search results pages, though dynamic, were treated as normal, everyday content by many people. Links were treated as normal, everyday content by many people.

The First Golden Age of Search Engines was not obsessed with linkage. But Inktomi changed all that and in a way once we became fully aware of Inktomi’s susceptibility to link manipulation the First Golden Age died. Of course, the Dot-com Meltdown on Wall Street was the real cause of the failure of so many search services. People were operating their companies mostly on venture capital and IPOs and when they failed to monetize their services enough to pay the bills, everything died. But in the world of search everything died more quickly because of link manipulation.

We invented link farms to manipulate Inktomi search results. It was a necessary evil because many reasonably good and useful sites were incapable of getting into Inktomi’s main index. Some sites would get into the main index one month and drop out the next. Inktomi recrawled the Web every 28-30 days. The changeouts happened like clockwork. Inktomi offered paid submission so that business sites would be recrawled each month but business operators hated knowing that guys like me could get in and stay in without having to pay for the privilege.

Why was that?

One of the chief reasons was search results pages. You used to see them in Inktomi all the time. Spammers learned early on how to get their pages crawled through search results pages. You see, you could not submit a site directly to Inktomi but you could submit a site through most of its partners. Canada.com, Hotbot, and a few search engines you probably never heard of were popular submission points. The submission tools generally worked well enough to get sites crawled but didn’t guarantee inclusion in the primary index.

One of the reasons why guaranteed inclusion was so elusive was that Inktomi intentionally placed lower value (less trust) in directly submitted sites. Pages that were found from crawls through the main index were given precedence. Although it may not seem obvious, trust has always been a major issue with Web search. The search engines felt the only way they could trust content was if they found the content through normal Web crawling.

And since Inktomi was crawling all the major search services, people learned that they could link to their Inktomi results on search engine A and get crawled through search engine B. What was really cool, however, was that there were other services to work with. So you could link to your search results on Excite, Lycos, NBCi, Google, and Altavista and Inktomi would crawl them. And those search engines also crawled search results. By crawling those results, the engines algorithmically concluded that they were finding good pages.

Problem was, search results pages kept changing. Yes, we linked favorably to directories that listed our sites. That was a bit of a trick because you didn’t want to link to a page that would also list your competitors (although once a search engine got into a directory it would eventually find those competitive listings anyway). Link farming became a popular alternative because people at least had better control over whose sites they were being associated with. The earliest link farms cared about whom they let in. The very first professional link farm service went out of business only after the owner gave up fighting with the spammers.

We were managing our reciprocal links directly and indirectly through search results. If we knew about 10 sites that linked to us and they were not listed in Inktomi, we could submit them to Infoseek, let Infoseek index them, and then submit the Infoseek search results page be found by an Inktomi crawler. We borrowed a page out of the spammers’ books and set up our own hallways and crawlways that pointed to directory pages and search results pages which listed our own sites. If you were clever, you could create little mini-indexes of your sites on each search engine and directory.

Inktomi, Google, and Altavista ate up these pages. They loved the links, they loved the fact they were finding the links through links, and as they found more links to our pages those pages became more firmly entrenched in search results. Paid submission? We didn’t need no stinkin’ paid submission. We were tightly linked by some of the most important sites on the Web.

But those days are gone. Google still indexes search results but as soon as someone reports them the results are taken out. And Google seems to be improving its ability to figure out if a page full of links should be trusted or not. So a logical question would be, “Should we continue to use search results pages for optimization?”

The answer is not simple. Search results pages are still legitimate content. People look at tens of millions of search results pages every day. But the search results pages are not necessarily the final stopping point for those people. While a SERP can answer a question (like define query operator), Google understandably fears the continuing link manipulation that has been and continues to be effected through search results pages.

In SEO forums and blogs, it is often preferred that people link to search results pages rather than directly to specific Web sites, so as to prevent or reduce the effect of linkage. Where feasible, some people just grab screen captures of search results and upload those but not everyone can do that in an online discussion. And images are not yet universally accepted across the Usenet and similar news groups.

Search results links lead to dynamic content. Dynamic content is generally less useful than static content. Dynamic content changes because it is being edited, added to, or losing stuff. The search result link you provide today may be totally irrelevant tomorrow (it’s a lot like Wikipedia — a really unreliable source of information because you don’t know what will be there the next time someone clicks on a link to it).

Now, if Google is concerned about link manipulation occurring through search results, should we assume that all search results no longer pass value? While the formulaic SEO answer may vary from day to day, the long-term answer is — in my opinion — an unqualified “yes”. Optimization works best when you don’t think about the possible effect an idea may have on search results. Search engine optimization is first and foremost grounded in the optimal organization of content for presentation. In that respect, you can use search results pages to validate your points and guide your visitors through further Web searches.

In fact, millions of Web sites do this every day with so-called “on-site search”, where they load search tools onto their Web sites and let the search engines help people navigate through their content. Those search results pages are rarely linked to and are even less rarely likely to be found in general search results. But as I have said before “building referrals from a search engine can be as simple as working the search engine into your content”. The more useful you make your predetermined internal search links, the more likely those links will be treated as proper content by people.

Never mind whether they’ll ever be crawled and indexed or pass link anchor text. The point is that if you can get the search engine to tell people about your pages in a relevant context, you need to be doing that. It even works for Supplemental Pages because you can tell people to search Google for ‘four foundations of SEO theory’ on the SEO Theory site and relevant content comes up. You can say I should be linking to the direct page with keywords like “content optimization theory”, “search results optimiation theory”, “brand search optimization theory”, etc., but the link to the search results page demonstrates my point.

You can optimize your search results by treating them like dynamic content. Just remember that dynamic content is not very reliable. I cannot make a case for dynamic content being very valuable (neither can Google, which alone is the best reason why they should not be favoring Wikipedia in their search results).

Whether you can get a search engine to crawl search results, or allow those search results to pass value, is not as important as whether you can use those search results to help your visitors. The more uses you find for search results pages, the more valuable those pages become, the more worthwhile your optimization efforts have been.

In other words, if you stop treating search results pages like a cheap spam trick and start treating them with some respect, you’ll find you get far more bang for your buck with less effort than it takes to master the Spam Trick of the Week.