• Register
• Log in

SEO Theory - SEO Theory and Analysis Blog

Algorithm analysis, Web community relationship analysis, SEO practices and techniques, industry news, etc.

• Home
• About SEO Theory
• SEO Glossary

Google adds a third index for site search

Posted by Michael Martinez on May 12, 2008 in Intermediate SEO

Matt Cutts mentioned on his blog that Google Custom Search Engines now have a special site search index.

This seems like a major step in the right direction as site search has been a real issue for me. Site search is the third navigation tool that every Web site designer (and SEO) MUST take into consideration. You have your on-site navigation, your HTML sitemap, and site search. On larger sites, people are more likely to use site search than either on-site navigation or HTML sitemaps, so you need really good site search.

Most Web site operators don’t have the resources to install and maintain a fully indexing site search resource (although I personally feel that Google Search Appliance is an adequate solution for those people who can afford it). The rest of us have to rely upon third-party search solutions, and there are still a few alternatives to the major search engines. Unfortunately, most third-party solution providers whose tools I have tested entail strict limits (5,000 pages of free indexing being one of those limits).

I think moderately sized sites can benefit from the third-party site search tools but when you work with volume you need industrial-strength site search. It behooves a major search engine to provide that capacity as its own index benefits from crawling and indexing large sites.

I’m not saying that Ask, Google, Live, and Yahoo! should include every page on the Web in their Main Web search indexes — I’m just saying they should acknowledge that their users often use their systems as site search tools. So I don’t have a problem with Google Custom Search Engine using a third index just for site search. In fact, that might make their service more scalable for both them and me.

The less work I have to do as a Webmaster to make my site searchable while giving my visitors access to the content they want, the better off both my visitors and I tend to be. When I run out of time for updating indexes or site search maintenance functions, my site search offers stale content to my visitors. That’s not a good thing. I’ve watched my site search statistics shoot through the roof as people look for new content they know is there but it’s not appearing in site search.

Reducing the number of steps a Webmaster has to go through to get site content indexed is therefore a major plus on Google’s part. But having only just learned about this enhancement to the service I haven’t had time to test it. There is more to site search than just inclusion. You also need to be concerned with relevance. In fact, in site search relevance is even more important than in general Web search. In general Web search people usually have choices so if they don’t like the first result they can try another. In site search when relevance is low the second result tends to be just as bad as the first.

Even the Google Custom Search Engine has typically had severe relevance problems (it doesn’t offer nearly as much diversity as Web search does and that degrades the quality of the user search experience). I created one of the first SEO search engine Custom Search Engine tools soon after Google announced their CSE service. To this day its results tend to favor certain sites over others, diminishing its usefulness.

Site search needs to pay as much attention to clustering and diversity as main Web search. However, it also needs to be configurable. Where major search engines really let down Web site operators is in not allowing us to set priorities for site search. I feel that Google, Yahoo!, and Live all provide platforms (their Webmaster consoles) that can be expanded to provide for some site search optimization. Site search is not competitive — you cannot invade my site search and I cannot invade yours. It would be more helpful to everyone involved if the search engines relaxed some of their filters for site search.

I’m willing to take on some administrative responsibility in order to offer improved site search and network search. Network search, unlike site search, does have to allow for some competitiveness and it could be much more easily abused than site search. Nonetheless, I feel there are enough diverse networks out there to justify some development in network search. I’ve never found a third-party tool that even came close to meeting my needs for network search.

I don’t mind sharing the burden of deciding what should be indexed and how to manage in-site and in-network relevance. I’m not asking for a back door into Main Web Search. What I’m asking for is an improved user experience and more options for Webmasters to help leverage major search engines into their content. In fact, I would be happy to leverage new, previously unknown search engines into my content if they didn’t force me to manually reindex, limit my indexing, or require that I run their ads or show their results on their pages.

Give me a solution where we can both benefit, where my users can find what they want quickly, and where I can adapt my relevance settings to changes in consumer sentiment. That may be a lot to ask but it’s an important part of search technology that has been underdeveloped.

Google’s CSE site search index is, as I said above, a step in the right direction. But more steps are needed and there is a great opportunity here for someone to step forward and carve out a new search engine if the major search services don’t rise to the challenge and meet the need.