Evolving Web infrastructure impacts search optimization

Netcraft’s June 2008 Web server survey shows an interesting trend has developed over the past two years.

Domain name registrations exploded around March 2006. At the same time a slight increase in active domain names took a slightly sharp upturn that has been pretty steady over the past 27 months.

However, starting around late 2005 (about six months earlier) the Apache server market share went into significant decline. Microsoft server share began climbing rapidly around March 2006. One has to wonder what happened in the September 2005 - March 2006 period, but the significant curves appear after March 2006.

Apache continues to lose market share, Microsoft continues to gain market share, and even when Google appears on the scene in March 2007 the best that Google is able to accomplish is to deflect Microsoft’s growth into a flat line.

Google seems to have taken more market share from Apache than from Microsoft, which fed on Apache like a lion mawling a baby elephant. Sun Microsystems’ market share looks like it has bottomed out. Their industrial-strength solutions will probably always be needed but the small to mid-sized business sector seems to have fallen in love with the instability and complexity of Microsoft Web hosting.

Okay, that was a biased remark but I’m a UNIX guy working for a Microsoft customer, so there are days when I feel a little frustrated.

My point, however, is that the perception of low quality with which Microsoft server technology has been associated for many years seems to now be a phantom of the past. Microsoft has made significant gains in server market share and it has successfully held on to those gains. Well done, Microsoft.

Google’s server technology is still a new force in the marketplace and it remains to be seen whether Google can become a major part of the market. Right now, they are a second-tier player. The real dogfight remains between the Apache community and Microsoft’s technologists.

However, if you look at Netcraft’s bottom chart (titled “Totals for Active Servers Across All Domains June 2000 - June 2008″) you’ll see that Apache has actually bounced back from the losses it suffered at Microsoft’s hands.

The Netcraft data suggests that there is a host startup curve — a timeframe of varying length between the registration of the domain and the activation of the Web server. But there is a second part of the curve that falls between the activation of the server and the launch of the site.

There could be several reasons for these gaps, some of them not so obvious. One thing to consider (which is to say that I don’t see how Netcraft accounts for it) is the amount of load balancing that occurs between multiple data centers. Then there are server transitions, transitions between hosting services, and other changes in infrastructure that affect live sites which might give the false impression of my so-called startup curve.

Still, search engine optimizers can learn several things from monitoring server market share data. First, you need to be aware that there are more than just 2 or 3 different kinds of server technologies out there. If you know how to fix basic SEO issues in Apache and IIS, you may still find yourself scrambling for the help documents when your client shows up with some other server technology.

Also, many of us Apache-centric sysadmins have never bothered to learn the Microsoft technology. I could force my way through if I had to, but I generally prefer working with UNIX-based architectures. That is my personal bias but it limits my ability to troubleshoot non-Apache Web serving issues.

A key skill for any successful search optimizer is troubleshooting Web server issues. A misconfigured server can screw up your URL structures, fail to serve pages to search robots, and do other things that make a site very unhelpful in the search environment. A good SEO has to understand that these are the types of off-page factors that do negatively impact rankings (they cannot help boost rankings, only hurt them).

Many people scoff at playing what-if games, but if you’re sure you want to stay involved with the search industries for the next 5-10 years you need to be paying attention to things like browser market share, server market share, etc. Knowing which technologies are dominating and invading the Web space helps you stay competitive and marketable.

The day may come when my Apache knowledge isn’t worth much. The day may come when Microsoft replaces its IIS technology. Maybe Google will consume the whole industry. You never know.

But you need to stay on top of the technology, at least well enough not to niche yourself in a skills market that has a limited lifespan. I’ve walked that path and believe you me it is no picnic when people tell you that your skills are no longer needed. I was fortunate enough to have developed a serious interest in a second profession before the first one burned out. Many other tech industry people were not so lucky.