As a mathematical statistician, I would counsel that it’s always worth making sure you understand all the variability in the field of study before designing your tests. The best analogy I have for the results of any given keyword SERP is a turbulent river. OK Google may be tuning its algorithm on a daily basis. Unfortunately the world is not standing still. Others may be creating web pages or may modify existing web pages so that even with a static algorithm rankings would change.

If you are aware of all these random factors, then you can determine what you might learn from any given test. Your assumption should be that there is lots of random noise in the system. So you should do adequate repetitions of any test and look at the average behavior.

Given all this, I believe it is almost impossible to reverse engineer the way Google does keyword searches. Thankfully they are trying to produce relevant pages for human beings. So perhaps your judgment of what is relevant to humans may be as important as whatever you ‘prove’ from your tests.

Most people don’t have the time or resources to perform a more precise test than that.

Think of how complex the Earth’s ecosystem is. It’s much larger, much more active, much more complex than a search engine. Nonetheless, we tag animals, watch plants grow, send planes into storm clouds, and capture large amounts of data for statistical analysis in thousands of ways.

With search engine results analysis the SEO community just needs to expand the scope of its investigative exercises a little bit. Instead of tracking one page with an obscure query people need to track 10 pages in active, natural queries.

The scientific method works just as well in documenting search engine behaviors as in documenting the behaviors of schools of fish, flocks of birds, herds of cattle, clans of meerkats, etc. You begin with observation and collect as much data as possible.

Unfortunately, the SEO community prefers to collect as little data as possible and to draw as definitive a conclusion as possible. That never works except by random chance, in which case you might as well draw explanations out of a hat. You’ll have as much accuracy with that method as with most SEO tests.

But if you tell & show ppl that you do all these little tests and give the illusion that what they represent is the truth and nothing but, you *are* doing the better marketing. And we all know who gets all the business. It is always the better marketeer, not the one ranting like a mad man although he in the end might even be right or better or both

My wife is a scholar who works with very large (30K to 300K) data sets about people’s work behavior, so while I am no statistician, I have watched a world class scholar work on large data sets. And what really strikes me is that without being able to construct a “why” around the behavior it’s pretty easy to get stuff wrong.

So, with humans, you can say, “Well, lower income people with lower education tend to start lower growth lower capital business for X, Y, and Z reasons.” And it makes sense.

With google moving pages round due to an algy with 200+ tweakable features, well, good luck with that!

-OT