Define:”SEO Method”

The SEO Method is easy to remember: experiment, evaluate, adjust. The adjusting part seems to be the easiest for most people. They just read about the latest techniques in blogs and forums and start changing their Web sites.

SEO experiments are not really that hard to do, but for some strange reason a lot of people struggle with the basic principles of experimentation. Maybe they didn’t take notes during science class. I’m not sure.

Here are a few ground rules to follow if you want to conduct SEO experiments. Keep in mind that following these rules won’t guarantee you the results you want. Predetermined vindication is generally not considered to be an acceptable result in a truly scientific community (unless you’re teaching well established principles).

Rules For SEO Experiments

1. Begin with a hypothesis. Make an assertion and test it.
2. Always use an existing query. Never use a new keyword or expression that no one has created before.
3. Always use queries that return at least 1,000 results. Never experiment on queries that return only a few results.
4. Always replicate your experiment. Never rely on only one set of results.
5. Toss out studies and experiments that are interrupted by announced search engine updates or algorithm changes
6. Look for evidence to support your conclusions in natural queries you have not experimented with
7. Repeat your experiments across at least two separate time frames

SEOs are impatient and quick to leap to unjustifiable conclusions. I have rarely seen anyone conduct proper results testing, and the most recent generation of popular gurus (going back at least four years) have been pretty sloppy in their research and testing.

SEO Method: Begin with a hypothesis

You cannot simply run a test to see what happens. The test will tell you nothing useful unless you’ve just landed on a distant planet and need to determine if the laws of physics as you know them work there. If you want to make an SEO experiment, form a hypothesis. You don’t have to believe it. Just state what you want to test in a short, simple statement.

“All links pass value” is a hypothesis. “Javascript links do not pass value” is a hypothesis. “What happens if I put Javascript links into my CSS-hidden navigation?” is not a hypothesis. It’s a question with many possible answers.

A hypothesis can be stated in the form of a question but it has to be a very specific question. Your experiment should be designed to give you a YES or NO answer. That doesn’t mean you’ll get a definitive answer, as the conditions of the experiment will qualify whatever answer you get.

You should base your hypotheses on observations you have made. The hypothesis proposes an explanation for an observed phenomenon.

SEO Method: Always use an existing query

I don’t know why, but SEOs have married themselves to the idea that if they are going to conduct an experiment on search results they should use only queries that no one else has thought of. SEO contests are really bad about this. “Let’s all use the new expression ‘quintilius expelimaxus resargus’ so that we can watch each other spam the search engine!”

If you want to study how search engines behave, you have to observe them in their natural conditions. The more natural the better. You can actually do comparative studies, where you examine technical queries, business queries, hobbyist queries, consumer queries, legal queries, etc. You need to define your query categories carefully and rigidly adhere to your classifications. Don’t worry about whether other people agree with your classifications, but each experiment should define the classifications it’s working with.

New queries are unnatural. A previously unpopulated keyword expression forces the search engine to make unnatural choices. When a previously unpopulated query naturally accrues content, that content is usually not optimized. It is usually not derived from one source. It is usually not constructed so as to “test” a search engine.

Artificial queries do not behave the same way natural queries behave.

SEO Method: Always use queries that return at least 1,000 results

A search engine will only throw all of its gears into motion if it has to pick and choose the best 1,000 possible results for a query. Counter-intuitive as that may seem, the process has a smoothing effect on search results. Your hypothesis may be constructed for underpopulated queries, but in that case you should have been studying (observing) those underpopulated queries before forming your hypothesis.

Your objective is to learn how to manipulate search engine results for popular, well-populated queries. You need to see what happens when you test your hypothesis on a natural query. Does the experiment support or challenge your hypothesis? An underpopulated query won’t tell you how a search engine handles synthetic adjustments in a competitive query.

SEO Method: Always replicate your experiment

The one most common flaw in all SEO experiments is that the tests are never repeated. You cannot learn anything useful or conclusive by conducting your test one time. But every SEO who makes an effort to “test” something stops after the first step and starts drawing conclusions.

This is why I am constantly rebutting so-called “SEO tests” on blogs and forums. People are wasting their time (and yours) by conducting their little experiments and drawing conclusions on the basis of the first set of observations. You need to observe, hypothesize, make a test, observe, make a test, observe, make a test, observe.

I often suggest to people who share bizarre connections (e.g., “I changed all the meta tags on my blog and lost all my rankings”) that they undo whatever change they made, evaluate the results, reimplement the change, evaluate, and then undo the change again. But that is really not a conclusive test. I just haven’t figured out how to persuade people to do it three times (since most people refuse to do it twice).

One iteration tells you nothing. Two iterations may tell you there is something to your hypothesis. Three iterations tell you there is a repetitive pattern, hence implying the cause-and-effect relationship you think you have identified.

It takes three licks to get to the chewy core of a Tootsie Pop.

SEO Method: Toss out studies and experiments that are interrupted by announced search engine updates or algorithm changes

Google announced its new search engine (which I have dubbed “Google 3.0″) in May 2007. Question: How many SEO tests in progress during the month of May 2007 were valid? Answer: None of them, as far as Google’s search results are concerned.

When a search engine rolls out the media and says, “Hey, we’re releasing a whole new infrastructure and algorithm today”, you just need to shut down your tests and wait until things settle down. All you do by looking at the results anyway is mislead and confuse yourself. There is no one in the industry — not Dr. Edel Garcia, not Shari Thurow, not Michael Martinez — no one in the industry — who can draw any valid conclusions from the results of SEO tests that begin on one side of an algorithm change and end on the other side.

SEO Method: Look for evidence to support your conclusions in natural queries you have not experimented with

Another favorite SEO screwup is to experiment with queries that SEOs are already optimizing for. Unless your hypothesis is something along the lines of, “Resolved: I can really confuse myself by experimenting on a query that I know has been optimized”, all you’re going to do is confuse yourself.

You have to be able to account for all the factors that might affect your test. A natural, well-populated query may bubble and churn on a regular basis but if you are reasonably sure no one is actively pursuing the top spot for that query then your test has the best possible chance of making a repeatable impact on the query.

SEO testers typically make no allowances for the disruptions caused by deliberate optimization. Remember, there are four causes of changes to search results:

1. What you do with your pages
2. What other people do with their pages
3. What the search engines do with their data
4. What people search for

When you are testing ideas on search results, you eliminate number 4 automatically. You eliminate number 2 by not telling anyone what you are doing and by working with unoptimized queries. Number 1 is your mechanism for figuring out what number 3 shows.

If you cannot normalize the data, filtering out possible causes from multiple optimization techniques, your data is mush, your conclusions are mush, and your SEO test is a waste of time.

SEO Method: Repeat your experiments across at least two separate time frames

An ideal experiment tests a hypothesis on three unoptimized, well-populated queries, waits a while, and the repeats the process. If your hypothesis is valid you should see the results of the first test replicated in the second test. Now, search engines do change things. And people start optimizing for new queries all the time. So the ‘do it twice’ rule doesn’t always help you the way it should, but you’re shooting at a moving target.

Since you don’t have the option of studying the search engine rats in your laboratory maze, you have to devote a lot of time to observing the search engine gorillas in their wild habitats. You don’t control the experiments completely. They may go awry. A search engine could change itself, your competitors could start dinking with your queries, people may just join the newest wave of Web site design and radically alter the content of your natural queries — things go wrong.

It’s a good idea to retest important hypothesis once or twice a year.

The best SEO experiments

I have often been impressed by people who started sharing ideas by saying, “I was reviewing my X last month and noticed something funny, so I decided to try Y to see what happened. Sure enough, I saw X”.

That sounds very similar to the style of testing I just trashed above, but the key distinction here is that people will randomly try simple ideas without overthinking them. They don’t go through all the elaborate setup that SEO testers usually resort to. They just go out and plunk a rock into the water to see what sort of ripples they can make.

Acting on the spur of the moment often produces better SEO test results than trying to set up a controlled test. Why? Because in SEO testing truly controlled tests are impossible. Remember, you have to observe the gorillas in their wild habitat. You cannot force the search engines to act like rats in a maze, rewarding them with food when they push the right lever, or otherwise “train” a search engine to prove you right or wrong.

A search engine is an immense non-biological system but it behaves in a very rational, almost biological manner. The problem is that you cannot trap it and subject it to any old test you want to.

You’re most likely to see consistent results by just trying the same thing over and over again without worrying about controlling the test. Don’t do something endlessly. Stop after a few attempts and see if you can identify consistent consequences. The simpler your test the less likely you’ve screwed it up. The less convinced you are that something will or will not work the less likely you’ll mislead yourself when you start forming conclusions.

Those first conclusions can be the hypotheses for more fornal tests. When you reach the stage where you’re ready to make a formal test, you should already have a body of observations to work with.

Should you share your test results? Maybe. I usually wait 1-2 years before sharing definitive conclusions. My early tests may be malformed and I may simply be wrong on a lot of points. I invest a lot of time in examining search engine results. That doesn’t mean I don’t make guesses about what is happening quickly. But I don’t confuse my pre-testing analysis with my post-testing analysis.

You can analyze natural search results and look for patterns in the data, but pattern analysis is much more complicated than formal testing. I do have pattern analysis experience but that still only leaves me with some gut-level interpretations until I get my hands on a lot of data.

Still, now you’re armed with a little more knowledge than you’ll have gleaned from the usual sources of SEO testing. The testing process is continuous, but new people are constantly buying SEO chemistry sets and trying to solve the Web’s problems without taking the classes and learning how and why to do things properly.

Always be skeptical of SEO test results, especially if they are announced on a blog or forum. The odds are pretty good that the tests were not well-conceived or managed properly.