Why your nofollow testing sucks

The SEO Method: Experiment, evaluate, adjust.

It seems like half the SEO community is trying to prove that “PageRank sculpting” is a viable optimization method. All the testing is a good sign that people are not willing to just take one pundit’s opinion over another’s. We want people to reserve judgement and to require proofs of claims that are made in blogs and forums. A natural consequence of demanding skepticism from a community is that you’ll see more people testing hypotheses.

There are two hypotheses making the rounds in the SEO community these days. First, some SEOs (like Rand Fishkin and Dan Thies) have championed the idea that you can improve a Web site’s search referral traffic by sculpting out underconverting pages. This is what everyone has been referring to as “sculpting PageRank” but what they’re really doing is decreasing the search visibility of pages that don’t provide for SEO-prioritized conversion.

Second, some SEOs are claiming or implying (or perhaps inferring from Fishkin and Thies’ comments) that you can swap out pages by channeling or directing your link flow toward “more important” pages on your site. By “swap out” I mean that PageRank sculptors appear to feel they can replace page A in the search results with page B simply by cutting off the flow of PageRank to page A.

Now, there are undoubtedly endless variations on these two ideas, and perhaps even some claims I haven’t seen yet. An increasing number of people are claiming to have tested rel=nofollow in a variety of ways and they are astounded, amazed, and impressed by the results — one way or another.

The problem is that the all the tests that have been openly described are invalid. It doesn’t take much effort to screw up an SEO test and invalidate all your findings. In fact, it’s because SEOs tend to commit the least amount of effort to their testing that their tests are usually useless and unproductive.

If you want to test how something impacts natural search results, you cannot construct an artificial query space. One of the most common SEO parlor tricks is to create a small number of domains or sub-domains that are “optimized” for a previously unindexed word or expression. Hence, you have a “clean” query result with only your own content. And when you look at how the search engine ordered your content you don’t take into consideration the fact that the content did not have to compete for ranking love with other content.

If you want to test how a search ranking algorithm works you MUST either qualify your analysis by saying, “Well, this appears to be what works for short query spaces” OR you MUST use a minimum of 1,000 indexed URLs so that the search engine has to throw all of its resources into organizing the query. And 1,000 URLs may not actually test a search engine’s full capability.

Short query space testing is like riding a lawn mower. Full, natural query space testing is like driving a Formula 500-class vehicle. If you want to look amateurish, then compare your lawn mower parlor tricks to what is happening in the Formula 500-class search results. I won’t be impressed but I’m sure someone else will be.

The easiest way to perform a full natural search test is to use site search. Now, your site may not have 1,000 pages that are relevant to “ixy quixy nixxy woo” but you should be able to make them relevant to that unique expression if you put some effort into the task. If nothing else, you’re forcing the search engine to make a more complex choice than if you ask it to figure out how to order 10 results.

Of course, where Google is concerned you cannot simply work with 1,000 search results. After all, relevance is not Google’s priority. Google divides the Web into the Main Web Index and the Supplemental Results Index. Pages in the SRI are not fully indexed and their links are not treated the same as pages in the MWI. If you want to create a valid Google test you have to choose from six environments:

1. Short query space with only Main Web Index results
2. Short query space with only Supplemental Index results
3. Short query space with mixed MWI and SRI results
4. Full query space with only MWI results
5. Full query space with only SRI results
6. Full query space with mixed MWI and SRI resulst

Keeping in mind that Main Web Index pages trump Supplemental Index pages, AND that your test expression may not be unique enough for Google to index your Supplemental pages for it, your chances of creating a valid test that mixes MWI and SRI or that only looks at SRI content plummets to almost 0. It is not easy to get Google to show you anything interesting with Supplemental Index content because in many cases Google doesn’t know those pages exist in its database.

So that means if you really want to learn something about Google’s algorithm you have to run with the big boys: Wikipedia, CNN, every Web spammer and his script-kiddie following, every know-it-all SEO pundit and his disciples, and low-class Internet marketers who rely on affiliate farms and multi-level marketing schemes. And let’s not even get into the real large content sites like eBay, Amazon, HP, et. al. They’re out there, too, sucking up Main Web Index space and flooding query spaces you never even imagined.

It is impossible to isolate search engine algorithmic factors. Anyone who thinks they have managed to do that in the past 8 years is fooling him or herself. You MUST make adjustments in all your testing and analysis for at least 100 other factors, possibly as many as 900 other factors, in order to figure out anything useful about any one factor.

Just because a lot of people cloud the blogs and forums with statements like, “We’ve done extensive testing and our results indicate that meta tag spam doesn’t work” doesn’t mean their tests were valid. It’s easier for everyone to do it the wrong way than for anyone to do it the right way. The odds of any of us actually isolating any algorithmic factor in our SEO testing are infinitismally small. You cannot do it. I cannot do it. No one in the industry can do it. It is statistically impossible to do.

Now, if truth and reality are not enough for you, let’s take a look at people who drop comments on blogs in order to test “Nofollow” and its effectiveness. Some of these folks have expressed surprise that their comments and/or the sites they link to show up for their “unique” anchor text. Why might that happen? Oh, let me count the ways:

1. They link check on Yahoo! and think this has something to do with Google
2. They use Google’s link command and think this reports useful information
3. They forget that spammers will scrape and republish blog content WITHOUT nofollow
4. They assume that no one else has linked to their content with their “unique” anchor text (this is a great way to mess with someone’s mind — just use “noindex,noarchive” on your pages and no one can find your stealth links)
5. They think that RSS feeds use nofollow
6. They forget that search engines index links in RSS feeds

Okay, I’m tired. There are too many possible, plausible explanations for why someone’s blog comment might actually pass value when the blogger clearly implements rel=nofollow.

The bottom line here is that no one has published a viable test for determining whether PageRank sculpting actually works. You cannot measure PageRank and you have no knowledge of how its awarded, passed around, and received but you have convinced yourself that PageRank sculpting works because you made some changes on your site and you saw changes in the search results.

Why do search results change?

1. You do something with your site
2. Someone else does something with their site
3. You and someone else do something with your sites
4. Search engines do something with their data (or algorithms)
5. You do something with your site and search engines do something
6. Someone else does something with their site and search engines do something
7. You and someone else do something with your sites and search engines do something
8. People change the way they search
9. People change the way they search and you do something with your site
10. People change the way they search and someone else does something with their site
11. People change the way they search and you and someone else do something with your sites
12. People change the way they search and search engines do something
13. People change the way they search, you and someone else do something with your sites, and search engines do something

In short, there are Four SEO Influences — there are four reasons why your search results change:

1. You do something with your site
2. Someone else does something with their site
3. The search engines do something with their data
4. People search for something different (or, “People change the way they search”)

This theorem draws upon something called The Law of Influence (or, The Law of Influences). The law simply states that “an influence reveals itself through its effects”. Jesus put it this way: “You will know the tree by the fruit it bears”.

Using the Law of Influence, you can easily figure out a cause-and-effect relationship by removing a supposed influence from a situation. If nothing changes in the situation then you know you have eliminated one possible influence (in your search to find an actual influence). But once you bring about a change in a situation you have to stop and ask yourself, “Did this change occur because I removed one hypothetical influence or because I removed 100 hypothetical influences”?

To isolate a single influence you have to gently remove your first influence and, if the removal fails to bring about a change, you then have to reinsert the influence into the situation before removing the next hypothetical influence. And it sometimes happens (or perhaps often) that in a very complex situation with many possible influences that it is a combination of influences rather than any singular influence that causes an effect.

Well, I haven’t yet found anyone who has published that kind of tedious testing data about search engine results on the Web. Maybe someone’s “premium access only” Web site (I can think of several such sites) offers extensive data based on that kind of careful scientific evaluation but I seriously doubt the likelihood of such data being shared. Most companies don’t have the patience for that kind of testing so SEOs have to arrange to do it on their own time for the most part.

For these and many other reasons that I don’t have time to explore today (perhaps another time), I extend absolutely no credibility to anyone in the industry who claims they have found positive results from “Sculpting PageRank”. You can achieve as much with your life by bottling ether (the stuff scientists once suggested lay between the stars) as by sculpting PageRank. Someone may buy your bottled ether — they may even believe there is ether in the bottle — and you’ll reap the rewards of ripping people off. But if you believe your own nonsense you’re just fooling yourself as well.