The SEO Method is to experiment, evaluate, and adjust. The search engine optimization industry implements the idea on a daily basis but the quality of the experimentation thus far has been very, very poor. The recent tests discussed by SEOmoz (“Do Tweets Still Effect (sic) Rankings”) and SEO.com’s Christian Greiner (“Testing the First Link Priority Rule”) are good examples of how NOT to conduct SEO tests.
Here is a brief list of some of the flaws in both tests.
SEOmoz Tweet Test
- The control example (“Test 1”) failed to disclose SERP details such as location, login status, time of day, how many times the query was checked, etc.
- The Firehose Off (“Test 2”) used a nonsense term
- Test 2 pointed to an accidentally “NoIndexed” page
- Test 2 used a URL shortener
- Test 2 pointed to a previously unused page URL
- Test 2 failed to disclose SERP details…
- The test URL appeared in several LinkedIn accounts
- The reported conclusion did not refer to the hypothesis
SEO.com Link Test
- The test invokes a nonsense “rule” (First Link Priority)
- The test uses two nonsense terms
- The test uses a previously unused page URL
Neither test was sophisticated and both fail to prove or disprove anything
The first test documents an effect that most people probably observed but because of its flaws fails to justify the conclusions offered (e.g. “We believe [the LinkedIn links] had a minimal influence on the experiment…”).
The second test begins with a false statement: “…when the search engine spider crawls a page on your website and comes across two links pointing to the same page, it will only consider the anchor text of the first link and disregard the second.”
This idea was first put forward by Rand Fishkin in early 2008. I published a counter example in July showing that his blanket assertion was invalid.
The takeaway from the back-and-forth experimentation on this subject should have been “SOMETIMES the 2nd link passes anchor text”. When Debra Mastaler asked Matt Cutts to confirm or challenge Rand’s assertion Matt replied (in 2 comments):
…Typically if the anchortext on the two links is identical, we would probably drop one of those links….This is the sort of thing where people can run experiments to see whether different anchortexts flow in various ways….
Now, before we all get out our nightsticks and start beating each other black and blue, let me point out that whatever may or may not have been “true” in 2008 is no longer relevant. Why? Because Google has changed its infrastructure (the Caffeine Update) and introduced up to 1500 algorithmic changes since then (they expect to roll out about 500 changes this year).
Is It Possible to Conduct a Valid SEO Experiment?
The short and simple answer is a heavily qualified “Yes”. There are two huge, major problems with the vast majority of SEO experiments. First, they are usually very poorly defined. Second, they are being conducted across a dynamic environment without regard for how that environment behaves.
If you don’t understand the environment where you are conducting your experiments, you don’t have any hope of conducting valid experiments, much less drawing reasonable conclusions.
But the sheer sloppiness of the methods being employed just dooms all these experiments to failure from the beginning. For example, if you are attempting to determine whether a site that uses “rel=’nofollow'” on its links can be passing value to its destinations, you MUST factor out secondary sites that pick up those links and repeat them.
So to assert that LinkedIn links don’t carry much if any weight is absolute nonsense. Maybe they don’t — but the lack of data showing how effective or ineffective LinkedIn links are invalidates the conclusion, which introduces a new factor as an aside.
What Makes SEO Experimentation So Challenging?
These types of experiments fail because people don’t take into consideration the dynamic nature of the search environment. Examples of aspects of the dynamic search environment that directly impact SEO experimentation include:
- Lag times between page fetches and subsequent search engine processes
- Changes in algorithmic processes (on average 2 per day)
- Differences in source and destination page quality
- Differences between how natural and artificial terms are handled by the search indexes
- Adustments to the value search algorithms apply to factors based on the presence or absence of similar documents in the index
- Variations in search results influenced by observer location, time, login status, etc.
Most of these issues cannot be documented much less factored out by the crude tests that people are performing. Furthermore, a simple experiment in and of itself tells you nothing significant no matter how clean your data is because its results could be a fluke. The dynamic environment will change the mix.
An SEO experiment whose results are replicated across multiple instances from different perspectives may provide data that is relevant only for a certain time because the search algorithms may be adjusted to change the effect.
An SEO experiment whose results appear to be consistent over a long period of time may be too narrowly defined, not taking into consideration different conditions that may be equally persistent.
An SEO experiment that attempts only to show that a predicted effect occurs does not in any way prove that a contrary effect will NOT occur. You have to challenge the hypothesis that the contrary effect will occur by trying to make it occur under as many possible conditions as possible. When you fail to make the contrary effect occur, then you’re justified in saying, “Our experimentation failed to produce a contrary effect.”
Remember, you CANNOT PROVE A NEGATIVE but any contrary example limits the validity of your conclusions. And a counter example flat out proves you wrong if you make a definitive assertion that only one effect will occur under any conditions.
Using the Scientific Method for SEO Testing
The Scientific Method has four parts:
- Observation and description of a phenomenon or group of phenomena.
- Formulation of a hypothesis to explain the phenomena.
- Use of the hypothesis to predict the existence of other phenomena, or to predict quantitatively the results of new observations.
- Performance of experimental tests of the predictions by several independent experimenters and properly performed experiments.
Let’s take the First Link Principle and apply the scientific method to it.
First, you have to describe the phenomenon: Given 2 or more links on a page pointing to the same destination, Google appears to allow only the first link’s anchor text to pass to the destination.
Second, you have to form a hypothesis to explain the phenomenon: The Google link anchor text processing software discards anchor text from the 2nd link onward.
Third, you have to make a prediction: If I create X pages with 2 links each pointing to the same destination, none of the 2ndary links will pass their anchor text to their destinations.
Fourth, you run the necessary experiments to test your prediction (and others should run them too).
If in the course of any of your (or other people’s) experiments something outside the prediction of 2ndary anchor text being discarded occurs (that is, if in one or more iterations the 2ndary anchor text is passed) you have to determine what happened and explain it.
Assuming all these links use unique expressions, are the expressions all indexable by the search engine?
Assuming all these links use indexable unique expressions, were all the first link expressions indexed by the search engine?
Assuming all the indexable unique first link expressions were indexed by the search engine, were they all passed to their destinations?
Assuming all the indexable unique first link expressions were indexed and passed to their destinations, then you have to run your tests again to see if the contrary example happens under the same conditions.
You sabotage your experiment by changing the conditions of the test to ensure failure of repetition.
Equally so you sabotage your experiment by changing the conditions of the test so as NOT to replicate the conditions of someone else’s test.
How Many Conditions Do We Need To Test?
The short answer is “all conditions”. However, you can only test as many conditions as you can test. The credibility of your testing declines as you eliminate conditions from your experiment. If someone else can produce contrary results by changing the conditions of the test, then your experiment is at best inconclusive and at worst a sham.
One test does not an experiment make. An experiment should look at as many conditions as possible. You should make an effort to isolate conditions as much as possible.
That doesn’t mean create a distinct nonsense term for every page on a new Website. New Websites are just one condition. What about old, previously existing Websites?
If you’re using nonsense terms for your SEO tests then you’re not stress-testing the search algorithms. Weak search stress may produce different results from strong search stress simply because the search algorithms have to take more factors into consideration when dealing with large document sets.
If you’re using link-poor Websites then they won’t behave the same way as link-rich Websites (and vice versa). You have to run your tests against real terms on existing link-rich Websites in order to show what will happen for real terms on existing link-rich Websites. Maybe you’ll get the same results as for your nonsense terms on link-poor new Websites, but assuming that you’ll get the same results reduces your experimentation to a sham.
In Conclusion, a Final Word
Every day I see people on SEO blogs and forums make definitive statements that are either not true or for which there is no proof of their truthfulness. These are standard, widely held SEO beliefs, such as “only the first link matters”, “Pagerank accounts for X% of relevance”, and “Google’s Panda update was directed at content farms”. A lot of this nonsense is picked up by the news media and regurgitated to the public.
This is what comes from allowing everyone to run around doing whatever they please without holding them accountable to some kind of standards. I’m not talking about certifications. I’m talking about the lack of a baseline for determining whose performance is acceptable and whose performance is sub-par.
In an environment where anyone can make an unscientific assertion, back it up with invalid tests, and toss in absurd disclaimers that everyone winks at, you cannot expect to find much high quality information.
And, frankly, there ISN’T much high quality information coming from our community.
Follow SEO Theory
Do you want more than just reposts of the week's SEO discussions and news?
Get the LARGEST weekly SEO newsletter now ...
God I love you for this post. I’ve been writing a paper on linkbuilding for the past 3 month, including a case study on the effect of different “common linkbuilding techniques” (social bookmarks, press releases, article directories and web2.0 props/freeblogs) and it is a freakin’ pain in the ass to create a somewhat meaningful test environment.
So for example, I had to buy 4 new domains, having the same name (www.name1.tld, http://www.name2.tld, ..), create content, that should have the same “onPage factors” (keyword density, title, headings, images, … – good luck doing that without creating duplicate content) in order to eliminate onpage effects as much as possible.
Of course I can’t state I took everything into account (especially since you don’t know what really counts) – but when I see tests like “how-do-tweets-influence-search-rankings-an-experiment-for-a-cause” I could start crying. Conclusion “The tweeted page ranks #1 for ‘ending hunger in sierra leone'” – guess what, that’s its page title. The title of the other one is ‘fighting hunger in sierra leone'” so I bet my a** the first one would have outranked the second one even without tweets, simply because the page title is an exact match for the query.
Oh btw. you got yourself a new RSS subscriber 😉
Cheers
Pascal
Very good post Michael! I think the problem with poor quality of information stems mostly from the fact that a. almost everybody in the SEO business feels the need to have their own blog and rehash the same information and b. most people rather read 10 posts or other short articles which excludes scientific articles. That isn’t neccesarily a bad thing as long as people remember that just because it says so online doesn’t make it true.
Besides that most people involved in the industry either work for an agency or do their work inhouse. Most of these people neither have the time or the resources to do proper scientific research.
By the way this and other articles of yours kinda reminds me of that famous quote “Half the time I spend on SEO is wasted; the trouble is I don’t know which half.” 😉
Sure, SEOMoz are not very clinical in their testing and I’m not too keen on their assertiveness on “facts” – especially when being read by their new readers… or ok, let’s face it, their comment section is always abundant with bottom feeders from the ‘Moz community, whom have been around for years. At times, certain things they preach could potentially be dangerous for the health of a website, in the wrong hands.
However, I like to view their tests, like many SEO tests; with a pinch of salt – looking at it from a broard brush stroke POV can sometimes point you in the right direction for your own results testing.
While it may be hard/impossible to grasp in full why something is happening, if the general direction is positive, then I’m sure many will try it.
“And, frankly, there ISN’T much high quality information coming from our community.”
Only a few people/sites offer any decent insight anymore, this is probably a good thing for my productivity levels…
To be honest I’m pleased that people believe certain SEO “celebrities” without testing anything or questioning as it gives me a head start! 🙂
Yet another fabulous post, Michael – thanks!
I particularly like how you so succinctly outlined the problems that arise because of the dynamic nature of the search environment. One can`t possibly claim that a given factor has influenced a search engine ranking one way or another without comparing the result for a keyword position and tested page against a keyword position and untested page: that is, without a control. Yet there are numerous difficulties in using control pages, not the least of which is the dynamic nature of the search engine environment itself (as we all know, search engine rankings change all the time without any webmaster intercession at all).
Further to your concluding remarks, as soon as I encounter any numerically-qualified result of an “experiment” I am immediately wary. “A 301 redirect results in a 10% loss in passed PageRank.” Really? These are the sorts that, back in the good ‘ole days, told us with confidence that ideal keyword density for a target phrase was 6%. Or 7%. Or 4%. Or whatever nonsense number that was tacked on as a result of a single, malformed “experiment.”
Pascal – “Conclusion “The tweeted page ranks #1 for ‘ending hunger in sierra leone’” – guess what, that’s its page title. The title of the other one is ‘fighting hunger in sierra leone’” so I bet my a** the first one would have outranked the second one even without tweets, simply because the page title is an exact match for the query.”
Is this true?! Wow. I know SEOmoz are really pushing social at the moment but thats a joke!
Good Stuff!
Look at my comments here:
“do-tweets-still-effect-rankings”
Such non-sense, I tore that experiment a new butt hole.
well said…I do SEO tests all the time and I always get different results. Coming from computer science background with a passion for algorithms, I see most SEO tests conducted by the “SEO experts” seems to be more self promoting than actually providing real value.
Cheers
Al Sefati
Great read. However, I think you need to catchup to modern times. You’re sounding like a member of the “old guard”, like maybe you “don’t get it”, perhaps you “refuse to adapt”, or maybe just “aren’t in tune” with modern reality.
“Why SEO Experiments Are Almost Always Invalid” is easily answered — because SEO experiments are marketing gimmicks. Done. Everybody knows that.
Today nothing matters… it’s all a sham. If you’re under 30 and not classically trained (lawyer, doctor, etc) there are no “good jobs” waiting for you with great pay and benefits. No one waiting to give you a spot with good pay and respect, just because you are who you are. It sucks the way things have changed (they say).
Anyone can be an SEO. Business cards cost like $10 these days. Design standards are so low, anyone can make them. Business website? Practically free. Content? Just copy it from some other “seo firm” website, change a few words to make it legitimately your own unique creation. Imagery? Cheap stock photos or even better, just crop someone else’s image to create a perfectly legit “unique version” all your own, free of copyright problems.
And if you’re really special and “creative”, and you “get it”, you can project a strong opinion around the edges of SEO issues and you’ll look like a promising service provider in little more than a weekend. With enough underemployed but awesome friends who also “get it”, you can garner social media support for your awesomeness. Let’s face it… most Powerpoint decks are built in a few days, so if you spend a full week (because you have nothing else to do), yours is likely to look professional and polished.
Next? Haha there is no next. Churn, learn, and burn. In short order you’ll be writing blog posts about “picking a niche”, creating clever names for the niche, taking side work on “only the niche part”, pitching speaking opportunities at Pubcon and SMX, and maybe even writing a “Niche Topic A for Dummies” book.
In a few years, if you play it right, you’ll be Under 30 with claims of “multiple Internet startups” behind you. And I’ll still not want to hire you, except as token C-suite “exec” for a startup destined to ride the bubble, profit from investment, or profit from the backs of underpaid, under-protected “knowledge workers” who only took the job because they saw you at the helm, and you looked so convincingly real.
It’s not pretty, but for now, it’s reality.
+1 for professional certifications.
The swamp of SEO voodoo providers will die slow. My agency has grown from $0 to $1,000,000 from referrals only. The #1 sales obstacle is “I’ve been burnt before…” and 4 months later they are selling for me. Thanks to the pool of shit-SEO providers, let them raise awareness to create a larger demand, and the pro’s will clean up.
I feel like every Donkey turned into a social media guru over the last 3 years! Bastards.
Michael,
Great subject, I’ve always appreciated your posts and the information you bring to the SEO community. You make some very valid points and while I can’t speak to the SEOmoz test, I would like to defend my own.
I agree that my test should be taken with a grain of salt. In fact, I specifically prefaced my post by stating that this test was in no way exhaustive, but it did contribute to the body of evidence available on the subject. Well knowing all the weaknesses of the test I performed, I still believe this statement is true.
There was a lot more I could have done to make my research sounder. Likewise, there is a lot that I simply could not have done, no matter how many resources were at my disposal. The algorithms are simply too complex to perform any SEO test with complete confidence in the findings.
The truth is my educational background is in Sociology, and I believe most sociologists have come to a realization that although they may never be able to perform a “perfect” test (one which accounts for all variables) the research put out still advances our understanding of society.
In the end I had to decide whether my post would benefit or mislead the SEO community. I believe, and still believe, it is useful and I hope it can be used in some future Meta-Analysis on the subject.
Edit: when I say certification I do not mean TopSEO or Google Certification junk. I’m talking series 7/Bar exam level requirements, regulated, nonprofit, authentication for SEO and SEM providers and agencies.
Christian,
I’ve never read your post, but if you look at majority of the comments on the SEOmoz post there is a lot of “yes men” who’ll misinterpret, and disseminate false information. Fueling the “myth” of snake-oil SEO providers.
I respect anyone who tries to educate. I don’t think the problem is the publisher, it’s the people.
Christian, I would rather you publish something for others to respond to than you not publish anything at all. The lack of standards that I deplore applies to me just as much as it applies to anyone else.
In my opinion, if more SEO bloggers presented case studies, we would gradually improve our communal analyses of the processes with which we work.
You keep blogging and sharing.
Anyone in SEO knows that you look at these tests and look for a basic pattern or gesture. These tests provide nothing more then a mere hint to what happens behind the magic curtains of Google even in their invalidness. But it is a hint non the less.
But alas, because of your SEO superiority, it is obvious that these lame posts and invalid tests have never once sparked an idea or made you think of something to improve your already vast SEO skills.
Tthank you for the explanation .. I want to continue to learn many things about SEO. Because the fact is, SEO is the dynamic
Thanks for this. As someone relatively new to SEO, but well-versed in the scientific method, many of the things I’ve read about search in the past year or so has bothered me for lack of scientific rigor.
It’s also the reason I have concerns about the increasing emphasis on social signals in search. Just because an idea is popular or endorsed by someone “famous” does not make it more true. Or, for that matter, true at all.
This post really garnered alot of discussion! Well done! I was involved in market research as one of my first marketing jobs and although I did not set up experiments, I do understand a little bit about the scientific method and the lack of validity in many so called “experiments” performed in the SEO community.
When I teach students or consult with clients, I notice that there are some basic truths to getting to the top of the pages, quality content, focusing on one theme or keyword phrase per page and links back to that page with good anchor text.
I don’t read about the results of many experiments, because as this article indicates they are typically not valid. The bottomline is getting results and that is what matters to my students and clients, period. This boils down to analyzing each website individually and taking action based on that analysis.
@Christian: “In fact, I specifically prefaced my post by stating that this test was in no way exhaustive, but it did contribute to the body of evidence available on the subject.”
Consistently flawed tests != evidence. Just flawed testing with bias towards Twitter being a factor by groups that are biased in that direction (whether on purpose or not).
“And, frankly, there ISN’T much high quality information coming from our community.”
Those that truly know what they are doing, who spend the time to complete proper, valid experiments just aren’t sharing is all…and how can you blame them?
I am all for teaching, but at the end of the day, said tests cost money and lots of valuable time. If you are not testing, you are not an SEO.
I say let the half-wits confuse each other with theories and ill-conceived experiments. Like Anthony said above…As long as people like this exist, my business is going to continue to thrive.
Very Great Post, Thanks
SEOmoz reste un vecteur de poids dans le SEO, mais leur expériences sont relatives, ils parlent de gros changements, pourtant j’ai des sites depuis 2000 qui sont toujours en première position dans Google sur plusieurs termes clés.
Great article Michael. We do a lot of testing in-house and a lot of our SEO team conduct their own experiments. Very rarely, if at all are their tests properly defined and executed leaving the results somewhat “inaccurate”. Great to see you bringing the scientific method into the SEO world and I only hope others who are heavily involved with SEO read this post and have their eyes opened as I have.
Thank you!
Michael, love the post and have always loved your work. But what you are basically saying here is that SEO testing IS in fact impossible. OK, so it is possible in that it is also possible for humans to fly to other galaxies through space. It is 99.9999999999% impossible, but there is that tiny sliver of possibility.
Its like you said, the environment is just too damned dynamic with way too many variables for an SEO test to be truly scientific. So while you say it is possible, in reality it really isn’t.
I believe that you stated in the comments that if SEOs provided more case studies then that would help, and I agree. But even then case studies fall by the wayside because the environment changed daily! Its all just madness.
Also, love John Andrews comment. Bold, straight up, and 1000% right on.