On the goodness of self-linking and why O’Reilly is wrong

by Michael Martinez on August 19, 2008

Tim O’Reilly has put forth a suggestion to major content providers:

Ensure that no more than 50% of the links on any page are to yourself. (Even this number may be too high.)

Ensure that the pages you create at those destinations are truly more valuable to your readers than any other external link you might provide.

That’s just crazy talk.

O’Reilly points out how he and others have become concerned with the self-promotional links that news providers (and search engines) have given to their own content, implying that the news media and search media are somehow morally obligated to link out to other Web sites.

Search engines like Google have come under increasing fire for promoting their own content ahead of other people’s content. Believe it or not, this is not the first time a major search service has allowed its own content organization to encroach upon the user experience. Who else did it in the past? I could give you a list of names but think about this: those search engines no longer dominate the field.

Newspapers are laying off more and more of their staff writers. I was a bit saddened to learn that the Chicago Tribune has just laid off Michael Martinez, their Los Angeles bureau chief. I don’t know the guy but I’ve followed his online career while watching my own.

There is actually another journalist named Michael Martinez and he was laid off by the Mercury News in March. I’ve followed “Mike” Martinez’ online career for a while, too (and I wondered for a while if he and “Michael” were one and the same guy but I think not since Mike Martinez is now apparently writing for the Belleville News-Democrat).

Tim O’Reilly seems to feel that news sites should mostly link out to original source material and that search sites should promote the content originators. However, though I’ve pointed out many times through the years that search engines would not exist without people like us (content originators), I’ve also pointed out that there is more to the Web than search engine listings — and, in fact, there is more to Web site marketing than search engine optimization.

If you’re obtaining more than 20% of your traffic from search you’re doing it wrong. Now, I base that 20% on my own experience at sharing and receiving traffic from other Web sites. Maybe I’m just an anomaly, but frankly I have always been disappointed by people who dwell on a single search engine. There was a time when it was imperative for many business site operators to capture a listing on the first page of Yahoo!’s directory.

Oh! the philosophical arguments we had back then. Those people just could not see the value in building linking relationships with other sites, particularly sites that were relevant to their industries. “Why should I exchange links with my competitors?” they would demand to know. For the record, I have never actually encouraged anyone to exchange links with competitive sites (although I have conceded it might be one way to get some traffic).

When you’re selling horse shoes, you would do well to encourage every horse shoe enthusiast with a blog or a traditional Web site that discusses horse shoes to link to you. You encourage people to link to your site by providing content that they want to link to. Reading what other people write about their interests helps you figure out what kind of content you can provide to attract their attention.

When you’re organizing information for other people, however, you have to deal with an entirely different set of issues. For one thing, simply linking out to a Web site might (gasp!) send your traffic to that other site. I’ve had plenty of fruitless discussions with Traffic Hoarders through the years. These people never seem to come to grips with the reality that when you’ve grown tired of their entrapping Web site, you’re just going to close the browser window and move on with your life. They never grasp the fact that your discretionary income will be divided among many providers of goods and services, and that it’s not their money.

Web traffic entrapment has been around for years, particularly in the business community, and what Mr. O’Reilly and his philosophical comrades in words are seeing is nothing new. In fact, these sites are not merely practicing traffic hoarding, they’re also practicing PageRank hoarding. Google has become one of the worst PageRank hoarders over the past 18 months (because they have yet to strip themselves of the ability to confer PageRank despite their many violations of their Webmaster guidelines).

Google doesn’t need all that PageRank. It can provide itself with plenty of favorable positioning in, around, and above organic search results (and it does). I think the idea that Google feels its PageRank is somehow helping it is ridiculous. Google seems to be hoarding PageRank mostly to protect its supplemental results index — or, rather, to keep people like you and me from using Google resources to promote our pages out of the Supplemental Results Index into the Main Web Index.

It’s Google’s PageRank, so Google can do whatever it likes with it. Apparently, PageRank is becoming a more and more rare commodity, as Matt Cutts told domainers in 2008 that pages don’t simply get PageRank just for being pages — he said this notion is a “misconception” (perhaps in response to an idea I used to promote that suggested you only needed to create more pages to increase your PageRank).

Since Google is no longer handing PageRank to pages just for being pages (as was implied by the original BackRub paper), and since Google has moved a large portion of the Web into the non-value-passing Supplemental Results Index, and since Google is presenting more of its own content diretcly to searchers — is it any wonder that other sites might do the same thing?

However, newspapers and newsmagazines at least owe it to their readers to point out the historical coverage they have provided on specific topics. Many news site queries now appear in Google’s search results precisely because people find it useful to link to those query results (I’ve linked to site search queries for many years in many discussions on many Web sites). The news organizations that O’Reilly feels should not be linking to their own content are doing nothing different from what Amazon.com does: packaging the information they provide in a consumer-friendly format.

Web sites are under no moral obligation to link out to any other site. I note that Mr. O’Reilly did not take Wikipedia to task. Of course, he seems to believe that Wikipedia is a good thing — and you may notice that in the same article he writes “PageRank might be thought of as a way of getting millions of readers to work on the slushpile of web content, and promoting the best material to the top, where it can become professionalized.” Alas! If only the Web were as naive as Google and Tim O’Reilly want it to be.

Since Mr. O’Reilly feels compelled to pick and choose who he criticizes for self-linking (Wikipedia, of course, does allow outbound links but it violates his 50% guideline), this is clearly not a topic with an intuitively obvious high moral ground. Web sites, in fact, need to devote more of their links to internal content as they increase in size — otherwise the older content falls out of search engine listings. Web forums in particular are prone to link dementia, where the search engines can no longer find and follow the links to older discussions.

Many bloggers have learned, much to their dismay, that old posts tend to fall out of the search indexes, too (mostly because they don’t practice good internal linking). So there are certainly sound technical reasons for embedding more internal links than external links in your own content — not so much to hoard PageRank but as to help the search engines recrawl your older content and keep it available in the search indexes.

But there are also consumer-oriented reasons. A large Web site with a lot of content about a given topic needs to not only organize the data so that people can browse it, it needs to show people that the data is readily available. Links are pathways between pages, and they provide informational value when directed at internal content. To simply say, “We include an archive of older articles on our site about search market share” without including the link is a waste of both the reader’s time (how do you get to the archive?) and the content creator’s time.

The more information we share, the more we need to organize that information and the more we need to let both people and search engines know that the information has been organized.

The inevitable isolation between searcher and original content is compensated for by the continual creation of new original content outside the large self-promoting sites. That is, if people cannot find the original content through the reporting site, many of them will go looking for it on the Web. In doing so they will find new resources that they feel are more helpful to them.

That is, every Web site’s hoarding techniques tend to be self-defeating. Call that the Princess Leia Principle: “The more you tighten your grip, Governor Tarkin, the more systems will slip through your fingers”. People get tired of being trapped on Web sites and they go find new sites that link out liberally. In doing so they elevate new sites to popular status. This is how the Web has worked since the beginning. It’s not about to change just because Google can’t figure out the difference between good content and Web spam.

Every site benefits from self-linking to a certain point. That point, call it the User Inconvenience Point, is determined by the demand to know more that on-site content creates. The skimpier the content is, the more difficult it is to read the content, the more obtrusive the advertising becomes — in short, the less incentive the user has to enjoy the content experience, the more likely the user will seek detailed information elsewhere.

But most users don’t seek detailed information in the first place. Most users are not likely to click on every link, hoping to learn more at every opportunity about a specific topic. They don’t have time or they are only interested in seeing what is out there, waiting to be discovered. We’re becoming used to filing ideas away for future research, if we have the time and the need.

Hence, we need to create briefer, more memorable, more compelling content that acts as a doorway or bridge to the deeper content — and, again, large content Web sites have more incentive to link to their own personally vetted information archives than to become new generation Web directories that won’t help people shape their thoughts. A new archive is considered a more reliable source of information than, say, a personal Web site (even though the personal Web site might, actually, be created by a leading scholar in the field).

All of which is to say that a balance exists between useful self-promotion and self-defeating self-promotion. It is a good idea for large content sites to show people to their internal directories and categories because those information hubs are helpful. It is also a good idea for Web sites to link out to other resources, but since most people don’t have time to follow every link, it is better for the user experience if the news sites focus more on organizing the information they gather and less on promoting other Web sites.

The bottom line here is really the quality of the user experience. It is not easy to provide a consistent level of quality in your outbound links because, frankly, you can never know which outside source of information is the best. You can, however, work to improve your own presentation and organization so that eventually other people come to feel you have done a great job of organizing what you know — a task you can do better than anyone else will do for you, if you put your resources into the task.

We need more self-linking, not less, and we don’t need to put any limits on it. As long as people are foolish enough to believe they can hoard PageRank they’ll hurt their search optimization efforts. But as long as information archives continue to improve their organization and presentation for visitors, we’ll all be better off.

{ 0 comments… add one now }