• Register
• Log in

SEO Theory - SEO Theory and Analysis Blog

Algorithm analysis, Web community relationship analysis, SEO practices and techniques, industry news, etc.

• Home
• About SEO Theory
• SEO Glossary

Doctor, Doctor, I’ve got a bad case of Web-page flu!

Posted by admin on December 14, 2006 in SEO Theory

Back in the day before people heard about Google, I and a few other SEO know-it-alls used to tell people that a great alternative to search engine optimization was to build up lots of inbound links from other Web sites. The traffic and visibility those links created for sites were, in some well-documented situations, more valuable than all the search referrals a particular industry experienced.

And then the news media heard about this guy, Adam Mathes, who apparently came up with the brilliant idea of pointing lots of links at blogs to manipulate Google’s search results. Link farmers were already doing that with static HTML content, and profiting handsomely, but the search engine optimization community was — until that time (about late 2001) — still skeptical of the practice of link farming.

Suddenly, with the mainstream news media talking about a technique most SEOs didn’t understand, the SEO community abandoned common sense and launched itself upon a course of self-destruction and link-bludgeoning tactics that are still going strong today. Instead of building up relationships between relevant content providers, search engine optimizers began emailing every unsuspecting Webmaster in sight, asking for links, explaining the value of linking relationships, talking about Toolbar PR, etc.

My email spam shot through the roof. I used to get more link requests than I did advertisements for Viaggatha-something. I rejoiced the day Xenite.Org’s Toolbar PR dropped from a 6 to a 5. My spam was cut in half.

All the search engine optimizers wanted links but they weren’t willing to create the content to host the links. Every possible short-cut imaginable was promoted in newsletters, forums, and SEO blogs. White papers touting the advantages of “link building” versus traditional content generation explained how you could shoot to the top of search results in no time.

The misleading facts of the link building frenzy were never mentioned. Those facts were inconvenient and annoying. And the most inconveniently annoying fact was that neither Google nor any other search engine has ever figured out a way to put more than one site into the top position of search results. Before Adam Mathes shot off his clueless mouth and gave the media a now-forgotten sound-bite, there was only 1 number 1 search result slot on all the search engines. Today, there is still only 1 number 1 search result slot on all the search engines.

The link building frenzy gave a competitive advantage to people who jumped on the bandwagon first. What link-building SEOs have yet to acknowledge (and perhaps they haven’t figured it out yet) is that many of us were already there. I don’t just mean link farmers. I mean large content sites that had great linkage. I used to create a sub-site on Xenite.Org, go out and promote it to a niche community, and then bring in traffic for that sub-site.

Do you sense anything missing from that formula? It was very frustrating and not highly productive. It shows you how naive I once was about Web marketing and promotion.

I never bothered to cross-promote my content. It didn’t occur to me that people who were interested in Farscape might also be interested in, say, The Lord of the Rings. I only had a general purpose science fiction and fantasy site, but I was overwhelmed by the need to create content about specific topics and promoting that content to specific communities. One day, as I jumped from one topic to the next in my own personal browsing, it occurred to me that there must be other people out there who, like me, had multiple interests. In fact, I realized, I occasionally ran into some of those folks in multiple forums.

Thus was born the everlasting Xenite.Org principle of “Thou shalt cross-promote all generally related topics!”

I soon realized that science fiction and fantasy television shows and movies are closely related with the entertainment industry in general, so as I added more general entertainment content I added more cross-promotional links between related sites devoted to a diverse array of subjects. My traffic zoomed as people discovered the depth of Xenite.Org. My inbound linkage and non-search referrals also zoomed.

Suddenly, it all seemed like child’s play. All you had to do was link between relevant content pages and people would follow the links.

And then Google came onto the scene. Yes, I was doing this before Google became big. And I was telling people to do this before Google became big. That doesn’t make me a genius. It just means I stumbled down the right alley in my drunken SEO stupor at the right time.

Problem is, things change. Although I have never had to look back and worry about linkage, I’ve watched traffic patterns change through the years. I get more and more direct requests for pages on Xenite.Org and across my network. Where are all these page requests coming from?

Let me change the subject.

Back in 1998 Netscape launched a service called MyNetscape. It was a great service. You’d take your little Netscape browser to their site, log in, and you had a start page. You could position groups of links on your start page. Netscape called these groups of links “channels”. Where did the channels come from? From me. That is, Netscape had created a huge directory of “commercial” or “professional” channels with lots of great content, but they also allowed people to upload their own channel feeds. They used the XML specification that they developed and called Rich Site Summary (it has now been rebranded by 2nd generation marketers as “Really Simple Syndication”).

I created several RSS feeds and fed them to Netscape. Netscape accepted those feeds and included them in its directory. Magical things happened. Among those magical things was a small trickle of traffic. A few people subscribed to my feeds and visited my Web sites. I thought, “Hey, this is pretty cool.” So I looked around for other places to submit my feeds.

Back in the day, there were quite a few services mimicking or competing with Netscape’s MyNetscape. Sadly, only Yahoo!’s MyYahoo! remains, and it was never very useful in the same way Netscape was as far as promoting new content. Still, I submitted my XML feeds to dozens of directories and start page services and syndicators. Yes, I saw lots of traffic come in.

But those services are gone now. And yet, every month thousands of robots hit my RSS files (which I still try to update even though I have to do it by hand — this ain’t no blog feed). I recognize some of the robots as coming from blog search services (even though I don’t ping them — those services must have crawled XMLTree and other feed directories in the old days).

But who do those thousands of other robots belong to?

Um…could it be these are the people who make those direct page requests?

Maybe. I’m not sure. I’m not set up to figure that out, although it would be a simple task to modify the links in the RSS files to do a little tagging and redirection so I could establish a footprint.

For years I have assumed that many of these robots come from scrapers — spammers who steal my content, republish it, and make money off the ads. I used to complain loudly and frequently about such scrapers because sometimes their pages outranked mine in the search results (they were encapsulating my optimized text with yet more on-topic text — algorithmically, their pages thus looked more original and relevant than my own).

But the scraper pages have almost all vanished, if they are not entirely gone. I’m not sure what to make of that. Did the spammers leave their bots on autopilot? Or do people really subscribe to my feeds?

I still occasionally see backlinks to Xenite.Org that come from 2nd-generation aggregator sites — call them the children or grandchildren of MyNetscape’s customized feed service. These sites, like domain parkers and cheap directories, place ads in the margins and basicallly hope to monetize my linkage. I assume I’m not getting any link love from those feed-driven pages, but I do occasionally get referrals.

The question is, should I do anything more to optimize my feeds? I’d rather not have to increase the workload for hand-crafted XML feeds. I suspect there are tools out there which I have ignored or never heard of that might simplify my task (or maybe I just need to bite the bullet and slap a content management system around Xenite).

All this feed-based activity, however, has made me wonder for the past couple of years if the desktop aggregator is not the new battleground for Web site promotion. We’ve been riding the wave of feed-based link love for years (well, I have — the rest of you may be out there spamming me with your emails though I have filters in place). I have debated, and occasionally pondered in forum and blog posts, how I should evaluate the RSS requests that Xenite gets. Are they equivalent to page views?

The page view, according to some sources, is dying in part because of technologies like Ajax (which doesn’t work with the concept of a “page view”) and in part because of aggregator technologies. People see your content in aggregators and still read your blogs without ever clicking on the links. That is why, I understand, some services now include advertising in their RSS feeds.

I once wondered how I might do that and then realized that all my feeds really are advertisements. At least, that was the original intention. Now they are content. So my advertising copy has become content copy, and my Web pages are still just plain, ordinary old ugly Web pages.

Xenite doesn’t hurt for traffic. We get plenty of traffic. Some topics have lost audience, but I add new content every year, track new trends and interests all the time, and try to keep Xenite visible in some way.

But as aggregation becomes more popular, and more sophisticated, will I have to reinvent my Web pages just so people will see them?

This isn’t about search engine optimization, but it impacts search engine optimization because redesigning Web content requires rethinking the optimization issues.

There is a finite amount of visibility that any page achieves in a search engine. If we quantify that visibility (call it the Value Search Function), we can say it has a value of VS(p) = Search Engine Visibility + Page Usefulness. Search Engine Visibility is the product of your position in the search results and the number of queries performed for all keywords to which your page is relevant. Page Usefulness is an arbitrary measurement on a scale of, say, 1 to 5, of how useful your page is to visitors. Some people will consider it to be useless, some will consider it to be very useful. This value has to reflect the overall value.

Most pages have a usefulness of 1 (with 1 being the least and 5 being the most). A Page Usefulness of 1 is not a bad thing. Spam has a Page Usefulness of 0. But the typical blog post doesn’t really offer much to most people so it’s the least useful type of content.

VS(p) therefore is not very much impacted by Page Usefulness. But Page Usefulness has some impact because — as long as some accurate information about your page is shown in search results — it can increase your VS(p) slightly. A simple URL listing (as Google provides for currently unindexed pages) provides no Page Usefulness value.

But we can now speak in terms of a Value Aggregator Function VA(p) which represents your Aggregator Visibility + your Page Usefulness. Aggregator Visibility is the sum total of all RSS feeds that include your page. There is no correlation between Aggregator Visibility and Search Visibility. And there’s the rub. If you optimize for Aggregator Visibility, do you sacrifice anything for Search Visibility?

I don’t think anyone is yet in a position to answer the question. But as more and more sites struggle to generate page-level traffic through traditional search (and the volume of daily searches masks the declining trend of page-level traffic) Webmasters are being forced (once again) to look at alternative means of generating traffic.

Cross-promotion between related content pages is still being touted, but I think that a whole new science of Aggregator Optimization is beginning to emerge because promoters have finally begun to realize that a growing segment of the online population is relying more on embedded feeds than an organic search. Trusted feed sources can direct people to new content that is useful to them. Page Usefulness, after all, is independent of the value conferred by either search or feed aggregation. A useful page is still a useful page.

It’s just that getting traffic to a page is now a more complex task than it once was for several reasons:

1. The explosion of Web content makes achieving critical visibility more difficult for newer sites.
2. The emergence of feed-based traffic requires aggregator optimization, thus depriving traditional search markets of some optimization resources.
3. Internal cross-promotion becomes more valuable if it is optimized for branding rather than for session extension (retaining the visitor longer per visit).
4. As search engines scrutinize linkage more closely, people are linking more conservatively, thus reversing the pre-Google trend of increasing inter-site connectivity.

In short, our Web pages are sick — sick with a viral affliction that strong linkage alone won’t stave off. Sites that are positioned to achieve high visibility through RSS feeds will suffer the least harm. Sites that have little to no aggregator visibility are the most likely to suffer harm. They’ll lose not only traffic but also brand value.

Even Xenite.Org has Web-page flu. I said our traffic is good. In fact, it’s very good. We get 100,000 qualified visitors a month.

Of course, a year ago we were getting 100,000 qualified visitors a month.

And two years ago we were getting 100,000 qualified visitors a month.

Do you see a trend here? I do. We’ve hit a plateau. Now, while many of the old science fiction fan sites we used to compete with no longer get as much traffic as we do (and in their heydays they got more traffic than we did), we’re not really growing our market share. Lack of growth eventually leads to implosion. I’ve been looking for new ways to build value into Xenite for the online SF community, but it may be that a steady stream of 100,000 monthly visitors is all I’ll ever get.

I don’t know.

What I do know is that if Xenite is hitting a glass ceiling, other older sites (that have not yet gone into decline) must be hitting glass ceilings, too. And new sites will struggle even more to match our Search Visibility because we have lots of content, lots of links, and dominate lots of search results.

But is it that we’re not growing traffic or is it that we’re not accurately measuring the traffic we get? Does Web-page flu really come down to metrics-analysis flu? Are the metrics even worse than many Webmasters say they are?

Metrics are the basis of marketing research. You cannot optimize for search engines or anything else without some sort of persistant measurements. Metrics tell you where you were yesterday versus where you are today so you can decide where you want to be tomorrow. Even Alexa provides at least that kind of value.

Search engine optimization relies upon metrics as much as any other aspect of marketing. And every month I see new discussions that question the value of traditional metrics. SEOs are struggling to define new baselines and ways to measure them. This is an important issue to the entire Webmastering community, but especially to the Web promotion community, because our resources are limited. We need to know where to look for the information that tells us just how sick our Web pages really are.

I’ll come back to this topic in the future.