Many historical Internet marketing perspectives have been written through the years, but all the SEO histories I have read leave me feeling something is lacking. They don’t look at the contributions made by pioneers and the forces of evil. With the dark side of SEO, there is no light side.
Today, search engine optimization is all about achieving the most productive placements for content. In some contexts, you actually are not concerned with the number 1 placement. But in principle it’s really about creating visibility for one of three reasons: to make money, vanity, or to help other people find useful and valuable information.
In its infancy, the Internet was created to share information without the hindrance of commercial and vanity priorities. But the Internet was not the only networking solution developed in the 1970s and 1980s. University systems began linking their computers together to create statewide networks. Other multi-agency entities also created large networks, at considerable expense, which enabled thousands of people to share resources without ever seeing each other.
Soon after email was devised people began sharing public messages in transient discussion channels. News groups and Internet Relay Chat became popular information sharing resources. But their data never lasted very long. Servers were continually dumping data as precious disk space ran out. To compensate for the lack of server space and for the transient nature of news groups, Frequently Asked Question documents were created to informally codify and index the most valuable information discussion communities could share.
In the early 1990s, as the Internet became more of a public resource through commercial access points, whole libraries of F.A.Q.s guided newcomers through the process of learning about thousands of topics. News group client tools incorporated basic text scanning features to help people find critical information.
During the 1980s and 1990s private bulletin board services became popular. Anyone could set up a dialup BBS if they had a PC, a modem, and a phone line. I had a friend who ran a BBS out of his living room. People were constantly dialing in and browsing the file archives, the message areas, and reading his online news and announcements. He operated a living-room wide web that was freely accessible to anyone else who could bear the expense of logging in.
Commercial bulletin board services like Genie, Compuserve, and Dialnet offered industrial-grade BBS service to commercial and residential subscribers. Users could exchange email, post messages in public discussion groups, engage in real-time chat, and create file archives as well as search their respective services for information. Libraries and archives had very limited indexing abilities compared to the directories and search engines that would eventually replace them, but the elements of online search and the need for search optimization were in place before we had a World Wide Web.
Of course, in 1989 Tim Berners-Lee developed software that converted the Internet into a worldwide BBS. The chief innovation Berners-Lee provided was the formalization of segmenting a portion of the Internet into “Web documents” constructed in HyerText Markup Language. But the Web was viewed as only one aspect of the Internet. Berners-Lee wanted to make both the search for information and its organization and presentation more user-friendly. He was merely improving the quality of the user experience.
By the mid-1990s, fully functional Web browsers gave you the ability to read email, browse news groups, and view Web pages as well as retrieve files from FTP archives. In 1993 the first search engines, including Aliweb, came online. Yahoo! came online by 1994. And about that time Jim Wilson created Virtual Promote.
Most search engine optimizers can trace their online legacy back to Virtual Promote, but there were other forebears. For example, the early dialup BBS world was actually commercialized by intrepid distributors of what remains one of the most popular online commodities today: pornography. Before there was a Web, online pornographers had to create visibility for their BBS systems through offline channels as well as online channels.
Pornography migrated to the news groups, which allowed binary content although often for only the briefest of times in specially designated groups. The BBS operators uploaded their teaser images to news servers in the hope of attracting new customers. The World Wide Web only made it easier for them to reach larger audiences and increase their cash flow.
Traditional Web advertising began in 1994 when Hotwired and other Web sites launched with advertising from major corporations. People began organizing banner networks, soe for free and some for pay, and click-fraud quickly followed. Pay-per-view banner ads were gamed by people who draw unsuspecting visitors to their sites and then launched as many windows as possible. Pay-per-click banner ads were gamed by people who tricked visitors into clicking on ads, or triggering click-emulation software.
1994 also saw the birth of news group and email unsolicited bulk advertising, forever changing the way people exchange messages. Laurence Canter and Martha Siegel destroyed the non-commercial Internet and created the commercial Internet with the press of a button. Today, 90% of all emails exchanged are considered to be worthless, unwanted spam messages — although people click on approximately 4% of those messages. News groups continue to be plagued by unsolicited bulk messages, as are Web forums, mailing lists, blogs, and search engines.
Yahoo! began monetizing search in 1995. For the year of 1996, 15.2 million households spent $1.3 billion and advertisers spent $301 million online.
1996 was the year people began selling their services for what in 1997 John Audette called “search engine optimization” for the first time. Popular search engines at the time included Altavista (launched by Digital Equipment Corporation), Hotbot (powered by Inktomi), and Infoseek. WebConnect created unique tracking URLs to track click-throughs for Web advertisements. GM created the largest Web presence by expanding its content to 38,000 pages.
Two other significant developments occurred in 1996. Jim Wilson launched his search engine forums and two Stanford University students began research into using citation-based weighting to influence automated document indexing. Wilson’s growing Web community spun off thousands of other Web communities and resources: forums, niche directories, competitive SEO communities, and even a few new search engines. The PPC alternative to what we now call “organic search” was launched with Wilson’s help by a small company then called Goto — who later rebranded as Overture and were acquired by Yahoo!
By summer 1998, when I joined the Virtual Promote community, the basic concepts of Web site optimization had already been established. Some old school spam techniques — such as embedding keywords in HTML comments and the keywords meta tags — were just then beginning their decline. By 2000 these spam techniques were no longer effective. On the other hand, some spam techniques were just being devised in those days.
Doorway pages, hallway pages, crawl trees, and link popularity networks were all being used by 1998, but we had not yet invented link farms. Nor had the full power of outbound linkage yet been documented. At the time, outbound linkage made a site into a useful resource and it helped to build non-search traffic. Actually, outbound linkage still does that today.
Yahoo! directory listings were considered vital to online success, and getting a “sunglasses” (cool site) icon placed beside your listing was deemed a great coup (I still have one). People agonized over the fact they could not submit titles starting with “AAA” and they cursed Yahoo! for not allowing them to use keywords more than 2 times in their descriptions. Category placements were also sources of angst and consternation.
Rich Skrenta and Bob Truel created the Gnuhoo directory in 1998 and by the end of the year Netscape acquired the service and rebranded it as the Open Directory Project. A small search engine using citation-based weighting for its indexing was launched by two Stanford University students named Sergey Brin and Larry Page. At the time, the Yahoo!-Inktomi combination was slaughtering all competition except for Altavista, but Altavista was being slaughtered by doorway pages and Amazon listings.
In 1999 search engine optimization was wrangling with the clash of the commercial Web and the non-commercial Internet. Many newly born online marketers refused to acknowledge the facts that the Internet was not created for their benefit, that most people were not searching for products and services to buy, and that search engines didn’t really benefit much from indexing monetized sites. In fact, the search engines were actively delisting most monetized sites (because they were mostly doorways for banner farms, affiliate stores, and pornography).
Based on many frustrating conversations I had in 1999, back then typical business site operators seeking help in SEO forums:
- Objected to exchanging links with other business sites
- Saw no value in linking out to informational Web sites or creating mini-directories for their visitors
- Believed all traffic had to be delivered to their sites for free
- Believed “if you build it they will come”
- Felt there was endless opportunity for anyone who had an idea to make money on the Internet
- Doubted the usefulness of putting their telephone numbers and addresses on their Web sites
- Expressed great skepticism about the value of building forums and online communities
- Thought there was no harm in posting bulk unsolicited announcements about their Web sites in news groups, forums, and mailing lists
Today, things are a little different.
Things began to change for the SEO world around 2000. Community fragmentation set in as the “old guard” became unpopular with many newer SEOs. The question of “SEO ethics”, which had been around for years, began to creep into many discussions and Humpty Dumpty took a great fall. Open hostility between some search optimizers and search engines occasionally flared up in media articles and often in online discussions.
And then the Dot-Com Meltdown happened. The Internet Bubble burst. Thousands of online businesses failed. Many people bailed out of search optimization, for a while if not forever. But after the storm had passed and people began picking up the pieces, new commercial incentives began creeping back onto the Internet. Some of the shaky survivors of the Meltdown solidified their successes. The old affiliate-and-advertising model of generating revenue was briefly replaced by a new generation of actual service-and-product pioneers. But their needs spurred a new investment in search optimization.
The golden age of link spam was born in late 1999 or early 2000 with the creation of automated link farming, reciprocal linking, and link scraping. What, you never heard of link scraping? Maybe you heard of people who sent little robots out to scour the Internet for sites just like theirs to include in their brand new directories. They would add those links to their directories and send you little emails asking for links back. Link scrapers were filtered out quickly. Link farmers lasted a while longer and link reciprocators are still active today (although how much reciprocation actually helps is a matter of considerable debate).
Doorway pages were transformed into link gateways. Now instead of holding just one red flag link, a doorway page could be used to link to many other pages. Hallway pages and crawl trees became less important as doorway networks started to create their own mini-webs. They faded into the natural Webscape so quickly the search engines had no idea of what hit them.
Nonetheless, by 2002 Googe had become a major search engine resource, Altavista was on its last legs, Yahoo! was struggling to hang on to its leading market share and develop new monetization sources, and SEO had devolved for many people to two techniques: link spamming and directory placement. Directory listings were considered mandatory for any serious business because they appeared to earn significant weighting in both Inktomi and Google. Everyone else had to resort to link spam, including link farms and reciprocal linking.
And then coined the expression ‘Google bombing’, completely ignorant of the fact that it had already been formalized as link farming and link scraping. But now Google Bombing (also called Link Bombing) became the so-called optimization method of choice for many people. They slaughtered blog comments, forum discussions, guest books, and other free link sources with unwanted bulk postings of worthless links.
Meanwhile, from the years 2001 through 2003 or thereabouts the major search engines competed on the basis of “index size”. The search engines kept one-upping each other until Google finally hit the 1 billion page mark. Then no one really cared any more. But Inktomi’s old two-index system (which gave rise to link farms in the first place) had only just barely died a much-needed death when Google decided in 2003 to introduce the Supplemental Index for pages it didn’t know what else to do with.
The golden age of Link Spam was fast coming to an end, though, because in early 2004 Google introduced what became called the Sandbox Effect. For more than a year people thought Google was penalizing new Web sites when in fact it was simply distrusting easy-to-get links. The age of Link Trust was born but Google only reluctantly shared information about what it was doing for nearly 2 years. Yahoo! quietly demoted its directory to a secondary status after acquiring Inktomi, the Open Directory Project became the focus of increasing abuse and criticism, and people stopped emphasizing the importance and value of optimizing directory listings.
Now the spammers were using RSS feeds and DMOZ databases to create fake directories for the purpose of carrying PPC ads (which Google had begun distributing in direct competition with Yahoo!). And the SEO community divided itself into “white hats” (organic optimizers and PPC managers) and “black hats” (dirty rotten no-good search engine spammers who, not coincidentally, managed to figure out all the new techniques that organic optimizers needed to know months or years ahead of the general SEO community).
By early 2005 it was obvious that PPC ads were the primary reason for the creation of the majority of spam. The search engines had become trapped on a treadmill of their own creation. Trust has become increasingly important. People now speak in terms of owning trusted domains, acquiring trusted links, passing trust, and even valuing trust. Some trusted sites are more valuable than others to some people.
But the age of Link Trust is, I think, about to close. I believe we have already begun to see the advent of Trust Spam. As search engines now provide both increased value and incentive for people to monetize their trusted content, it will become increasingly difficult to distinguish between merely editorially chosen links and influenced editorially chosen links. The Federal Trade Commission entered the game in late 2006 by requiring full disclosure for compensated endorsements, but what about uncompensated endorsements? If you draw a line between compensation and anything else, people will find ways to benefit from the “anything else”.
Link baiters may see a golden age of creating useful, inventive content for a while but alerady some link baiters are starting to look formulaic. And we have learned that not all link baiters are true link baiters. Some, if not all link baiters, have relied upon seeding social linking Web sites, even manipulating their resources. That isn’t true link baiting. Nor is it true search engine optimization. It’s just another form of spam. Eventually, we’ll have White Linkers and Black Linkers, White Baiters and Black Baiters.
The commoditization of Trust follows upon the commoditization of click-throughs, displays, and links. It’s all about putting money into someone’s pocket. That is because the quest for achieving online visibility has shifted its emphasis from helping people to generating an income. Even vanity has now been monetized in a hundred different ways. SEO theory is all about creating visibility and value for monetizable content.
Legitimate benevolent content still needs optimization, too. And there are, in fact, some optimizers who help non-profit institutions optimize their sites. However assisting non-profits with optimization is not necessarily the sole means of income for these SEO firms. They still have to pay the bills and non-profit organizations aren’t usually search engine dependent.
Some Trust will undoubtedly be misused, and whether we end up speaking about Black Trust versus White Trust (although I would prefer Red Trust versus Gold Trust) remains to be seen. I think we can safely say, however, that search engine optimization has evolved to a much more sophisticated set of skills than people had to master 8-10 years ago. Deep SEO Theory — what I call the “second tier” stuff — is far more concerned with relationships, networking, and organizing information than the first-tier SEO theory that still dominates the industry today.
You tend to move into the realm of second-tier SEO when you stop thinking about what works today and start asking yourself, “What will I need to do tomorrow?” Some people take the position of, “If you worry about the future, you’ll miss out on the present”. For most SEOs that is most likely true. But in a competitive world, those who are prepared for the inevitable changes in strategies, demands, and technologies tend to survive the best.
Evolution, of course, is not about “survival of the fittest” but rather the “survivors of the calamities of the ages”. In search engine optimization, there will be calamities every year. Those who learn to adapt and survive will succeed. If you’re going to be serious about SEO Theory, then you really do need to study its history. Know where it came from. Find out who has benefitted from it. Understand why the search engines continually strive to stay one or two steps ahead of the SEOs.
{ 0 comments… add one now }
You must log in to post a comment.