• Register
• Log in

SEO Theory - SEO Theory and Analysis Blog

Algorithm analysis, Web community relationship analysis, SEO practices and techniques, industry news, etc.

• Home
• About SEO Theory
• SEO Glossary

How click management works

Posted by admin on December 20, 2006 in SEO Theory

Ask.com has now apparently patented a variation of DirectHit’s technology. Ask bought DirectHit several years ago and the old click-popularity search engine hasn’t been seen much since then. Which is just as well.

DirectHit was always a stupid idea because even though intuitively it does make sense that as people evaluate Web pages a consensus of opinion will form, there are several critical flaws in judging relevance, quality, and importance on the basis of a concensus of opinion.

To begin with, most opinions are too poorly informed to provide any valid guide to quality. If 1,000 people look at a Web site that offers wrong information in a very compelling format, the odds are very good that most of those people will be convinced the Web site is correct. Call this Wikipedia Syndrome, although in reality it’s been a problem with human judgement for far longer than we’ve had an Internet.

There is some value or merit in measuring quality on the basis of popularity, but the value should not be mispackaged as both Google and DirectHit have mispackaged the value by labeling it as a quality measurement. Click-throughs on listings in a search results page don’t indicate quality of search results. They reflect the fact that, given 10 search results, people are most likely to click on the first of the 10 simply because it’s first. They assume the first result is first because it’s a more relevant result (to their query) than anything else, not because the search engine had to make an arbitrary judgement and pick something to be first.

If popularity is the basis for determining quality, then the only quality that can be measured is the value that a community places on a known resource. That is, subjective value does not quantify relevance or accuracy or thoroughness. What the search engines need to do is quantify relevance, accuracy, and thoroughness because that is what most people (wrongly) believe the search engines are trying to do for them.

But a more serious issue with DirectHit’s click-counting quality measurement is that the technology to manipulate click-throughs existed before DirectHit existed. That technology was developed to simulate page-views for banner farm networks. A banner farm was a page that was packed with as many pay-per-view and later pay-per-click banners as possible. All these pages ever had were banners. People made a lot of money off of banner displays and banner clicks without ever worrying about search engines.

The reason that people made money is they simulated page views (with banner loads) and click-throughs. Now, the banner advertising networks caught on to this manipulation pretty quickly, and as real human eyes and real human clicks became more necessary to keep money flowing to banner farm operators, they jumped into the the then new practice of search engine manipulation.

It didn’t require much of an evolutionary leap in spam page design to create doorway pages that ranked highly in search engines and which trapped users in a constant stream of autoloading pages. The spammers came right out of the pornography and gaming Web industries, where autoloading pages and automated rebills of credit cards were standard tactics. People actually went to jail for doing stuff like that.

But just as the search engine industry began to fight back against spam DirectHit came on the scene and the spammers were able to convert their click management networks to influence search engine rankings. DirectHit licensed its technology to several popular search engines. Back in those days it was common for search engines to run off of mashups based on Inktomi, DirectHit, and Netscape’s Open Directory Project. Some search engines continued to rely on their own technologies, but these three services undermined legitimate competent search technology.

The Open Directory was once a great thing because it proved to be almost as popular and influential as Yahoo! and was easier to get into (it was free and editors were responsive, friendly, and courteous). Inktomi was a nightmare to work with because if you didn’t have lots of inbound links even if you could get into the main index you quickly dropped out. And DirectHit had many people fooled into thinking its results could not be manipulated.

But, you see, I knew people who were manipulating DirectHit. I figured out how to manipulate Inktomi and they figured out how to manipulate DirectHit. Inktomi just required lots of crawl pages hosted on different domains. The way to stay indexed in Inktomi was just to have the service crawl your pages on a weekly or daily basis. The more crawl pages you had out there, the more often Inktomi found your pages. I regularly submitted crawl pages to Inktomi partners every week and my pages stayed in the index.

Manipulating Inktomi’s relevance was a little trickier because each partner site could tweak the way they served results (and many did). But just getting into the index often proved to be an advantage over competitor sites.

Manipulating DirectHit required a small bit of retooling of the old click-management software. The programs had to be distributed across multiple servers hosted in separate NOCs. They used randomized pools of IP addresses and spoofed user agents. Each instance of the software was given a queue of queries to run and URLs to click on. It didn’t matter how deeply the URLs were buried in the results. The software scraped the search results, looked for the URls, and then dug deeper until it found them.

And then the software clicked through to the desired page, waited 1-3 minutes (randomly determined), and “hit the BACK” button so that it could go back to DirectHit and run another query.

I was told that thousands of DirectHit’s users were in fact shadow users spoofed by sotfware running on a network of servers spread out across the country.

Anyone can do this. The problem is that it is expensive. My source told me that his clients were large companies who wanted to ensure that their corporate pages came up first for brand names. Now, in today’s search markets, you should be able to get corporate sites to rank for brands with a minimum of on-page optimization and inbound linkage. The search engines are smarter than their predecessors.

Still, paid advertising has never quite lost its gnatty little problem of click manipulation. Many merchants have struggled with making affiliate programs cost-effective. Click-simulating software can click on embedded affiliate links just as easily as it can click on banner links and DirectHit search result links. One proposed counter-measure was to start placing links in Javascript; another counte measure required joining large affiliate management networks that had the resources to look for patterns of fraud.

Still, the pay-per-click affiliate model collapsed under the weight of click management. Affiliate programs switched en masse to commission-based compensation. Now click-throughs don’t matter. Only sales. But Goto.com (renamed Overture and since then rebranded by Yahoo! under its own name) made click management relevant again.

Now malicious clicks have become a favored tactic since they can be used to deplete competitors’ balances quickly. Once the highest bidder drops out of the competition paying for legitimate click-throughs becomes less expensive. Many merchants complained in forums of seeing their click budgets vanish in minutes without their realizing any sales.

But even as the PPC networks cleaned up the malicious clicking, click managers turned their software onto third-party pages hosting ads. And now they can supposedly read Javascript, thanks to Mozilla. I have been told (but have not attempted to verify this) that the Mozilla page rendering software (or at least older versions of it) can be utilized by scraper robots. Hence, once the Javascript links are rendered into a virtual screen for the robots, they can click on Javascript links.

Assuming a network of servers in multiple NOCs is still available to spoof user agents and serve multiple IP addresses, massive click manipulation is not immediately identifiable. A smart robot will click on both paying links and non-paying links, spending a little more time at the non-paying links, to make itself loook like a natural surfer.

Cookies can be manipulated, Javascript can be read and cilcked, user agents can be faked, browser types can be faked, virtually everything that a normal human surfer does can be faked. We have the technology. We can build bionic spambots.

So it concerns me when search engines claim they have a handle on click management. I haven’t seen any compelling evidence to support these claims.

Neither have I seen compelling evidence to support the claims of widespread click-fraud. Both the search engines and the click auditors are working largely in a vacuum with respect to how automated click management happens. Neither group has apparently publicly investigated the small community of self-proclaimed click managers.

I find it doubtful that the average AdSense spammer is running click-manipulating robots. Most of the publicly vocal spammers demonstrate a consistent lack of technical capability. They are not real programmers who have extensive module integration and network load management experience. I just don’t feel they are capable of designing the software necessary to manage clicks.

AdSense spammers rely upon random volume. They create thousands of Web pages and spread them across the search engines, waiting for occasional click-throughs. With 10,000 pages getting an average of 3 long-tail visitors a day, a typical spammer can generate 30,000 page views a day, 900,000 page views a month. That could represent $1500-2000 in monthly revenue. With 100,000 pages, 3 visitors per day per page can easily scale up to $15-20,000 monthly revenue.

If anyone is still using click-management networks, they are probably doing so to manipulate Yahoo! search results (as Yahoo! counts click-throughs though they don’t say how much or if those click-throughs influence results) and possibly to engage in malicious PPC clicking to deplete competitor accounts. Yet despite the economies of scale, I have seen little evidence to indicate the AdSense spammers are using robots like this.

If Ask.com has been relying on click-throughs all these years, they may have been targeted by click management networks but it may be they have dodged the bullet. But if they are about to implement a new click counting methodology, then as they increase their market share the incentive to manipulate their clicks will increase. My feeling is that if Ask maintains its present growth rate it will control about 10% of the search market by the end of 2007.

Click-spoofing has been used in other contexts through the years. Click spoofers have allegedly manipulated Alexa rankings, Amazon review feedback, niche directory click counts, and competitor landing page strategies (by driving ghost traffic to less well designed landing pages).

I know that, in the past, there were people out there selling these services. I also know that small-scale click-spoofing software used to be available (at least, people talked about it). But I don’t know anyone who today is claiming to be manipulating clicks.

I just wonder what has happened to this sector of the industry. I suspect it has evolved, but I don’t know that it has survived at all. Still, if Google and Ask want to be taken seriously they need to look at click management and share what they find. They are in a position to know better than most people, but they haven’t done anything to indicate they do know better than most people.