Swarm Theory And Web Communities

The National Geographic Society has probably done more to popularize the idea of Swarm Theory than any other media group, either online or offline. They caused a huge stir among Swarm Theory scientists in July 2007 with their Swarm Theory article, which explored the science from a layman’s point of view in depth.

Swarm Theory traces its roots back to the 1940s in a conjecture by John Von Neumann that there might be a self-replicating automaton. British mathematician John Conway simplified Von Neumann’s solution with “The Game of Life” (which has no connection to the Milton Bradley board game of the same name).

Conway’s game uses a small set of very simple rules for turning cells on or off within a grid structure. The game launched a revolution in artificial intelligence and the study of cellular automata. However, it would not be until scientists began figuring out ways to replicate ant behavior in business scheduling processes that people would see the connection between the simple math rules of “The Game of Life” and Swarm Theory.

Swarm Theory proposes that large groups of simple intelligences can solve complex problems very efficiently. You might stop to ask if that means the human brain is a swarm of simple intelligences working together. The question has indeed occurred to a number of people.

Swarm Theorists have studied the behaviors of ant colonies, bee hives, fish schools, bird flocks, and herds of animals to deduce the basic behaviors common to all such “social” groups. Generally speaking, the animals are only required to mimic the behaviors of their neighbors, to respond to stimuli from their neighbors, and to share vital stimuli with their neighbors.

Swarm Theory tells us that as long as the members of a group behave responsibly (do whatever they do without going rogue) the group tends to benefit in a number of ways. What all the groups have in common is that they invest resources in large scale experimentation, winnowing out undesirable choices from desirable choices.

Here are three examples of how the process works:

1. Ants find the richest food source by following the strongest pheromone trail, which is laid down by the largest number of ants following the same path.
2. Bees choose a new hive location on the basis of which potential new hive location first attracts 15 advocates.
3. Fish and Elk evade predators by fleeing in as many directions as possible, thus overwhelming the predators’ abilities to choose from many targets.

Swarms behave in very specific ways although we’re still learning about the fundamentals of the theory. For example, whereas for years Swarm Theorists have maintained that swarms have no real leaders, recent work shows that swarms follow leaders who take risks. Intuitively, it makes sense that occasional risk-taking would be rewarded by a swarm because it may help a swarm find a new safe haven, a new food source, or an escape path.

Swarm members are only expected to perform a few tasks. Their level of intelligence does not matter as long as they act like a swarm. For example, one experiment showed that trying to organize a swarm is counter-productive. When 81 volunteers attempted to solve a Sudoku puzzle, they completed 90% of the work within about 7 minutes but spent another 8 minutes executing a spectacular failure after 1 person decided to take charge.

1 person can certainly solve a problem like a Sudoku puzzle, but the experiment indicated that letting everyone figure out their place without direction would provide a solution sooner.

Life doesn’t always imitate science, probably because we do spend a lot of time trying to solve problems by ourselves rather than together. The lesson to be learned from Swarm Theory seems to be that extremely complex challenges (such as propagating a species, ensuring species survival, etc.) are most efficiently solved by engaging in a massive amount of simple, low-level experimentation and then propagating the best choices throughout a group.

Maybe we can cure cancer sooner by funding more research.

But what does all that have to do with the Searchable Web Ecosystem, you might reasonably ask. Think about how broad your reach is into the SEO Web. The SEO Web consists of all the blogs, forums, personal Web sites with articles and tutorials, and all the free article and press release sites that publish information about search engine optimization.

If a site has information about SEO techniques, methods, and/or metrics then it is part of the SEO Web. It’s a big Web. Several people have attempted to document the SEO Web in several innovative ways.

Okay, none of them are particularly innovative but they only have to be different from each other enough to provide us with some options. The three sites I link to are, in my opinion, the most successful SEO Web hubs. That provides some validation for the much-maligned PageRank Theory, which is based on Citation Analysis. That is, I’m editorially linking to/recommending three Web sites.

Now here is where Swarm Theory enters the equation. You can also recommend three SEO hub sites within your own blog, forum, tutorial, etc. So can your friends, their sisters’ cousins’ best friends’ mothers, and your neighborhood street person.

When enough people make recommendations (with or without nofollow, which is irrelevant in this context), a tendency toward consensus forms. Unfortunately, we’re self-aggrandizing human beings and we’re not likely to actually form a consensus but when enough people look at a Web site they’ll either Yea or Nay it to untold heights of popularity or the nether regions of anonymity.

In other words, we don’t choose our hubs on the basis of what works best, but rather on the basis of what most people recommend. I don’t necessarily endorse Sphinn, AllTop, or The Big List — I’m just telling you they exist and that I feel they are pretty good SEO hubs (allowing for some variation in quality). Why I like them is irrelevant. I have spoken.

Now it’s your turn. And then it will be your friends’ turns. You only get to speak, not debate. Debate is non-speak. In other words, the debate doesn’t occur at the individual level but rather at the swarm level. The group evaluates the pros and cons of all the available Web sites and picks the hubs it is most likely to benefit from as a group.

That might explain why the SEO community is so in love with Google. It’s not that Google is such a great search engine as the SEO community simply derives more value from optimizing for Google than for optimizing for other search engines.

All that means is the other search engines have not provided us as much value as Google, but that factor can change at any time. If any of the factors in the equation change then the results of the equation change. And the changes can be described in the simplest terms possible.

• Google can increase the value it provides to SEOs.
• Google can decrease the value it provides to SEOs.
• Live can increase/decrease the value it provides to SEOs.
• Yahoo! can increase/decrease the value it provides to SEOs.

And so on. Every time a factor changes, someone will check out the new options. If there is some enthusiasm for the new options that someone will tell other people, some of whom will also check out the new options.

A critical mass may be reached where so many people check out the new options that the whole swarm starts moving in a new direction. We’ve seen this happen many times through the years with the SEO community. Only a few years ago you needed to be listed in the Yahoo! directory. Now it’s a nice link to have if you can get it. Once it was important to get into DMOZ. Now it’s hardly worth the trouble to submit to DMOZ.

Altavista was the king of the search engines before Google. Infoseek was the king before Altavista.

Change is the only constant in our industry and it happens all around us on a daily basis. Every time a new search engine is launched people investigate it. Through the past 12 years I have used more search engines and directories than most SEOs could name — but the vast majority of them have never developed large audiences. Too few people recommended them.

Opinions may be influenced through the Theory of Viral Propaganda, in which large groups reach a consensus by making simple choices (every member of the group must choose between A and B). When the group has reduced its choices/choosers to two groups, one sub-group for A and one sub-group for B, the larger group usually wins out (or a splinter group breaks away).

Swarm Theory offers us another way to look at how the SEO community chooses its methodologies. Instead of recommending specific Web sites as SEO hubs, I might instead suggest that you go to site X to obtain some value-passing links. You, in turn, might consider that to be dubious advice but then you see Rand Fishkin and Aaron Wall talk about site X and you start to think, “Hm. There must be something in this site X deal.”

It’s not that Michael Martinez, Rand Fishkin, and Aaron Wall are that influential. It’s that the more agreement you perceive among the people whose opinions matter to you, the more likely you are to agree with that perceived agreement (even though we may not be agreeing for any reason other than that we are each following in the footsteps of three other influencers).

In fact, both Rand Fishkin and Aaron Wall have great reputations for promoting ideas and resources put forth by other people. You could easily lump many other SEO pundits in with them. Most people in the SEO community tell you about their latest discoveries and why they like them. Every day I read articles on SE Roundtable, Search Engine Journal, Search Engine Watch, Search Engine Guide, and a few other sites where people tell us about new things to check out.

Swarms fundamentally rely upon the process of discovery, testing, and comparing test results to make critical decisions. Some ants will leave the colony and go roaming around with no apparent purpose in mind. These “patrollers” convey to the rest of their colony that the area is safe (because they made it back alive). Some ants check with multiple patrollers before leaving the colony to go forage for food. These ants come back and tell other ants that they found food (and some of the other ants will then follow their trail back to the food location to see if there is more food).

Search engine optimizers form communal decisions about linking resources (DIGG, Stumble, Twitter, Wikipedia, forums, blogs, Delicious, etc.) on the basis of their shared successes and failures. Some rogues and young whipper-snappers are unaware of the communal decision but anyone interested in Best Practices SEO eventually tunes in to the buzz and figures out where you can and should not invest your time in building links.

You’ll do okay as long as you don’t try to out-think the group. If you do try to out-think the group, you had better be on to the right solution (for the moment). Otherwise, you’ll be ignored by most people in the group. That’s not a sign of your failure as a leader. It just means you are not sitting at a pivotal moment where you have an opportunity to become a key influencer.

By the time an idea is endlessly repeated on every SEO blog you’ve never heard of, it’s usually one of the fundamental principles or a really very bad idea that has gone viral. SEO memes are among the worst bad ideas that go viral because people don’t stop to think about what they are doing. That’s the point of the meme — you’re participating in a social behavior where you repeat the social behavior of the person before you and pass the social behavior on to one or more other people.

SEO memes often turn out to be toxic because they are intentionally manipulative and I’ve watched whole blog communities incur some pretty nasty penalties simply because the bloggers thought it was okay to participate in the memes.

Not every meme is bad, mind you. It’s just that they are high risk behavior and any group that depends on high risk behavior won’t survive long. The larger the group is the more conservative its choices tend to be. Risky choices are made only when one or more members becomes desperate. When you have nothing to lose you have everything to gain, and if you benefit from a risky decision your neighbors may choose to follow you.

What happens next depends on how acceptable the risk seems to other members of the group. The fewer people (taking risks) who incur any sort of penalty, the more reassured other members of the group become. Nonetheless, some SEO “best” practices of the past are now frowned upon because the search engines became disenchanted with them. Hence, even when the majority of the group follows an early risk-taker’s lead, there is no guarantee the group will benefit in the long-term from the risky behavior.

And every setback, especially the larger ones, results in more conservative group-think. We collectively become more cynical about risk-taking with our search engine optimization. We stay with the swarm, where it’s safe.

However, in a swarm — as on the Web — it’s impossible to see what the crowd is really doing. The leaders of the swarm may be flying toward the pretty red colors not realizing they’re about to fly into an erupting volcano, but the rest of the group — having no idea of what lies ahead — is just following along.

In a swarm you do as your neighbors do, giving them just enough room to be experimental for your benefit and staying close enough that you can capitalize on their changes in direction.

In a swarm you share information (I failed doing this, I succeeded doing that) and information comes back to you (I also failed doing this, I also succeeded doing that) to help the group make good choices.

In a swarm you may perform only a small number of functions: patrolling (alerting the swarm to potential problems), engineering (you build new infrastructure for the swarm), foraging (you find new resources for the swarm), scouting (you experiment), defense (justifying or explaining the group’s existence). In each of your roles you have a viral function to fulfill as well: you must share information with other members of the group.

Every task performed benefits the group only if it serves to validate a previously performed task or if it is validated by subsequent tasks. It could be said that the group benefits indirectly from failure by not following the path of failure, but the swarm really follows the path of unpenalized success over any length of time.

To just survive, watching what your neighbors do and doing what they do usually works best in a swarm but every swarm loses members. That holds true with the SEO community as well. Some people start taking risks too soon and they suffer the same consequences as the risk-takers they follow.

Science has yet to decide whether swarm behavior is a normal part of human behavior, but I suspect that we’ll find that swarm theory documents a long-existing aspect of human behavior.

In the end, maybe what sets us above all other social creatures on this Earth — including the ecosystem changing creatures like rats, beavers, grazing cattle, ants, and bees — is the fact that we (and only we) have become rational enough to create weapons that are capable of obliterating nearly all life on Earth.