Support Russ Jones’ robots.txt proposal

Posted by Michael Martinez on August 7, 2007 in General

Russ Jones suggested on the Google Cache that search engines honor a disallow-by-keyword directive in robots.txt. This is a great idea. If someone injects spam pages into your site that use keywords in the URLs, you should be able to tell the spiders to ignore those pages — thus rendering the offensive and unwanted content ineffective.

This is far more useful than “rel=’nofollow’” or spam-tattling in Webmaster Central. The search engines can enlist Webmasters’ assistance by letting the Webmasters tell the search engines what should not be considered legitimate content.

Would it require some thought and tweaking? Sure. But I would do this in a heartbeat if people were injecting spam into my own sites.

Great idea, Russ!

If you have a blog or a forum where you discuss Webmaster issues, I encourage you to discuss Russ’ idea with your visitors. Let the search community know about this proposal. Let’s give it some visibility and see what happens.

Comment

Log in or Register to post a comment.

More

Read more posts by Michael Martinez

About the Author

Michael Martinez is the Director of Search Strategies for Visible Technologies, Inc. A former moderator at SEO forums such as JimWorld an Spider-food, Michael has been active in search engine optimization since 1998 and Web site design and promotion since 1996. Michael was a regular contributor to Suite101 (1998-2003) and SEOmoz (2006).

Diary of a Mad SEO Gooroo Google outlaws the Web: Sites no longer allowed to link