Menu Menu

Coding Tips

Noindex vs. Robots.txt

Learn from the Expert

Robots Meta Tag - When the Robots.txt Just Won't Work

by Jerry West
Updated April 15, 2005

Update: The only major search engine to consistently read the "meta tag" is Inktomi. It is suggested that you create a Robots.txt file instead - as it has been proven to be more effective.

The Robots Meta Tag allows HTML authors to indicate to visiting robots if a document may be indexed or used to harvest more links. No server administator action is required.

There are only a handful of search engine robots which read the robots.txt file. Most of them are secondary search engines.

Proper usage to index a page and follow all links:

<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">

Proper usage to index a page but not follow all links:

<META NAME="ROBOTS" CONTENT="INDEX, NO FOLLOW">

Proper usage to exclude a page from being indexed, but have the links followed:

<META NAME="ROBOTS" CONTENT="NO INDEX, FOLLOW">

Proper usage to exclude a page from being indexed or links followed:

<META NAME="ROBOTS" CONTENT="NO INDEX, NO FOLLOW">

The robots meta tag is NOT case sensitive.

Click for more info on meta tags and their proper use on your website.

------
© 2000 - 2007, WebMarketingNow.com
Jerry West is the Director of Internet Marketing for WebMarketingNow. He has been consulting on the web since 1996 and has assisted hundreds of companies gain an upper-hand over their competition. Visit Web Marketing Now for the latest in marketing tips that are tested and proven.

The above article can be reproduced on your site or e-zine as long as the signature file.

Article Search Phrases: robots meta tag, robots.txt