Noindex vs. Robots.txt
Learn from the Expert
Robots Meta Tag - When the Robots.txt Just Won't Work
by Jerry West
Updated April 15, 2005
Update: The only major search engine to consistently read the "meta tag" is Inktomi. It is suggested that you create a Robots.txt file instead - as it has been proven to be more effective.
The Robots Meta Tag allows HTML authors to indicate to visiting robots if a document may be indexed or used to harvest more links. No server administator action is required.
There are only a handful of search engine robots which read thefile. Most of them are secondary search engines.
Proper usage to index a page and follow all links:
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
Proper usage to index a page but not follow all links:
<META NAME="ROBOTS" CONTENT="INDEX, NO FOLLOW">
Proper usage to exclude a page from being indexed, but have the links followed:
<META NAME="ROBOTS" CONTENT="NO INDEX, FOLLOW">
Proper usage to exclude a page from being indexed or links followed:
<META NAME="ROBOTS" CONTENT="NO INDEX, NO FOLLOW">
The robots meta tag is NOT case sensitive.
Click for more info on meta tags and their proper use on your website.
© 2000 - 2007,
Jerry West is the Director of Internet Marketing for WebMarketingNow. He has been consulting on the web since 1996 and has assisted hundreds of companies gain an upper-hand over their competition. Visit Web Marketing Now for the latest in marketing tips that are tested and proven.
The above article can be reproduced on your site or e-zine as long as the signature file.
Article Search Phrases: robots meta tag,