Robots Meta Tag – When the Robots.txt Just Won’t Work

Update: The only major search engine to consistently read the “meta tag” is Inktomi. It is suggested that you create a Robots.txt file instead – as it has been proven to be more effective.

The Robots Meta Tag allows HTML authors to indicate to visiting robots if a document may be indexed or used to harvest more links. No server administator action is required.

There are only a handful of search engine robots which read the robots.txt file. Most of them are secondary search engines.

Proper usage to index a page and follow all links:

<META NAME=”ROBOTS” CONTENT=”INDEX, FOLLOW”>

Proper usage to index a page but not follow all links:

<META NAME=”ROBOTS” CONTENT=”INDEX, NO FOLLOW”>

Proper usage to exclude a page from being indexed, but have the links followed:

<META NAME=”ROBOTS” CONTENT=”NO INDEX, FOLLOW”>

Proper usage to exclude a page from being indexed or links followed:

<META NAME=”ROBOTS” CONTENT=”NO INDEX, NO FOLLOW”>

The robots meta tag is NOT case sensitive.

Click for more info on meta tags and their proper use on your website.