Robots.txt File – No Website Should Be Without One
by Jerry West
Updated April 15, 2005
The Robots Exclusion Protocol --
by Jerry West
When a search engine spider or robot visits a web site if first checks for the presence of afile. If this file is found, the search engine spider or robot will analyze the contents of the file for:
The Robots Exclusion Protocol is a method that allows website administrators to indicate which parts of their site should NOT be visited by a search engine robot.
There can only be one Robots Meta Tag.file per domain. If you have users with sub-domains you must either merge all information to the one file or instruct your users to use the
Thefile is case sensitive and you should use all lowercase letters.
What To Put Into thefile
The "" file usually contains a record looking like this:
In the above example, three directories are excluded. You need to separate the "Disallow" line for each directory.
A good source is: The Robots Text Pages.
If you wish to check the syntax of yourfile, visit:
- if it is present, search engines will obey it
- without a file Google will not index your site as deep
- you cannot exclude "bad sites" using a file as bad sites ignore the file
- exclude your images folder to not allow the search engines (like Yahoo! and Google) to grab your images for their image directory
© 2000 - 2005,
Jerry West is the Director of Internet Marketing for WebMarketingNow. He has been consulting on the web since 1996 and has assisted hundreds of companies gain an upper-hand over their competition. Visit Web Marketing Now for the latest in marketing tips that are tested and proven.
The above article can be reproduced on your site or e-zine as long as the signature file.
Article Search Phrases:, robots txt, exclude search engines, disallow search engines