Robots.txt
This file is very important as it tells robots which files they are allowed to look at. A basic robots.txt file uses two main variables — User-agent and Disallow.
User-agent allows you to specify an exact bot and tell it what to do. For example, you could allow Google’s spider to see all of your site but disallow the Wayback Machine.
There is is a list of common bots maintained at the Web Robots Database. There is also a wildcard using an asterisk (*) which means it applies to all bots.
Disallow is the command that tells the robot which files not to view.
Example of robots.txt file:
User-Agent: *
Disallow: /report/
What this is doing is telling all bots not to view the report folder.
You can use the google webmaster account to create a Robots.txt file, if in doubt us this one.
|