It is nice when search engines frequently visit your site and index your content. But there are instances that you happen to have a sensitive data on your site and you don’t want the search engines to index these pages. Also there are times that you want to save your bandwidth therefore you have to exclude your images from indexing. The best way to inform the search engines which files and folders to avoid is to use robots.txt.
Robots.txt is a text file that you put on your site to tell search robots which pages you would like them not to visit. Though it is not mandatory, search engines generally obey what they are asked not to do.
Robots.txt must be placed on the main directory. It is very important that you put your robots.txt on the right location. Spiders do not search the entire site for the file named robots.txt before indexing. Instead they look for it on the main directory.
How to create Robots.txt file
The structure of robots.txt is pretty simple.
User-agent:
Disallow:
User-agent are search engine crawlers.
Disallow lists the files and folders to be excluded from indexing
Save your file with robots.txt as its filename. Filename is case sensitive so make sure that it is all written in lowercase.
Examples:
User-agent: *
Disallow:/
The above syntax means all search engines are not allowed to index any files or folders on site.
User-agent: *
Disallow: /sample folder/
The syntax above means that all search engines are allowed to index any files or folders on site except for the sample folder.
User-agent: Googlebot
Disallow: /
Syntax above simply means that Googlebot is not allowed to access any files on the website
Doubt if how your robots.txt work? You may check your robots.txt file by using Robots.txt checker http://tool.motoricerca.info/robots-checker.phtml.
To learn more about robots.txt visit googlewabmaster.
Note: With regards to website security, Robots.txt is not the best option since bad spiders could still index your website folders if they want to. Robots.txt is somewhat like a sign on your door that reads “please do not enter”.
See more of my posts at: archilbandada.net76
