History of The perfect robots.txt
Version 3 | Current version | |
---|---|---|
A robots.txt is only needed if you want to exclude a search engine or other service from some directory on your service. Bitweaver doesn't create a robots.txt file by default. If you need one, you have to create it. This file can only be in the root directory. If you cannot create such a file, use meta tags. There are pages online that discuss its value and its usage, or let you check for validity. DevelopmentThis is an example of a robots.txt during testing or developing a site. Add services like browsershots.org, validators, link checkers etc. Pages that list user agent strings tell you the possible values for the "user-agent:" part. In the following example, all user agents except one will be excluded, so no search engine will index your content before launch:/robots.txt
Live sitesWhen your site is live, you might want to exclude some services, so that nobody can submit your site there. (Doesn't make too much sense with browsershots.org, it's just an example. Also, anyone could just write their own service and crawl your site anyway.) The important part is where you exclude all (*) user agents from certain Bitweaver-specific directories:/robots.txt
See also: The perfect .htaccess file | A robots.txt is only needed if you want to exclude a search engine or other service from some directory on your service. Bitweaver doesn't create a robots.txt file by default. If you need one, you have to create it. This file can only be in the root directory. If you cannot create such a file, use meta tags. There are pages online that discuss its value and its usage, or let you check for validity. DevelopmentThis is an example of a robots.txt during testing or developing a site. Add services like browsershots.org, validators, link checkers etc. Pages that list user agent strings online resume writer tell you the possible values for the "user-agent:" part. In the following example, all user agents except one will be excluded, so no search engine will index your content before launch:/robots.txt
Live sitesWhen your site is live, you might want to exclude some services, so that nobody can submit your site there. (Doesn't make too much sense with browsershots.org, it's just an example. Also, anyone could just write their own service and crawl your site anyway.) The important part is where you exclude all (*) user agents from certain Bitweaver-specific directories:/robots.txt
See also: The perfect .htaccess file |