What is a robots.txt file?
A robots.txt file is a text file that you put in the root directory of your site that tells a search engine which pages and directories it should NOT index.
Why you should create a robots.txt file for your sites?
There are several good reasons for creating robots files for your sites and blogs.
1. You don't want some files indexed because you don't want your visitors to be able to access them through search engine searches. For example, you don't want your viewers to access download pages, ebooks or software you're selling without purchasing them. Another example is that you may not want some images on your site indexed in an image search engine.
2. You don't want some pages indexed by one or more search engines. There might be several reasons for this.
For example, WordPress automatically creates some duplicate files and you might want to block those duplicate pages from being indexed to avoid Google's duplicate content issue. You may also want to prevent pages (e.g.auto-generated pages) from showing up in the search engine results that don’t add value for users.
3. Using a robots file saves you bandwidth each time a spider visits your site.
4. Adding unnecessary files and directories from being indexed removes clutter from your web statistics and makes it easier to analyze your statistics.
How do you create a robots file?
A robots file is a text file that you create in a text editor like notepad or other. Simply lists the pages and directories you don't want indexed. Save the file as a robots.txt file and upload it to the main directory of your site or blog. That's all there is to it!
Here's a video explaining how to create a robots.txt file.
Here's an example of the content of a robots.txt file
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: */trackback
Disallow: /category/
Disallow: /tag/
Disallow: /category/uncategorized
Disallow: /login/
Disallow: /author/
If using WordPress, the files you want to block might vary depending on the theme you're using. If you're not technically inclined, you might want to ask your webmaster to create a robots.txt files for you and help you determinate which files you should block from search engines.