How do I configure Robots.txt in Magento?

Question

I'm trying to setting up de robots.txt on our Magento store. Actually, I have no idea of the best way to set up the robots.txt.

Answer 1

Writing robots.txt is tricky. Robots defines the architecture of your website for crawlers.

Normally i use the following robots for my website

# Website Sitemap
Sitemap: www.example.com/sitemap.xml


User-agent: Googlebot-Image
Disallow: /
Allow: /media/catalog/product/

# Crawlers Setup
User-agent: *

# Directories
User-agent: *
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /media/
Disallow: /pkginfo/
Disallow: /shell/
Disallow: /skin/
Disallow: /var/

# Paths (clean URLs)
User-agent: *
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /customer/
Disallow: /customize/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /wishlist/

# Files
User-agent: *
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /api.php
Disallow: /get.php
Disallow: /mage


# Paths (no clean URLs)
User-agent: *
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?p=*&
Disallow: /*?SID=

There is a very nice article from inchoo regarding robots.txt. The link of the article is Inchoo Robots.txt article

EDIT

As per the new guidelines, it is not recommended to hide your js/css/images from google bots. Hence disallowing them will be wrong. Please remove below tags from the robots mentioned above

Disallow: /js/
Disallow: /media/ 
Disallow: /skin/

Answer 2

Simply, create a file robots.txt in the root path of your installation like this:

# Sitemap
Sitemap: http://example.com/sitemap.xml

# Google Images
User-agent: Googlebot-Image
Disallow: /
Allow: /media/catalog/product/
Allow: /media/wysiwyg/

# Crawlers Setup
User-agent: *

# Directories
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /magento/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/

# Paths (clean URLs)
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /onestepcheckout/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/

# Files
Disallow: /api.php
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /get.php
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /mage
Disallow: /README.txt
Disallow: /RELEASE_NOTES.txt

# Paths (no clean URLs)
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?p=*&
Disallow: /*?SID=
Disallow: /*?dir*
Disallow: /*?mode*
Disallow: /*?limit*

# System files
Disallow: /CVS
Disallow: /*.git$
Disallow: /*.svn$
Disallow: /*.idea$
Disallow: /*.sql$
Disallow: /*.tgz$

# PHP
Disallow: /cgi-bin/
Disallow: /cleanup.php
Disallow: /apc.php
Disallow: /memcache.php
Disallow: /phpinfo.php

Answer 3

Creating and configuring robots.txt is very easy. You can create it with simple text editor. find the best example here http://www.robotstxt.org/robotstxt.html that how to create robots.txt file.

Once you have created Robots.txt file it is supposed to submit at root directory of your webstore. For an example if your store domain is www.example.com you should put robots.txt file under the domain root where you also have app directory. So it sits and accessed like www.example.com/robots.txt

Answer 4

Put robots.txt under root directory of your Magento installation

here is the default rules robots.txt have

User-agent: *
Disallow: /

---- this will simply block your whole site to index and crawl by search engines, so better you define folders and files you want to avoid to be looked by search engines.

Here is how you can define the folders and files which you dont want to crawled by search engines

User-agent: *
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /README.txt
Disallow: /sample.html
Disallow: /sample.php
Disallow: /sample.conf
Disallow: /sample_svn.

You can also define search engines in robots.txt

User-agent: Googlebot
User-agent: Slurp
User-agent: msnbot 
User-agent: Mediapartners-Google*
User-agent: Googlebot-Image 
User-agent: Yahoo-MMCrawler

Here is how you can define which search engine to allow to crawl the site

User-agent: *
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: Yahoo-slurp
Disallow: 

User-agent: Msnbot
Disallow:

Either way you can generate your robots.txt online with just putting some basic information related to your site. http://www.robotsgenerator.com/