1

I am planning to use the following robots.txt:

## robots.txt for Store
 
## GENERAL SETTINGS
 
## Enable robots.txt rules for all crawlers
User-agent: *
 
## Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
## Set a custom crawl rate if you're experiencing traffic problems with your server.
# Crawl-delay: 30
 
## Magento sitemap: uncomment and replace the URL to your Magento sitemap file
##Sitemap: http://mystore.com/sitemap.xml
 
## DEVELOPMENT RELATED SETTINGS
 
## Do not crawl development files and folders: CVS, svn directories and dump files
Disallow: /CVS
Disallow: /*.svn$
Disallow: /*.idea$
Disallow: /*.sql$
Disallow: /*.tgz$
 
## GENERAL MAGENTO SETTINGS
 
## Do not crawl Magento admin page
Disallow: */admin/
 
## Do not crawl common Magento technical folders
Disallow: /app/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /shell/
Disallow: /var/
 
## Do not crawl common Magento files
Disallow: /api.php
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /get.php
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /README.txt
Disallow: /RELEASE_NOTES.txt
 
## MAGENTO SEO IMPROVEMENTS
 
## Do not crawl sub category pages that are sorted or filtered.
##Disallow: */*?
Disallow: */*?dir*
Disallow: */*?dir=desc
Disallow: */*?dir=asc
Disallow: */*?limit=all
Disallow: */*?mode*
Disallow: */*?p=*&

# Paths (no clean URLs)
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
 
## Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
Disallow: */index.php/
 
## Do not crawl links with session IDs
Disallow: */*?SID=
 
## Do not crawl checkout and user account pages
Disallow: */checkout/
Disallow: */onestepcheckout/
Disallow: */customer/
Disallow: */customer/account/
Disallow: */customer/account/login/
 
## Do not crawl search pages and not-SEO optimized catalog links
Disallow: */catalogsearch/
Disallow: */catalog/product_compare/
Disallow: */catalog/category/view/
Disallow: */catalog/product/view/

## Do not crawl some specific CMS pages & others
Disallow: */terms-of-sale/
Disallow: */terms-of-use/
Disallow: */privacy-policy/
Disallow: */sales/order/
Disallow: */storecredit/info/
Disallow: */invitation/
 
## SERVER SETTINGS
 
## Do not crawl common server technical folders and files
Disallow: /cgi-bin/
Disallow: /cleanup.php
Disallow: /apc.php
Disallow: /memcache.php
Disallow: /phpinfo.php
 
## IMAGE CRAWLERS SETTINGS
 
## Extra: Uncomment if you do not wish Google and Bing to index your images
# User-agent: Googlebot-Image
# Disallow: /
# User-agent: msnbot-media
# Disallow: /

But this is for multi store environment. I have following stores (as sub-directory):

http://mystore.com/
http://mystore.com/us/
http://mystore.com/uk/
http://mystore.com/de/

And robots.txt will be uploaded to the root folder.

The main thing I am concerned here is the use of * (wild card operator) for multi store environment. For example:

    Disallow: */*?dir*
    Disallow: */*?dir=desc
    Disallow: */*?dir=asc
    Disallow: */*?limit=all
    Disallow: */*?mode*
    Disallow: */*?p=*&

Are the robots.txt's contents fine for multi-store (esp. the one with * at first)?

  • 2
    No idea about robots.txt but if you want different robots.txt just rewrite them in htconf/htaccess to different targets :-) – Fabian Blechschmidt May 27 '14 at 10:20
  • @FabianBlechschmidt that would work as well. I am not sure how to do that in ngnix server – MagePsycho May 27 '14 at 11:24
  • Thankfully someone already asked and got an answer ;-) stackoverflow.com/a/18038165/1480397 – Fabian Blechschmidt May 27 '14 at 11:58
1

I have addressed this issue with a quick little bit of PHP

https://github.com/joekersh/magento-multistore-robots

.htaccess

RewriteRule ^robots.txt$ /robots.php [NC,L]

robots.php

<?php

// Find VirtualHost name and define as variable

$url = $_SERVER['SERVER_NAME'];

// Define multi-store websites url

$website1 = "website1url";
$website2 = "website2url"; 
$website3 = "website3url"; 
$website4 = "website4url";

// Test for requested website then display relevant robots.txt content

if (strpos($url,$website1) !== false) {
    include 'rbts-depend/website1.php';
}

if (strpos($url,$website2) !== false) {
    include 'rbts-depend/website2.php';
}

if (strpos($url,$website3) !== false) {
    include 'rbts-depend/website3.php';
}

if (strpos($url,$website4) !== false) {
    include 'rbts-depend/website4.php';
}



?>

rbts-depend/website1.php

<?php

$content = 
"User-agent: *
# INSERT ROBOTS CONTENT HERE"
;

echo $content;
?>

and so on...

1

At least with some crawlers, wildcards are supported, so the file should work.

From https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

Google, Bing, Yahoo, and Ask support a limited form of "wildcards" for path values. These are:

* designates 0 or more instances of any valid character
$ designates the end of the URL

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.