Magento robots.txt for multi store

Question

I am planning to use the following robots.txt:

## robots.txt for Store
 
## GENERAL SETTINGS
 
## Enable robots.txt rules for all crawlers
User-agent: *
 
## Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
## Set a custom crawl rate if you're experiencing traffic problems with your server.
# Crawl-delay: 30
 
## Magento sitemap: uncomment and replace the URL to your Magento sitemap file
##Sitemap: http://mystore.com/sitemap.xml
 
## DEVELOPMENT RELATED SETTINGS
 
## Do not crawl development files and folders: CVS, svn directories and dump files
Disallow: /CVS
Disallow: /*.svn$
Disallow: /*.idea$
Disallow: /*.sql$
Disallow: /*.tgz$
 
## GENERAL MAGENTO SETTINGS
 
## Do not crawl Magento admin page
Disallow: */admin/
 
## Do not crawl common Magento technical folders
Disallow: /app/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /shell/
Disallow: /var/
 
## Do not crawl common Magento files
Disallow: /api.php
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /get.php
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /README.txt
Disallow: /RELEASE_NOTES.txt
 
## MAGENTO SEO IMPROVEMENTS
 
## Do not crawl sub category pages that are sorted or filtered.
##Disallow: */*?
Disallow: */*?dir*
Disallow: */*?dir=desc
Disallow: */*?dir=asc
Disallow: */*?limit=all
Disallow: */*?mode*
Disallow: */*?p=*&

# Paths (no clean URLs)
Disallow: /*.js$
Disallow: /*.css$
Disallow: /*.php$
 
## Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
Disallow: */index.php/
 
## Do not crawl links with session IDs
Disallow: */*?SID=
 
## Do not crawl checkout and user account pages
Disallow: */checkout/
Disallow: */onestepcheckout/
Disallow: */customer/
Disallow: */customer/account/
Disallow: */customer/account/login/
 
## Do not crawl search pages and not-SEO optimized catalog links
Disallow: */catalogsearch/
Disallow: */catalog/product_compare/
Disallow: */catalog/category/view/
Disallow: */catalog/product/view/

## Do not crawl some specific CMS pages & others
Disallow: */terms-of-sale/
Disallow: */terms-of-use/
Disallow: */privacy-policy/
Disallow: */sales/order/
Disallow: */storecredit/info/
Disallow: */invitation/
 
## SERVER SETTINGS
 
## Do not crawl common server technical folders and files
Disallow: /cgi-bin/
Disallow: /cleanup.php
Disallow: /apc.php
Disallow: /memcache.php
Disallow: /phpinfo.php
 
## IMAGE CRAWLERS SETTINGS
 
## Extra: Uncomment if you do not wish Google and Bing to index your images
# User-agent: Googlebot-Image
# Disallow: /
# User-agent: msnbot-media
# Disallow: /

But this is for multi store environment. I have following stores (as sub-directory):

http://mystore.com/
http://mystore.com/us/
http://mystore.com/uk/
http://mystore.com/de/

And robots.txt will be uploaded to the root folder.

The main thing I am concerned here is the use of * (wild card operator) for multi store environment. For example:

    Disallow: */*?dir*
    Disallow: */*?dir=desc
    Disallow: */*?dir=asc
    Disallow: */*?limit=all
    Disallow: */*?mode*
    Disallow: */*?p=*&

Are the robots.txt's contents fine for multi-store (esp. the one with * at first)?

Answer 1

I have addressed this issue with a quick little bit of PHP

https://github.com/joekersh/magento-multistore-robots

.htaccess

RewriteRule ^robots.txt$ /robots.php [NC,L]

robots.php

<?php

// Find VirtualHost name and define as variable

$url = $_SERVER['SERVER_NAME'];

// Define multi-store websites url

$website1 = "website1url";
$website2 = "website2url"; 
$website3 = "website3url"; 
$website4 = "website4url";

// Test for requested website then display relevant robots.txt content

if (strpos($url,$website1) !== false) {
    include 'rbts-depend/website1.php';
}

if (strpos($url,$website2) !== false) {
    include 'rbts-depend/website2.php';
}

if (strpos($url,$website3) !== false) {
    include 'rbts-depend/website3.php';
}

if (strpos($url,$website4) !== false) {
    include 'rbts-depend/website4.php';
}



?>

rbts-depend/website1.php

<?php

$content = 
"User-agent: *
# INSERT ROBOTS CONTENT HERE"
;

echo $content;
?>

and so on...

Answer 2

At least with some crawlers, wildcards are supported, so the file should work.

From https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

Google, Bing, Yahoo, and Ask support a limited form of "wildcards" for path values. These are:

* designates 0 or more instances of any valid character
$ designates the end of the URL