We are running magento but we need to have the Store View Code in our url i.e:
www.mydomain.com/shop/ if shop is our store view code.
Do I need to reflect this in our robots.txt.
For example at the moment I have:
Disallow: /review/ Disallow: /sendfriend/ Disallow: /tag/ Disallow: /wishlist/
etc but I keep getting bots crawling our wishlist. See the below example from our logs.
"GET /shop/wishlist/index/add/product/2765/form_key/PSS5nZ8XUWUVfn15/ HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
Is it simply a case of adding "shop" to all our Disallow rules? How do I stop the bots crawling these pages?