robots.txt and website path

Question

We are running magento but we need to have the Store View Code in our url i.e:

www.mydomain.com/shop/ if shop is our store view code.

Do I need to reflect this in our robots.txt.

For example at the moment I have:

Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/

etc but I keep getting bots crawling our wishlist. See the below example from our logs.

"GET /shop/wishlist/index/add/product/2765/form_key/PSS5nZ8XUWUVfn15/ HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Is it simply a case of adding "shop" to all our Disallow rules? How do I stop the bots crawling these pages?

Answer 1

Absolutely!

You should just add /shop/ before all of your disallow rules:

Disallow: /shop/review/
Disallow: /shop/sendfriend/
Disallow: /shop/tag/
Disallow: /shop/wishlist/