We are running magento but we need to have the Store View Code in our url i.e:

www.mydomain.com/shop/ if shop is our store view code.

Do I need to reflect this in our robots.txt.

For example at the moment I have:

Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/

etc but I keep getting bots crawling our wishlist. See the below example from our logs.

"GET /shop/wishlist/index/add/product/2765/form_key/PSS5nZ8XUWUVfn15/ HTTP/1.1" 302 - "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

Is it simply a case of adding "shop" to all our Disallow rules? How do I stop the bots crawling these pages?

1 Answer 1



You should just add /shop/ before all of your disallow rules:

Disallow: /shop/review/
Disallow: /shop/sendfriend/
Disallow: /shop/tag/
Disallow: /shop/wishlist/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.