I'm trying to update my robots.txt file for magento installed in a folder
However from testing on Google Webmaster all files seem to open and none is getting blocked by the robots.txt file. By some hit and trail method I've reached a conclusion that it's not working because my Magento is installed in a folder rather than root. Below are examples of my testing:
Disallow: /app/- Allowed (doesn't work)
Disallow: /shop/app/- Blocked (works)
However for all Disallows I cannot prefix
/shop as some disallows are within some folders. I know I can somehow use
/ and the robots.txt will work but not sure how to use it correctly.