Robots.txt - Magento not in root folder

Question

I'm trying to update my robots.txt file for magento installed in a folder public_html/shop/.

I've updated my robots.txt and .htaccess file based on solutions provided on below post:
Google Ignore's Magento's robots.txt

However from testing on Google Webmaster all files seem to open and none is getting blocked by the robots.txt file. By some hit and trail method I've reached a conclusion that it's not working because my Magento is installed in a folder rather than root. Below are examples of my testing:

• Fail: Disallow: /app/ - Allowed (doesn't work)
• Pass: Disallow: /shop/app/ - Blocked (works)

However for all Disallows I cannot prefix /shop as some disallows are within some folders. I know I can somehow use * or / and the robots.txt will work but not sure how to use it correctly.

Answer 1

If you installed Magento 1.x in the shop directory, you can use the robots.txt from this answer which you already found and prefix all those entries with /shop.

i.e. /app/ => /shop/app/

However, this message seems to indicate a different problem:

Fail Disallow: /app/ - Allowed (doesn't work)

It seems like you have a different application installed at document root level or at least a directory app next to shop and did not block it. This has nothing to do with your Magento installation in shop.

If there's nothing, the message is a false alert and you don't need to worry about it.