1

We are currently evaluating to optimize our robots.txt file. Our SEO Manager requested to remove specific stuff that Magento puts there by default.

Those values are:

-# Files
-Disallow: /cron.php
-Disallow: /cron.sh
-Disallow: /error_log
-Disallow: /install.php
-Disallow: /LICENSE.html
-Disallow: /LICENSE.txt
-Disallow: /LICENSE_AFL.txt
-Disallow: /STATUS.txt

Developers don't want to remove those positions as there might be sensitive content in those files, SEO Manager says that its just a recommendation for Bots and that they'll crawl the files most likely anyway. Additionally to that some SEO Agency told our SEO Manager that there is a SEO influence in general when removing those positions (???).

Lets come to the question:

Are there any influences to the shop if we remove those files, and if so, how might they change the total SEO of the shop.

Improve this question
3

Remove them from robots.txt, and instead make sure that they are not even accessible from the web, because they should not!

Files that can be removed:

  • /install.php
  • /LICENSE.html
  • /LICENSE.txt
  • /LICENSE_AFL.txt
  • /STATUS.txt

Files that should be protected

  • /cron.php
  • /cron.sh
  • /error_log (if it exists)

If you use Apache with the latest default .htaccess file from Magento, the cron scripts already should be protected.

Test if you can access the URLs, and if it is the case, configure your webserver to deny access or send a 404 response. Examples how to do that can be found here: Recommended method to protect /downloader?

Background

For the PHP files, the reason they should not be accessible is clear: cron.php should only be executed on the command line (or via cronjob) and install.php should not be used as soon as Magento is successfully installed.

The text files are a minor security risk because they make it easy to find out your Magento version, and if they are indexed by search engines, people can search for these files to find specific Magento versions. So if you have a version with known vulnerabilities and expose that through these files, you're one of the first to be hacked.

Now somebody added these entries to robots.txt with good intentions, probably with this scenario in mind. But if the files are accessible from the web, the problem still remains. Even worse, people can search for robots.txt with these entries and then look for exactly the forbidden files. Because no bot is forced to adhere to the rules in robots.txt.

TL;DR: Removing the lines in robots.txt is better for security, as long as you also restrict access to those files, which you should anyway.

Improve this answer
1

In my opinion, these files have no impact on SEO. These files do not appear in sitemaps. These files will not be indexed. In the other way, you show interested person what you're trying to hide - just for an example ... if I find install.php there I can try to load the page and see Magento's default error ...

If there is no good reason for it, I would not add it ...

Improve this answer
0

According to this "Developers don't want to remove those positions as there might be sensitive content in those files,"

Maybe it was misunderstanding there. You don't remove them but only remove them from indexation.

If your SEO specialist want add this directive to file , agree for that.

There is a lot of opinions about this, but if you dont try you will not know. Just check after few dayes that your position dont go to much.

Google recognize magento and other CMS systems and he/it crawl directories that he want anyway.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.