We are currently evaluating to optimize our robots.txt file. Our SEO Manager requested to remove specific stuff that Magento puts there by default.
Those values are:
-# Files
-Disallow: /cron.php
-Disallow: /cron.sh
-Disallow: /error_log
-Disallow: /install.php
-Disallow: /LICENSE.html
-Disallow: /LICENSE.txt
-Disallow: /LICENSE_AFL.txt
-Disallow: /STATUS.txt
Developers don't want to remove those positions as there might be sensitive content in those files, SEO Manager says that its just a recommendation for Bots and that they'll crawl the files most likely anyway. Additionally to that some SEO Agency told our SEO Manager that there is a SEO influence in general when removing those positions (???).
Lets come to the question:
Are there any influences to the shop if we remove those files, and if so, how might they change the total SEO of the shop.