We are currently evaluating to optimize our robots.txt file. Our SEO Manager requested to remove specific stuff that Magento puts there by default.
Those values are:
-# Files -Disallow: /cron.php -Disallow: /cron.sh -Disallow: /error_log -Disallow: /install.php -Disallow: /LICENSE.html -Disallow: /LICENSE.txt -Disallow: /LICENSE_AFL.txt -Disallow: /STATUS.txt
Developers don't want to remove those positions as there might be sensitive content in those files, SEO Manager says that its just a recommendation for Bots and that they'll crawl the files most likely anyway. Additionally to that some SEO Agency told our SEO Manager that there is a SEO influence in general when removing those positions (???).
Lets come to the question:
Are there any influences to the shop if we remove those files, and if so, how might they change the total SEO of the shop.