After upgrading my site from magento 1.938 to 1.942, google and bing spiders began crawling useless URLs such as
Google and bing are crawling these URLs in large numbers, and they have never stopped, which consumes a lot of server resources.
I have added the following command to
robots.txt a long time ago.
Disallow: /cms/ Disallow: /wishlist/
But this seems to be useless.
Before I upgraded to magento 1.942, I never found out that google and bing spiders would crawl these URLs in
Online Customer tab.
Why is this happening, how to prohibit the crawl of these two URLs?
I have checked the robots.txt in google robots.txt Tester. There is no error in my robots.txt
robots.txt Tester showed these two URLS has been blocked.