0

After upgrading my site from magento 1.938 to 1.942, google and bing spiders began crawling useless URLs such as

https://www.example.com/cms/index/noCookies/

and

https://www.example.com/wishlist/index/add/product/101/form_key/QRh31BjtGTfy2Ur9

enter image description here

Google and bing are crawling these URLs in large numbers, and they have never stopped, which consumes a lot of server resources.

I have added the following command to robots.txt a long time ago.

Disallow: /cms/ 

Disallow: /wishlist/

But this seems to be useless.

Before I upgraded to magento 1.942, I never found out that google and bing spiders would crawl these URLs in Online Customer tab.

Why is this happening, how to prohibit the crawl of these two URLs?

I have checked the robots.txt in google robots.txt Tester. There is no error in my robots.txt

robots.txt Tester showed these two URLS has been blocked.

Here's screenshot enter image description here

0

you can just block these locations from bots with user agent filter:

location ~ ^/(wishlist|customer|catalog/product_compare|tag/product/list|cms/index/noCookies)  {
    if ($http_user_agent ~* "Baiduspider|Googlebot|bingbot|rogerbot|Yahoo|Yandex") { return 410; }
    try_files $uri $uri/ /index.php$is_args$args;
    }

add any location.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.