0

Daily we get in our webservers logs hundreds of thousands of lines from bots who are visiting these paths:

/catalog/category/view/id/$1

/catalog/product/view/id/$i

where i is between 1 to 54000

As immediatelly measures:

  1. We redirect these paths to a 404 error using in .htaccess file. As long as we use URL's rewrites nothing is affected inside Magento.

  2. We catch the IP's and block them for a week using Fail2Ban.

Please let me know if there are any issues in time using method 1 (we did not re-index the Catalog URL's yet to see what is happening). If there are any other methods you are welcomed to reply.

0
##### Block unwanted Crawler Bots that clog your server #####

 RewriteCond %{HTTP_USER_AGENT} MJ12bot
 RewriteRule .* - [F]
 RewriteCond %{HTTP_USER_AGENT} 80legs [NC]
 RewriteRule ^ - [F]

Add this to your htaccess. You can also add following codes

##### Search Engine redirects and rewrites for SEO purposes #####

<IfModule mod_rewrite.c>
 #RewriteCond %{HTTP_HOST} !^www.yourdomain.com$ [NC]
 #RewriteRule ^(.*)$ http://www.yourdomain.com/$1 [R=301,L]

 ##### Redirect away from /index.php and /home  
 ##### Warning: This index.php rewrite will prevent Magento 
 ##### Connect from working. Simply comment out the 
 ##### following two lines of code when using Connect.
 ##### Please note - http://www. if not using www simply use http://

 RewriteCond %{THE_REQUEST} ^.*/index.php
 RewriteRule ^(.*)index.php$ http://www.yourdomain.com/$1 [R=301,L]

    ##### Please note - http://www. if not using www simply use http://
 redirect 301 /home http://www.yourdomain.com

 Options +FollowSymLinks
 RewriteEngine on
 RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
 RewriteCond %{REQUEST_URI} !^/(media|skin|js)/
 RewriteCond %{REQUEST_FILENAME} !-f
 RewriteCond %{REQUEST_FILENAME} !-d
 RewriteCond %{REQUEST_FILENAME} !-l
 RewriteRule .* index.php [L]
</IfModule>

also ensure that your htaccess has following codes

# Directories
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
#Disallow: /js/
#Disallow: /lib/
Disallow: /magento/
#Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/

# Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
#Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/

# Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt

# Paths (no clean URLs)
#Disallow: /*.js$
#Disallow: /*.css$
Disallow: /*.php$
Disallow: /*?SID=

This should help

| improve this answer | |
  • Using robots.txt is not efficient at this stage. We have a long list of bots updated regularly. Most of those records in the log are coming from known browsers, they look like normal visitors. Blocking the paths and banning the IP's I think is the best solution for the moment. I was interested knowing if this rewrite of the paths could affect in some ways Magento core. – ADDISON74 Sep 28 '17 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.