1

How to prevent(block) bad bots traffic with .htaccess for Magento 2 ? What you recommend ? I do not know how to do this.
And which codes in .htaccess I need to modify.
Or Do we have any other alternative other than .htaccess ?

1

2 Answers 2

1

There are two ways to block bots trying to access your site. One is through the robots.txt file, and the other is through the .htaccess file.

As you might have guessed from the title of this post, I’m going to be focusing on the second one. First, though, let’s talk about robots.txt. What is a robots.txt file?

A robots.txt file is a text file you put in the root director of your server. Its purpose is to give guidelines to bots that want to access your site. You can use it to block bot access, either to specific bots or to all bots. So why not use it?

enter image description here The .htaccess file is a configuration file that is used by the Apache web server software. It’s a lot more like a security guard at the front gate, actively stopping potential robbers. Except in this case, the security guard has the ability to see whether or not the person trying to enter is coming from RobberHome, is wearing a shirt that says “I’m a robber”, or otherwise identifies itself.

What this means is that the .htaccess file can actively block most bots, but not all bots. In particular, the botnet bots – slaved computers from normal users – are generally not blocked by default. This is because those are regular user computers, using regular user software. If you block them, you’re blocking humans. For most other bots, though, the .htaccess file is ideal.

enter image description here

Note that using the .htaccess file can only be done if your web server is running Apache. If you’re using Nginx, Lighttpd, or one of the other niche server architectures, you’ll have to find that software’s way of blocking bots.

Identifying Bots to Block

First of all, a word of warning. Be very careful when you’re blocking bots through the .htaccess file. One typo and you can end up blocking the entire Internet. Obviously you don’t want that.

The first thing you want to do is back up your current .htaccess file. In the case of an error that blocks traffic you don’t want blocked, you can restore the old file to revert the changes until you can figure out what went wrong. enter image description here

The second thing you want to do is figure out how to find your own access logs. With Apache, you need to use a Linux/Unix command to access the log file. You can read about how to do that in this guide.

Using that guide, you will generate a log file that shows server access with quite a bit of detail. It will show you the IP address used to access the server, the identity of the client machine if available, the user ID of the machine if it used authentication, the time of the request, whether it was accessed by HTTP, the status code the server returned, and the size of the object requested. This will, likely, be a huge file.

The log file will have data on all of your regular users, and all of your bot access. Some bots, like the Google bots, will identify themselves through their user agent information. Bad bots sometimes identify themselves, but often just have certain characteristics that flag them as non-human. They might be using an out of date version of a browser commonly known to be exploited. They might come from known spam addresses or domains.

This guide is pretty good at helping you identify which log entries are bad bots and which are either good bots or good users.

Generally, if a bot is only accessing your site once a month, you don’t necessarily need to worry about it. You can block it if you like, but it isn’t necessarily going to save you might time or effort. Your primary goal should be to block the bots that visit constantly and have a negative impact on the performance of your server.

Be very careful when blocking by IP address or IP range. It’s easy to see a lot of bots coming from something like 168...*, with a variety of different numbers in the stars, and think “I can just block all of those! Block the entire /8 range!” The problem is, a /8 range in IPv4 is 16,777,216 different IP addresses, many of which may be used by legitimate users. You could block a massive amount of legitimate traffic with one overly-broad rule.

Most entries in a .htaccess file won’t block via IP address, simply because an IP address is too easy to change via proxies. Most will use user agent names, specific recurring IP addresses from bots that don’t care to change, or domains generally used to host spambots or hacker tools.

Using The .htaccess File

There are three ways we’re going to use to block bots through the .htaccess file. The first is the most common, using the user agent of the bot to block it. This is generally reliable, as normal users won’t accidentally have a bot user agent.

In your .htaccess file, you first want a line that says “RewriteEngine on”. This line makes sure that any following rewrite lines will work rather than being parsed as comments.

Next, you can add “RewriteCond %{HTTP_USER_AGENT} \” as its own line. This enables a rewrite condition based on user-agent. You have two options here; you can either add a ton of different user agents after this one line, or you can add one user agent and then add the line again. For example:

RewriteCond %{HTTP_USER_AGENT} \
12soso|\
192\.comagent|\

1noonbot|\
1on1searchbot|\
3de\_search2|\ [NC,OR]

Or:

RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^$ [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Acunetix [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^binlar [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [NC,OR]

Both work fine. With the first example, you will need to add an additional RewriteCond line every ~500 or so entries. This is because the longer the line with one command, the harder it is for Apache to parse. Breaking it up into individual entries makes it more cluttered but perhaps easier to read. Regardless, you can use either method.

The NC and OR bits at the end are rewrite flags. NC means “nocase”, which means the entry is not case-sensitive. It means “12soso” and “12Soso” are treated the same way. OR means “this or that”, as in, the bot will be blocked as long as it matches one or another of the entries on the list, as opposed to “AND”, which would be all of them.

After your list of bots here, you need to specify the rewrite rule. All of this is just the first part of a two-part clause: if the URL matches this, then… The second part is what happens. Add “RewriteRule .* – [F,L]” on its own line.

What this does is redirects any incoming traffic from the bot user agent to a blocked page. Specifically, it sends the 403 Forbidden code. The [F] is Forbidden, and the [L] is a code indicating that the rewrite rule should be applied immediately rather than after the rest of the .htaccess file is parsed.

The other two methods are blocking based on HTTP referrer, and blocking based on IP address.

To block by HTTP referrer, use “RewriteCond %{HTTP_REFERRER}” as the starting line, use the domain of the exploitative referrer like www1.free-social-buttons.com, and use the [NC,OR] block. Add the same RewriteRule line afterwards. You’ll end up with something like this:

RewriteCond %{HTTP_REFERER} www4.free-social-buttons\.com

RewriteRule ^.* - [F,L]

Finally, you can simply block based on IP address. If you notice one particular IP address is particularly detrimental, spamming your site a hundred times an hour or whatever, you can block it. Just write “Deny from ...”, where the stars are the IP address. It will look like “Deny from 173.192.34.95”, possibly with a /28 or something at the end to block a range.

0

To prevent bad bot traffic in Magento 2 using .htaccess, you can add the following code to your .htaccess file:

<IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTP_USER_AGENT} ^.*(bot1|bot2|bot3).*$ [NC]
    RewriteRule .* - [F,L]
</IfModule>

Replace bot1, bot2, bot3, etc. with the user agents of the bad bots you want to block. This code will return a 403 Forbidden response to any requests from these bots.

Alternatively, you can use the robots.txt file to disallow bad bots from crawling your site:

User-agent: bot1
Disallow: /

User-agent: bot2
Disallow: /

Replace bot1, bot2, etc. with the user agents of the bad bots you want to block.

Another alternative is to use security plugins or services specifically designed to block bad bot traffic, such as Cloudflare, Sucuri, or Akamai.

These services offer more advanced bot protection features and can be easier to configure than modifying .htaccess directly.

Let know if this fix your issues.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.