Magento Product Data Import causing blocked IP

Question

It seems that since applying patch SUPEE-7045, we are having our office IP address blocked by our hosting provider whenever we do a product data import (via CSV, using Dataflow Profile).

Watching the progress of records being processed, it seems to pause at/after each 6th record, before continuing.

Checking Error Log via our cPanel, we see the following:

• mod_hive matched URI example.com/.../xyz.csv/ (repeated 6 times)

• mod_hive client reached the maximum connections(6) per URI example.com/.../system_convert_gui/batchRun/key/.../home/.../public_html/index.php for 10 seconds, referer: example.com/index.php/.../system_convert_gui/run/id/3/key/.../files/import-20160216014458-1_XXX.csv/

The above is repeated X number of times, e.g. repeated 8 times if we had 48 products.

Our first occurrence of the issue saw our 200-product import start error-ing out at around 40-something records; subsequent imports have been 58 products, and have completed successfully, with the blocking happening anytime within 90 seconds of import finishing.

Our hosting provider say they start blocking IP addresses if they see 20 or so multiple connections, given it is a sign of potential hacking. Not sure how the 20 vs 6 adds up, but does seem related.

They suggested to remove "admin" from our admin URL (we had some codified prefix + "admin"), which we tried but still got blocked.

They say nothing has changed in their server settings. We have not changed anything either, and we've successfully imported 100s of products in the past. It seems only since that Magento patch we have these problems.

Does anyone have a clue as to if Magento connection behaviour has changed (generally, or specifically to dataflow)? What code could I be fixing? I've never even heard of Hive before now (referring to that "mod_hive" in the error log).

Have read through the patch notes, and although some items appear semi-related, nothing strikes me as the right place to start looking.

Answer 1

You are on a system running mod_hive. It is a management system for shared servers. It has some various things it is able to do, one is load restriction

The Maximum Load Limit for the server is defined in the /usr/local/apache/conf/includes/1h.conf file.

For example:

<IfModule mod_hive.c>
  MaxLoadLimit 30.0
</IfModule>

The purpose of this option is to prevent execution of dynamic content when the average server load is too high. This way you are preventing server overloads.

Important: You can change this value according to your needs. However, please note that it is of utmost importance to not set the value too low. Otherwise you are risking to render websites on the server inaccessible. This value is low enough and it is advisable to consider only increasing it in case a server is temporarily keeping too high load and not lowering below the default value at any circumstances.

Magento imports are creating extra load on the system and mod_hive restricts the connection. As to why Magento would suddenly start making multiple requests after the patch, somebody will have to dig into that to see if:

1. Magento really is requiring more connections after the patch or
2. you were in an incipient zone of usage which finally pushed the shared host over its limits.

As to the solution, if this connection restriction is not being tripped by customers and only gives issues with certain admin functions, then whitelisting probably is the answer.

Expect shared hosting to eventually give out on you as you start requiring more resources, Magento doesn't play well with others.