Yet another bulk product import issue

Question

We are planning to import a few thousands of products with custom options in Magento CE 1.9. We use SOAP for that, CSV is not an option since additional attributes may store html, json or serialized arrays. WSDL cache is enabled, index is disabled before and restored after bulk import/update.

Attribute media is another story but for the simple product creation, the ws needs 5 s, 10s in total if we take account of the custom options import. We can live with that but the real issue is this duration (10s) keeps raising in time to reach 50+s for the 200th product! Re indexing reset this value to 10s.

What is going on, has anyone encountered this and eventually find a solution?

MySQL and Apache are on the same server with 8 vCPUs and 16 GB RAM.

Answer 1

I wouldn't create products over the API as that approach will be slow (if I recall the API product creation will load up the model and save that way, which results in lots of overhead). I'd highly suggest this open source extension (I am not affiliated with the author in any way, its also free). http://avstudnitz.github.io/AvS_FastSimpleImport/

It wraps the stock Import/Export module but allows you to run the import programatically (using PHP arrays). So if your input file format is JSON like you stated, you simply json_decode. Its also still extremely fast.

I've used this library for my clients that have hundreds of thousands of products (my largest project had close to 500k products), updating nightly.

Couldn't recommend this extension more.