Best practices/methods for adding 100k products in Magento 2?

Question

I have to add 100k products by the end of this week. Most of them are simple products but 30% are configurable products which each have 8 simple products, total 100,000 SKUs.

I used Magmi (in Mage 1) which worked quite well for importing CSV files, but with no updated version for Mage 2 it seems most people use default Import functionality in admin. But is there a better extension which store owners are using? I've done some searches with no luck, hoping to find some direction.

Lastly is it ever recommended to add products programmatically? I would rather go this route as I'm well versed in PHP but am still somewhat new to Magento. Can experienced store owners/developers give me advice on the best way to add 100k products with best practices in mind? Thank you

Answer 1

If you haven't got any product customizations and your products just work with attribute values set then in theory the import /export should work. By that I mean nothing dynamic which requires additional lookups as you insert the data.

However as you say you are working in volume you've got to factor resource restrictions such as memory and script timeouts.

Worse case scenario your data import fails and you don't know where you are in the process.

Sounds like you know what you are doing but you might want to take a look at this:

https://www.alexcorradi.org/blog/a-guide-on-how-to-import-export-products-in-magento-2

I've briefly looked into extending default functionality which I read in the docs has been written in a way to make extending easier. I'm not sure whether they achieved what they wanted

Eg

https://github.com/magento/magento2/blob/2.3/app/code/Magento/AdvancedPricingImportExport/Model/Import/AdvancedPricing.php

That's for tier pricing

So as for what I do and what I recommend.

I've created a base class for importing csv. Which in turn builds a queue. I've then processed these queue differently depending on my requirements. Good news is that of something falls over you've got that queue.

I've only got a couple of repos online (I've done more bit takes some thought and planning to make modular) but no doubt I'll add more as I do more and more different types of data work.

Anyway basically I build an import queue which I can process either in browser or via console scripts and therefore on cron.

It's all very bespoke so far

Base classes

https://github.com/DominicWatts/CsvUpload

Tier pricing (requires base)

https://github.com/DominicWatts/TierPriceUpload

Of course there are paid extensions out there. Trouble is I've always got very specific data requirements.