31
votes

I need to see how Magento will perform with 1M SKUs; but I'm struggling to find a large data set of sample data to download - or find a feasible method of generating the feed for import (and the import process itself).

  1. Does anyone know where I could download a large dataset of dummy data for import (or a sensible means to generate and import it)?
  2. What issues do you forsee with having a catalogue size of 1M+ products?
  3. Is there a way to share a single products DB with multiple independent stores (different companies)?

locked by philwinkle Oct 16 '15 at 23:20

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. See the help center for guidance on writing a good question.

Read more about locked posts here.

36
votes

tl;dr -> "Can Magento handle 1M products", the answer is yes, but with some considerations. At this scale one would assume you have volume to support a decent investment into infrastructure and personnel to merchandise a catalog of this proportion.

First:

Magento CE sample data, as you may have seen, has only a handful of products from varying categories. The EE sample data has more, and has them separated by store type.

You can download CE sample data here. You will have to download EE sample data from within your MagentoCommerce.com account if you have EE.

You will find, however, that this is not hundreds or even thousands of products. I would advise that you import products into the database - a good exercise to get a handle on how this process works. This can be done via Magento's Dataflow or via the API import - information on how to do this at scale is readily available on the internet.

A word of caution - Dataflow is notoriously slow, so it may take a fair amount of time to import a catalog the size that you request. To my knowledge there is not a sample catalog in the wild with hundreds of thousands or millions of products that exists.


Edit 1/7/14:

@ryaan_anthony on Twitter released a MySQL stored procedure that will generate hundreds of thousands of products https://gist.github.com/ryaan-anthony/6290973


Some reading on Magento API and Dataflow:

http://www.magentocommerce.com/knowledge-base/entry/introduction-to-magento-dataflow

http://www.magentocommerce.com/api/soap/catalog/catalog.html

Second:

Product, URL Rewrite, and Inventory Indexing are the major issues when running a catalog this size. Catalog search can be fairly slow, too, but can be mitigated if you use Apache Solr (integration provided native to EE). There are CE plugins for Solr - Sonassi has one, and others can be found via Google.

I've managed catalogs in the 700k-range, which is still a good deal fewer than 1M, and indexing can take hours upon hours. This has been addressed in Enterprise 1.13. I highly recommend you take a hard look at Enterprise Edition at this scale. Is this possible with CE? Absolutely; but the indexing improvements in EE 1.13 specifically are tailored to this sort of situation.

Third:

Multi-store is native to Magento; you can set up different top-level categories and websites. They don't all have to share the same catalog -- you can choose which products to share across sites, or decide to keep your catalog segregated. More info here:

http://www.magentocommerce.com/knowledge-base/entry/overview-how-multiple-websites-stores-work

The more stores, store views you have in Magento, the more index entries and the more your flat catalog may bloat to the point that flat catalog may actually be a performance drain. Again, Sonassi has a ton of information about this here on Magento.SE and on their site. You will want to search some of Sonassi's answers on Magento.SE for handling/scaling Magento when you get into this realm of product management.

Every person's install is different - you need to constantly test, refine, implement tweaks to find what settings work best for your catalog, in your situation.

  • Hello there! Thank you very much for all this info. – Gabriele May 17 '13 at 13:23
  • The DB is built automatically by a system connected to many editor who regularly update our DB. We provide the final DB and updates to bookstores and now we want to offer a complete e-commerce solution to our clients. I made it to import all the data via Magmi. It's fantastic and perfect for us. As far as Indexing, I'll go for the Solr solution. I can't use MultiStores since I need to provide full admin access to my clients. Thank you again! – Gabriele May 17 '13 at 13:30
  • Interesting that you failed to mention the consideration of hosting, db optimization, alternatives to or enhancements for dataflow, the use of clone instead of factory instantiation for large data processing, cache & performance optimization and other performance options for optimizing magento for a catalog of this size. Waiting several hours for indexing sounds painful... why not run a cluster, or use mysql proxy to process the indexing and let a DB table sync when its finished? Just some basic thoughts... there are more advanced methods available as well. – mprototype Dec 24 '13 at 20:53
  • @mprototype feel free to add your own answer as you see fit. – philwinkle Dec 25 '13 at 3:56
7
votes

Use ApiImport for importing such a large amount of products. It is based on ImportExport and really fast... I've managed up to 500k (indexed) simple products per hour on a virtual machine.

Just run tests/benchmark_import_api.php. Edit that file to remove the entity types (and subtypes) that you don't need. You might also want to set USE_API to false for faster results.

4
votes

We have used http://www.icecat.biz/en/ in the past to extract product feeds for loading in sample data. There are a few Magento extensions too, but they nevery worked for us so we landed up writing most of our import scripts.

4
votes

to get one million+ product into magento. write simple php script which generate magmi supported product import csv file with different kinds of product types. Then use the magmi to import them

http://sourceforge.net/apps/mediawiki/magmi/index.php?title=Magmi_Wiki

  • Magmi is a csv importer, right? So I have to feed Magm with csv files containg the catalog, right? – Gabriele May 13 '13 at 11:45
  • 1
    yes, in the wiki there is documentation, how should format your csv for the product import and then make the profile with web-interface and use cli command to import it do /usr/bin/php magmi.cli.php -profile=custom_options -mode=create -CSV:filename="${x}"; done – sutha kathir May 14 '13 at 9:00
  • CSV is one of the datasources Magmi can use. Keep in mind that Magmi has a datapump interface that you can inject data into, sans CSV files. – Axel Aug 7 '13 at 20:44
3
votes

Not really a full answer as it seems others have already addressed most of your questions, just a few things to add:

1) I've had this laying around: Almost One Million Random Magento Products in ten CSVs You could also give http://beta.generatedata.com/ a try.

2) As Philwinkle mentioned already: indexing, data flow and search is the largest hurdle to overcome with such a large data set. EE1.13 does a better job of handling such large data (MySQL Triggers, considering all product/category status, etc), but keep in mind its still an initial release (x.0.0) at this time, I tend to wait a few releases to let others take on the burden of bug finding before considering it for a production environment. Infrastructure and optimization is key. Future upgrading is also something else to consider, as ALTER TABLE are not combined during upgrades and can take hours/days to perform the upgrade on the DB:

Some further reading on the topic of indexing on a large database:

3) The easiest way to share data between two Magento stores would be via a REST/SOAP request to the other companies Magento API. The alternative would be to simply dump the catalog from one company and allow the other to pick it up and parse it, it may be much faster than going through the API with 1+ million products.

  • 1
    1) I will takea look at it. 2) Yes, I went for Magmi in CE. We'll see how it will perform. 3) Yes I think dumping data and import in new store will be our choice, unless we'll find a way to share a common products DB between all the e-stores. Thanksa lot B00mer! – Gabriele May 17 '13 at 13:34
3
votes

We just worked on a project with 1.2m (no attributes and especially only one store view) products using magento 1.7.x and here are some of the experiences we had:

  1. Actually importing the products are quite fine, I think our initial import took something like 1.5h

  2. When doing the reindex our disk io would suffer extremely, solution was to get a good amount of ram (32gb ram amazon ssd instance). Optimize the innodb settings where we put the innodb pool memory allocation to a bit over the size of the database and especially changing the temporary table buffer from default 16mb to 128mb, this is really what saved our reindexing process.

  3. Cache, using only APC cache for fast cache, files for slow cache, turning off unnecessary logging and modules together with flat table and a couple of other optimizations makes the server deliver the product pages html (not the whole page) in 200ms. On our todo list is varnish cache.

  4. We where fighting and killing a lot of deadlock issues (some in admin still remain), possibly a newer version of Magento will not give these problems according to forums.

I will say there really are issues with 1.2m products, its not something I'd recommend doing without having the proper team and resources in place however if you have the time you can make it work.

I don't know what other platform would do better job.

2
votes

Always good this one, yes Magento CE & EE can (from experience not theory using supplied datasets) although obviously EE is better for indexing. Magmi is fine however when you come to reindex for the initial load you will have a serious problem. On top of that you then have maintenance where if 3% of products change daily you need to update 30,000 products with auto index, you will not be able to perform a daily reindex. This all comes down to two things, clustered hosting and delta enabled supplier onboarding, which are the domains of enterprise companies.

People seem to think the job ends when the products are loaded, however that is when the hard work starts. If you have too many stores, pricing tiers then your hosting needs to double, so for all intents and purposes 95% have no chance implementing it, 99% have no chance maintaining it. Millions of products equals Medium to Large Enterprise - if your consultants do not have this experience expect the infrastructure to collapse medium to long term.

0
votes

Magmi is great for importing large numbers of products also. http://sourceforge.net/apps/mediawiki/magmi/index.php?title=Magmi_Wiki

We are working on a development for a client at the moment who have 2.2 million SKUs the initial import was done using Magmi.

Not the answer you're looking for? Browse other questions tagged or ask your own question.