3 added 218 characters in body
source | link

Edit 1/7/14:

@ryaan_anthony on Twitter released a MySQL stored procedure that will generate hundreds of thousands of products https://gist.github.com/ryaan-anthony/6290973



Edit 1/7/14:

@ryaan_anthony on Twitter released a MySQL stored procedure that will generate hundreds of thousands of products https://gist.github.com/ryaan-anthony/6290973


2 added 231 characters in body
source | link

tl;dr -> "Can Magento handle 1M products", the answer is yes, but with some considerations. At this scale one would assume you have volume to support a decent investment into infrastructure and personnel to merchandise a catalog of this proportion.

First:

Magento CE sample data, as you may have seen, has only a handful of products from varying categories. The EE sample data has more, and has them separated by store type.

You can download CE sample data here. You will have to download EE sample data from within your MagentoCommerce.com account if you have EE.

You will find, however, that this is not hundreds or even thousands of products. I would advise that you import products into the database - a good exercise to get a handle on how this process works. This can be done via Magento's Dataflow or via the API import - information on how to do this at scale is readily available on the internet.

A word of caution - Dataflow is notoriously slow, so it may take a fair amount of time to import a catalog the size that you request. To my knowledge there is not a sample catalog in the wild with hundreds of thousands or millions of products that exists.

Some reading on Magento API and Dataflow:

http://www.magentocommerce.com/knowledge-base/entry/introduction-to-magento-dataflow

http://www.magentocommerce.com/api/soap/catalog/catalog.html

Second:

Product, URL Rewrite, and Inventory Indexing are the major issues when running a catalog this size. Catalog search can be fairly slow, too, but can be mitigated if you use Apache Solr (integration provided native to EE). There are CE plugins for Solr - Sonassi has one, and others can be found via Google.

I've managed catalogs in the 700k-range, which is still a good deal fewer than 1M, and indexing can take hours upon hours. This has been addressed in Enterprise 1.13. I highly recommend you take a hard look at Enterprise Edition at this scale. Is this possible with CE? Absolutely; but the indexing improvements in EE 1.13 specifically are tailored to this sort of situation.

Third:

Multi-store is native to Magento; you can set up different top-level categories and websites. They don't all have to share the same catalog -- you can choose which products to share across sites, or decide to keep your catalog segregated. More info here:

http://www.magentocommerce.com/knowledge-base/entry/overview-how-multiple-websites-stores-work

The more stores, store views you have in Magento, the more index entries and the more your flat catalog may bloat to the point that flat catalog may actually be a performance drain. Again, Sonassi has a ton of information about this here on Magento.SE and on their site. You will want to search some of Sonassi's answers on Magento.SE for handling/scaling Magento when you get into this realm of product management.

Every person's install is different - you need to constantly test, refine, implement tweaks to find what settings work best for your catalog, in your situation.

First:

Magento CE sample data, as you may have seen, has only a handful of products from varying categories. The EE sample data has more, and has them separated by store type.

You can download CE sample data here. You will have to download EE sample data from within your MagentoCommerce.com account if you have EE.

You will find, however, that this is not hundreds or even thousands of products. I would advise that you import products into the database - a good exercise to get a handle on how this process works. This can be done via Magento's Dataflow or via the API import - information on how to do this at scale is readily available on the internet.

A word of caution - Dataflow is notoriously slow, so it may take a fair amount of time to import a catalog the size that you request. To my knowledge there is not a sample catalog in the wild with hundreds of thousands or millions of products that exists.

Some reading on Magento API and Dataflow:

http://www.magentocommerce.com/knowledge-base/entry/introduction-to-magento-dataflow

http://www.magentocommerce.com/api/soap/catalog/catalog.html

Second:

Product, URL Rewrite, and Inventory Indexing are the major issues when running a catalog this size. Catalog search can be fairly slow, too, but can be mitigated if you use Apache Solr (integration provided native to EE). There are CE plugins for Solr - Sonassi has one, and others can be found via Google.

I've managed catalogs in the 700k-range, which is still a good deal fewer than 1M, and indexing can take hours upon hours. This has been addressed in Enterprise 1.13. I highly recommend you take a hard look at Enterprise Edition at this scale. Is this possible with CE? Absolutely; but the indexing improvements in EE 1.13 specifically are tailored to this sort of situation.

Third:

Multi-store is native to Magento; you can set up different top-level categories and websites. They don't all have to share the same catalog -- you can choose which products to share across sites, or decide to keep your catalog segregated. More info here:

http://www.magentocommerce.com/knowledge-base/entry/overview-how-multiple-websites-stores-work

The more stores, store views you have in Magento, the more index entries and the more your flat catalog may bloat to the point that flat catalog may actually be a performance drain. Again, Sonassi has a ton of information about this here on Magento.SE and on their site. You will want to search some of Sonassi's answers on Magento.SE for handling/scaling Magento when you get into this realm of product management.

Every person's install is different - you need to constantly test, refine, implement tweaks to find what settings work best for your catalog, in your situation.

tl;dr -> "Can Magento handle 1M products", the answer is yes, but with some considerations. At this scale one would assume you have volume to support a decent investment into infrastructure and personnel to merchandise a catalog of this proportion.

First:

Magento CE sample data, as you may have seen, has only a handful of products from varying categories. The EE sample data has more, and has them separated by store type.

You can download CE sample data here. You will have to download EE sample data from within your MagentoCommerce.com account if you have EE.

You will find, however, that this is not hundreds or even thousands of products. I would advise that you import products into the database - a good exercise to get a handle on how this process works. This can be done via Magento's Dataflow or via the API import - information on how to do this at scale is readily available on the internet.

A word of caution - Dataflow is notoriously slow, so it may take a fair amount of time to import a catalog the size that you request. To my knowledge there is not a sample catalog in the wild with hundreds of thousands or millions of products that exists.

Some reading on Magento API and Dataflow:

http://www.magentocommerce.com/knowledge-base/entry/introduction-to-magento-dataflow

http://www.magentocommerce.com/api/soap/catalog/catalog.html

Second:

Product, URL Rewrite, and Inventory Indexing are the major issues when running a catalog this size. Catalog search can be fairly slow, too, but can be mitigated if you use Apache Solr (integration provided native to EE). There are CE plugins for Solr - Sonassi has one, and others can be found via Google.

I've managed catalogs in the 700k-range, which is still a good deal fewer than 1M, and indexing can take hours upon hours. This has been addressed in Enterprise 1.13. I highly recommend you take a hard look at Enterprise Edition at this scale. Is this possible with CE? Absolutely; but the indexing improvements in EE 1.13 specifically are tailored to this sort of situation.

Third:

Multi-store is native to Magento; you can set up different top-level categories and websites. They don't all have to share the same catalog -- you can choose which products to share across sites, or decide to keep your catalog segregated. More info here:

http://www.magentocommerce.com/knowledge-base/entry/overview-how-multiple-websites-stores-work

The more stores, store views you have in Magento, the more index entries and the more your flat catalog may bloat to the point that flat catalog may actually be a performance drain. Again, Sonassi has a ton of information about this here on Magento.SE and on their site. You will want to search some of Sonassi's answers on Magento.SE for handling/scaling Magento when you get into this realm of product management.

Every person's install is different - you need to constantly test, refine, implement tweaks to find what settings work best for your catalog, in your situation.

1
source | link

First:

Magento CE sample data, as you may have seen, has only a handful of products from varying categories. The EE sample data has more, and has them separated by store type.

You can download CE sample data here. You will have to download EE sample data from within your MagentoCommerce.com account if you have EE.

You will find, however, that this is not hundreds or even thousands of products. I would advise that you import products into the database - a good exercise to get a handle on how this process works. This can be done via Magento's Dataflow or via the API import - information on how to do this at scale is readily available on the internet.

A word of caution - Dataflow is notoriously slow, so it may take a fair amount of time to import a catalog the size that you request. To my knowledge there is not a sample catalog in the wild with hundreds of thousands or millions of products that exists.

Some reading on Magento API and Dataflow:

http://www.magentocommerce.com/knowledge-base/entry/introduction-to-magento-dataflow

http://www.magentocommerce.com/api/soap/catalog/catalog.html

Second:

Product, URL Rewrite, and Inventory Indexing are the major issues when running a catalog this size. Catalog search can be fairly slow, too, but can be mitigated if you use Apache Solr (integration provided native to EE). There are CE plugins for Solr - Sonassi has one, and others can be found via Google.

I've managed catalogs in the 700k-range, which is still a good deal fewer than 1M, and indexing can take hours upon hours. This has been addressed in Enterprise 1.13. I highly recommend you take a hard look at Enterprise Edition at this scale. Is this possible with CE? Absolutely; but the indexing improvements in EE 1.13 specifically are tailored to this sort of situation.

Third:

Multi-store is native to Magento; you can set up different top-level categories and websites. They don't all have to share the same catalog -- you can choose which products to share across sites, or decide to keep your catalog segregated. More info here:

http://www.magentocommerce.com/knowledge-base/entry/overview-how-multiple-websites-stores-work

The more stores, store views you have in Magento, the more index entries and the more your flat catalog may bloat to the point that flat catalog may actually be a performance drain. Again, Sonassi has a ton of information about this here on Magento.SE and on their site. You will want to search some of Sonassi's answers on Magento.SE for handling/scaling Magento when you get into this realm of product management.

Every person's install is different - you need to constantly test, refine, implement tweaks to find what settings work best for your catalog, in your situation.