I'm interested in adding a very large range of crowd-sourced digital products (in time, perhaps 500,000+) to my Magento setup. Each would appear on about 10 websites (one per country/region) via a single Magento instance.

  1. From what I read elsewhere, resource requirements scale roughly in proportion with the number of websites, so 500,000 products on 10 websites requires similar resources to 5 million products on a single website. Is that correct?
  2. Is supporting this many products a realistic aim? (If so, are there companies out there successfully managing Magento catalogs this large?)
  • The quick answer is no, because indexes would take longer to build than there is time available to reasonably have operational churn. That might have changed with 1.13 EE and 1.8 CE, though.
    – parhamr
    Commented Dec 10, 2013 at 19:39
  • I disagree. I think that the architecture in 1.13EE specifically provides the ability to cherry pick your reindexing and cache clearing; you may be able to build a hardware infrastructure to support this. It would have to be highly optimized and I'm sure that you would probably limit personalization and maximize a reverse proxy - but I think it could be done.
    – philwinkle
    Commented Dec 10, 2013 at 20:23
  • I'm currently managing a site with 75,000 products across 14 stores on a single EE 1.13 install, not quite the same scale as your question, but also not on a crazy hardware infrastructure either - AWS m2.4xlarge. Bottleneck currently is the catalog_price_rules table rebuild at midnight takes about 1.5 hours, other indexes are fine. I'd say 0.5 million across 10 sites is quite doable with some planning.
    – Roscius
    Commented Dec 11, 2013 at 2:03
  • Also keep in mind there are ways to reindex only the products that have been modified in CE (through 3rd party extensions). The initial reindex would take a very long time though.
    – Axel
    Commented Dec 11, 2013 at 6:09
  • On CE it is complex but possible, EE makes it easier but it comes down to the number of deltas (changes) you have per day. The hosting architecture needs to be high end clusters costing $1,000s/mth and you will need special onboarding tools. We have seen and used the technology (well scaled versions of it) to do this, but you are going to need good budgets, only a handful have ever done something like this as it is classed as Medium Enterprise ($10-50million revenue) and Large Enterprise ($50million+ revenue) so you need architecture experience in these companies but also know Magento, rare.
    – user2935
    Commented Dec 11, 2013 at 18:33