4

I have imported more than 1M of products for two stores. I am trying to indexing those products but it is taking long time through Admin and SSH. "Catalog URL Rewrites, Product Flat Data, Catalog Search Index" this are taking so much time to complete. Is there is any way that i can perform indexing on those products ?

Thanks in Advance.

Improve this question
7
  • 1
    What unit of measure is a 'lacs'
    – philwinkle
    Commented Jul 2, 2013 at 5:03
  • @philwinkle 10 lacs means 1 million
    – Mufaddal
    Commented Jul 2, 2013 at 5:50
  • 1
    @philwinkle A Lac/Lakh is equal to 100,000. "10 Lacs" is 1,000,000 en.wikipedia.org/wiki/Lakh
    – Alana Storm
    Commented Jul 2, 2013 at 6:55
  • 1
    Believe it or not I actually googled! Thanks @alanstorm
    – philwinkle
    Commented Jul 2, 2013 at 12:17
  • 1
    What version of Magento are you using, and what do you consider a long time? Also, what type of hosting environment do you have the site running on?
    – davidalger
    Commented Jul 2, 2013 at 13:39

2 Answers 2

Reset to default
4

Magento has a notoriously slow indexing engine that recently received a much-needed refresh in the Enterprise Edition 1.13 (though, at the time of this writing it is currently being overhauled for some issues with SEO).

In this answer, I have provided some direction on the types of issues you typically encounter when managing a site of this magnitude:

Is Magento the right platform for 1M products?

While your store may have 1M unique products, based on the number of pricing rules, stores, store views, tiered prices, etc., your indexes may have > 1M index entries. I have seen index processes run for 10+ hours. At this size, the flat catalog is not helpful, and in some documented cases, it actually negatively impacts performance.

At this scale, one would hope that you have the financial backing or annual volume to consider the Enterprise solution, as it addresses these indexing issues specifically. Otherwise, while it is possible to run a Community Edition store with a catalog of this size, all of your hardware needs scale horizontally -- meaning, you need larger and larger hardware to accomodate.

At this time, there are no fixes planned for indexing improvement in Community Edition.

Edit:

There are some community modules which tout that they improve index speed - I have no personal experience with any of them:

http://www.magentocommerce.com/magento-connect/asynchronous-reindex-4388.html

http://sourceforge.net/apps/mediawiki/magmi/index.php?title=Magmi_Magento_Reindexer

Ecomdev has a freely-available indexer specifically for UrlRewrites:

https://github.com/IvanChepurnyi/EcomDev_UrlRewrite

Improve this answer
3
  • Thanks, But is there any way that i can boost the indexing process ? Because I have approximately more than 10M products.
    – kedar
    Commented Jul 3, 2013 at 9:28
  • I added some alternatives.
    – philwinkle
    Commented Jul 3, 2013 at 15:06
  • @philwinkle Speaking from experience of using them, the asynchronous re-index extension above, does't do what your after, it merely runs the re-indexing in the background, but if you have a large catalogue and are constantly increasing this, this creates a backlog that never clears and causes more issues than it solves. The Magmi reindexer "should" work, but I'm unsure of the php cli command to insert in..? I'm going to explore the Ecomdev url rewrite extension and report back.
    – MagentoMac
    Commented Feb 19, 2014 at 21:55
3

We have been running a large database with 1.5m plus SKU's and multiple stores as well. The one consistent thing is you will find a million opinions on what to do for performance. Here are some of the things that has helped us lower our indexing time:

  1. Dedicated hardware with separate web and DB nodes just like you have done.
  2. Hard drive speed and configuration is critical. We use Solid State Drives configured in a RAID 10 and that is where we got the biggest increase in indexing speed. You can mount those other drives for space for backups, etc as 3TB's in SSD's would be $$$.
  3. Focus on DB settings in MySQL. Tuning this depends on so many factors you really just need to try different things out.
  4. We also focused on fast RAM speeds and upgraded to faster core speeds on our processors.

The biggest bang for the buck though was with the SSD's configured in a RAID 10. It was well worth the investment.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.