2

A Client has mentioned to me a bug which is near impossible to re-produce thus only leaves theory based solutions.

I thought I would check to see if anyone else has stumbled upon this issue and with guidance or a solution.

Site Outline

our client has

  • 60,000 + products
  • over 40 sites (within single magento build)

When they run a re-index it takes up to 4 hours (yes our import script disabled re-index on save to help speed up the indexing process, this is due to number of products X number of stores)

The Issue:

Approximately 4 times in the past couple of months, ONE of the category pages shows 0 products, for a time of up to 1 day.

It appears re-indexing is still running when this happens.

It is difficult to reproduce (tried several scenarios) but is very important to resolve.

Has anyone else heard of / looked into / resolved this issue in the past?

My Theory:

1: They are viewing the page during the re-index process (re-index empties the SQL table then re-populates it, they happen to catch this as this time)

1a: Index Crashed during this step

1b: FPC has cached this version of the page during this process.

So I ask my friends, any more suggestions would be appreciated here.

Improve this question
2
  • Check the cron_schedule table and look for exceptions to see why the indexer didn't finish.
    – Rick Kuipers
    Commented Feb 4, 2013 at 15:30
  • Did you solve your problem? How? Please share the details with us. Thanks!
    – Fabian Blechschmidt
    Commented May 9, 2013 at 7:09

1 Answer 1

Reset to default
6 
60,000 products * 40 store views = 2,400,000 products

I'm honestly not surprised you've got catalogue consistency issues. Magento doesn't scale well at all with large catalogues combined with large numbers of store views.

Your issue is almost entirely caused by an index not being complete, so your theories of 1, 1a and 1b are all correct.

We've seen this on a LOT of stores where the catalogues are large and re-indexing is performed.

The fixes for most of these stores was to optimise the indexing process for each index. We swapped MySQL out for SOLR - which gives an almost immediate reduction in population time. There are some other aspects to SOLR that serve to benefit you, such as the index not being dropped entirely until the previous process is committed; and that updates again don't remove data until committed.

Although, with 2.4 Million products - you're still going to struggle, unless you make some significant application-level changes.

My advice would be to take a proper look at why you are re-indexing, and at what frequency - and try and find some alternatives to this process that can perform in real-time.

I think you'll struggle to get a definitive answer on SE because of how tailored the answer needs to be to your store/business.

Improve this answer
3
  • I'll second this - we've seen it with even smaller catalogues - 2000 products or so * 10 store views. Solr definitely helped but the index time for Solr is, in our experience, slower than the MySQL Fulltext index.
    – philwinkle
    Commented Feb 4, 2013 at 16:54
  • Hm ... but we can only switch in SOLR for the catalog_search index - what about the others?
    – Alex
    Commented Feb 5, 2013 at 21:43
  • That's far beyond the scope of this answer. Infrastructure and architecture planning is critical in large catalogue stores. A solution here is certainly not going to adequately or accurately help individual's issues - as the appropriate resolution isn't generic - its not 'one size fits all'. Its a case of reviewing the bottlenecks and store itself and making an intelligent decision based on that.
    – Ben Lessani
    Commented Feb 5, 2013 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.