How do you handle a large site with tons of traffic and tons of SKUs? The main point here is that running indexes in Magento is a huge weight on the server; especially when you have hundreds of concurrent users.

How is a business supposed to run indexes without disrupting the site? I'm looking for input from the warriors who have fought the battles and conquered big business. How do you do it?

Has anyone done anything "interesting" to overcome this?

2 Answers 2


From my experience there are 3 main problems:

  1. Running indexes in mode "Update on Save", so every time you change something, many of the indexes are reindexed. To fix it, this extension is useful - https://www.magentocommerce.com/magento-connect/fast-asynchronous-re-indexing.html (** EDIT: Links outdated, here is the Direct link **)
  2. Huge core_url_rewrite table. Useful fix - http://www.dnd.fr/2012/09/magento-patch-how-to-optimize-re-index-processing-time-for-url-rewrite/ (** EDIT: Links outdated, here is the GitHub link **)
  3. Big cache (many cache files) which slows add to cart, completing an order and saving a product. Useful fix - https://github.com/colinmollenhour/Cm_Cache_Backend_File + clearing the old cache periodically with cron job.
  1. Export all catalog_product_flat _1 tables with data and structure with ‘Disable foreign key checks’ enabled.
  2. Truncate all catalog_product_flat _1 tables.
  3. Ensure that whether products categories are all visible in Frontend.
  4. Login to Magento Admin Panel.
  5. Then go to ‘System’ menu and click on ‘Index Management’ submenu.
  6. At last click “Reindex Data” next to “Product Flat Data” index.

To figure out the reindexing issue for Category Products do the following actions:

  1. Log into phpMyAdmin Panel.
  2. Find the table ‘catalog_category_product_index’.
  3. Click ‘Export’ to obtain the ‘catalog_category_product_index’ table backup.
  4. Export only structure with “Disable foreign key checks” enabled.
  5. Truncate the table ‘catalog_category_product_index’.
  6. With the SQL that was exported, remove the constraints at the bottom of the export. Ensure to keep the “SET FOREIGN_KEY_CHECKS” statements.
  7. Click “SQL”, paste in modified SQL and click “Go”.
  8. Login to Magento Admin Panel.
  9. Then go to ‘System’ menu and click on ‘Index Management’ submenu.
  10. At last click “Reindex Data” next to “Product Flat Data” index.
  • Can you explain the purpose of these steps? This is probably not what you are doing every day instead of running the indexers the normal way, or is it? Commented Jun 20, 2016 at 6:21
  • No context here. Should be removed.
    – HenryHayes
    Commented Sep 21, 2023 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.