We (abc.com) have Magento Multiwebsite and multistoreview built in Magento CE 1.9.X. Each website has one store view. We are basically a marketplace system. We sale products from different magento systems. All the products of different magento website (magento1.com, magento2.com etc) are imported to our system using extension we install on different magento systems.
Currently we have 200,000 products and tie up with 100 different magento systems (Vendor)(For exmaple magento1.com to magento100.com).
If any product is changed in vendor site (magento1.com or magento100.com) then system will prompt them to sync those product with our magento system. If they do not sync then every night 1AM by cron job changed products would be synched.
We were facing problem in reindex mainly. When parallel requests are coming together from multiple vendors for sync then at the same time product import and reindex is going on and while import and reindex running together, we are facing Deadlock issue. We are using Fast Asynchronous Reindex as we can not run full reindex for 200,000+ products everytime.
To avoid this problem, we have come up with Queue system. So once import, reindex and sending sync status is done for one vendor and then only other vendor import, reindex and sending sync status will start. System is working but most of the time other vendor has to wait a long (Upto 24 hours) Products to be synched. For example 50 vendors has requested for 500 products each that means (500*50) = 25000 products. For import 500 products, reindex and sending sync status for each vendor will take close to 30 minutes. So 50th vendor has to waut (30*50) = 1500 minutes which is 25 hours.
How system will handle multiple imports + reindex together? What solution will be proposed so that system can handle request without deadlock and it should not take that much time to sync products? How we will ensure that new proposed system will handle more vendors as we grow and until how many vendors it will work fine (We may have 500 as well)