3

We (abc.com) have Magento Multiwebsite and multistoreview built in Magento CE 1.9.X. Each website has one store view. We are basically a marketplace system. We sale products from different magento systems. All the products of different magento website (magento1.com, magento2.com etc) are imported to our system using extension we install on different magento systems.

Currently we have 200,000 products and tie up with 100 different magento systems (Vendor)(For exmaple magento1.com to magento100.com).

If any product is changed in vendor site (magento1.com or magento100.com) then system will prompt them to sync those product with our magento system. If they do not sync then every night 1AM by cron job changed products would be synched.

Problem:

We were facing problem in reindex mainly. When parallel requests are coming together from multiple vendors for sync then at the same time product import and reindex is going on and while import and reindex running together, we are facing Deadlock issue. We are using Fast Asynchronous Reindex as we can not run full reindex for 200,000+ products everytime.

Temporary Solution:

To avoid this problem, we have come up with Queue system. So once import, reindex and sending sync status is done for one vendor and then only other vendor import, reindex and sending sync status will start. System is working but most of the time other vendor has to wait a long (Upto 24 hours) Products to be synched. For example 50 vendors has requested for 500 products each that means (500*50) = 25000 products. For import 500 products, reindex and sending sync status for each vendor will take close to 30 minutes. So 50th vendor has to waut (30*50) = 1500 minutes which is 25 hours.

Solution:

How system will handle multiple imports + reindex together? What solution will be proposed so that system can handle request without deadlock and it should not take that much time to sync products? How we will ensure that new proposed system will handle more vendors as we grow and until how many vendors it will work fine (We may have 500 as well)

Improve this question
2

1 Answer 1

Reset to default
1

use MAGMI to do the database based products upload for multiple vendors and index after uploading, with MAGMI you can easily do this

Mgami

Improve this answer
3
  • Hello Niranga, This is not one time process. We dont know when vendor will sync products to main site.
    – user45388
    Oct 5, 2016 at 11:28
  • Hi,, but we can set daily synchronization process or put a specific time to sync using cron jobs. for different vendors we can setup different ftp paths where app can easily update. also i have updated 670K products for a web site without any issue, and there is a daily syncing process.
    – Niranga Jayakody
    Oct 7, 2016 at 2:47
  • Yes, we have daily cron setup also. But products prices when vendors will change we dont know if sync not happen immediately will get price difference problem. We have almost 1lk+ products.
    – user45388
    Oct 7, 2016 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.