We have a multi-store setup with 16 stores and every store has it's own language, we have a lot of store-specific information. Most of our products are configurable products with lots of variants, so all in all we have about 6000 simple or configurable products.

We have a middleware in place that manages upload of categories, product and media via Magento2's synchronous REST API. This middleware has its own message queues and concurrency management. I played a bit around and ended up with about 5 concurrent workers as the best tradeoff between performance and Magento-deadlocks.

In the rare case that I have to upload all products, I have to perform about 6000 products * 16 stores = ~ 100.000 API calls which takes a lot of time, eventough our instance is running on an AWS t2.2xlarge instances. Usually it takes more then 24 hours.

I understand how utilizing bulk or asynchronous APIs can facilitate and optimize things when you don't have a message queue in place yet. But do you think switching over to async or bulk APIs could still signifantly optimize performance compared to our solution? I don't want to optimize just the time it takes to hand over a task to Magento but I want to optimize the time it takes a change actually is applied and becomes visible for the customer. If Magento's bulk/async APIs use the exact same internal APIs as the sync-API, I don't see any potential optimizations except saving network latency here. What do you think?

The management interface of our middleware

  • Any experiences? – Vale Aug 21 '19 at 8:53
  • Looks like no one has ever worked with the async endpoints? – Vale Sep 9 '19 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.