We have a multi-store setup with 16 stores and every store has it's own language, we have a lot of store-specific information. Most of our products are configurable products with lots of variants, so all in all we have about 6000 simple or configurable products.

We have a middleware in place that manages upload of categories, product and media via Magento2's synchronous REST API. This middleware has its own message queues and concurrency management. I played a bit around and ended up with about 5 concurrent workers as the best tradeoff between performance and Magento-deadlocks.

In the rare case that I have to upload all products, I have to perform about 6000 products * 16 stores = ~ 100.000 API calls which takes a lot of time, eventough our instance is running on an AWS t2.2xlarge instances. Usually it takes more then 24 hours.

I understand how utilizing bulk or asynchronous APIs can facilitate and optimize things when you don't have a message queue in place yet. But do you think switching over to async or bulk APIs could still signifantly optimize performance compared to our solution? I don't want to optimize just the time it takes to hand over a task to Magento but I want to optimize the time it takes a change actually is applied and becomes visible for the customer. If Magento's bulk/async APIs use the exact same internal APIs as the sync-API, I don't see any potential optimizations except saving network latency here. What do you think?

The management interface of our middleware

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.