We have a multi-store setup with 16 stores and every store has it's own language, we have a lot of store-specific information. Most of our products are configurable products with lots of variants, so all in all we have about 6000 simple or configurable products.
We have a middleware in place that manages upload of categories, product and media via Magento2's synchronous REST API. This middleware has its own message queues and concurrency management. I played a bit around and ended up with about 5 concurrent workers as the best tradeoff between performance and Magento-deadlocks.
In the rare case that I have to upload all products, I have to perform about 6000 products * 16 stores = ~ 100.000 API calls which takes a lot of time, eventough our instance is running on an AWS t2.2xlarge instances. Usually it takes more then 24 hours.
I understand how utilizing bulk or asynchronous APIs can facilitate and optimize things when you don't have a message queue in place yet. But do you think switching over to async or bulk APIs could still signifantly optimize performance compared to our solution? I don't want to optimize just the time it takes to hand over a task to Magento but I want to optimize the time it takes a change actually is applied and becomes visible for the customer. If Magento's bulk/async APIs use the exact same internal APIs as the sync-API, I don't see any potential optimizations except saving network latency here. What do you think?
