Does Magento 2 launch all RabbitMQ consumers only by cron?

Question

I am very surprised by the fact that I found that it seems that Magento 2 launches all RabbitMQ consumers only by cron.

I expected to see RabbitMQ as a separate application that accepts tasks from PHP / Magento code as input.

And later, RabbitMQ itself decides at what time to call the Magento 2 API and give some task for processing.

Can anyone clarify this question for me?

I expected to see RabbitMQ as a separate application that runs in a separate process and only receives tasks for execution and then calls the Magento API to process the task.

I had a better opinion of Magento 2 architects. But perhaps this is the only possible way to do it. Or not?

Answer 1

The integration of RabbitMQ with Magento 2 is more flexible than that.

Magento 2 can launch RabbitMQ consumers in two ways:

Command-line Interface (CLI): You can manually start a RabbitMQ consumer using the command bin/magento queue:consumers:start <consumer_name>. This method allows you to run the consumer as a separate process, continuously processing messages from the queue. You can also configure the consumer to run as a daemon or within a process manager like Supervisor.

Cron: Magento 2 can also launch RabbitMQ consumers through cron, which is useful for scheduling periodic tasks. In this case, the consumer runs as a separate process for a specified time, executing the queued messages and then stopping.

In both cases, RabbitMQ functions as an intermediary message broker between the PHP/Magento code and the Magento API. Magento 2 sends tasks to RabbitMQ, which in turn manages the queue and passes the tasks to the designated consumers for processing. The consumers then make API calls to Magento 2 to execute the tasks.

So, the architecture of Magento 2's RabbitMQ integration is closer to your expectation than you initially thought. The use of cron for running consumers is just one of the available options and is not the only method for managing RabbitMQ consumers in Magento 2.

** EDIT **

The integration between Magento and RabbitMQ is established through the Advanced Message Queuing Protocol (AMQP). RabbitMQ doesn't require its own PHP interpreter; instead, it communicates with Magento through the AMQP interface.

In the context of Magento and RabbitMQ, here's how the process works:

Magento publishes messages to RabbitMQ's exchange. RabbitMQ then routes these messages to the appropriate queue based on the defined binding rules. Magento consumers are responsible for consuming messages from the queues, processing the tasks, and executing the specified PHP classes. As you've mentioned, the initialization and the real launch of consumers happen on the Magento side. Magento's PHP application is responsible for running the consumers that process the messages from RabbitMQ.

To ensure smooth communication between Magento and RabbitMQ, you need to configure the message queue settings in the env.php file and set up the appropriate exchanges, queues, and bindings in RabbitMQ.

To summarize, RabbitMQ doesn't need a PHP interpreter; it works as a message broker that routes messages between Magento and its consumers. The actual processing of tasks using the specified PHP classes happens on the Magento side.

Answer 2

As Brent pointed out, you can process RabbitMQ queues by either CLI or the cron. There are advantages and disadvantages to each.

Consumers

CLI - This method is undoubtedly the preferred method, as it processes queue tasks immediately as they come in. You keep the process running (either as a daemon or with a process like Supervisor, as noted before), and they'll be processed immediately. You'll need this if you want real-time processing of your tasks.

Cron - If you have tasks that don't need to be executed right away, the Cron method will suffice. This is useful if you want to publish a message to a queue, but don't really care if they are immediately executed. They could be processed on the next Cron schedule tick, or even many minutes from now (if the queue fills up with too many messages). This could also be useful if you have so many messages to process, and need to rate-limit the number of messages processed every minute to prevent an over-traffic situation. But if you do this, some messages in the queue could be delayed for multiple batches.

Integration

RabbitMQ is integrated with Magento. RabbitMQ is a separate app, a "message broker" server, similar to Redis or MySQL. You can use RabbitMQ with Magento code and with PHP code -- it depends on your use-case, and you could use one RabbitMQ server for multiple apps, just like you can share a Wordpress database with a Magento database in separate tables.

If you have a queue designated for use by Magento, the related consumer process determines when to call the related Magento code.

Usage

You really only want to use RabbitMQ/queues when you have async tasks that you need processed (that is, you don't need to run them right "now").

Amazon does this when you place an order -- your credit card isn't processed right away, as that takes 2-3 seconds to execute. This is an unacceptable time frame due to their requirements and architecture, so they push your order and credit card info to a queue. It's processed some 20-25 minutes later, which is when you receive a followup email if your credit card is declined, as the order now waits for you to update the credit card info (and on the followup, a queue isn't used and processed in real-time, as you can't have the order status held up again).

Magento also utilizes RabbitMQ for bulk operations, which is a perfect application for this. For example, when you submit a bulk API request, it will be added to the queue and immediately respond to the request with a "success" response, basically meaning that the request has been received. However, the entire bulk operation has not yet succeeded, as it will take some time to execute. These requests happen async and will be processed as some point in the future.

This could also be useful for things like processing an email, or updating an order status. You don't need it consumed synchronously, and don't mind waiting either a few seconds or a few minutes for it to process. The items in the queue will be processed in the order that they come in, first-in, first-out (FIFO). This creates a reliable way to process a bunch of tasks, but in a systemized way that doesn't overload servers or require an immediate execution.