Magento 2.3.2 CE and Cron jobs, flock and RabbitMQ

Question

Last few days I was chasing an interesting bug in production system running Magento 2.3.2 CE Where as per community recommendation I run cron jobs like this

flock --timeout=0 /run/lock/magento_system_cron.lock  /usr/bin/php /var/www/html/bin/magento cron:run

After few executions of cron task it is suddenly gets locked and flock never releases this lock. The only way to release it is to either remove the lock file (which is incorrect, don't do it) or figure out related process IDs and kill them. One can figure out process IDs like this:

fuser -u /run/lock/magento_system_cron.lock

Once you know process IDs - do something like ps aux | grep <process-id-from-fuser-command>

Well, when you see processes that are involved there you would probably understand why it gets locked forever. Those processes are message consumers and they are supposed to sit and wait (they are actually blocked and effectively sleeping) till they receive a message via the message-queue mechanism and then process them.

Magento documentation is somewhat vague on this:

• https://devdocs.magento.com/guides/v2.3/config-guide/mq/rabbitmq-overview.html
• https://devdocs.magento.com/guides/v2.3/install-gde/prereq/install-rabbitmq.html
• https://devdocs.magento.com/guides/v2.3/config-guide/cli/config-cli-subcommands-cron.html

At the same time I noticed that cron jobs get blocked once I see this in logs:

main.INFO: Consumer "async.operations.all" skipped as required connection "amqp" is not configured. Unknown connection name amqp []

And googling for that was always leading me to the tip to disable the cron configuration for those message consumers, which didn't sound right to me:

// app/etc/env.php
// ...
'cron_consumers_runner' => array(
    'cron_run' => false,
    'max_messages' => 1000,
    'consumers' => array(
        'async.operations.all',
    )
)

See this for instance: https://community.magento.com/t5/Magento-2-x-Admin-Configuration/Consumer-async-operations-all-skipped-as-required-connection/td-p/129621

Related questions:

• Add RabbitMQ to existing Magento 2 EE installation
• Which core modules use RabbitMQ in Magento Open Source 2.3?

Somehow it became clear to me that something is not right with that message-queue setup as well with cron configuration. So I was looking through Magento docs multiple times and were not finding there what is supposed to consume messages that magento pushes into rabbitMQ, see this for example: https://www.rabbitmq.com/tutorials/tutorial-one-php.html

The question that I was asking myself all the time: "What am I missing here?" and "What is a recommended setup here?"

Answer 1

Flock not needed since Magento 2.3.2, because Magento already has own lock mechanism.
See https://devdocs.magento.com/guides/v2.3/install-gde/install/cli/install-cli-subcommands-lock.html#instgde-cli-lockconfig

Answer 2

Unfortunately googling that topic will only lead you to quite destructive answers which will cause you disabling message queues entirely.

The message:

main.INFO: Consumer "async.operations.all" skipped as required connection "amqp" is not configured. Unknown connection name amqp []

It is exactly what it says. You're missing amqp connection details.

The amqp connection details are the requirement of the Bulk API feature.

Cron or MySQL is not sufficient enough, the only way is setting up RabbitMQ.

You must install and configure RabbitMQ for the Bulk API feature to work.

Here's a complete walkthrough for Magento 2 BulkAPI enablement for CentOS/RHEL 7, 8.

Essentially, you need to install RabbitMQ and configure amqp details in the app/etc/env.php to match with the setup, e.g.:

    'queue' =>  [
        'amqp' =>  [
            'host' => '127.0.0.1', // IP address of RabbitMQ instance. Change it, if it is not running on the same machine with Magento
            'port' => '5672', // Port on which RabbitMQ running. 5672 is default port
            'user' => 'guest', // RabbitMQ user name
            'password' => 'guest', // RabbitMQ password
            'virtualhost' => '/' // The virtual host for connecting to RabbitMQ. The default is /.
        ],
    ],

If you don't care for Bulk API at all, disable the corresponding module (gets rid of the error as well):

php bin/magento module:disable Magento_WebapiAsync

Answer 3

So far I came to the following conclusions and possible answers to the question:

1. Remove flock, leave the rest of configuration as per default and hope for the best.

2. Keep flock, disable starting of message consumers per cron and manage that on per-consumer way using other tools

3. use separate cron jobs with per-consumer flock

4. use http://supervisord.org/index.html as per Magento docs recommendation

In either case it is a good idea to have RabbitMQ installed and configured for your Magento, cause it will be used either way.

You can get the list of all consumers you need to manage if you execute:

bin/magento queue:consumers:list

UPDATE:

what I ended up using the last months is this

flock --timeout=0 /run/lock/magento_mq_async_operations_all.lock  php bin/magento queue:consumers:start async.operations.all

here is the idea behind it:

• flock with timeout=0 exits immidiately if such process is already running
• if process is not running - it starts another one.
• message consumer is constantly sleeping and waiting for messages, eventually exiting if it reaches the max-messages or crashing.
• even if cron consumer has ever exited or crashed - latest in 1 minute it will be restarted by cron and I'm guarateed to have only 1 instance of message consumer

all the above covers my requirements pretty well.

meanwhile more people are aware of the problem and there is a reasonable suggestion towards magento: https://github.com/magento/architecture/pull/232/files#diff-81c5aa0b55a519b20c0ffd8b3f57b21b