Configuration of a Queue CRON job

Question

I have a MySQL-based queue that has been added by a custom extension and I'd like to know how to configure the CRON to run that queue process.

The Magento documentation has this to say about how to run a Magento 2 Queue using a CRON job:

After consuming all available messages, the command terminates. You can run the command again manually or with a cron job. … For example, you can append & to the command to run it in the background, return to a prompt, and continue running commands (e.g., bin/magento queue:consumers:start <consumer_name> &).

Based on the documentation, I would think you'd configure a CRON job like this:

* * * * * /var/www/stage/current/bin/magento queue:consumers:start <NAME OF QUEUE> &

However if I run just /var/www/stage/current/bin/magento queue:consumers:start <NAME OF QUEUE> & on the command line, the process runs for a long time (potentially indefinitely?), so I can't see how running that command every minute from the CRON job would work, as you'd have an ever-increasing number of processes running.

What is the recommended way to run a MySQL queue from the command line?

Answer 1

After talking with David Alger about this, I've determined the following:

The standard Magento CRON configuration will run Magento 2 queues without any special configuration of the CRON, so a special crontab configuration in order to run queues is not strictly necessary. However if the CRON is processing a long-running CRON job, your queue might not get processed every minute. So to ensure that the queue runs at least once a minute, you can configure it to run like this:

* * * * * cd <PATH TO MAGENTO> && ( ps x | grep [n]ameOfQueue >>/dev/null 2>&1 || bin/magento queue:consumers:start nameOfQueue)

Where nameOfQueue is the name of the queue that you're wanting to run. You can retrieve a list of all queues using this command:

bin/magento queue:consumers:list

Answer 2

Since Magento 2.2.0 you can configure running of consumers via app/etc/env.php

https://devdocs.magento.com/guides/v2.2/config-guide/mq/manage-mysql.html

Also, If you run consumers manually - set number of messages bigger than 0 because if you use 0 then consumers will work infinitely.

Answer 3

If the extension was meant to be run on a schedule there should be a crontab.xml file the extensions etc folder, this would then be run from the standard magento cron:run based on whatever is configured there. Running it directly from the system crontab though you'd never be able to prevent it overrunning without wrapping it in some other script.