Consumer "async.operations.all" skipped as required connection "amqp" is not configured. Unknown connection name amqp

Question

Since I upgraded from Magento 2.2.7 to Magento 2.3.1 I am getting this message in the system.log file:

main.INFO: Consumer "async.operations.all" skipped as required connection "amqp" is not configured. Unknown connection name amqp [] []

I get this error also on a fresh Magento 2.3.1 install.

Where could I configure or disable it?

Answer 1

Danila is above is right Solutions that tell you to do cron_run = 'false" will disable other queues you do want, like bulk attribute update or export file generation.

Option 1 disable packages

php bin/magento module:disable Magento_WebapiAsync
php bin/magento module:disable Magento_Amqp

I think it makes sense to disable both as the absence of Ampq allows modules to switch to mysql - but I haven't dug deep into that. Indeed if all you want is for the error message to vanish then Danila's option is all you need

Option 2 Tweak your queue cron, dont disable it

This is if you don't want to or can't take core modules off.

1. List your existing queues

   bin/magento queue:consumers:list

2. Add all of them except async to your cron consumer, in env.php. If there is functionality you never use, skip that one (but remember you did...)

'queue' => [
    'consumers_wait_for_messages' => 0,
],
'cron_consumers_runner' => [
    'cron_run' => true,
    'max_messages' => 2,
    'single_thread' => true,
    'consumers_wait_for_messages' => 0,
    'consumers' => [
        'product_action_attribute.update',
        'product_action_attribute.website.update',
        'exportProcessor',
        'inventory.source.items.cleanup',
        'inventory.mass.update',
        'inventory.reservations.cleanup',
        'inventory.reservations.update'
    ]
],

Most places have a higher max_messages but if you are not having rabbitmq you are possibly on a low budget host too and short queues more often are better. Up the number as you want.

Option 3 - take things out of Magento's cron and add them manually, queue per queue, to the host system cron

this implies

cron_run => false

and then putting things like bin/magento queue:consumers:start --single-thread --max-messages=20 inventory.source.items.cleanup in your cron (or running them manually if you do it rarely. If you have a shared server 'consumers_wait_for_messages' => 0 or consumers-wait-for-messages=1 is key for memory management

Answer 2

The answer from @Rafael Corrêa Gomes (and any to the point that mentions either 'cron_run' => false or 'consumers' => ['async.operations.all'] in the configuration are quite outrageous because they make many folks copy-paste and destroy their message queues from functioning.

cron_run set to false in configuration means that the message consumers are not going to be launched by the Magento cron. It means that you will have to run them using other means like SupervisorD or SystemD. And if not (and those answers don't mention that at all), you will have broken stuff beyond words "message queues", which includes, at very minimum, data export in Magento admin.

The documentation that is being referred to when posting such configuration has incorrect heading below this section with words:

Specific configuration

instead of

Sample configuration

It is a sample, and not a standard. Neither it is a recommendation to solve anything.

But, while the docs are at fault, it doesn't excuse skipping the lines and not reading further, then posting destructive solutions online.

The real solution is either disabling the Bulk API which most installations don't use, essentially:

php bin/magento module:disable Magento_WebapiAsync

Or, configure RabbitMQ if you want the bulk API (highly unlikely, depends on specific use case).

Answer 3

Edit the /app/etc/env.php file to configure the cron job consumers_runner.

<?php
return [

// ...

    'cron_consumers_runner' => [
        'cron_run' => true,
        'max_messages' => 20000,
        'consumers' => [
            'async.operations.all',
        ]
    ],

]; // return closing

To get the list of all your queue consumers that can be configured running:

php bin/magento queue:consumers:list

Reference: https://devdocs.magento.com/guides/v2.4/config-guide/mq/manage-message-queues.html#configuration

Answer 4

Short answer: Create config file in your custom module {module_vendor}/{module_name}/etc/queue_consumer.xml with following content:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/consumer.xsd">
    <consumer name="async.operations.all" queue="async.operations.all" connection="db"/>
</config>

Detailed answer: Magento has default implementation for message queue based on MySQL - Magento_MysqlMq. All default Magento modules, which requires message queue processing, uses this MySQL implementation (example: vendor/magento/module-inventory-catalog/etc/queue_consumer.xml) by connection="db" for their consumers. But for some reason Magento_WebapiAsync expects RabbitMQ implementation which is available by default in Magento Commerce (maybe this module was developer for Magento Commerce).

So solution is to move this consumer from RabbitMQ (which is unavailable) to MySQL-MQ (which is available by default)

Answer 5

I ran into this on Magento 2.3.1 and after some searching through the docs I found this link: https://devdocs.magento.com/guides/v2.3/config-guide/mq/manage-message-queues.html

Here it explains what it does and it says you can edit the /app/etc/env.php file to configure the cron job consumers_runner. I decided to add this "async.operations.all" consumer here.

Answer 6

For me Vladyslav Sikailo answer is right, but my xml is a bit different. I add attribute queue because Magento give me an error if missing.

This is my {module_vendor}/{module_name}/etc/queue_consumer.xml to force using db instead of amqp

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/consumer.xsd">
    <consumer name="async.operations.all" queue="async.operations.all" connection="db"/>
</config>

Answer 7

communication.xml

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Communication/etc/communication.xsd">
    <topic name="test.import.products" request="string" />
</config>

queue_consumer.xml

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/consumer.xsd">
    <consumer name="TestImportProducts" queue="test_import_products" connection="amqp" handler="Test\Products\Model\Consumer::processProduct"/>
</config>

queue_publisher.xml

    <publisher topic="test.import.products">
        <connection name="amqp" exchange="test-import-products" />
    </publisher>
</config>

queue_topology.xml

    <exchange name="test-import-products" type="topic" connection="amqp">
        <binding id="TestImportProducts" topic="test.import.products" destinationType="queue" destination="test_import_products"/>
    </exchange>
</config>

Answer 8

I went with this at the end of app/etc/env.php

'cron_consumers_runner' => array(
        'cron_run' => false,
        'max_messages' => 20000,
        'consumers' => array(
            'consumer1',
            'consumer2',
        )
    )