5

We're using magento's message queue with the mysql provider to push new orders to an external system. The external system is sometimes unreliable and failes, resulting in the queued job failing. Most of the time, the failures get resolved after a bit of time, so all we should do is retry the job and it should be fine.

Is there a way to automatically retry a failed job with a backoff period in magento or would we need to build that ourselves?

We are looking for a similar solution as the one Laravel provides.

2
+350

I don't know of any built in message queue retry on failure process for messages in Magento 2.

If you are using your own consumer I think you would have to build your own logic in the consumer to confirm the task executed successfully and create some kind of queue for failed messages that can be automatically processed and re-published after a given period.

A simple file queue might be enough whereby you store the json encoded data of the failed message in a text file and process this with a cron job every X minutes.

1
  • 1
    I see, thanks for the answer. Do you know if we could "abuse" magento's retry functionality for that? Aka just setting the status column to "retry required" or something like that? But I think that wouldn't get us the exponentional backoff part, so we might still need custom code. – kolaente Feb 18 at 16:01
1

For what it's worth, I have had to go the "custom" route to manage a messages state/retries etc. Magento does have functionality to retry a message using the MySQL driver, within the \Magento\MysqlMq\Model\Driver\Queue::reject (https://github.com/magento/magento2/blob/7c6b6365a3c099509d6f6e6c306cb1821910aab0/app/code/Magento/MysqlMq/Model/Driver/Queue.php#L127) function if $requeue is true and some other conditions are passed, a message will pushToQueueForRetry and process again up to a configured retry limit.

The issue is that the reject function is called with $requeue = false if an error occurs within the queue consumer, which is ironically where you'd most likely want to trigger the retry logic! See https://github.com/magento/magento2/blob/7c6b6365a3c099509d6f6e6c306cb1821910aab0/lib/internal/Magento/Framework/MessageQueue/Consumer.php#L236 ... The only solution I have at the moment is overriding/plugging into the default queue implementation so \Magento\MysqlMq\Model\Driver\Queue::reject can be handled differently (maybe always force a re-queue on failure). For example:

/**
 * Force `$requeue = true` in the case of a failure
 *
 * @param \Magento\MysqlMq\Model\Driver\Queue               $subject
 * @param \Magento\Framework\MessageQueue\EnvelopeInterface $envelope
 * @param bool                                              $requeue
 * @param string|null                                       $rejectionMessage
 * @return array<int, mixed>
 */
public function beforeReject(
    Queue $subject,
    EnvelopeInterface $envelope,
    bool $requeue = true,
    string $rejectionMessage = null
) : array {
    return [$envelope, true, $rejectionMessage];
}

That approach may work for you and might be a quicker way to achieve what you want without having to implement queue/retry state management.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.