4

I have a situation where I need to process a request that will create a queue of messages to be sent, and I need this messages to be processed and send continuously.

The current solution I have in my mind is to create a Magento cronjob that will check every n minutes if my script.php is running.

If it's running, all good. If not, it will start running it, and this is script will have a infinite loop that process all messages on the queue.

I don't know why, but something in this solution sound sketchy to me, so I'd like to know if there's a better solution for that. I don't want to use RabbitMQ or other linux-like software.

EDIT: for a better context, I'm trying to guarantee messages are processed and sent in a personal chatbot project that integrate Magento to Facebook Messenger and Telegram (https://github.com/blopa/Magento-Chatbot/)

Improve this question

1 Answer 1

Reset to default
3

It does not sound this bad. A cronjob is a good start. Instead of triggering a second script, you can do the processing as part of the cronjob. The easiest is to limit the number of processed messages low enough, that it is finished before it is run the next time. Running it every 5min should be often enough for most cases.

If the limit is not possible, because it sometimes needs more time, or because you need to process as much messages as possible in one run, you can use some kind of locking mechanism.

[UPDATE]

ok, as I now know more about the scope, we need to think a bit bigger. As you need a guarantee for messages to be processed you will need a queue system. This can be RabbitMQ(which is the most stable and tested solution), this can also be Kafka(probably no, this is really to much), or my favorit as a first step, a simple queue table which consinsts of only the necessary collums. (mysql scales quite well in general) The reason is, you must be able to split "accepting new messages" from processing them. The processing must be able to fail, without loosing the initiating message, so you are able to retry. From experience I know, that some "requests" will need far more then just a few seconds to process, but you should be able to accept new requests from other users while processing a request. Also, some Shops may want to scale the process up, because they have such a big audience, that a few 100 requests per minutes are not unlikely. Having a proper queue system allows to scale the processing part up. This means, you should also use an additional queue to schedule the results for sending out. Because the accepting/sending part is very hard to scale higher then 1 worker per chatservice.

Improve this answer
8
  • Thank you for your feedback, @Flyingmana. I'll use that for a chatbot, so messages must be sent out as fast as possible, that's why I'm thinking about using a infinite loop script, to make sure all messages are going to be processed ASAP. The cronjob would be like a safety measure to make sure the script is always running. I don't want to use RabbitMQ or other linux-like softwares.
    – Pablo
    Commented Sep 29, 2017 at 2:32
  • I see, that changes things a lot. What kind of Data do you need from magento, and how fast do you want responses to be? And are messages from users of interest? Is connecting/disconnecting a noticable/annoying action visible for users?
    – Flyingmana
    Commented Oct 2, 2017 at 14:03
  • There are plenty of different processes to be handled by Magento. Pretty much everything is written in here github.com/blopa/Magento-Chatbot
    – Pablo
    Commented Oct 2, 2017 at 14:18
  • This is a module developed by me, and currently it just get the messages request from client, process it, and respond, with no guarantee that that response will be sent if a exception occurs
    – Pablo
    Commented Oct 2, 2017 at 14:19
  • 1
    Thank you very much @Flyingmana. Just to make things clear, you mean something like this: i.imgur.com/BtzFXcS.jpg 1 - User send the message and that messages is saved on database. And that's it. 2 - A endless worker will consume all input messages and process them AND a endless worker will consume all output messages and send them back to the chat. My main fear is, what happens if a worker fail in some random exception? How can I guarantee it will be up and running any given time? I'm afraid of that because I never trully worked with threads in PHP before.
    – Pablo
    Commented Oct 22, 2017 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.