How best to call a custom long running PHP program over and over

Question

I wrote a PHP cli program that I call many times in a loop, each loop processing a set number of rows at one time until there is no more data.

It is a custom REST program that is the middle piece between Magento and a drop shipper.

Each run may take up to 25 minutes and there may be up to 400 runs.

When I had the process in one PHP program I ran into timeout and memory issues.

So now I have a bash script that calls the PHP program as many times as needed until there is no more data. The method works.

I am concerned that I am not following Magento standards by using a bash shell program.

The Magento cron system seems to only support php calls and not a bash call, so I have to use the OS cron.

Is there a better way for running a program over and over again?

Secondarily, is there a way to open a bash window from within the admin?

Answer 1

When dealing with scripts that run for a very long time, in PHP isn't really what PHP is designed for. I haven't run into a piece of software that is able to run hours on one end without running into memory problems.

When I had the process in one PHP program I ran into timeout and memory issues. These are the issues that all developers run in when they are building their first long-running scripts. No shame in that.

The tools that run very long usually make sure they do not spawn a lot of objects (reuse them) and do not create new variables in each loop. These programs are usually developed with the memory requirement as one of the most important requirements. This isn't easy, developing with a low memory requirements is a lot harder than doing everything in memory and just iterating through everything.

Your solution of using bash is fine, if it works and you do not need to deploy your installation to multiple servers each week, I see no harm in doing it this way.

Magento's cron is a system that allows you to define cron jobs without actually having to set up a cron each time on each server, you simply define it in Magento's XML config (when distributing modules this is very handy). It is more development friendly, but it doesn't fit every solution (like yours).

That being said, if you scripts runs for 25 minutes x 400 you are probably having troubles making changes, developing it further etc. The development iteration when adding features is very long making actual script improvements very difficult. Make sure you have figured this out, I have been burned by this more than one time.

Secondarily, is there a way to open a bash window from within the admin? Nope, that would be a potential security issue.

If you are looking to start the scripts from the Magento backend you could build a Magento cron job that exactly does the same as your bash script, you can even run shell scripts from there (http://php.net/manual/en/function.shell-exec.php). This way you could schedule the job with something like https://github.com/fbrnc/Aoe_Scheduler.