At the company I work for, our architecture is set up in such a way that we have hundreds of Magento stores, and a variable number of application servers (behind a load balancer), but only one centralized database and file server. This has proven good for cost, reliability and scalability, but it it's not good for launching the cron for a few reasons:
1) The central file server doesn't have PHP, so we can't launch cron.sh from there.
2) There's no way to decide which app-server should run the cron (we don't want the same cron to run multiple times.)
3) Even if we could denote which one should run the cron, we'd have to intelligently balance this out across the farm.
4) Even if we could balance it out, this balance will be constantly changing as we add or take away app servers during load spikes and peak seasons.
5) One other idea I had was to have a separate application server only for cron processing, but this seems wasteful, and also not scalable because you will eventually need a second server when you get enough crons (I think we are already at this point.)
As a workaround, we have a separate server which sends an HTTP request to each store's cron.php using curl to kick off the cron. This helps to distribute the load because the load balancer can decide which server runs it, but we are starting to see weird issues where some stores crons don't run (too late in schedule,) or run more than once, or get scheduled more than once. After reading this article which explains why this implementation is prone to race conditions it got me thinking about changing to the cron.sh method, but to do so seems very complicated with the current state of our platform.
Is anyone else out there running a distributed multi-server, multi-store Magento platform with some success? How do you launch your crons?