I run into this issue after upgrading from magento 188.8.131.52 to 184.108.40.206.
After tracking the logs, there was nothing wrong in the client behavior. He just navigated a couple of categories and suddenly it started failing.
Even though all pages were running properly with 256M of memory limit, for this particular client it was failing. so following everybody's advice I kept increasing memory until 4gb, and it didn't work... so something else is wrong.
It seemed like 2 potential problems could be happening:
- an infinite loop
- an extremely slow processing task
I tried lots of things since the magento, php and nginx logs weren't offering much information.
I found out that if you delete the client session from the DB or the var/sessions folder (depending on your implementation), the 500 error disappears. (though, it didn't explain if it was due to infinite loop or just slow processing).
In order to do that you can find the user's ip address in the server logs, and running a grep into the sessions folder or a query in the core_session table would take you to the client session.
As a potential ugly solution you could create a bash script that runs against the server logs looking for this error, find the session and delete it. Same thing you can achieve with loggly alerts, amazon CloudWatch. etc.
I kept investigating to answer to my question, and waited until the next client experienced the same issue, then I increased the max execution time to 6 minutes and memory to 6 gb with the hopes that finally magento would finish loading those requests (without running out of memory or execution time), and it worked. After finishing the request (taking several minutes) the 500 error disappeared for that client.
So it wasn't an infinite loop, it was a performance issue... even though it didn't seem so.
I implemented loggly and I'm waiting for it to happen again so I can activate the profiler immediately to find where the memory leak comes from. I'll update this answer if I'm successful.