Downloadable products - zip files cut off at just over 1GB mark

Question

We've had a problem with our Magento store for a long time now (used to be Magento 1.4, recently upgraded to 1.9.2) and after a few days pulling my hair out trying to find a solution without success, I thought I'd post here in case anyone has had similar problems or has any ideas what else to try.

Our Magento store sells some digital products - ranging in size from a few hundred MB to a couple of GB.

A lot of customers complain on the larger products that their downloads cut off at around the 1 gigabyte mark, and lots of testing ourselves over the last few days I've been unable to figure out the cause or a solution. I've been testing with 1.9GB file that has caused problems for many of our customers. (Note however that the download does not fail every time -- just more often than not!)

When the file is downloaded directly -- via the filesystem or a custom test .php script -- the download almost always finishes in its entirety.
When downloaded via Magento downloadable product link (testing via 'sample'), the download almost always stalls and then cuts off at around 1.08GB - 1055MB (different byte sizes each time -- have hex-inspected the end of the downloaded files and can't see any incorrect headers etc). This is the same thing that customers report, and I can see this in the log files too -- but not everyone experiences this, and not every time -- just the vast majority of times!

Our Magento 1.9.2 is running on a dedicated server, running Centos with nginx.

Things that I have tried changing in order to solve this problem are:

• disabling CloudFlare (but problem occurred before I ever used CloudFlare, and persists after it is disabled)
• disabling nginx gzip support
• increasing PHP.ini gc.max_session_lifetime setting
• increasing max_execution_time setting
• set the 'sample link' to use a custom php script that uses X-Accel-Redirect (script downloads OK when accessed directly, but commonly fails at ~50% when used through Magento)

I've attempted to inspect the HTTP traffic to find out exactly what's going on, but have been unable to. I also haven't been able to spot any pattern around when

When inspecting the request & response headers between the 'working' direct-download requests and the non-working Magento download requests, I can see some differences, for example:

• Magento sends Content-Type as application/zip, custom scripts send Content-Type as application/octet-stream;
• Magento sends Expires: header, custom scripts do not. (Expires: header says 1981, if that may make any difference!)
• Magento sets cookies, the custom scripts do not.
• Magento sends an X-Powered-By: PHP/5.4.27 header, the custom scripts do not;
• Custom scripts send an ETag: header, Magento does not;

Short of attempting to implement a replacement Downloadable class that hands file downloads off to nginx via X-Accel-Redirect -- and I don't know if that would even help or just be a waste of time? -- I don't know what to try next.

If anyone has any ideas where to look, light to cast on the technologies involved - or even better if you've had this problem yourself and know how to fix it!

Answer 1

In the end, I wrote a simple Magento module that uses X-Accel-Redirect to handle the downloads (extending Mage_Downloadable_DownloadController and overriding the _processDownload function). I haven't had a customer complain about a downloading problem since this was implemented.