17

If I'm logged into Magento's backend and perform some task that takes a long time (global search on large catalogs, long running dataflow, etc), my web browser will refuse to load other admin pages in that browser only. Why does this happen, and is there any known science for workarounds?

That is, if I

  1. Log into Magento's dashboard page

  2. Open a second tab with any Magento admin page

  3. Perform a long running global search (simulated with a call to sleep(30) at the start of globalSearchAction) in the first tab

  4. Attempt to reload the second tab

Expected Behavior: Second tab loads with page contents immediately

Actual Behavior: Second tab only loads once the long running global search completes

Does anyone know, specifically, why this happens? (My guess is Magento admin console requests lock some resource Magento needs to bootstrap, but I don't know what that is)

Does anyone know of a fix/workaround?

  • 1
    You, sir, have just sent me on a challenge! :) – davidalger May 15 '13 at 1:56
21

The problem is caused by a lock placed by the PHP session handler. So it's not Magento explicitly locking something and trying to block admin requests, but almost a side-effect per-se of file-based session storage.

A write lock is being placed on the session data file when it is opened by the initial (long-running) request, causing the second request to block until the lock is released when it calls session_start in Mage_Core_Model_Session_Abstract_Varien::start

This is 100% reproducible. I used the same method you did, adding a sleep(30) to the top of Mage_Adminhtml_IndexController::globalSearchAction

Worth noting is that this cannot be reproduced if you are using db session storage. After I found the root cause, I set a sandbox to db session storage and could no longer reproduce the issue. So the db session handlers Magento has seemingly do not use row-level locking to lock session writes. I find this interesting, because it has the potential for session data loss since the application is obviously not accounting for multiple threads writing to the same session. Note for Readers: I would never use db session storage in production to try and solve this, it's only good for overloading your MySql database.

I didn't try reproducing the behavior using memory based session storage systems such as Redis, but my guess is that locking the records in the session store was probably overlooked in these as well.

There are techniques which could be employed to avoid this like using session_write_close to release the lock before you start on a long-running job. But this would also prevent you from writing to the session since you've just closed it. So it's not likely to be readily implemented across the board in Magento, but could potentially be implemented on specific routes/controllers.

My technique for pinning this as the root cause was to enable the Xdebug profiler and exam the "cachegrind" file. Once the second request completed, I loaded the output file (~25 MB log) into MacCallGrind and drilled down into the trace following the path of calls where the inclusive time was 28 seconds or greater. This ultimately led me to the session_start call which took ~28 seconds to run, giving me a great point to research from.

EDIT: For the interested, I've posted a screenshot of the "cachegrind" file viewed in MacCallGrind on Twitter.

  • Unirgy's uRapidFlow module closes the session in order to avoid this issue, which is nice in a way, but it's frustrating in as much as it makes it difficult to provide feedback to the user afterwards. – Peter O'Callaghan May 15 '13 at 12:23
  • @davidalger — What session storage do you typically deploy with for client sites? – Alan Storm May 15 '13 at 14:36
  • 3
    Re: locking the PHP session file, this is less for for data integrity than it is for the integrity of the file-on-disk itself. Having multiple processes opening and writing to bits on disk (which is what a file is) will quickly lead to corruption of the data. MySQL, redis, and most databases are designed specifically to stay consistent, even if there's multiple writes at nearly the same time. i.e. it's not that locking was overlooked, it's that it's not as critically needed. – Alan Storm May 15 '13 at 14:42
  • @AlanStorm — A dedicated Memcached instance for a load-balanced installation, file-system for single application node clusters. File-system performs best where you don't have multiple nodes since you have no IP latency. – davidalger May 15 '13 at 14:47
  • Correct, locking the file is all about preventing corruption of data. However, holding the lock till the session is closed is about preventing data loss. If two processes load the session data, modify it and then write it out, one of them is going to have it's modifications overwritten. Not generally going to pose a critical problem, but could cause data loss and/or nasty difficult to debug problems. – davidalger May 15 '13 at 14:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.