5

We're running a Magento site, EE 1.14.1 and are having seemingly random problems with the site going down because all of the assets fail to load and the site reverts from our theme to the default enterprise theme (!!) -- all of our customized layout changes in our theme are not applied and the asset paths revert back to the default paths. The admin panel, similarly is unstyled and unusable.

Interestingly, when I look at the raw source of the page when the site is down, not only do the paths revert back to the default paths, but they are also missing the initial {skin|design|js} path, e.g: 

    <link rel="stylesheet" type="text/css" href="/frontend/enterprise/default/css/styles.css" media="all" />
    <link rel="stylesheet" type="text/css" href="/frontend/enterprise/default/css/widgets.css" media="all" />
    <link rel="stylesheet" type="text/css" href="/frontend/enterprise/default/aw_blog/css/style.css" media="all" />
    <link rel="stylesheet" type="text/css" href="/frontend/enterprise/default/css/print.css" media="print" />
   ...
    <script type="text/javascript" src="/mage/cookies.js"></script>

Furthermore, when the site is operating correctly, we have all of our assets minified and served from CloudFront. At first, I thought this may be the result of a CloudFront hiccup, but I don't think that this is the case.

We're also using Redis for FPC. Also, I can see the "hole punch" tags in the raw source when loading the page: 

<!--{GLOBAL_MESSAGES_aae3749ba9c2e308ffa9c240ac185959}--><!--/{GLOBAL_MESSAGES_aae3749ba9c2e308ffa9c240ac185959}-->

I have temporarily disabled CloudFront and am serving all assets from our server to see if that will address the issue. My next guess would be something with Redis that is causing the problem. Clearing our Redis cache invariably solves the problem, but it keeps recurring and never at the same time, but about once every day or so

I'm believe this has to do with applying the October 2015 SUPEE-6788 patch. We did not have this problem prior to applying this patch.

Has anyone experienced something similar or have ideas on how to best debug this problem? I'm kind of stumped - it's almost as if Magento completely throws out the configuration for this store.

Update 12/8/2015

Over the past couple of weeks, I have disabled Redis for the config cache and have been using file based caching. This has not fixed the problem. I also created a little observer to log all cache flushes in an admin file.

Thanks to @Persata and @Luke Rodgers's suggestions below (see comments), I was able to get a stack trace this afternoon - as they noted, it seems to occur shortly (anywhere from a second to 30 minutes) after a full page or block cache flush -- the result is a malformed configuration cache file. Thanks to Luke's patch that prevents the cache from being saved, it prevented the site from going down. Here's the trace: 

   /includes/src/__default.php(21686): Mage_Core_Model_Cache->save('<admin><design>...', 'config_global_s...', Array, NULL)
#1 /includes/src/__default.php(23473): Mage_Core_Model_App->saveCache('<admin><design>...', 'config_global_s...', Array, NULL)
#2 /includes/src/__default.php(23395): Mage_Core_Model_Config->_saveCache('<admin><design>...', 'config_global_s...', Array, NULL)
#3 /includes/src/__default.php(20960): Mage_Core_Model_Config->saveCache()
#4 /includes/src/__default.php(20882): Mage_Core_Model_App->_initModules()
#5 /app/Mage.php(684): Mage_Core_Model_App->run(Array)
#6 /index.php(87): Mage::run('', 'store')
#7 {main}

Unfortunately, I don't have the full output of the configuration that it is attempting to save but will modify my code accordingly. Interestingly, the no issues for over a week (with numerous cache flushes) before getting the trace this afternoon. Traffic was a little bit higher during this time, but not astronomical.

Improve this question
2
  • What do you get with no caching and no CDN?
    – Jonathan Hussey
    Nov 5, 2015 at 17:45
  • I will report back, @JonathanHussey - thanks. Right now everything is fine with both off. In the meantime, I double-checked all of Collin Mollenhour's ( github.com/colinmollenhour/Cm_Cache_Backend_Redis ) suggested tuning settings and everything looks okay. I noticed I was missing garbage collection script so I have added that to the mix for when I reactivate Redis.
    – Phil Birnie
    Nov 5, 2015 at 18:08

1 Answer 1

Reset to default
3

Edit #2: I decided to attach XDebug to our live server during one of the outages to debug the state of the config, and discovered that the store config had been cached incorrectly - specifically, the results of \Mage_Core_Model_Store::initConfigCache were all null values, which was leading to the issue with the paths we're experiencing. This meant that the database config wasn't being cached properly, since these values are all fetched from the DB inside \Mage_Core_Model_Config::loadDb.

This made me take a closer look at the GitHub I linked below, and following the steps it lists, I was able to replicate the issue. I attached XDebug to a local instance while running cron.php, and with a separate web request to the site's homepage, if the cache is cleared at the right point while the cron & the web request are functioning, it would create those null values for the store cache, which re-created the problem exactly.

After applying the Magento patch in that repo (PATCH_SUPEE-4755), I can no longer replicate the issue by running those processes at the same time - so, touch wood, I'm hoping this has solved the issue, and since applying the patch to our live server it has been stable.

This leaves something unanswered though - I still can't see anything in PATCH_SUPEE-6788 that would have caused this to happen. I'm going to keep digging and see if there's anything I can find.

Edit: Bad news, I've compared the config dump from during the outage against a config dump from the site being fully functional, and the only changes are things altered by my cronjobs that I'd expect to be different, otherwise there's nothing wrong. Next time I'll try and grab the layout config too.

I'm also in the exact same boat - EE 1.14.1, using Redis (3 separate instances, 1 for session, 1 for general cache, 1 for FPC), have recently applied the SUPEE-6788 patch and now we're experiencing the same issue you are, where the config seems to get corrupted / dumped and the store will load templates etc from an entirely different theme.

I've been over the SUPEE-6788 patch several times since this started happening, and I was unable to spot anything obvious in the changes that could be causing it, but our store has been stable for months, and now you're reporting the same issue, so I feel like this must be related to the patch somehow.

What is the server load like at the times when your issues start occurring? Ours appear to be around the time when we receive mass stock updates from our warehouse, and I'm wondering if it's due to heavy load - there are a lot of calls to our XMLRPC API, and that's something that did change with the patch - the new security measures in Zend/Xml/Security.php / Zend/Xmlrpc/Request.php / Zend/Xmlrpc/Response.php.

I was also looking at the following GitHub which suggests that the configuration cache can become corrupted at times of high server load - https://github.com/AmpersandHQ/magento-ce-ee-config-corruption-bug. I haven't applied the patch to our store, but I'm wondering if it's related and perhaps worthwhile trying it?

With the most recent occurrence of this issue I managed to take a dump of the configuration XML from the Redis server, I'll be looking over it for anything unexpected / peculiar, I'll report back with my findings.

Improve this answer
9
  • While I'm sorry to hear that you're dealing with it too, I'm glad to hear I'm not the only one. Thanks for reporting back. I also took a good look at the patch, but I'll walk through it one more time as well. I'll also open a support ticket and report back.
    – Phil Birnie
    Nov 6, 2015 at 1:03
  • Thanks for the updates @Persata - nice work! I still have a ticket in with Magento and gave them the information you provided above... I'll mark this as answered once confirming that this fixes it. Glad to hear you are nearing resolution!
    – Phil Birnie
    Nov 10, 2015 at 5:36
  • Just another update - in short, the above solution may not work for everyone. Same issue occurred again after employing the patch, but it's definitely a corrupt configuration storage as @Persata notes.
    – Phil Birnie
    Nov 18, 2015 at 2:46
  • 1
    Hey @PhilBirnie I recommend you put the following into an instance where you receive the 100 router match error. github.com/AmpersandHQ/… I know it's a bit scary, but it will A) prevent so many 100 router errors, B) give us some debug data so we can see what other path is causing this 100 router error! I'm the author of the write up and want to know more / may be able to help :)
    – Luke Rodgers
    Nov 18, 2015 at 11:42
  • That's quite concerning that the issue is still occurring, someone reports the same here after applying the patch also. Our store has been working fine since applying that patch though, which is even more perplexing. Did you hear anything useful back from Magento support, @PhilBirnie?
    – Persata
    Nov 20, 2015 at 10:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.