3

We have a multi server Magento installation using Redis as cache backend.

Every time I flush Redis (FLUSHDB or FLUSHALL) under traffic Redis runs into race conditions.

Calling

redis-cli -h xxx.xxx.xxx.xxx monitor | egrep set | cut -c 1-120

I get lots of lines like this

1444810735.659889 [0 int.ern.ip.3:55588] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES" "d" "gz:\...
1444810735.779509 [0 int.ern.ip.0:38044] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_PL" "d" ""gz:\...
1444810735.787633 [0 int.ern.ip.7:51360] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_SE" "d" ""gz:\...
1444810735.795483 [0 int.ern.ip.4:33149] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_EU" "d" ""gz:\...
1444810735.803374 [0 int.ern.ip.4:33142] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_CZ" "d" ""gz:\...
1444810735.831377 [0 int.ern.ip.1:40537] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_UK" "d" ""gz:\...
1444810735.840627 [0 int.ern.ip.7:51359] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_EU" "d" ""gz:\...
1444810735.847732 [0 int.ern.ip.7:51356] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_AT" "d" ""gz:\...
1444810735.854708 [0 int.ern.ip.1:40535] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_FR" "d" ""gz:\...
1444810735.864083 [0 int.ern.ip.7:51345] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_PL" "d" ""gz:\...
1444810735.872110 [0 int.ern.ip.0:38046] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_CH" "d" ""gz:\...
1444810735.880313 [0 int.ern.ip.4:33145] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_CZ" "d" ""gz:\...
1444810735.887541 [0 int.ern.ip.4:33148] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_EN" "d" ""gz:\...
1444810735.897101 [0 int.ern.ip.0:38044] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_FI" "d" ""gz:\...
1444810735.904982 [0 int.ern.ip.4:33149] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_AT" "d" ""gz:\...
1444810735.912321 [0 int.ern.ip.4:33142] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_PL" "d" ""gz:\...
1444810735.919390 [0 int.ern.ip.7:51360] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_EN" "d" ""gz:\...
1444810735.929803 [0 int.ern.ip.7:51359] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_AT" "d" ""gz:\...

again and again. Sometime after some minutes the global caches could be written and everything is ok. But under heavy traffic I have to take the shop offline and warm the cache manually.

My Redis config:

   <backend>Cm_Cache_Backend_Redis</backend>
    <backend_options>
        <server>xxx.xxx.xxx.xxx</server>                          <!-- or absolute path to unix socket for better performance -->
        <port>6379</port>
        <database>0</database>
        <persistent></persistent>
        <force_standalone>0</force_standalone>                   <!-- 0 for phpredis, 1 for standalone PHP -->
        <automatic_cleaning_factor>0</automatic_cleaning_factor> <!-- Disabled by default -->
        <compress_data>1</compress_data>                         <!-- 0-9 for compression level, recommended: 0 or 1 -->
        <compress_tags>1</compress_tags>                         <!-- 0-9 for compression level, recommended: 0 or 1 -->
        <compress_threshold>20480</compress_threshold>           <!-- Strings below this size will not be compressed -->
        <compression_lib>gzip</compression_lib>                  <!-- Supports gzip, lzf and snappy -->
    </backend_options>

Any idea to prevent such race conditions?

  • BTW: We now tested with phpredis and Credis lib from CM. Same thing. – waltavista Oct 26 '15 at 13:55
  • I am wondering if this is more suited for a server group? – brentwpeterson Nov 16 '15 at 16:22
  • It also happened if we use only one server of our cluster. If you have a lot of traffic it ends in race condition. Also with one server we have to take the shop offline for a short moment and warm config cache. We will definitely try the answer below and give feedback asap. – waltavista Nov 20 '15 at 15:30
3

The 'cache is corrupt' there is to short time to rebuild the core config cache, it's to big.

More information or explanation and the credits; https://github.com/AmpersandHQ/magento-ce-ee-config-corruption-bug

The fix is easy, in app/code/core/Mage/Core/Model/Config.php

change the function: public function init($options=array())

To :

$cacheLoad = $this->loadModulesCache();
if ($cacheLoad) {
    return $this;
}
// Cache fix 
$this->_useCache = false;
// Fix End

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.