How to prevent Redis cache race conditions

Question

We have a multi server Magento installation using Redis as cache backend.

Every time I flush Redis (FLUSHDB or FLUSHALL) under traffic Redis runs into race conditions.

Calling

redis-cli -h xxx.xxx.xxx.xxx monitor | egrep set | cut -c 1-120

I get lots of lines like this

1444810735.659889 [0 int.ern.ip.3:55588] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES" "d" "gz:\...
1444810735.779509 [0 int.ern.ip.0:38044] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_PL" "d" ""gz:\...
1444810735.787633 [0 int.ern.ip.7:51360] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_SE" "d" ""gz:\...
1444810735.795483 [0 int.ern.ip.4:33149] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_EU" "d" ""gz:\...
1444810735.803374 [0 int.ern.ip.4:33142] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_CZ" "d" ""gz:\...
1444810735.831377 [0 int.ern.ip.1:40537] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_UK" "d" ""gz:\...
1444810735.840627 [0 int.ern.ip.7:51359] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_EU" "d" ""gz:\...
1444810735.847732 [0 int.ern.ip.7:51356] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_AT" "d" ""gz:\...
1444810735.854708 [0 int.ern.ip.1:40535] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_FR" "d" ""gz:\...
1444810735.864083 [0 int.ern.ip.7:51345] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_PL" "d" ""gz:\...
1444810735.872110 [0 int.ern.ip.0:38046] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_CH" "d" ""gz:\...
1444810735.880313 [0 int.ern.ip.4:33145] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_CZ" "d" ""gz:\...
1444810735.887541 [0 int.ern.ip.4:33148] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_EN" "d" ""gz:\...
1444810735.897101 [0 int.ern.ip.0:38044] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_FI" "d" ""gz:\...
1444810735.904982 [0 int.ern.ip.4:33149] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_AT" "d" ""gz:\...
1444810735.912321 [0 int.ern.ip.4:33142] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_PL" "d" ""gz:\...
1444810735.919390 [0 int.ern.ip.7:51360] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_EN" "d" ""gz:\...
1444810735.929803 [0 int.ern.ip.7:51359] "hmset" "zc:k:my_cache_prefix_CONFIG_GLOBAL_STORES_AT" "d" ""gz:\...

again and again. Sometime after some minutes the global caches could be written and everything is ok. But under heavy traffic I have to take the shop offline and warm the cache manually.

My Redis config:

   <backend>Cm_Cache_Backend_Redis</backend>
    <backend_options>
        <server>xxx.xxx.xxx.xxx</server>                          <!-- or absolute path to unix socket for better performance -->
        <port>6379</port>
        <database>0</database>
        <persistent></persistent>
        <force_standalone>0</force_standalone>                   <!-- 0 for phpredis, 1 for standalone PHP -->
        <automatic_cleaning_factor>0</automatic_cleaning_factor> <!-- Disabled by default -->
        <compress_data>1</compress_data>                         <!-- 0-9 for compression level, recommended: 0 or 1 -->
        <compress_tags>1</compress_tags>                         <!-- 0-9 for compression level, recommended: 0 or 1 -->
        <compress_threshold>20480</compress_threshold>           <!-- Strings below this size will not be compressed -->
        <compression_lib>gzip</compression_lib>                  <!-- Supports gzip, lzf and snappy -->
    </backend_options>

Any idea to prevent such race conditions?

Answer 1

The 'cache is corrupt' there is to short time to rebuild the core config cache, it's to big.

More information or explanation and the credits; https://github.com/AmpersandHQ/magento-ce-ee-config-corruption-bug

The fix is easy, in app/code/core/Mage/Core/Model/Config.php

change the function: public function init($options=array())

To :

$cacheLoad = $this->loadModulesCache();
if ($cacheLoad) {
    return $this;
}
// Cache fix 
$this->_useCache = false;
// Fix End