Cm_Cache_Backend_Redis - Suspected config cache invalidation

Question

I have a cluster with 8 webserver nodes and a redis cache instance running on a separate server.

During high load, the throughput on the redis server goes through the roof, approximately 35-40 MB/s (megabytes). I inspected the redis commands with MONITOR on the redis instance and I discovered that there were a disproportionate amount of EVALSHA calls against the redis cache with the config tags.

This means, the web servers were sending data to the redis cache at a pace of 14 MB/s.

If I disable the config cache in the backend, the received bytes counter on the redis server came all the way down to 1 MB/s, which is reasonable.

I suspect that the config cache is getting invalidated somehow.

Have any of you seen anything like this before?

These are my config settings:

<cache>
  <backend>Cm_Cache_Backend_Redis</backend>
  <backend_options>
    <server>10.254.253.1</server> <!-- or absolute path to unix socket -->
    <port>6380</port>
    <persistent></persistent> <!-- Specify unique string to enable persistent connections. E.g.: sess-db0; bugs with phpredis and php-fpm are known: https://github.com/nicolasff/phpredis/issues/70 -->
    <database>0</database> <!-- Redis database number; protection against accidental data loss is improved by not sharing databases -->
    <password></password> <!-- Specify if your Redis server requires authentication -->
    <force_standalone>0</force_standalone>  <!-- 0 for phpredis, 1 for standalone PHP -->
    <connect_retries>2</connect_retries>    <!-- Reduces errors due to random connection failures; a value of 1 will not retry after the first failure -->
    <read_timeout>10</read_timeout>         <!-- Set read timeout duration; phpredis does not currently support setting read timeouts -->
    <automatic_cleaning_factor>0</automatic_cleaning_factor> <!-- Disabled by default -->
    <compress_data>1</compress_data>  <!-- 0-9 for compression level, recommended: 0 or 1 -->
    <compress_tags>1</compress_tags>  <!-- 0-9 for compression level, recommended: 0 or 1 -->
    <compress_threshold>2048</compress_threshold>  <!-- Strings below this size will not be compressed -->
    <compression_lib>snappy</compression_lib> <!-- Supports gzip, lzf, lz4 (as l4z) and snappy -->
    <use_lua>1</use_lua> <!-- Set to 1 if Lua scripts should be used for some operations -->
  </backend_options>
</cache>

Answer 1

Is the first load on the page also slow ? So is the cache also generated again?

I have seen before that there was a special character in the config cache ( copy / paste from Word ) that made the config cache always invalid.

Answer 2

We've seen the same behavior on one site under high load. Flicking the site into maintenance mode momentarily allows the cache to normalize and carry on correctly afterwards, but we've not been able to establish a root cause.

Answer 3

I had the same problem at some point, after a lot of time spent investigating it seems that the config that magento was building was invalid. i linted all the config.xml files and they were fine, but then I found some weird entry in the core_config_data table, holding utf data. Try dumping your config xml (the one that magento builds) and also look for suspicious data in the configuration table.

Answer 4

Have you tried disabling use_lua? Since the EVALSHA1 is in relation to it. According to the Colins Github config.xml example, standard is disabled. Also, a good set of examples can be found here.

<use_lua>0</use_lua> <!-- Set to 1 if Lua scripts should be used for some operations -->

If you require the use of LUA scripts than look into lua_max_c_stack as this threshold may be encountered causing flushes.

• https://github.com/colinmollenhour/Cm_Cache_Backend_Redis/blob/master/Cm/Cache/Backend/Redis.php#L87

You may also consider trying a different compression like lzf or gzip as snappy may be causing the odd behaviour, especially if its a non standard compression built-in to linux. Disabling compression all together may help as well.

Another possibility is with the high load your running out of memory. Make sure your maxmemory setting isn't too low for the needed high load traffic.

The recommended "maxmemory-policy" is "volatile-lru". All tag metadata is non-volatile so it is recommended to use key expirations unless non-volatile keys are absolutely necessary so that tag data cannot get evicted. So, be sure that the "maxmemory" is high enough to accommodate all of the tag data and non-volatile data with enough room left for the volatile key data as well.

Also if you have not implemented a garbage collection cron tasks, it is also recommended. Especially since Redis likes to hold on to its cache.

<?php PHP_SAPI == 'cli' or die('<h1>:P</h1>');
ini_set('memory_limit','1024M');
set_time_limit(0);
error_reporting(E_ALL | E_STRICT);
require_once 'app/Mage.php';
Mage::app()->getCache()->getBackend()->clean('old');
// uncomment this for Magento Enterprise Edition
// Enterprise_PageCache_Model_Cache::getCacheInstance()->getFrontend()->getBackend()->clean('old');

• https://github.com/colinmollenhour/Cm_Cache_Backend_Redis#example-garbage-collection-script-magento

Last bit, have not used this but recently discovered it, may not hurt to give it a go to see if it helps offer anymore insight into the problem:

• https://github.com/steverobbins/Magento-Redismanager