Redis Session Slowness

Question

I've enabled redis for session, have the latest extension and lib code from github, and I'm noticing some requests take around 30 seconds - and this is happening pretty frequently, and it's always redis which is to blame.

Here are 3 transaction traces from newrelic, they look pretty much the same to me: https://i.sstatic.net/7ceh1.png

I am using the following settings in redis:

    <redis_session>
            <host>127.0.0.1</host>
            <port>6379</port>
            <password></password>
            <timeout>2.5</timeout>
            <persistent></persistent>
            <db>2</db>
            <compression_threshold>2048</compression_threshold>
            <compression_lib>lzf</compression_lib>
            <log_level>4</log_level>
             <!-- maximum number of processes that can wait for a lock on one session; for large production clusters, set this to at least 10% of the number of PHP processes -->
            <max_concurrency>12</max_concurrency>
            <!-- seconds to wait for a session lock in the frontend; not as critical as admin -->
            <break_after_frontend>5</break_after_frontend>
            <break_after_adminhtml>30</break_after_adminhtml>
            <!-- Bots get shorter session lifetimes. 0 to disable -->
            <bot_lifetime>7200</bot_lifetime>
    </redis_session>   

At the moment, only one frontend server is being used for magento.

Thanks!

Answer 1

It could be a dozen different things, and without a full overview of your entire server at that point in time, it isn't going to be possible to give a definitive answer.

It could be,

• redis-server process at 100% CPU
  • Due to a flush to disk of in-memory data for persistence
  • Due to key expiration during an eviction
  • Due to a heavy process being executed
• Server side bottleneck
  • If its a VPS/Cloud then it could be that another guest on the hypervisor is consuming resources, forcing yours to hang (tip. Don't use a VPS/Cloud if you want reliability and predictability)
  • OOM condition due to swapping etc. or exchange of memory from buffers/cache to userspace
  • High CPU load starving the Redis process
  • No available TCP states (ie. table full)
  • No available sockets
  • No available file descriptors
• Multi purposing Redis
  • Using the same Redis instance (not database) for both cache and sessions will cause this type of behaviour. If you are using Redis for cache too, then edit your init script to instantiate two separate demons on two different ports/sockets.

The list could go on and on. What you need is full and proper monitoring of your entire server, so that you can see exactly what is happening to make a diagnosis.

That includes graphing every application with Munin, historical logging with the Atop daemon, logging and centralisation of log data into a single dashboard.

New Relic is a nice tool, whilst useful for some identification of issues, its not necessarily the best tool to try and have full visibility of the cause of issues.

---

Easiest place to start would be to check if Redis needs more memory,

redis-cli
> info

Compare the peak memory with the limit you have defined.

---

NB. Your hosting provider should be able to answer this question in an instant, I would definitely suggest asking for their input.

Answer 2

Depending on your version (I'm using 1.8 here) Cm_RedisSession_Model_Session will sleep() on certain conditions:

Line 325

// Detect dead waiters
if ($tries == 1 /* TODO - $tries % 10 == 0 ? */) {
    $detectZombies = TRUE;
    // TODO: allow configuration of sleep period?
    usleep(1500000); // 1.5 seconds
}

Line 364:

if ($tries >= $this->_breakAfter+self::FAIL_AFTER) {
    // ...
    if ($this->_logLevel >= 5) {
        // ...
        Mage::log(
            sprintf(
                "%s: Giving up on read lock for ID %s %s",
                $this->_getPid(),
                $sessionId,
                $additionalDetails
            ),
            Zend_Log::NOTICE, self::LOG_FILE
        );
    }
    break;
}
else {
    // TODO: configurable wait period?
    sleep(1);
}

Create an Xdebug profile and see if this is where things are hanging up. Both of these lines have to do with getting a lock on the session.

I had this issue with a client where sleep() was getting called 20 to 30 times. We ended up switching to DB sessions. Alternatively, you could try upgrading Cm_RedisSession_Model_Session with the latest