Magento Extremely Slow When Cache "Full"

Question

We are running Magento 1.9.2.1 with Lesti_Fpc on an adequately sized managed server. Initially, we used the default file cache, which was fine. But after the catalog grew (although I think ~8000 products is not too bad) and crawlers became more aggressive, the site became slow as soon as the cache was getting a bit bigger. When the cache was cleared, everything was running quick again.

We tried to switch to APC as a cache backend via the following entry in the local.xml:

<global>
    <cache>
        <backend>apc</backend>
        <prefix>MYSHOP_</prefix>
    </cache>
</global>

But this made the problems even worse. I then read that Cm_Cache_Backend_File is made for this problem and integrated it via:

<global>
    <cache>
        <backend>Cm_Cache_Backend_File</backend>
    </cache>
</global>

This feels a bit better, but the problem is still the same. To keep the cache small and tidy, I also integrated Aoe_CacheCleaner, but this also does not help. Still, as soon as the cache is cleared, everything is running quick again.

EDIT:
Based on the answer by infabo, I also activated Cm_Cache_Backend_File for the FPC with the file app/etc/fpc.xml and the following content:

<?xml version="1.0"?>
<config>
    <global>
        <fpc>
            <lifetime>86400</lifetime>
            <backend>Cm_Cache_Backend_File</backend>
        </fpc>
    </global>
</config>

I am sure this makes sense, but it also does not solve the problem.

I know that the general solution to this problem seems to be Redis (or maybe alternatively Memcached) as a cache backend, but unfortunately, it is not available on our managed server. Switching to another hosting company is not (yet) an option.

I investigated a lot now, but I do not have any more idea. Maybe anyone else can help?

Answer 1

I investigated even more and I think I finally solved the issue. So what can you do to analyse such an issue?

1. In order to have a good idea when the cache becomes too large and if the problem is really the cache size, add a cronjob which calls the following script e.g. each 15 minutes:

   #!/bin/bash
   
   # this script is an attempt to check if the full cache is the real performance problem
   # it should be called regularly as a cronjob
   
   cache_dir="/html/shop/var/cache/"
   log_file="/html/cache_log"
   
   line=$(date)
   line="$line Size of cache directory: $(du -hs $cache_dir)"
   echo "$line" >> $log_file
   
   line=$(date)
   line="$line Total cache tags: $(find $cache_dir'cm-tags/' -type f | wc -l)"
   echo "$line" >> $log_file
   

   Then you can analyse the contents of /html/cache_log to see how the cache size develops, when your page becomes too slow and if the root cause is really the cache.

2. Analyse your cache files. Therefore, it is quite helpful to write all cache files into a log file with e.g.:

   ls -R /html/shop/var/cache > /html/cachefiles
   

   Have a look at this file and the file names within. What kind of cache files are there? Is there anything suspicious? In my case, there were a whole lot of cache files containing AMSHOPBY in their file name - a reference to the Amasty Improved Navigation (Amasty_Shopby) extension. It was creating a whole lot of cache files. Some of them looked quite weird to me. Disabling the Amasty Improved Navigation cache solved the problem instantly. I contacted their support with a detailed description and the support was really good. The caching strategy was quickly completely overhauled and is now much better. They promised to integrate the patch in the next version of the extension, so every version > 2.8.3 should be fine.

Good luck in finding the root cause for your big cache!

Answer 2

Have you tried Cm_Cache_Backend_File as a backend in fpc.xml too? Maybe give that a try. I would give Aoe_Profiler a shot too. If you are able to reproduce the "slowdowns" on a staging-copy - go and profile your slow requests there. Otherwise you can do it on production (i strictly do not recommend it, but if you dare, you can configure the profiler to just be enabled when the GET parameter is set and go ahead)