The performance benefits of the full page cache in Magento Enterprise are fairly well known. What may not be quite so well known is that for the full benefit of this to be realized, it must be fully populated and hot, particularly on large product sets where you don't have just a few pages thus making use of organic traffic to prime it speedy enough.

Magento includes a built-in cronjob to crawl the site and warm the FPC early in the morning.

I've seen and heard of issues caused by early morning jobs taking too long to run, blocking other jobs from running, and would like to know what others use or would suggest be used to do this. A couple ideas I have are:

  • Put together a shell script to crawl every page in the generated sitemap file.
  • Use a separate crontab entry and a short PHP script to bootstrap Magento and execute the crawler process directly.

Any thoughts and/or experience on this is welcome!

locked by philwinkle Oct 16 '15 at 23:20

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

Read more about locked posts here.

  • 1
    Actually you can call the Enterprise crawler from a seperate file and use your servers crontab to trigger it so it won't be in the way. – Toon Van Dooren Dec 3 '13 at 21:46

You could use siege in combination with the sitemap.xml file, like MageSpeedTest does.

curl http://yourmagentostore.com/sitemap.xml | sed 's/\<url\>/\<url\>\n/g' | grep 0.5 | sed 's/.*loc>\(.*\)<\/loc.*/\1/g' > urls.txt
curl http://yourmagentostore.com/sitemap.xml | sed 's/\<url\>/\<url\>\n/g' | grep 1.0 | sed 's/.*loc>\(.*\)<\/loc.*/\1/g' >> urls.txt

Then run

siege -i -c 1 -t 7200s -f urls.txt

Content sourced from here.

  • You can also add a delay between requests using –delay – Ben Lessani - Sonassi May 16 '13 at 21:56
  • Note: These sed commands do not work on Darwin, but do on CentOS. – davidalger May 18 '13 at 0:05
  • 1
    This doesn't guarantee every url will be "warmed". siege will randomly select URLs to hit from the file, but will not necessarily visit every URL. – Joe Constant Jul 24 '13 at 7:04

We just don't - at all. Ever. We'll say this over and over again but

Caching != Performance

Your site needs to be fast without the addition of FPC (or Varnish for that fact). There is always going to be a time when the content isn't primed (your scenario above).

On an unloaded store, page load times with FPC shouldn't be that much more impressive than non-FPC; Magento is quite happily capable of < 400ms page load times on standard caches (on category/product/search pages). FPC will bring that down to < 80ms - but comes with caveats.

  1. Stock/price information is out of date until invalidation or TTL expiry
  2. New items/more relevant search is out of date until invalidation or TTL expiry


Why reliance on FPC (or Varnish) is A Bad Idea

If you're looking to continually ensure caches are primed manually, there's likely a few reasons

  1. You don't have enough natural footfall to keep the caches primed (see 'Where FPC is useful')
  2. Your site is too slow without them

You can't cache everything

If you take a store with just 5 categories, nested 2 levels deep, 5 filterable attributes, 5 attribute options each and 1000 products; that is a lot of possible combinations.

25 options to choose from, picking one up to 5 times in a row - I'm no statistician, but I'm aware that is ... (assuming the number of attribute options doesn't decrease completely)

25 possible URLs on the first selection
20 possible URLs on the second selection
15 possible URLs on the third selection
10 possible URLs on the fourth selection
5  possible URLs on the fifth selection

5^5 = 3,125 possible combinations (for top level categories)
5^4 = 625 possible combinations (for 2nd level categories)

Ok, the above is not a likely scenario, as I would imagine, within 3 clicks - the number of available products would have decreased sufficiently for the customer to find their product. So even if it were ...

25 possible URLs on the first selection
10 possible URLs on the second selection
3 possible URLs on the third selection

5^3 = 125 possible URL combinations 

Then times that by 5 categories, that is 625 URLs. At this stage, we're talking about a tiny catalogue, and completely ignoring all the product URLs.

We're also not factoring in that if you had nested categories with is_anchor on, its going to exponentially increase.

So to crawl that volume of pages - you've either got to hope that your page load times are nice and low to begin with, so that it is a quick lightweight process (thus defeating the purpose of the crawl) - or that you have enough time for it to complete before the TTL expires.

If your pages had a page load time of 0.4s and you had a 8 core CPU - then ...

625 * 0.4 = 250 / 8 = 31 seconds

0.5 minutes, not bad - but lets imagine you had 2s page load times

625 * 2 = 1250 / 8 = 156 seconds

But if you took the maximum possible scenario

3,750 * 2 = 7,500 / 8 = 937 seconds ~ 15 minutes

So that's your production server, under 100% CPU load for 15 minutes. You would reduce the crawl speed proportionately to the TTL that you want.

So if you want the content to have a 3600s TTL, the crawl could be 4 times slower - ie. only 25% CPU dedicated to the crawl. That's a lot of resource just to keep category content primed - we haven't even factored in products, search terms or additional store views at this stage

In fact, just looking at the sheer size of combinations in the catalog_url_rewrites table (which isn't even factoring in parameters from the layered navigation) will give an idea as to how many URLs you could end needing to crawl.

Every store will certainly be different, but what I'm trying to strike home is that crawling the site to prime FPC isn't practical. Just ensure your store is fast to begin with.

Where FPC is useful

Where the benefits of FPC come into play is on a heavily loaded store - where you have genuinely high levels of traffic and the caches are naturally and continually primed by sheer foot-fall alone.

FPC then comes into play by reducing infrastructure overheads on commonly requested content - cutting down on those repeated calls to the Magento backend.

So we've found that FPC is great to deploy when you've got very high traffic levels - not to reduce page load time - but to reduce resource usage.

Who cares, I still want to crawl

Well, then you've got two options

  1. Crawl from a template (Eg. sitemap)
  2. Extract links page by page and crawl each

And there are many utilities to do both of these, these are some I know of

  1. mage-perftest
  2. HTTrack
  3. Nutch
  4. Sphider
  5. Crawler4j

Using Mage-Perftest

You can crawl your store with Mage-Perftest pretty easily, first download it

wget http://sys.sonassi.com/mage-perftest          (64bit) OR
wget http://sys.sonassi.com/mage-perftest-i386     (32bit)
chmod +x http://sys.sonassi.com/mage-perftest*

Then define the crawl process using the Magento sitemap (you can customise this by making a sitemap of any URLs, provided the urls are wrapped in <loc></loc> tags). The following command will read all the URLs from the sitemap file, then crawl (PHP only) the URLs over the course of 1440 minutes (1 day). If the server exceeds 20% CPU or a load average of 2 - the crawl will pause temporarily.

./mage-perftest -u www.example.com -s www.example.com/sitemap.xml -r auto -b -d 1440 -z -a 20 -l 2  

If you have 1000 URLs, crawled over 1 day, that will be approx. 1 request every 86 second(s) ~ target of 0.011 RPS

  • You're opening line is very true… page caching is not the way to achieve performance. I know this. You don't know how many times I've told clients the same thing. I'll be honest, I've never setup a site where we have a crawler priming the FPC before, and have only seen it used once where a client enabled it in the admin…slowing things since they file-based cache tags. The main reason I'm asking is because I'm exploring ideas related to this based on some research in Nexcess' white paper. For insanely high traffic sites, priming the cache after flushing it early in the morning can be critical – davidalger May 17 '13 at 13:26
  • 1
    I respect Nexcess - but their white paper focuses very heavily on caching to achieve performance - rather than ensuring the environment is already performant and that the code is clean, fast and efficient. We provide Varnish for our customers - but don't advocate its use until required. Only then as a means to reduce infrastructure costs - ie. when ~94% of non-converting/checkout traffic is consuming CPU cycles. Caching makes for nice artificial benchmark stats - but means nothing in reality if the TTLs are too long (stale content) - or there isn't enough traffic to keep it primed. – Ben Lessani - Sonassi May 17 '13 at 14:13
  • 1
    For insanely high traffic sites - we've got a few, and trying to keep a cache hot through artificial crawling is pointless - the natural traffic does that just fine. If anything, crawling just removes resources that would otherwise be used by customers. – Ben Lessani - Sonassi May 17 '13 at 15:14
  • I beg to differ on their white paper focusing on using caching for performance. They were showing how much throughput a 2 + 1 cluster could achieve. They didn't even touch on page load times in it, only transaction throughput. The hardware they have is just about as optimized as you can get… and yes, I realize the effects of TTLs on cached content. Just to re-iterate, I'm not looking to achieve performance here, we already have that. What this would be exploring is ways to bypass lags/drops in throughput due to early morning cache flushing, i.e. before the normal traffic picks up. – davidalger May 17 '13 at 15:31
  • 1
    I'm confused then. If your store is already fast - but falls over when you clear the cache. Either a) Don't clear the cache in the morning, do it the night before and let the search crawl bots (google/bing etc.) do the priming for you or b) get the right infrastructure. If your store hinges on FPC/Varnish to prevent lag/slowdown - then it sounds like you're running on a knife-edge ... – Ben Lessani - Sonassi May 17 '13 at 15:50

I'll save my full rant for a blog post one these days, but in the meantime have a peak at my little cache warmer wfpc.

Testing performance

You can test the performance of your Magento site

./wfpc -t http://mymagentosite.com/sitemap.xml

Finished testing your Magento site performance
Total download time (in seconds)   : 5.0269110202789
Total download time (formatted)    : 0:0:5.026
Average page time (in milliseconds): 502.69110202789

FPC Warming

And you can warm the FPC, which will hit every URL in sitemap.xml.

./wfpc -w http://mymagentosite.com/sitemap.xml

You can also put a delay between requests if you like, here's a 1 second delay between requests.

./wfpc -w -d=1 http://mymagentosite.com/sitemap.xml

The test mode only hits 10 URLs randomly, so once you've warmed your FPC, you can run the test mode to find out how much of a difference the FPC makes!


Personally, I think a warmer makes sense... On a small site with about 40 pages, download time is cut roughly in half by the FPC. On a large site with nearly 40,000 products using Lesti_FPC with APCu as the backend, I'm using a little over 200MB for the cache, which frankly is nothing on the 8GB production server.

Not the answer you're looking for? Browse other questions tagged or ask your own question.