We just don't - at all. Ever. We'll say this over and over again but
Caching != Performance
Your site needs to be fast without the addition of FPC (or Varnish for that fact). There is always going to be a time when the content isn't primed (your scenario above).
On an unloaded store, page load times with FPC shouldn't be that much more impressive than non-FPC; Magento is quite happily capable of
< 400ms page load times on standard caches (on category/product/search pages). FPC will bring that down to
< 80ms - but comes with caveats.
- Stock/price information is out of date until invalidation or TTL expiry
New items/more relevant search is out of date until invalidation or TTL expiry
Why reliance on FPC (or Varnish) is A Bad Idea
If you're looking to continually ensure caches are primed manually, there's likely a few reasons
- You don't have enough natural footfall to keep the caches primed (see 'Where FPC is useful')
- Your site is too slow without them
You can't cache everything
If you take a store with just 5 categories, nested 2 levels deep, 5 filterable attributes, 5 attribute options each and 1000 products; that is a lot of possible combinations.
25 options to choose from, picking one up to 5 times in a row - I'm no statistician, but I'm aware that is ... (assuming the number of attribute options doesn't decrease completely)
25 possible URLs on the first selection
20 possible URLs on the second selection
15 possible URLs on the third selection
10 possible URLs on the fourth selection
5 possible URLs on the fifth selection
5^5 = 3,125 possible combinations (for top level categories)
5^4 = 625 possible combinations (for 2nd level categories)
Ok, the above is not a likely scenario, as I would imagine, within 3 clicks - the number of available products would have decreased sufficiently for the customer to find their product. So even if it were ...
25 possible URLs on the first selection
10 possible URLs on the second selection
3 possible URLs on the third selection
5^3 = 125 possible URL combinations
Then times that by 5 categories, that is 625 URLs. At this stage, we're talking about a tiny catalogue, and completely ignoring all the product URLs.
We're also not factoring in that if you had nested categories with
is_anchor on, its going to exponentially increase.
So to crawl that volume of pages - you've either got to hope that your page load times are nice and low to begin with, so that it is a quick lightweight process (thus defeating the purpose of the crawl) - or that you have enough time for it to complete before the TTL expires.
If your pages had a page load time of 0.4s and you had a 8 core CPU - then ...
625 * 0.4 = 250 / 8 = 31 seconds
0.5 minutes, not bad - but lets imagine you had 2s page load times
625 * 2 = 1250 / 8 = 156 seconds
But if you took the maximum possible scenario
3,750 * 2 = 7,500 / 8 = 937 seconds ~ 15 minutes
So that's your production server, under 100% CPU load for 15 minutes. You would reduce the crawl speed proportionately to the TTL that you want.
So if you want the content to have a 3600s TTL, the crawl could be 4 times slower - ie. only 25% CPU dedicated to the crawl. That's a lot of resource just to keep category content primed - we haven't even factored in products, search terms or additional store views at this stage
In fact, just looking at the sheer size of combinations in the
catalog_url_rewrites table (which isn't even factoring in parameters from the layered navigation) will give an idea as to how many URLs you could end needing to crawl.
Every store will certainly be different, but what I'm trying to strike home is that crawling the site to prime FPC isn't practical. Just ensure your store is fast to begin with.
Where FPC is useful
Where the benefits of FPC come into play is on a heavily loaded store - where you have genuinely high levels of traffic and the caches are naturally and continually primed by sheer foot-fall alone.
FPC then comes into play by reducing infrastructure overheads on commonly requested content - cutting down on those repeated calls to the Magento backend.
So we've found that FPC is great to deploy when you've got very high traffic levels - not to reduce page load time - but to reduce resource usage.
Who cares, I still want to crawl
Well, then you've got two options
- Crawl from a template (Eg. sitemap)
- Extract links page by page and crawl each
And there are many utilities to do both of these, these are some I know of
You can crawl your store with Mage-Perftest pretty easily, first download it
wget http://sys.sonassi.com/mage-perftest (64bit) OR
wget http://sys.sonassi.com/mage-perftest-i386 (32bit)
chmod +x http://sys.sonassi.com/mage-perftest*
Then define the crawl process using the Magento sitemap (you can customise this by making a sitemap of any URLs, provided the urls are wrapped in
<loc></loc> tags). The following command will read all the URLs from the sitemap file, then crawl (PHP only) the URLs over the course of 1440 minutes (1 day). If the server exceeds 20% CPU or a load average of 2 - the crawl will pause temporarily.
./mage-perftest -u www.example.com -s www.example.com/sitemap.xml -r auto -b -d 1440 -z -a 20 -l 2
If you have 1000 URLs, crawled over 1 day, that will be approx. 1 request every 86 second(s) ~ target of 0.011 RPS