What would be your recommendations about warming a varnish cache? The particular case is a site that has not too many visits, but from time to time we need to flush the cache (the reasons for this would be a whole different matter).

So, the particular request is: "we want to warm the cache as soon as a flush is done". We are just looking for information and found the standard method of using crawlers and/or sitemaps-based scripts, however, we don't want to be too aggressive and harm the server in the process.

Any advice, experience, opinion, etc. that you share would be very appreciated.

Thank you!

  • Crawler with a time delay is about all I could say really. Monitor to see what your servers can handle. Apr 21, 2016 at 8:48

1 Answer 1


I came across this recently and thought it would be helpful to you.

A crawler is really the only way to go about it as I mentioned in my comment to your question, here's a pre-baked one from the Magento Turpentine plugin group.


It gets the URLs from the sitemap and crawls them.

And, in case the Github page ever goes AWOL.


# Nexcess.net Turpentine Extension for Magento
# Copyright (C) 2012  Nexcess.net L.L.C.
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# GNU General Public License for more details.
# You should have received a copy of the GNU General Public License along
# with this program; if not, write to the Free Software Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

TMP_URL_FILE="/tmp/urls_$(cat /proc/sys/kernel/random/uuid).txt"
PROCS="${PROCS-$(grep processor /proc/cpuinfo | wc -l)}"

echo '<root/>' | xpath -e '*' &>/dev/null

if [ $? -eq 2 ]; then
    XPATH_BIN='xpath -e'

if [ -z "$SITEMAP_URL" ]; then
    cat <<EOF
Usage: $0 <sitemap URL>
    Warm Magento's cache by visiting the URLs in Magento's sitemap
        $0 http://example.com/magento/sitemap.xml

    exit 1

echo "Getting URLs from sitemap..."

curl -ks "$SITEMAP_URL" | \
    $XPATH_BIN '/urlset/url/loc/text()' 2>/dev/null | \
    sed -r 's~http(s)?:~\nhttp\1:~g' | \
    grep -vE '^\s*$' > "$TMP_URL_FILE"

echo "Warming $(cat $TMP_URL_FILE | wc -l) URLs using $PROCS processes..."

cat "$TMP_URL_FILE" | \
    xargs -P "$PROCS" -r -n 1 -- \
        siege -b -v -c 1 -r once 2>/dev/null | \
    sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' | \
    grep -E '^HTTP'
cat "$TMP_URL_FILE" | \
    xargs -P "$PROCS" -r -n 1 -- \
        siege -H 'Accept-Encoding: gzip' -b -v -c 1 -r once 2>/dev/null | \
    sed -r 's/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g' | \
    grep -E '^HTTP'

rm -f "$TMP_URL_FILE"

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.