0

On our server we are running multiple Magento installs for multiple clients, served by NGINX and PHP-FPM. We haven't used Varnish before, but as we are moving to Magento 2 and Varnish is supported out of the box, we are looking for a way to roll this out.

Previously, with Turpentine on Magento 1, is was assumed you use 1 Varnish instance for 1 Magento install.

Now, we're talking about Magento 2 and Varnish.

Can we run multiple Magento 2 installations on the same server and use only 1 Varnish instance? The goal is to keep server management as easy as possible, including our monitoring.

Just like we can use 1 NGINX and 1 MySQL shared for all Magento's, can we use 1 Varnish server?

Magento 2 generated a default VCL config file, so I assume we'd need to change it so it works well for multiple installs.

Also, flushing the cache should only purge the right Magento install, not all installs at once.

What would be the best way to tackle this?

I know some people are using docker, but this makes monitoring more complicated and I'm looking for the simplest solution here.

The default VCL for Varnish 4 that Magento 2 generates is

vcl 4.0;

import std;
# The minimal Varnish version is 4.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'

backend default {
    .host = "localhost";
    .port = "8080";
}

acl purge {
    "localhost";
}

sub vcl_recv {
    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        if (!req.http.X-Magento-Tags-Pattern) {
            return (synth(400, "X-Magento-Tags-Pattern header required"));
        }
        ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        return (synth(200, "Purged"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass shopping cart, checkout and search requests
    if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
        return (pass);
    }

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Remove Google gclid parameters to minimize the cache objects
    set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
    set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
    set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"

    # static files are always cacheable. remove SSL flag and cookie
        if (req.url ~ "^/(pub/)?(media|static)/.*\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)$") {
        unset req.http.Https;
        unset req.http.X-Forwarded-Proto;
        unset req.http.Cookie;
    }

    return (hash);
}

sub vcl_hash {
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # For multi site configurations to not cache each other's content
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }

}

sub vcl_backend_response {
    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    # cache only successfully responses and 404s
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 0s;
        set beresp.uncacheable = true;
        return (deliver);
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # validate if we need to cache it and prevent from setting cookie
    # images, css and js are cacheable by default so we have to remove cookie also
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
        if (bereq.url !~ "\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|gz|tgz|bz2|tbz|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)(\?|$)") {
            set beresp.http.Pragma = "no-cache";
            set beresp.http.Expires = "-1";
            set beresp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
            set beresp.grace = 1m;
        }
    }

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
        beresp.http.Surrogate-control ~ "no-store" ||
        (!beresp.http.Surrogate-Control && beresp.http.Vary == "*")) {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }
    return (deliver);
}

sub vcl_deliver {
    if (resp.http.X-Magento-Debug) {
        if (resp.http.x-varnish ~ " ") {
            set resp.http.X-Magento-Cache-Debug = "HIT";
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}
5

Yes, you can run multiple Magento 2 shops with single Varnish instance.

For single Magento installation with multiple storefronts

No changes needed at all

For multiple Magento installations on the same server (unsupported)

There is no easy solution that doesn't involve hacking Magento core code. You will at minimum have to amend changes from here.

Then you will need to adjust your VCL file so that the hash includes Host header value. Uncomment in the VCL:

# For multi site configurations to not cache each other's content
# if (req.http.host) {
#     hash_data(req.http.host);
# } else {
#     hash_data(server.ip);
# }

Further, to make sure that the cache flush is done against particular Magento cache, change:

ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);

to:

ban("obj.http.X-Host ~ " + req.http.host + " && "obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);

Finally, add to top of vcl_backend_response:

set beresp.http.X-Host = bereq.http.host;

Full VCL

  • Wow, looks great. Though I'm not exactly sure how this would fit in Magento 2's suggested VCL. Does this also mean you use 1 VCL for Varnish or create multiple VCLs that will be combined? – Wouter Feb 12 '17 at 13:15
  • Magento 2's suggested VCL is only that - suggested. It can't enforce you into writing a better (suitable for your needs) VCL :) – Danila Vershinin Feb 12 '17 at 20:50
  • In case you're going to host only Magento 2 stores on the server - then this single modified VCL will work just fine. If, however, you will host other website types, you will have to dig deeper and introduce support for something like "virtual hosts for Varnish" allowing you to have different caching logic for each site: getpagespeed.com/server-setup/varnish/varnish-virtual-hosts – Danila Vershinin Feb 12 '17 at 20:55
  • This config works well for multiple magneto 2 websites on the same server. Question for Daniel: lets say I want to add a wordpress or owncloud to the same server and I want varnish just to pass it trough without chasing. What would you recommend: A)a separate varnish vhost for that wordpress or owncloud website B) a basic "if" statement in default.vcl is ok? (It is just 1 wp website, the rest of websites on the server are m2) – Mark May 26 '18 at 2:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.