0

My Magento 2 setup with Varnish normalises user agents to pc/mobile/bot. It is configured to store different cached versions per user agent.

The caching per user agent works great, but it seems to be caching a separate version for users with/without recent history as well. I'm assuming this has something to do with cookies being set, but I really have no idea how to tackle the issue. Is anyone familiar with this issue?

Expected result for page:

  • Cached version for mobile
  • Cached version for pc
  • Cached version for bot

Actual result for page:

  • Cached version for mobile new visitors' 1st page visit (tested with incognito mode)
  • Cached version for mobile with recent activity (eg. browsing through the site, having previously visited, etc.)

  • Cached version for pc new visitors' 1st page visit (tested with incognito mode)
  • Cached version for pc recent activity (eg. browsing through the site, having previously visited, etc.)

  • Cached version for bot new visitors' 1st page visit (tested with incognito mode)
  • Cached version for bot recent activity (eg. browsing through the site, having previously visited, etc.)

UPDATE 1: Here's the VCL file and screenshots of the headers.

# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 6
vcl 4.0;

import std;
# The minimal Varnish version is 6.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'

backend default {
    .host = "#######";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/pub/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

acl purge {
    "localhost";
    "####";
}


include "devicedetect.vcl";


sub vcl_recv {

    call devicedetect;
    

    if (req.restarts > 0) {
        set req.hash_always_miss = true;
    }

    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
        # has been added to the response in your backend server config. This is used, for example, by the
        # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
        if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
            return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
        }
        if (req.http.X-Magento-Tags-Pattern) {
          ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        }
        if (req.http.X-Pool) {
          ban("obj.http.X-Pool ~ " + req.http.X-Pool);
        }
        return (synth(200, "Purged"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass shopping cart, checkout and search requests
    if (req.url ~ "/checkout" || req.url ~ "checkout" || req.url ~ "/onestepcheckout" || req.url ~ "/catalogsearch") {
        return (pass);
    }

    # Bypass health check requests
    if (req.url ~ "/pub/health_check.php") {
        return (pass);
    }

    # Set initial grace period usage status
    set req.http.grace = "none";

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|webp|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unknown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Remove all marketing get parameters to minimize the cache objects
    if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
        set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

    # Static files caching
    if (req.url ~ "^/(pub/)?(media|static)/") {
        # Static files should not be cached by default
        return (pass);

        # But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
        #unset req.http.Https;
        #unset req.http.X-Forwarded-Proto;
        #unset req.http.Cookie;
    }

    # Authenticated GraphQL requests should not be cached by default
    if (req.url ~ "/graphql" && req.http.Authorization ~ "^Bearer") {
        return (pass);
    }


    return (hash);
}



sub vcl_backend_fetch {

    if (bereq.http.X-UA-Device) {
        set bereq.http.User-Agent = bereq.http.X-UA-Device;
    } 

}



sub vcl_hash {

    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # For multi site configurations to not cache each other's content
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }

    if (req.url ~ "/graphql") {
        call process_graphql_headers;
    }

}


sub process_graphql_headers {
    
    if (req.http.Store) {
        hash_data(req.http.Store);
    }
    
    if (req.http.Content-Currency) {
        hash_data(req.http.Content-Currency);
    }
}


sub vcl_backend_response {

    if (bereq.http.X-UA-Device) {
        if (!beresp.http.Vary) { # no Vary at all
            set beresp.http.Vary = "X-UA-Device";
        } elsif (beresp.http.Vary !~ "X-UA-Device") { # add to existing Vary
            set beresp.http.Vary = beresp.http.Vary + ", X-UA-Device";
        }
    }
    set beresp.http.X-UA-Device = bereq.http.X-UA-Device;

    set beresp.grace = 3d;

    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # cache only successfully responses and 404s
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 0s;
        set beresp.uncacheable = true;
        return (deliver);
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    # validate if we need to cache it and prevent from setting cookie
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
    }

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
       beresp.http.Cache-Control ~ "no-cache|no-store") ||
       beresp.http.Vary == "*") {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }

    return (deliver);
}



sub vcl_deliver {

    if (obj.hits > 0) {
        set resp.http.X-Cache = "cached";
    } else {
        set resp.http.X-Cache = "uncached";
    }

    if ((req.http.X-UA-Device) && (resp.http.Vary)) {
        set resp.http.Vary = regsub(resp.http.Vary, "X-UA-Device", "User-Agent");
 
    }

    if (resp.http.X-Magento-Debug) {
        if (resp.http.x-varnish ~ " ") {
            set resp.http.X-Magento-Cache-Debug = "HIT";
            set resp.http.Grace = req.http.grace;
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    # Not letting browser to cache non-static files.
    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "-1";
        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
    }

    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

sub vcl_hit {
    if (obj.ttl >= 0s) {
        # Hit within TTL period
        return (deliver);
    }
    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 300s > 0s) {
            # Hit after TTL expiration, but within grace period
            set req.http.grace = "normal (healthy server)";
            return (deliver);
        } else {
            # Hit after TTL and grace expiration
            return (restart);
        }
    } else {
        # server is not healthy, retrieve from cache
        set req.http.grace = "unlimited (unhealthy server)";
        return (deliver);
    }
}

Headers of a 1st visit (with cache hit) New visitor

Headers of a visit with recent browse activity on site (with cache hit) Already visited

4
  • First, why do you want to have separate cache versions for pc/mobile/bot? Nowadays all themes being responsive, there should be no difference. Second, please post your modified VCL. – Danila Vershinin Mar 23 at 18:44
  • Please share your VCL file and provide some example HTTP request & response payload. Please include request & response headers. This will help me understand the current behavior and how VCL might be causing unwanted cache variations. Please also check if your Magento setup returns Vary headers. – Thijs Feryn Mar 24 at 7:36
  • @DanilaVershinin My client has a responsive theme, but their former developer added an additional extra lightweight theme for mobile devices. Painful to see, but I'll have to accept it for now. As for the bots... we use Mirasvit Cache Warmer module. It logs all visited pages to a table, and uses the table to warm frequently visited pages. To avoid bots from flooding the table with irrelevant pages, it uses user agents to exclude bot visits from the table. Unfortunately that means a separate cache for bots. – Jason Mar 25 at 10:58
  • @ThijsFeryn & @ DanilaVershinin I have edited my post. Included VCL (default magento VCL with custom changes) + screenshots of the headers. The site runs on Apache with Varnish in front of it. And Nginx as SSL reverse proxy in front of Varnish. – Jason Mar 25 at 11:05
1

Your problem is related to unnecessary cache variations.

VCL is responsible for creating cache variations through the following code:

sub vcl_hash {

    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # For multi site configurations to not cache each other's content
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }

    if (req.url ~ "/graphql") {
        call process_graphql_headers;
    }

}

On top of the the URL and Host header variations, this custom vcl_hash logic will also create cache variations based on the X-Forwarded-Proto header, which makes total sense. It ensures there's a variation for http:// and for https://, which avoids ending up in a redirect loop.

There's also the X-Magento-Vary cookie which may also create variations.

We must not forget that HTTP has built-in support for cache variations through the Vary header. Varnish applies these variations on top of what is in vcl_hash.

In your case that's:

Vary: User-Agent, Accept-Encoding

The Accept-Encoding part makes sense: you need to make sure clients that don't support compression (Gzip, Brotli) get a plain text version.

However, the User-Agent variation makes no sense what so ever. Because there's no validation or limitation, there's going to be an uncontrollable amount of variations.

Please get rid of Vary: User-Agent.

In terms of device variations, there's already the X-UA-Device header that is used inside vcl_backend_response, which should do the trick.

Long story short: Vary: User-Agent is causing too many cache variations, wich results in a low hit rate.

2
  • In my VCL, I am setting User-Agent to match X-UA-Device (mobile/pc/bot), so I guess Vary'ing on the User-Agent is safe. The issue I'm facing is that it seems to be making 2 cache versions per (normalised) User-Agent; 1 for first-time visitors, 1 for visitors that have visited another page before. The difference between the two, is that first-time visitors don't have any cookies set (until they click to go to another page). Therefore based on your useful info, I'd say the missing/present "X-Magento-Vary" cookie causes the extra cache version. I guess I'll have to look into Magento's behaviour.. – Jason Mar 26 at 9:34
  • 1
    Ah, I'm really thankful for your detailed explanation Thijs. The fact that you explained that the X-Magento-Vary cookie creates new variations as well, pointed me towards the solution. Apparently one of my Magento modules sets a blank X-Magento-Vary cookie on default - and obviously - "first time" visitors don't have this cookie yet. Means Varnish' behaviour is all fine, but Magento's behaviour isn't. Modified the module and now it's no longer setting X-Magento-Vary for no reason. – Jason Mar 26 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.