Magento2 Varnish issues with customer session and site header

Question

Magento version: C.E 2.3.5 Varnish: varnishd (varnish-4.1.1 revision 66bb824)

I am facing a very strange issue. In the admin panel, Stores -> Configuration -> Advanced -> System -> Cache -> Varnish (recommended)

Now, varnish is enabled on my application.

1. I log into a customer account
2. Access some category pages, homepage, product pages. In my header, I see "Welcome ".
3. I logout from the application
4. Now when the system automatically redirects to hompage (after 5 seconds of logout), in the header, I still see the dropdown with options "My acount", "wish list", logout.
5. If I access the already opened pages like, category pages, product pages, the same issue appears. 

Expected result: In the header, the labels, "Sign In  or Create Account" should appear after logout.

Actual Result: The system serves the cached version of the page, with a dropdown to logout for an already logged out user.

I am using a docker setup: Below is the vcl configuration file:

# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 6
vcl 4.0;

import std;
# The minimal Varnish version is 6.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'

backend default {
    .host = <docker service name>;
    .port = "80";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/pub/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

acl purge {
    <docker service name>;
}

sub vcl_recv {
    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
        # has been added to the response in your backend server config. This is used, for example, by the
        # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
        if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
            return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
        }
        if (req.http.X-Magento-Tags-Pattern) {
          ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        }
        if (req.http.X-Pool) {
          ban("obj.http.X-Pool ~ " + req.http.X-Pool);
        }
        return (synth(200, "Purged"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass shopping cart, checkout and search requests
    if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
        return (pass);
    }

    # Bypass health check requests
    if (req.url ~ "/pub/health_check.php") {
        return (pass);
    }

    # Set initial grace period usage status
    set req.http.grace = "none";

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unknown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Remove all marketing get parameters to minimize the cache objects
    if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
        set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

    # Static files caching
    if (req.url ~ "^/(pub/)?(media|static)/") {
        # Static files should not be cached by default
        return (pass);

        # But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
        #unset req.http.Https;
        #unset req.http.X-Forwarded-Proto;
        #unset req.http.Cookie;
    }

    return (hash);
}

sub vcl_hash {
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # For multi site configurations to not cache each other's content
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }
    

    if (req.url ~ "/graphql") {
        call process_graphql_headers;
    }
}

sub process_graphql_headers {
    if (req.http.Store) {
        hash_data(req.http.Store);
    }
    if (req.http.Content-Currency) {
        hash_data(req.http.Content-Currency);
    }
}

sub vcl_backend_response {

    set beresp.grace = 3d;

    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # cache only successfully responses and 404s
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 0s;
        set beresp.uncacheable = true;
        return (deliver);
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    # validate if we need to cache it and prevent from setting cookie
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
    }

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
       beresp.http.Cache-Control ~ "no-cache|no-store") ||
       beresp.http.Vary == "*") {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }

    return (deliver);
}

sub vcl_deliver {
    if (resp.http.X-Magento-Debug) {
        if (resp.http.x-varnish ~ " ") {
            set resp.http.X-Magento-Cache-Debug = "HIT";
            set resp.http.Grace = req.http.grace;
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    # Not letting browser to cache non-static files.
    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "-1";
        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
    }

    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

sub vcl_hit {
    if (obj.ttl >= 0s) {
        # Hit within TTL period
        return (deliver);
    }
    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 300s > 0s) {
            # Hit after TTL expiration, but within grace period
            set req.http.grace = "normal (healthy server)";
            return (deliver);
        } else {
            # Hit after TTL and grace expiration
            return (restart);
        }
    } else {
        # server is not healthy, retrieve from cache
        set req.http.grace = "unlimited (unhealthy server)";
        return (deliver);
    }
}

The headers after running below command on home page

**curl -I <domain url>**

HTTP/1.1 200 OK
Date: Thu, 18 Jun 2020 14:42:27 GMT
Server: ====server name======
X-Powered-By: ===php version====
Set-Cookie: PHPSESSID=471bd6665575857ac82; expires=Thu, 18-Jun-2020 17:42:27 GMT; Max-Age=10800; path=/; domain=<domain name>; HttpOnly
Expires: Fri, 19 Jun 2020 14:42:28 GMT
Cache-Control: max-age=86400, public, s-maxage=86400
Pragma: cache
X-Magento-Tags: store,cms_b,cat_c,cat_c_4,cat_c_5,cat_c_6,cms_p_2,cms_b_header_menu,cms_b_page_top_usp_right,cms_b_home_content4,cat_c_p_2,cat_p_29,cat_p,cat_p_28,cat_p_30,cat_p_31,cat_p_26,cat_p_27,amasty_label_4,cms_b_footer_top_section4,cms_b_footer_bottom_above,cms_b_footer_bottom_links
Content-Length: 63720
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
X-UA-Compatible: IE=edge
Connection: keep-alive
Content-Type: text/html; charset=UTF-8

How can I check the headers during login and logout?

Did anyone faced any similar issues?

Answer 1

Request & response headers using Chrome developer tools

After you logout, please log the headers from the next request. Don't do this using curl, because curl is stateless, by default it doesn't keep track of cookies unless you explicitly associate a cookie store.

You could use Chrome developer tools to get request & response headers for this request.

Enable x-magento-debug

In vcl_deliver there is code to return debugging information. If you enable x-magento-debug, probably from within Magento, we'll see more debug information about caching in the response headers.

Varnishlog

The varnishlog binary will help us figure out the behavior of Varnish when these requests happen.

Please send me the output of the following command:

varnishlog -g request -q "ReqUrl eq '/page'"

Please replace /page in this expression with the URL of the actual page you're debugging. Please only run this varnishlog command exactly when you can reproduce the error in your browser.

The output will contain all headers, all metadata, and the exact behavior of Varnish.

One piece of information that varnishlog cannot display by default, is the composition of the hash key. I would advise you to run Varnish with the following runtime parameter:

-p vsl_mask=+Hash

This parameter should be added to the runtime parameters that are registered with your varnishd programme. This is either done is Systemd, or in Docker, or any other please you use to manage your Varnish process.

This will allow us to see the Hash log lines in the varnishlog output, and will help us understand if there are certain cache variations that are causing this behavior.

Conclusion

The goal is to reproduce the error from a web browser, and send as much information as possible. Extra debugging is possible in Varnish by enabling the x-magento-debug flag in Magento.

varnishlog will also provide a great deal of information.

Good luck!