Varnish applied to Magento 2 Instance?

Question

I have installed the Varnish and configured it to my local instance. And I have created the logs before Varnish and after Varnish as below.

And the front-end response headers.

My question is when I saw in the Final Verification from Magento docs the important headers are below as per standards.

X-Magento-Cache-Control: max-age=86400, public, s-maxage=86400
Age: 0
X-Magento-Cache-Debug: MISS

But when I run the same command I got the first image response.

I'm confusing here. Is Varnish applied to my instance? as a reference to the above images. Please suggest me.

# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 6
vcl 4.0;

import std;

backend default {
    .host = "10.10.10.120";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

# Add hostnames, IP addresses and subnets that are allowed to purge content
acl purge {
    "localhost";
    "127.0.0.1";
    "::1";
}

sub vcl_recv {
    # Remove empty query string parameters
    # e.g.: www.example.com/index.html?    
    if (req.url ~ "\?$") {
        set req.url = regsub(req.url, "\?$", "");
    }

    # Remove port number from host header
    set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
    
    # Sorts query string parameters alphabetically for cache normalization purposes    
    set req.url = std.querysort(req.url);
    
    # Remove the proxy header to mitigate the httpoxy vulnerability
    # See https://httpoxy.org/    
    unset req.http.proxy;

    # Add X-Forwarded-Proto header when using https
    if (!req.http.X-Forwarded-Proto && (std.port(server.ip) == 443)) {
        set req.http.X-Forwarded-Proto = "https";
    }
    
    # Reduce grace to 300s if the backend is healthy
    # In case of an unhealthy backend, the original grace is used
    if (std.healthy(req.backend_hint)) {
        set req.grace = 300s;
    }
    
    # Purge logic to remove objects from the cache
    # Tailored to Magento's cache invalidation mechanism
    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
            return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
        }
        if (req.http.X-Magento-Tags-Pattern) {
          ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        }
        if (req.http.X-Pool) {
          ban("obj.http.X-Pool ~ " + req.http.X-Pool);
        }
        return (synth(200, "Purged"));
    }
    
    # Only handle relevant HTTP request methods
    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "PATCH" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          return (pipe);
    }
    
    # Only cache GET and HEAD requests
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Don't cache the checkout page
    if (req.url ~ "/checkout") {
        return (pass);
    }

    # Don't cache the health check page
    if (req.url ~ "^/(pub/)?(health_check.php)$") {
        return (pass);
    }

    # Collapse multiple cookie headers into one
    std.collect(req.http.Cookie);

    # Remove tracking query string parameters used by analytics tools
    if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
        set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

    # Don't cache the authenticated GraphQL requests
    if (req.url ~ "/graphql" && req.http.Authorization ~ "^Bearer") {
        return (pass);
    }

    return (hash);
}

sub vcl_hash {
    # Add a cache variation based on the X-Magento-Vary cookie
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    } else {
        hash_data("");
    }
    
    # Create cache variations depending on the request protocol
    hash_data(req.http.X-Forwarded-Proto);

    # Create store and currency cache variations for GraphQL requests
    if (req.url ~ "/graphql") {
        hash_data(req.http.Store);
        hash_data(req.http.Content-Currency);
    }
}

sub vcl_backend_response {
    # Serve stale content for three days after object expiration
    # Perform asynchronous revalidation while stale content is served
    set beresp.grace = 3d;

    # All text-based content can be parsed as ESI
    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    # Allow GZIP compression on all JavaScript files and all text-based content
    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }
    
    # Add debug headers
    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # Only cache HTTP 200 & HTTP 404 responses
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
        return (deliver);
    # Don't cache private responses
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    # Remove the Set-Cookie header for cacheable content
    # Only for HTTP GET & HTTP HEAD requests
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
    }
   
    # Don't cache content with a negative TTL
    # Don't cache content for no-cache or no-store content
    # Don't cache content where all headers are varied
    if (beresp.ttl <= 0s ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
       beresp.http.Cache-Control ~ "no-cache|no-store") ||
       beresp.http.Vary == "*") {
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }

    return (deliver);
}

sub vcl_deliver {
    # Add debug headers
    if (resp.http.X-Magento-Debug) {
        if (obj.uncacheable) {
            set resp.http.X-Magento-Cache-Debug = "UNCACHEABLE";
        } else if (obj.hits) {
            set resp.http.X-Magento-Cache-Debug = "HIT";
            set resp.http.Grace = req.http.grace;
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    # Don't let browser cache non-static files
    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "-1";
        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
    }
    
    # Cleanup headers
    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

After updating the default.vcl file getting below error

• Trying 10.10.10.120...
• TCP_NODELAY set
• connect to 10.10.10.120 port 80 failed: Connection refused
• Failed to connect to 10.10.10.120 port 80: Connection refused
• Closing connection 0 curl: (7) Failed to connect to 10.10.10.120 port 80: Connection refused

Answer 1

You can test if Varnish is configured and enabled from the command line using curl:

curl -I -H "host: www.myshop.com" 127.0.0.1:80

X-Varnish: 762904605
X-Magento-Cache-Debug: HIT

Note - in production mode the X-Magento-Cache-Debug headers are disabled.

The default Magento 2 Varnish VCL may unset the debug headers - for troubleshooting comment out

unset resp.http.X-Magento-Debug;
unset resp.http.X-Magento-Tags;

Answer 2

Varnish is definitely installed based on your logs.

However there is some inconsistent behavior that might be triggered by your VCL.

The Set-Cookie header

The before log show a Set-Cookie header which shouldn't be cached. This is part of Varnish's built-in VCL behavior and makes total sense: when you're changing state by setting a cookie, the result shouldn't be cached.

Due to the Set-Cookie header, Varnish will trigger its hit-for-miss behavior and cache the decision not to cache the response for the next 2 minutes or until the next response can be cached.

This behavior is correctly reflected by the Age: 0 header. This header indicates that Varnish has been caching the response for zero seconds.

The Cache-Control header

What does not make sense is that a Cache-Control: max-age=86400, public, s-maxage=86400 is emitted. When a Set-Cookie header appears, a Cache-Control: private, no-cache, no-store header should be sent instead.

If you would call the setcookie() function in PHP, PHP would automatically set that no-cache header. I'm assuming something else is getting in the way.

The X-Varnish header

The X-Varnish: 493911 header refers to the Varnish Shared Memory Log transaction id. You can use this transaction id to ready the corresponding logs.

Run the following command to figure out what happened for that transaction:

varnishlog -d -q "vxid == 493911"

Please note that the Varnish Shared Memory Logs are stored in a circular buffer. When the buffer is full, content is overwritten. This means that logs aren't forever. The default size of the circular memory buffer is 80 MB.

The X-Magento-Tags header

The X-Magento-Tags header is used to add metadata to the cached object so that cache invalidation based on one of these tags can take place.

Magento uses this to efficiently remove multiple objects from the cache when updates in the backend takes place.

Normal cache invalidation happens based on the URL, but using tag-based invalidation is more efficient here.

Your VCL file?

Several other people have requested to see your VCL file. Please attach it to your response.

I would also advise you to have a look at https://www.varnish-software.com/developers/tutorials/configuring-varnish-magento/#optimized-magento-vcl-file. This is the official Magento 2 VCL file that is optimized by Varnish Software.

Ensuring Varnish runs on port 80

Please run sudo systemctl edit --full varnish to edit your systemd configuration for Varnish and check that the -a parameter is set to port 80 as explained in https://www.varnish-software.com/developers/tutorials/installing-varnish-ubuntu/#modifying-the-listening-port-and-cache-size

Ensuring your webserver runs on port 8080

You also have to make sure that your webserver is no longer running on port 80 if it is hosted on the same machine as Varnish.

See https://www.varnish-software.com/developers/tutorials/installing-varnish-ubuntu/#5-configure-the-web-server-to-work-with-varnish for more information on how to do this for Apache and Nginx.

I'm assuming you use an Ubuntu or Debian system. If you're running CentOS, please have a look at the following tutorial https://www.varnish-software.com/developers/tutorials/installing-varnish-centos/#5-configure-the-web-server-to-work-with-varnish

Restarting services

Once you've set the ports to the correct values, please restart the services.

If your using Apache, run the following command:

sudo systemctl restart apache2 varnish

If you're not running Debian or Ubuntu, but CentOS, please run the following command to restart Apache & Varnish:

sudo systemctl restart httpd varnish

If you're using Nginx instead of Apache, run the following command:

sudo systemctl restart nginx varnish

Answer 3

If you look at the screen shot : X-varnish : if it has a single value then its a Cache Miss case. if has two values with some space in between then the page is delivered from varnish.

In your case there is just one value hence the page is not delivered from varnish cache.

Generally if one of the blocks is not cacheable due to it being set as cacheable false then the page will not be delivered from varnish cache.