0

I would like to purge a specific URL from a multi varnish Server enviroment. The extra is that we have multiple Domains running over the same Varnish.

curl -X PURGE -H 'X-Magento-Tags-Pattern: .*' -H 'Host: https://www.domain.de' http://12.34.56.78/path/to/product.html

The varnish server is configured like this in "sub vcl_hash"

# For multi site configurations to not cache each other's content
if (req.http.host) {
    hash_data(req.http.host);
} else {
    hash_data(server.ip);
}

i am watching varnish like varnishlog -g request -q 'ReqMethod eq "PURGE"'

*   << Request  >> 3096028
-   Begin          req 3096027 rxreq
-   Timestamp      Start: 1692698848.940666 0.000000 0.000000
-   Timestamp      Req: 1692698848.940666 0.000000 0.000000
-   VCL_use        reload_20230821_145932_15180
-   ReqStart       11.22.33.44 54242 a0
-   ReqMethod      PURGE
-   ReqURL         /path/to/product.html
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Accept: */*
-   ReqHeader      Host: https://www.domain.de
-   ReqHeader      X-Magento-Tags-Pattern: .*
-   ReqHeader      X-Forwarded-For: 11.22.33.44
-   VCL_call       RECV
-   VCL_acl        MATCH purge "11.22.33.44"
-   VCL_return     synth
-   VCL_call       HASH
-   VCL_return     lookup
-   Timestamp      Process: 1692698848.940763 0.000097 0.000097
-   RespHeader     Date: Tue, 22 Aug 2023 10:07:28 GMT
-   RespHeader     Server: Varnish
-   RespHeader     X-Varnish: 3096028
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     OK
-   RespReason     Purged
-   VCL_call       SYNTH
-   RespHeader     Content-Type: text/html; charset=utf-8
-   RespHeader     Retry-After: 5
-   VCL_return     deliver
-   RespHeader     Content-Length: 242
-   Storage        malloc Transient
-   Filters
-   RespHeader     Accept-Ranges: bytes
-   RespHeader     Connection: keep-alive
-   Timestamp      Resp: 1692698848.940817 0.000151 0.000054
-   ReqAcct        114 0 114 220 242 462
-   End

All looks fine, but as the result of the PURGE Request, that every site of the same domain and also any other domains are purged, like i someone has cleaned the full cache

Does anybody see what i am missing?

Update: VCL Configuration:

# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 6
vcl 4.0;

import std;
# The minimal Varnish version is 6.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'

backend default {
    .host = "12.34.56.78";
    .port = "7080";
    .first_byte_timeout = 600s;
}

acl purge {
    "11.22.33.44";
}

sub vcl_recv {

    if (req.restarts > 0) {
        set req.hash_always_miss = true;
    }
    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
        # has been added to the response in your backend server config. This is used, for example, by the
        # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
        if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
            return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
        }
        if (req.http.X-Magento-Tags-Pattern) {
          ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        }
        if (req.http.X-Pool) {
          ban("obj.http.X-Pool ~ " + req.http.X-Pool);
        }
        return (synth(200, "Purged"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass shopping cart, checkout and search requests
    if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
        return (pass);
    }

    # Bypass health check requests
    if (req.url ~ "/pub/health_check.php") {
        return (pass);
    }

    # Set initial grace period usage status
    set req.http.grace = "none";

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unknown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Remove all marketing get parameters to minimize the cache objects
    if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|__hs[a-z]+|_bta_[a-z]+|_)=") {
    set req.http.X-Original-Url = req.url;
        set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+|__hs[a-z]+|_)=[-_A-z0-9+()%.]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

    # Static files caching
    if (req.url ~ "^/(pub/)?(media|static)/") {
        # Static files should not be cached by default
        return (pass);

        # But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
        #unset req.http.Https;
        #unset req.http.X-Forwarded-Proto;
        #unset req.http.Cookie;
    }

    return (hash);
}

sub vcl_hash {
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # For multi site configurations to not cache each other's content
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }
    

    if (req.url ~ "/graphql") {
        call process_graphql_headers;
    }
}

sub process_graphql_headers {
    if (req.http.Store) {
        hash_data(req.http.Store);
    }
    if (req.http.Content-Currency) {
        hash_data(req.http.Content-Currency);
    }
}

sub vcl_backend_response {

    set beresp.grace = 3d;
                                        

    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # cache only successfully responses and 404s
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 0s;
        set beresp.uncacheable = true;
        return (deliver);
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }else {
        set beresp.ttl = 604800s;
    }

    # validate if we need to cache it and prevent from setting cookie
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
    }

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
       beresp.http.Cache-Control ~ "no-cache|no-store") ||
       beresp.http.Vary == "*") {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }

    return (deliver);
}

sub vcl_deliver {
    if (obj.hits > 0) {
             set resp.http.X-Cache = "HIT";
    } else {
             set resp.http.X-Cache = "MISS";
    }
    if (resp.http.X-Magento-Debug) {
        if (resp.http.x-varnish ~ " ") {
            set resp.http.X-Magento-Cache-Debug = "HIT";
            set resp.http.Grace = req.http.grace;
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    # Not letting browser to cache non-static files.
#    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
#        set resp.http.Pragma = "no-cache";
#        set resp.http.Expires = "-1";
#        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
#    }

    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

sub vcl_hit {
    if (obj.ttl >= 0s) {
        # Hit within TTL period
        return (deliver);
    }
    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 300s > 0s) {
            # Hit after TTL expiration, but within grace period
            set req.http.grace = "normal (healthy server)";
            return (deliver);
        } else {
            # Hit after TTL and grace expiration
            return (restart);
        }
    } else {
        # server is not healthy, retrieve from cache
        set req.http.grace = "unlimited (unhealthy server)";
        return (deliver);
    }
}
2
  • Can you please the complete VCL file to the question. There is probably a lot more logic related to cache invalidation that is currently missing. Once that information is shared in the question, I can provide a more educated explanation. Aug 22, 2023 at 10:27
  • @ThijsFeryn i added the configuration of Varnish
    – Luco
    Aug 22, 2023 at 10:46

1 Answer 1

0

The current situation

Based on your VCL, the PURGE request method is used to remove objects from the cache using the X-Magento-Tags-Pattern request header or X-Pool request header.

The headers are leveraged in the ban() function to add these patterns to the ban list. The ban lurker thread is responsible for removing objects from the cache that match these ban expressions.

See https://www.varnish-software.com/developers/tutorials/ban/ for a tutorial about banning.

Here's the if statement that is used for that:

if (req.http.X-Magento-Tags-Pattern) {
    ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
}
if (req.http.X-Pool) {
    ban("obj.http.X-Pool ~ " + req.http.X-Pool);
}

As you can see there is no context that refers to any specific host. A matched object will apply to every domain you have.

Host-based banning

If you want to be host-specific, a slight VCL modification will add host context to the bans.

First you need to add the Host header value to the object as a header. This adds request information to the response, which will be stored in the object. At the same time it strips the header off again at delivery time:

sub vcl_backend_response {
    set beresp.http.X-Host = bereq.http.Host;
}

sub vcl_deliver {
    unset resp.http.X-Host;
}

You can then match that value in your ban() function. The if-statement we referred to earlier, now looks like this:

if (req.http.X-Magento-Tags-Pattern) {
    ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern + " && obj.http.X-Host == " + req.http.Host);
}
if (req.http.X-Pool) {
    ban("obj.http.X-Pool ~ " + req.http.X-Pool + " && obj.http.X-Host == " + req.http.Host);
}

You can trigger the bans using the same curl call:

curl -X PURGE -H 'X-Magento-Tags-Pattern: .*' -H 'Host: www.domain.de' http://12.34.56.78/path/to/product.html

But keep in mind that the Host header is an FQDN and not a URL. Please don't include http://. Please also keep in mind that the URL itself doesn't matter. You can do the purge on any URL you want, it will have the same impact.

Use purges for individual invalidations

If you're interested in removing individual objects from the cache based on the URL (like the hypothetical http://12.34.56.78/path/to/product.html example, you can also use traditional purges in Varnish.

This is a different mechanism than bans, but is not mutually exclusive in your VCL configuration.

The only difference would be removing the following if-statement from your VCL:

if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
    return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
}

And replacing it with the following if-statement:

if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
    return (purge);
}

This would allow you to run the following curl call:

curl -X PURGE -H 'Host: www.domain.de' http://12.34.56.78/path/to/product.html

And would result in www.domain.de/path/to/product.html being removed from the cache. Because we didn't specify X-Magento-Tags-Pattern or X-Pool, the URL and Host header are used for invalidation. If you do specify any of these headers, banning based on tags is used.

See https://www.varnish-software.com/developers/tutorials/purge/ for a purging tutorial.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.