Varnish Cache won't clear on Magento 2.4.2

Question

I'm running Magento 2.4.2 and have recently added Varnish to it. The site also has Cloudflare in front of it.

The server is running Plesk (Ubuntu) and Varnish is running in Docker within Plesk.

I've followed https://dx3webs.com/configure-varnish-for-magento-2-on-plesk/ to the letter and the site is now using Varnish but I cannot get Varnish to flush whene doing a Magento Cache Flush.

I've added http_cache_hosts with the site ip and port 80 but no success.

My env.php does have cache rules in it, pointing at Redis, which is what I used to use before Varnish but removing those doesn't help.

The hosting config is Nginx and apache but nothing I do gets Varnish to clear.

The only way to refresh it currently is to restart Varnish everytime I make a change to content.

I'm guessing I need to remove the sections from env.php that refer to 'cache' => [ and 'page_cache' => [

I have added (ip redacted) the following to env.php

'http_cache_hosts' => [ [ 'host' => 'XX.XX.XX.XXX', 'port' => '80' ] ]

but I can't see what else I can add / remove / configure to get flush to clear Varnish.

VCL

# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 6
vcl 4.0;

import std;
# The minimal Varnish version is 6.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'

backend default {
    .host = "77.68.90.73";
    .port = "7080";
    .first_byte_timeout = 600s;
}

acl purge {
    "77.68.90.73";
}

sub vcl_recv {
    if (req.restarts > 0) {
        set req.hash_always_miss = true;
    }

    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
        # has been added to the response in your backend server config. This is used, for example, by the
        # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
        if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
            return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
        }
        if (req.http.X-Magento-Tags-Pattern) {
          ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        }
        if (req.http.X-Pool) {
          ban("obj.http.X-Pool ~ " + req.http.X-Pool);
        }
        return (synth(200, "Purged"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass shopping cart and checkout
    if (req.url ~ "/checkout") {
        return (pass);
    }

    # Bypass health check requests
    if (req.url ~ "/health_check.php") {
        return (pass);
    }

    # Set initial grace period usage status
    set req.http.grace = "none";

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unknown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Remove all marketing get parameters to minimize the cache objects
    if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
        set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

    # Static files caching
    if (req.url ~ "^/(pub/)?(media|static)/") {
        # Static files should not be cached by default
        return (pass);

        # But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
        #unset req.http.Https;
        #unset req.http.X-Forwarded-Proto;
        #unset req.http.Cookie;
    }

    # Authenticated GraphQL requests should not be cached by default
    if (req.url ~ "/graphql" && req.http.Authorization ~ "^Bearer") {
        return (pass);
    }

    return (hash);
}

sub vcl_hash {
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }
    

    if (req.url ~ "/graphql") {
        call process_graphql_headers;
    }
}

sub process_graphql_headers {
    if (req.http.Store) {
        hash_data(req.http.Store);
    }
    if (req.http.Content-Currency) {
        hash_data(req.http.Content-Currency);
    }
}

sub vcl_backend_response {

    set beresp.grace = 3d;

    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # cache only successfully responses and 404s
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 0s;
        set beresp.uncacheable = true;
        return (deliver);
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    # validate if we need to cache it and prevent from setting cookie
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
    }

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl = 0s) {
        # Hit within TTL period
        return (deliver);
    }
    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 300s > 0s) {
            # Hit after TTL expiration, but within grace period
            set req.http.grace = "normal (healthy server)";
            return (deliver);
        } else {
            # Hit after TTL and grace expiration
            return (restart);
        }
    } else {
        # server is not healthy, retrieve from cache
        set req.http.grace = "unlimited (unhealthy server)";
        return (deliver);
    }
}

env.php

 [
        'frontName' => 'XXXXXXXXX'
    ],
    'crypt' => [
        'key' => 'XXXXXXXXXXXXXXX'
    ],
    'db' => [
        'table_prefix' => '',
        'connection' => [
            'default' => [
                'host' => 'localhost',
                'dbname' => 'XXXXXXXXXXX',
                'username' => 'XXXXXXXXXX',
                'password' => 'XXXXXXXXX',
                'model' => 'mysql4',
                'engine' => 'innodb',
                'initStatements' => 'SET NAMES utf8;',
                'active' => '1'
            ]
        ]
    ],
    'directories' => [
        'document_root_is_pub' => true
    ],
    'session' => [
        'save' => 'redis',
        'redis' => [
            'host' => '127.0.0.1',
            'port' => '6379',
            'password' => '',
            'timeout' => '2.5',
            'persistent_identifier' => '',
            'database' => '3',
            'compression_threshold' => '2048',
            'compression_library' => 'gzip',
            'log_level' => '4',
            'max_concurrency' => '25',
            'break_after_frontend' => '30',
            'break_after_adminhtml' => '30',
            'first_lifetime' => '600',
            'bot_first_lifetime' => '60',
            'bot_lifetime' => '7200',
            'disable_locking' => '0',
            'min_lifetime' => '60',
            'max_lifetime' => '2592000'
        ]
    ],
    'cache' => [
        'frontend' => [
            'default' => [
                'id_prefix' => 'tbl_',
                'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                'backend_options' => [
                    'server' => '127.0.0.1',
                    'database' => '4',
                    'port' => '6379',
                    'password' => '',
                    'compress_data' => '1',
                    'compression_lib' => '',
                    'preload_keys' => [
                        'tbl_EAV_ENTITY_TYPES:hash',
                        'tbl_GLOBAL_PLUGIN_LIST:hash',
                        'tbl_DB_IS_UP_TO_DATE:hash',
                        'tbl_SYSTEM_DEFAULT:hash'
                    ]
                ]
            ],
            'page_cache' => [
                'backend' => 'Magento\\Framework\\Cache\\Backend\\Redis',
                'backend_options' => [
                    'id_prefix' => 'tbl_',
                    'server' => '127.0.0.1',
                    'port' => '6379',
                    'database' => '5',
                    'compress_data' => '0'
                ]
            ]
        ]
    ],
    'resource' => [
        'default_setup' => [
            'connection' => 'default'
        ]
    ],
    'x-frame-options' => 'SAMEORIGIN',
    'MAGE_MODE' => 'production',
    'cache_types' => [
        'config' => 1,
        'layout' => 1,
        'block_html' => 1,
        'collections' => 1,
        'reflection' => 1,
        'db_ddl' => 1,
        'eav' => 1,
        'customer_notification' => 1,
        'config_integration' => 1,
        'config_integration_api' => 1,
        'full_page' => 1,
        'translate' => 1,
        'config_webservice' => 1,
        'compiled_config' => 1,
        'vertex' => 0,
        'google_product' => 1,
        'ox_megamenu' => 1,
        'ox_googlemap_locations' => 1
    ],
    'install' => [
        'date' => 'Thu, 03 May 2018 20:02:35 +0000'
    ],
    'system' => [
        'default' => [
            'dev' => [
                'debug' => [
                    'debug_logging' => '0'
                ]
            ]
        ]
    ],
    'downloadable_domains' => [
        'XXXXXXXXXXX'
    ],
    'queue' => [
        'consumers_wait_for_messages' => 0
    ],
    'cron_consumers_runner' => [
        'cron_run' => true,
        'max_messages' => 2000,
        'consumers' => [
            'product_action_attribute.update',
            'product_action_attribute.website.update',
            'exportProcessor',
            'codegeneratorProcessor'
        ]
    ],
    'http_cache_hosts' => [
        [
            'host' => '77.68.90.73',
            'port' => '80'
        ]
    ]
];

Docker config

CONTAINER ID   IMAGE                  COMMAND                  CREATED       STATUS         PORTS                                                NAMES
cf57636333bc   varnish:6.4            "docker-varnish-entr…"   8 days ago    Up 2 minutes   127.0.0.1:32782->80/tcp, 127.0.0.1:32783->8443/tcp   varnish-tuckers-live

I have found the problem, see my answer below that fixes it.

Answer 1

Can you please share your full VCL file.

Please also share the output of sudo varnishlog -g request -q "ReqMethod eq 'PURGE'", which should be kept running until you executed the cache flush.

The logs will give us a better understanding of the situation, along with the VCL file.

UPDATE

Based on the input that was provided, we now know Varnish runs inside a Docker container. In order to see the logs for PURGE requests, simply run the following command:

docker exec -ti varnish-tuckers-live varnishlog -g request -q "ReqMethod eq 'PURGE'"

The output shows what happens when a PURGE request is made. If no output appears when performing the cache flush in Magento, it means something is wrong in Magento and Varnish isn't reached.

Answer 2

The solution was to ignore the link I posted above and to follow these instructions very carefully.

https://talk.plesk.com/threads/magento-2-plesk-varnish-setup-in-a-docker.362442/

Specifically be thorough on the part that mentions creating /root/varnish/conf etc.

That allows the Varnish container to operate across all vhosts you have in Plesk rather than being domain specific.

I still had to add the following to env.php before it would work using the port number I used in the container.

'http_cache_hosts' => [
        [
            'host' => '127.0.0.1',
            'port' => 'XXXXX'
        ]
    ]

Hope this helps other people out.