Magento 2.1 Varnish Cache not Purge

Question

I have a server with NGINX / Varnish 4 & Magento 2.1, I have setup everything need for varnish and Magento 2.1 in Stores > Configuration > Admin. Varnish is working.

My problem is that if I try to Flush Magento cache, the cache is not clearing. Varnish cache is purged only after restarting via ssh. On Magento developer documents I found somewhere that if /var/cache/ is empty then everything is working. In my case cache folder is not empty, I tried to delete but it comes back.

my server IP replaced by "server.ip"

env.php

 'http_cache_hosts' => array(
    array (
        'host' => 'server.ip',
        'port' => '6082',
    )
),

default.vcl

   vcl 4.0;

import std;
# The minimal Varnish version is 4.0
 # For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'

 backend default {
.host = "server.ip";
.port = "8080";
}

  acl purge {
"server.ip";
 }

  sub vcl_recv {
if (req.method == "PURGE") {
    if (client.ip !~ purge) {
        return (synth(405, "Method not allowed"));
    }
    if (!req.http.X-Magento-Tags-Pattern) {
        return (synth(400, "X-Magento-Tags-Pattern header required"));
    }
    ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
    return (synth(200, "Purged"));
}

if (req.method != "GET" &&
    req.method != "HEAD" &&
    req.method != "PUT" &&
    req.method != "POST" &&
    req.method != "TRACE" &&
    req.method != "OPTIONS" &&
    req.method != "DELETE") {
      /* Non-RFC2616 or CONNECT which is weird. */
      return (pipe);
}

# We only deal with GET and HEAD by default
if (req.method != "GET" && req.method != "HEAD") {
    return (pass);
}

# Bypass shopping cart, checkout and search requests
if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
    return (pass);
}

# normalize url in case of leading HTTP scheme and domain
set req.url = regsub(req.url, "^http[s]?://", "");

# collect all cookies
std.collect(req.http.Cookie);

# Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
        # No point in compressing these
        unset req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
        set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
        set req.http.Accept-Encoding = "deflate";
    } else {
        # unkown algorithm
        unset req.http.Accept-Encoding;
    }
}

# Remove Google gclid parameters to minimize the cache objects
set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"

# static files are always cacheable. remove SSL flag and cookie
    if (req.url ~ "^/(pub/)?(media|static)/.*\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)$") {
    unset req.http.Https;
    unset req.http.X-Forwarded-Proto;
    unset req.http.Cookie;
}

return (hash);
}

sub vcl_hash {
if (req.http.cookie ~ "X-Magento-Vary=") {
    hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
}

# For multi site configurations to not cache each other's content
if (req.http.host) {
    hash_data(req.http.host);
} else {
    hash_data(server.ip);
}

# To make sure HTTP users don't see SSL warning
if (req.http.X-Forwarded-Proto) {
    hash_data(req.http.X-Forwarded-Proto);
}

}

 sub vcl_backend_response {
if (beresp.http.content-type ~ "text") {
    set beresp.do_esi = true;
}

if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
    set beresp.do_gzip = true;
}

# cache only successfully responses and 404s
if (beresp.status != 200 && beresp.status != 404) {
    set beresp.ttl = 0s;
    set beresp.uncacheable = true;
    return (deliver);
} elsif (beresp.http.Cache-Control ~ "private") {
    set beresp.uncacheable = true;
    set beresp.ttl = 86400s;
    return (deliver);
}

if (beresp.http.X-Magento-Debug) {
    set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
}

# validate if we need to cache it and prevent from setting cookie
# images, css and js are cacheable by default so we have to remove cookie also
if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
    unset beresp.http.set-cookie;
    if (bereq.url !~ "\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|gz|tgz|bz2|tbz|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)(\?|$)") {
        set beresp.http.Pragma = "no-cache";
        set beresp.http.Expires = "-1";
        set beresp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
        set beresp.grace = 1m;
    }
   }

 # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
if (beresp.ttl <= 0s ||
    beresp.http.Surrogate-control ~ "no-store" ||
    (!beresp.http.Surrogate-Control && beresp.http.Vary == "*")) {
    # Mark as Hit-For-Pass for the next 2 minutes
    set beresp.ttl = 120s;
    set beresp.uncacheable = true;
}
return (deliver);
}

sub vcl_deliver {
if (resp.http.X-Magento-Debug) {
    if (resp.http.x-varnish ~ " ") {
        set resp.http.X-Magento-Cache-Debug = "HIT";
    } else {
        set resp.http.X-Magento-Cache-Debug = "MISS";
    }
} else {
    unset resp.http.Age;
}

unset resp.http.X-Magento-Debug;
unset resp.http.X-Magento-Tags;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.Link;
}

curl -I -v --location-trusted http://server.ip/

 < HTTP/1.1 200 OK
 HTTP/1.1 200 OK
 < Date: Sun, 19 Mar 2017 19:01:06 GMT
 Date: Sun, 19 Mar 2017 19:01:06 GMT
 < Content-Type: text/html; charset=UTF-8
 Content-Type: text/html; charset=UTF-8
 < Keep-Alive: timeout=60
 Keep-Alive: timeout=60
 < X-Content-Type-Options: nosniff
 X-Content-Type-Options: nosniff
 < X-XSS-Protection: 1; mode=block
 X-XSS-Protection: 1; mode=block
 < X-Frame-Options: SAMEORIGIN
 X-Frame-Options: SAMEORIGIN
 < X-Magento-Cache-Control: max-age=86400, public, s-maxage=86400
 X-Magento-Cache-Control: max-age=86400, public, s-maxage=86400
 < Pragma: no-cache
 Pragma: no-cache
 < Expires: -1
 Expires: -1
 < Cache-Control: no-store, no-cache, must-revalidate, max-age=0
 Cache-Control: no-store, no-cache, must-revalidate, max-age=0
 < Age: 578
 Age: 578
 < X-Magento-Cache-Debug: HIT
 X-Magento-Cache-Debug: HIT
 < Connection: keep-alive
 Connection: keep-alive

curl -X PURGE

     <!DOCTYPE html>
  <html>
  <head>
<title>400 X-Magento-Tags-Pattern header required</title>
  </head>
  <body>
<h1>Error 400 X-Magento-Tags-Pattern header required</h1>
<p>X-Magento-Tags-Pattern header required</p>
<h3>Guru Meditation:</h3>
<p>XID: 65764</p>
<hr>
<p>Varnish cache server</p>
 </body>
 </html>

Answer 1

Finally I fixed myself the issue.

env.php

'http_cache_hosts' => 
 array (
 0 => 
   array (
  'host' => '127.0.0.1',
  'port' => '80',
  ),
),

default.vcl

if (req.method == "PURGE") {
if (client.ip !~ purge) {
    return (synth(405, "Method not allowed"));
}
if (!req.http.X-Magento-Tags-Pattern) {
    return (synth(400, "X-Magento-Tags-Pattern header required"));
}
ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);

# If all Tags should be purged clear
# ban everything to catch assets as well
if (req.http.X-Magento-Tags-Pattern == ".*") {
  ban("req.url ~ .*");
}

return (synth(200, "Purged"));
}

Answer 2

Adding something really important to George George's answer:

Run: bin/magento config:set system/full_page_cache/caching_application 2 (otherwise the purge request won't be sent!)

Then: bin/magento setup:config:set --http-cache-hosts=127.0.0.1:6081

Then change default.vcl to add this:

if (req.method == "PURGE") {
    if (client.ip !~ purge) {
        return (synth(405, "Method not allowed"));
    }
    # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
    # has been added to the response in your backend server config. This is used, for example, by the
    # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
    if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
        return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
    }
    if (req.http.X-Magento-Tags-Pattern) {
      ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
    }
    if (req.http.X-Pool) {
      ban("obj.http.X-Pool ~ " + req.http.X-Pool);
    }

    ###
    ### THIS IS WHAT YOU MUST ADD
    ###

    # If all Tags should be purged clear
    # ban everything to catch assets as well
    if (req.http.X-Magento-Tags-Pattern == ".*") {
      ban("req.url ~ .*");
    }

    ###
    ### END OF THIS IS WHAT YOU MUST ADD
    ###

    return (synth(200, "Purged"));
}

Answer 3

netstat -tulpn | grep varnish

see which port is actually used by Varnish service. i guess your settings to Varnish interface port is wrong.

Varnish by default is running admin on 127.0.0.1:6082 so obviously you send cache purge requests to nowhere...

check varnish.params you will see these:

# Default address and port to bind to. Blank address means all IPv4
# and IPv6 interfaces, otherwise specify a host name, an IPv4 dotted
# quad, or an IPv6 address in brackets.
VARNISH_LISTEN_ADDRESS=127.0.0.1
VARNISH_LISTEN_PORT=80

# Admin interface listen address and port
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082

you need cache purge sent to 127.0.0.1:80 where is your varnish is listening.

Answer 4

If you're on a dev machine try port 80 in env.php

Answer 5

In addition to direct edit the env.php reference the accepted answer, you could also use the command below, which is simpler:

magento setup:config:set --http-cache-hosts=127.0.0.1

The parameter format must be <hostname or ip>:<listen port>, where you can omit <listen port> if it’s port 80.

You can then purge Varnish hosts when you refresh the Magento cache (also referred to as cleaning the cache) in the Magento Admin or using the command line.

More details could see the official guide Configure Magento to purge Varnish