Magento2.4.5 Varnish alway so 503 backend fetch fail

Question

I have setup Vanish 6 on Magento2.4.5 but it alway show 503 backend fetch fail. My nginx config:

upstream fastcgi_backend {
  server   unix:/run/php/php8.1-fpm.sock;
}

server {
    server_name mysite.com www.mysite.com *.mysite.com;
    listen 8080;
    set $MAGE_ROOT /var/www/html/ins;
    set $MAGE_MODE production; # or developer

    access_log /var/log/nginx/mysite.com-access.log;
    error_log /var/log/nginx/mysite.com-error.log;

    include /var/www/html/ins/nginx.conf.sample;
}

server {

    listen 443 ssl http2;
    server_name nikitagossain.com www.mysite.com *.mysite.com;

    ssl_certificate /root/ssl/development_multi_domain_new_2022_to_2023_second_cert_chain.crt;
    ssl_certificate_key /root/ssl/development_multi_domain_new_2022_to_2023_second.key;
    ssl_protocols              TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers               'AES128+EECDH:AES128+EDH:!aNULL';
    ssl_session_cache    shared:SSL:10m;
    ssl_session_timeout 24h;
    keepalive_timeout 300s;

    location / {
        proxy_pass http://127.0.0.1;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Ssl-Offloaded "1";
        proxy_set_header      X-Forwarded-Proto https;
        proxy_set_header      X-Forwarded-Port 443;
        #proxy_hide_header X-Varnish;
        #proxy_hide_header Via;
        proxy_set_header X-Forwarded-Proto $scheme;

    }

}

The command: varnishlog -g request -q "ReqUrl eq '/'" Result:

    Begin          req 58 rxreq
-   Timestamp      Start: 1671967669.126018 0.000000 0.000000
-   Timestamp      Req: 1671967669.126018 0.000000 0.000000
-   VCL_use        boot
-   ReqStart       127.0.0.1 46472 a0
-   ReqMethod      GET
-   ReqURL         /
-   ReqProtocol    HTTP/1.0
-   ReqHeader      Host: mysite.com
-   ReqHeader      X-Forwarded-Host: mysite.com
-   ReqHeader      X-Real-IP: 182.180.107.169
-   ReqHeader      X-Forwarded-For: 182.180.107.169
-   ReqHeader      Ssl-Offloaded: 1
-   ReqHeader      X-Forwarded-Proto: https
-   ReqHeader      X-Forwarded-Port: 443
-   ReqHeader      X-Forwarded-Proto: https
-   ReqHeader      Connection: close
-   ReqHeader      cache-control: max-age=0
-   ReqHeader      sec-ch-ua: "Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"
-   ReqHeader      sec-ch-ua-mobile: ?0
-   ReqHeader      sec-ch-ua-platform: "Linux"
-   ReqHeader      upgrade-insecure-requests: 1
-   ReqHeader      user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
-   ReqHeader      accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
-   ReqHeader      sec-fetch-site: none
-   ReqHeader      sec-fetch-mode: navigate
-   ReqHeader      sec-fetch-user: ?1
-   ReqHeader      sec-fetch-dest: document
-   ReqHeader      accept-encoding: gzip, deflate, br
-   ReqHeader      accept-language: en-PK,en-US;q=0.9,en;q=0.8
-   ReqHeader      cookie: _gid=GA1.2.61049282.1671455531; hubspotutk=f3b6df9e00be29ddedfc7eb34fd8eade; _clck=lo69rm|1|f7o|0; private_content_version=a1b7dfdfe60b99b881915cecc447647c; __hstc=31723169.f3b6df9e00be29ddedfc7eb34fd8eade.1671455543898.1671864564688.167189458746
-   ReqUnset       X-Forwarded-For: 182.180.107.169
-   ReqHeader      X-Forwarded-For: 182.180.107.169, 127.0.0.1
-   VCL_call       RECV
-   ReqHeader      grace: none
-   ReqURL         /
-   ReqUnset       accept-encoding: gzip, deflate, br
-   ReqHeader      Accept-Encoding: gzip
-   VCL_return     hash
-   VCL_call       HASH
-   VCL_return     lookup
-   VCL_call       MISS
-   VCL_return     fetch
-   Link           bereq 60 fetch
-   Timestamp      Fetch: 1671967669.126376 0.000358 0.000358
-   RespProtocol   HTTP/1.1
-   RespStatus     503
-   RespReason     Backend fetch failed
-   RespHeader     Date: Sun, 25 Dec 2022 11:27:49 GMT
-   RespHeader     Server: Varnish
-   RespHeader     content-type: text/html; charset=utf-8
-   RespHeader     Retry-After: 5
-   RespHeader     X-Varnish: 59
-   RespHeader     Age: 0
-   RespHeader     Via: 1.1 varnish (Varnish/6.2)
-   VCL_call       DELIVER
-   RespHeader     X-Magento-Cache-Debug: MISS
-   RespHeader     Pragma: no-cache
-   RespHeader     Expires: -1
-   RespHeader     Cache-Control: no-store, no-cache, must-revalidate, max-age=0
-   RespUnset      Age: 0
-   RespUnset      Server: Varnish
-   RespUnset      X-Varnish: 59
-   RespUnset      Via: 1.1 varnish (Varnish/6.2)
-   VCL_return     deliver
-   Timestamp      Process: 1671967669.126402 0.000384 0.000026
-   Filters        
-   RespHeader     Content-Length: 279
-   RespHeader     Connection: close
-   Timestamp      Resp: 1671967669.126494 0.000476 0.000092
-   ReqAcct        1662 0 1662 293 279 572
-   End            
**  << BeReq    >> 60        
--  Begin          bereq 59 fetch
--  VCL_use        boot
--  Timestamp      Start: 1671967669.126158 0.000000 0.000000
--  BereqMethod    GET
--  BereqURL       /
--  BereqProtocol  HTTP/1.0
--  BereqHeader    Host: nikitagossain.com
--  BereqHeader    X-Forwarded-Host: nikitagossain.com
--  BereqHeader    X-Real-IP: 182.180.107.169
--  BereqHeader    Ssl-Offloaded: 1
--  BereqHeader    X-Forwarded-Proto: https
--  BereqHeader    X-Forwarded-Port: 443
--  BereqHeader    X-Forwarded-Proto: https
--  BereqHeader    sec-ch-ua: "Google Chrome";v="107", "Chromium";v="107", "Not=A?Brand";v="24"
--  BereqHeader    sec-ch-ua-mobile: ?0
--  BereqHeader    sec-ch-ua-platform: "Linux"
--  BereqHeader    upgrade-insecure-requests: 1
--  BereqHeader    user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
--  BereqHeader    accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
--  BereqHeader    sec-fetch-site: none
--  BereqHeader    sec-fetch-mode: navigate
--  BereqHeader    sec-fetch-user: ?1
--  BereqHeader    sec-fetch-dest: document
--  BereqHeader    accept-language: en-PK,en-US;q=0.9,en;q=0.8
--  BereqHeader    cookie: _gid=GA1.2.61049282.1671455531; hubspotutk=f3b6df9e00be29ddedfc7eb34fd8eade; _clck=lo69rm|1|f7o|0; private_content_version=a1b7dfdfe60b99b881915cecc447647c; __hstc=31723169.f3b6df9e00be29ddedfc7eb34fd8eade.1671455543898.1671864564688.167189458746
--  BereqHeader    X-Forwarded-For: 182.180.107.169, 127.0.0.1
--  BereqHeader    grace: none
--  BereqHeader    Accept-Encoding: gzip
--  BereqProtocol  HTTP/1.1
--  BereqHeader    X-Varnish: 60
--  VCL_call       BACKEND_FETCH
--  VCL_return     fetch
--  FetchError     backend default: unhealthy
--  Timestamp      Beresp: 1671967669.126262 0.000104 0.000104
--  Timestamp      Error: 1671967669.126266 0.000108 0.000004
--  BerespProtocol HTTP/1.1
--  BerespStatus   503
--  BerespReason   Service Unavailable
--  BerespReason   Backend fetch failed
--  BerespHeader   Date: Sun, 25 Dec 2022 11:27:49 GMT
--  BerespHeader   Server: Varnish
--  VCL_call       BACKEND_ERROR
--  BerespHeader   content-type: text/html; charset=utf-8
--  BerespHeader   Retry-After: 5
--  VCL_return     deliver
--  Storage        malloc Transient
--  Length         279
--  BereqAcct      0 0 0 0 0 0
--  End   

My setup vcl file export from magento2 with varnish version6

# VCL version 5.0 is not supported so it should be 4.0 even though actually used Varnish version is 6
vcl 4.0;

import std;
# The minimal Varnish version is 6.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'SSL-OFFLOADED: https'

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/pub/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

acl purge {
    "localhost";
}

sub vcl_recv {
    if (req.restarts > 0) {
        set req.hash_always_miss = true;
    }

    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
        # has been added to the response in your backend server config. This is used, for example, by the
        # capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
        if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
            return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
        }
        if (req.http.X-Magento-Tags-Pattern) {
          ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        }
        if (req.http.X-Pool) {
          ban("obj.http.X-Pool ~ " + req.http.X-Pool);
        }
        return (synth(200, "Purged"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }
     # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass customer, shopping cart, checkout
    if (req.url ~ "/customer" || req.url ~ "/checkout") {
        return (pass);
    }

    # Bypass health check requests
    if (req.url ~ "^/(pub/)?(health_check.php)$") {
        return (pass);
    }

    # Set initial grace period usage status
    set req.http.grace = "none";

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unknown algorithm
            unset req.http.Accept-Encoding;
        }
    }
    # Remove all marketing get parameters to minimize the cache objects
    if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
        set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

    # Static files caching
    if (req.url ~ "^/(pub/)?(media|static)/") {
        # Static files should not be cached by default
        return (pass);

        # But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
        #unset req.http.Https;
        #unset req.http.SSL-OFFLOADED;
        #unset req.http.Cookie;
    }

    # Bypass authenticated GraphQL requests without a X-Magento-Cache-Id
    if (req.url ~ "/graphql" && !req.http.X-Magento-Cache-Id && req.http.Authorization ~ "^Bearer") {
        return (pass);
    }

    return (hash);
}

sub vcl_hash {
    if ((req.url !~ "/graphql" || !req.http.X-Magento-Cache-Id) && req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # To make sure http users don't see ssl warning
    if (req.http.SSL-OFFLOADED) {
        hash_data(req.http.SSL-OFFLOADED);
    }


    if (req.url ~ "/graphql") {
        call process_graphql_headers;
    }
}

sub process_graphql_headers {
    if (req.http.X-Magento-Cache-Id) {
        hash_data(req.http.X-Magento-Cache-Id);

        # When the frontend stops sending the auth token, make sure users stop getting results cached for logged-in users
        if (req.http.Authorization ~ "^Bearer") {
            hash_data("Authorized");
        }
    }

    if (req.http.Store) {
        hash_data(req.http.Store);
    }
 if (req.http.Content-Currency) {
        hash_data(req.http.Content-Currency);
    }
}

sub vcl_backend_response {

    set beresp.grace = 3d;

    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # cache only successfully responses and 404s that are not marked as private
    if (beresp.status != 200 &&
            beresp.status != 404 &&
            beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    # validate if we need to cache it and prevent from setting cookie
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
    }

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
       beresp.http.Cache-Control ~ "no-cache|no-store") ||
       beresp.http.Vary == "*") {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
   }

   # If the cache key in the Magento response doesn't match the one that was sent in the request, don't cache under the request's key
   if (bereq.url ~ "/graphql" && bereq.http.X-Magento-Cache-Id && bereq.http.X-Magento-Cache-Id != beresp.http.X-Magento-Cache-Id) {
      set beresp.ttl = 0s;
      set beresp.uncacheable = true;
   }

    return (deliver);
}
sub vcl_deliver {
    if (resp.http.x-varnish ~ " ") {
        set resp.http.X-Magento-Cache-Debug = "HIT";
        set resp.http.Grace = req.http.grace;
    } else {
        set resp.http.X-Magento-Cache-Debug = "MISS";
    }

    # Not letting browser to cache non-static files.
    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "-1";
        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
    }

    if (!resp.http.X-Magento-Debug) {
        unset resp.http.Age;
    }
    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

sub vcl_hit {
    if (obj.ttl >= 0s) {
        # Hit within TTL period
        return (deliver);
    }
    if (std.healthy(req.backend_hint)) {
        if (obj.ttl + 300s > 0s) {
            # Hit after TTL expiration, but within grace period
            set req.http.grace = "normal (healthy server)";
            return (deliver);
        } else {
            # Hit after TTL and grace expiration
            return (restart);
        }
    } else {
        # server is not healthy, retrieve from cache
        set req.http.grace = "unlimited (unhealthy server)";
        return (deliver);
    }
}

Can anyone help me this issue?

Answer 1

From varnishlog

FetchError     backend default: unhealthy

Assuming you're using the /pub directory as your entry point, try removing that directory from your vcl file

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .first_byte_timeout = 600s;
    .probe = {
        .url = "/health_check.php";
        .timeout = 2s;
        .interval = 5s;
        .window = 10;
        .threshold = 5;
   }
}

Restart nginx and varnish and see if that helps. If not, post relevant more varnishlog.

Tom

Answer 2

Let's do some extra debugging to figure out why Varnish would consider your backend unhealthy.

Step 1: did you reload your VCL?

As suggested in other answers: it makes sense to switch the .url parameter of your health checking probes from /pub/health_check.php to /health_check.php for more recent Magento installations.

See https://www.varnish-software.com/developers/tutorials/configuring-varnish-magento/#fixing-the-backend-health-checks-for-magento-24.

After performing that change, did you reload your VCL configuration?

You can do this using one of the following commands:

• Either run sudo varnishreload
• Or run sudo systemctl varnish reload

Step 2: check the current health of your backends

As described in https://www.varnish-software.com/developers/tutorials/troubleshooting-varnish/#vsl-backend-health-checks, there are commands to get a real-time status update on backend health.

Run the following command:

sudo varnishlog -g raw -i Backend_health

If the status is either Back healthy or Still healthy, your problem is solved. Otherwise go to the next step in the debugging process.

Step 3: run the health check locally via curl

Access your server via SSH and run the following curl command locally:

curl -v -H "Host: mysite.com" http://localhost:8080/health_check.php

The output of the health check as well as the status code will give a clear indication of the Magento health check.

Please also consider the timeout you configured in your probe. If the health check returns the proper output with an HTTP 200 status code, it could still make the health probe fail because it took longer than 2 seconds.

You could do a timing check by adding the time command in front of the curl command. Here's an example:

time curl -v -H "Host: mysite.com" http://localhost:8080/health_check.php

Step 4: taking the TLS context into account

You mentioned that the health check issue only appears for HTTPS traffic.

Since Varnish doesn't support TLS natively, you're forced to terminate the TLS connection separately. In your case that's done via Nginx.

As far as Varnish is concerned, all traffic to the backend is done over plain HTTP. Technically it doesn't really make a difference whether the original connection was made over HTTP or HTTPS.

However, maybe the Magento health checks are sensitive to this. That's why you can also test how the health check responds if you add the forwarded headers with the TLS meta information;

Here's an example test command based on the headers from your varnishlog output:

curl -v -H "Host: mysite.com" \
-H "X-Forwarded-Host: mysite.com" \
-H "X-Real-IP: 182.180.107.169" \
-H "X-Forwarded-For: 182.180.107.169" \
-H "Ssl-Offloaded: 1" \
-H "X-Forwarded-Proto: https" \
-H "X-Forwarded-Port: 443" \
http://localhost:8080/health_check.php

And you can again add a timing check to the command by prepending the time command to your curl command.

Step 5: how to fix?

If the /health_check.php page outputs an error, that error should provide the contextual information you need to fix the issue.

If you can't even connect to http://localhost:8080/health_check.php, there might be something wrong with your webserver configuration or Magento.

If the health check page doesn't respond within 2 seconds, consider increasing the VCL health probe .timeout attribute.