1

We have a Magento 2.4.0 store. Most of our products are configurable. And some of them are having a lot of options. We also use swatch options.

We have discovered some links which return:

Error 503 Backend fetch failed

Backend fetch failed

Guru Meditation:

XID: 56884378

Varnish cache server

My Nginx configuration file:

autoindex       off;
    sendfile        on;
    tcp_nodelay     on;
    keepalive_timeout 25;
    client_body_timeout 15;
    client_header_timeout 15;
    send_timeout 15;
    types_hash_max_size 2048;
    server_tokens off;
    client_max_body_size 64M;
    client_body_buffer_size 128k;
    client_header_buffer_size 16k;
    large_client_header_buffers 4 16k;
    fastcgi_read_timeout 300;
    proxy_read_timeout 300;
    fastcgi_buffer_size 256k;
    fastcgi_buffers 16 256k;

    open_file_cache max=50000 inactive=30s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 2;


    gzip on;
    gzip_vary   on;
    gzip_disable "MSIE [1-6]\.";
    gzip_static on;
    gzip_min_length   1400;
    gzip_buffers      16 8k;
    gzip_http_version 1.1;
    gzip_comp_level 6;
    gzip_proxied    any;
    gzip_types text/plain text/css text/xml application/javascript  application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml ;

My vhost file:

fastcgi_index index.php;
        fastcgi_pass golfcoukbackend;
        fastcgi_buffers 16 256k;
        fastcgi_buffer_size 256k;
        fastcgi_read_timeout 600s;
        fastcgi_connect_timeout 600s;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
        location ~* .(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ {
           add_header Cache-Control "public";
           add_header X-Frame-Options "SAMEORIGIN";
           expires +1y;
        }
  }

Nginx seems that does not have the issue.

Varnish default.vcl file:

# Varnish configuration
vcl 4.0;

import std;
# The minimal Varnish version is 6.0
# For SSL offloading, pass the following header in your proxy server or load balancer: 'X-Forwarded-Proto: https'


backend default {
    .host = "10.2.37.9";
    .port = "8080";
    .first_byte_timeout = 600s;
#    .probe = {
#        .url = "/health_check.php";
#        .timeout = 2s;
#        .interval = 5s;
#        .window = 10;
#        .threshold = 5;
#   }
}


acl purge {
    "127.0.0.1";
    "myip";
    "10.2.37.9";
}

sub vcl_recv {

    # For multiple backends, uncomment the below. Ensure you are using the right
    # selection code depending on the type of loadbalancing being used.
    # set req.backend_hint = prod_web.backend();

    # Remove this if using multiple backends.
    set req.backend_hint = default;

    # To Exclude a Domain uncomment the below 
    #if (req.http.host ~ "exampledomain.com") {
    #return (pass);
    #}

    if (req.restarts > 0) {
        set req.hash_always_miss = true;
    }

    # Exclude domain example
    if (req.http.host ~ "exampledomain.com") {
     return (pass);
    }

    # Exclude URI example
     if (req.url ~ "^/.well-known/") {
      return (pass);
    }

    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        # To use the X-Pool header for purging varnish during automated deployments, make sure the X-Pool header
        # has been added to the response in your backend server config. This is used, for example, by the
        # capistrano-magento2 gem for purging old content from varnish during its deploy routine.
        if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
            return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
        }

        if (req.http.X-Magento-Tags-Pattern) {
          ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        }

        # ban everything to catch assets as well
        if (req.http.X-Magento-Tags-Pattern == ".*") {
          ban("req.url ~ .*");
        }

        if (req.http.X-Pool) {
          ban("obj.http.X-Pool ~ " + req.http.X-Pool);
        }
        return (synth(200, "Purged"));
    }

    # HTTP -> HTTPS
    if (req.http.X-Forwarded-Proto !~ "https") {
        set req.http.location = "https://" + req.http.host + req.url;
        return (synth(750, "Permanently moved"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass shopping cart, checkout and search requests
    if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
        return (pass);
    }

    # Bypass health check requests
    if (req.url ~ "/health_check.php") {
        return (pass);
    }

    # Set initial grace period usage status
    set req.http.grace = "none";

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unknown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Remove all marketing get parameters to minimize the cache objects
    if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
        set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
        set req.url = regsub(req.url, "[?|&]+$", "");
    }

    # Static files caching
    if (req.url ~ "^/(pub/)?(media|static)/") {
        # Static files should not be cached by default
        return (pass);

        # But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
        #unset req.http.Https;
        #unset req.http.X-Forwarded-Proto;
        #unset req.http.Cookie;
    }

    return (hash);
}

sub vcl_hash {
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # For multi site configurations to not cache each other's content
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }
    

    if (req.url ~ "/graphql") {
        call process_graphql_headers;
    }
}

sub process_graphql_headers {
    if (req.http.Store) {
        hash_data(req.http.Store);
    }
    if (req.http.Content-Currency) {
        hash_data(req.http.Content-Currency);
    }
}

sub vcl_backend_response {

    set beresp.grace = 3d;

    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # cache only successfully responses and 404s
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 0s;
        set beresp.uncacheable = true;
        return (deliver);
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    # validate if we need to cache it and prevent from setting cookie
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
    }

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
       beresp.http.Cache-Control ~ "no-cache|no-store") ||
       beresp.http.Vary == "*") {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }

    return (deliver);
}

sub vcl_deliver {
    if (resp.http.X-Magento-Debug) {
        if (resp.http.x-varnish ~ " ") {
            set resp.http.X-Magento-Cache-Debug = "HIT";
            set resp.http.Grace = req.http.grace;
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    if (obj.hits > 0) {
        set resp.http.X-Cache = "HIT";
        set resp.http.X-Cache-Hits = obj.hits;
    }
    else {
        set resp.http.X-Cache = "MISS";
    }

    # Not letting browser to cache non-static files.
    if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "-1";
        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
    }

    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

sub vcl_synth {
    if (resp.status == 750) {
        set resp.http.location = req.http.location;
        set resp.status = 301;
        return (deliver);
    }
}

#sub vcl_hit {
#    if (obj.ttl >= 0s) {
        # Hit within TTL period
#        return (deliver);
#    }
#    if (std.healthy(req.backend_hint)) {
#        if (obj.ttl + 300s > 0s) {
            # Hit after TTL expiration, but within grace period
#            set req.http.grace = "normal (healthy server)";
#            return (deliver);
#        } else {
            # Hit after TTL and grace expiration
#            return (restart);
#        }
#    } else {
        # server is not healthy, retrieve from cache
#        set req.http.grace = "unlimited (unhealthy server)";
#        return (deliver);
#    }
#}

Varnish runs with default hdr_len

If I am right our issue is related to the Magento 2 bug with configurable products and big headers which are coming when there are configurable products with a lot of variations.

https://github.com/magento/magento2/issues/23834

I have tried a solution described in the above link which disables plugin product_identities_extender. This fixes the problem but not in all our products...

Some still return Backend fetch failed

varnishlog -g request -q "ReqUrl ~ '^/masters-rain'" output:

  << Request  >> 4121486
  Begin          req 4121485 rxreq
  Timestamp      Start: 1637777394.195388 0.000000 0.000000
  Timestamp      Req: 1637777394.195388 0.000000 0.000000
  VCL_use        boot
  ReqStart       10.2.37.9 60676 a0
  ReqMethod      GET
  ReqURL         /masters-rain-wedge-bag-cover
  ReqProtocol    HTTP/1.0
  ReqHeader      X-Real-IP: my.ip
  ReqHeader      X-Forwarded-For: my.ip
  ReqHeader      X-Forwarded-Proto: https
  ReqHeader      X-Forwarded-Port: 443
  ReqHeader      Host: www.example.co.uk
  ReqHeader      Connection: close
  ReqHeader      accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
   ReqHeader      user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.1 Safari/605.1.15
  ReqHeader      accept-language: en-GB,en;q=0.9
  ReqHeader      accept-encoding: gzip, deflate, br
  ReqHeader      cookie: _dc_gtm_UA-example-1=1; _ga=GA1.3.1098407753.1637181974; _gid=GA1.3.734412942.1637623269; PHPSESSID=e9pbvu084k13675ka6vpkukdp2; form_key=2qfq1dIqSyhkL8ho; form_key=2qfq1dIqSyhkL8ho; mage-cache-sessid=true; mage-messages=; section_data_ids=%7B%22
   ReqUnset       X-Forwarded-For: my.ip
   ReqHeader      X-Forwarded-For: my.ip, 10.2.37.9
   VCL_call       RECV
   ReqHeader      grace: none
   ReqURL         /masters-rain-wedge-bag-cover
   ReqUnset       accept-encoding: gzip, deflate, br
   ReqHeader      Accept-Encoding: gzip
   VCL_return     hash
   VCL_call       HASH
   VCL_return     lookup
   VCL_call       MISS
   VCL_return     fetch
   Link           bereq 4121487 fetch
   Timestamp      Fetch: 1637777397.649357 3.453968 3.453968
   RespProtocol   HTTP/1.1
   RespStatus     200
   RespReason     OK
   RespHeader     Server: nginx
   RespHeader     Date: Wed, 24 Nov 2021 18:09:57 GMT
   RespHeader     Content-Type: text/html; charset=UTF-8
   RespHeader     Pragma: cache
   RespHeader     Cache-Control: max-age=86400, public, s-maxage=86400
   RespHeader     Expires: Thu, 25 Nov 2021 18:09:54 GMT
   RespHeader     ALEX-TEST-HEADER: 37230
   RespHeader     X-Magento-Tags:
   RespHeader     X-Content-Type-Options: nosniff
   RespHeader     X-XSS-Protection: 1; mode=block
   RespHeader     X-Frame-Options: SAMEORIGIN
   RespHeader     Vary: Accept-Encoding
   RespHeader     Content-Encoding: gzip
   RespHeader     X-Varnish: 4121486
   RespHeader     Age: 0
   RespHeader     Via: 1.1 varnish (Varnish/6.5)
   VCL_call       DELIVER
   RespUnset      Age: 0
   RespHeader     X-Cache: MISS
   RespUnset      Pragma: cache
   RespHeader     Pragma: no-cache
   RespUnset      Expires: Thu, 25 Nov 2021 18:09:54 GMT
   RespHeader     Expires: -1
   RespUnset      Cache-Control: max-age=86400, public, s-maxage=86400
   RespHeader     Cache-Control: no-store, no-cache, must-revalidate, max-age=0
   RespUnset      X-Magento-Tags:
   RespUnset      Server: nginx
   RespUnset      X-Varnish: 4121486
   RespUnset      Via: 1.1 varnish (Varnish/6.5)
   VCL_return     deliver
   Timestamp      Process: 1637777397.649415 3.454027 0.000058
   Filters
   RespHeader     Accept-Ranges: bytes
   RespHeader     Content-Length: 22193
   RespHeader     Connection: close
   Timestamp      Resp: 1637777397.649481 3.454092 0.000065
   ReqAcct        1133 0 1133 436 22193 22629
   End

 << BeReq    >> 4121487
  Begin          bereq 4121486 fetch
  VCL_use        boot
  Timestamp      Start: 1637777394.195457 0.000000 0.000000
  BereqMethod    GET
  BereqURL       /masters-rain-wedge-bag-cover
  BereqProtocol  HTTP/1.0
  BereqHeader    X-Real-IP: my.ip
  BereqHeader    X-Forwarded-Proto: https
  BereqHeader    X-Forwarded-Port: 443
  BereqHeader    Host: www.example.co.uk
  BereqHeader    accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  BereqHeader    user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.1 Safari/605.1.15
  BereqHeader    accept-language: en-GB,en;q=0.9
  BereqHeader    cookie: _dc_gtm_UA-70723715-1=1; _ga=GA1.3.1098407753.1637181974; _gid=GA1.3.734412942.1637623269; PHPSESSID=e9pbvu084k13675ka6vpkukdp2; form_key=2qfq1dIqSyhkL8ho; form_key=2qfq1dIqSyhkL8ho; mage-cache-sessid=true; mage-messages=; section_data_ids=%7B%22
  BereqHeader    X-Forwarded-For: my.ip, 10.2.37.9
  BereqHeader    grace: none
  BereqHeader    Accept-Encoding: gzip
  BereqProtocol  HTTP/1.1
  BereqHeader    X-Varnish: 4121487
  VCL_call       BACKEND_FETCH
  VCL_return     fetch
  BackendOpen    60 default 10.2.37.9 8080 10.2.37.9 48270 reuse
  Timestamp      Bereq: 1637777394.195563 0.000105 0.000105
  Timestamp      Beresp: 1637777397.638441 3.442983 3.442878
  BerespProtocol HTTP/1.1
  BerespStatus   200
  BerespReason   OK
  BerespHeader   Server: nginx
  BerespHeader   Date: Wed, 24 Nov 2021 18:09:57 GMT
  BerespHeader   Content-Type: text/html; charset=UTF-8
  BerespHeader   Transfer-Encoding: chunked
  BerespHeader   Connection: keep-alive
  BerespHeader   Set-Cookie: form_key=2qfq1dIqSyhkL8ho; expires=Wed, 24-Nov-2021 19:09:54 GMT; Max-Age=3600; path=/; domain=www.example.co.uk; secure
  BerespHeader   Set-Cookie: PHPSESSID=e9pbvu084k13675ka6vpkukdp2; expires=Wed, 24-Nov-2021 19:09:54 GMT; Max-Age=3600; path=/; domain=www.example.co.uk; secure; HttpOnly
  BerespHeader   Set-Cookie: PHPSESSID=e9pbvu084k13675ka6vpkukdp2; expires=Wed, 24-Nov-2021 19:09:54 GMT; Max-Age=3600; path=/; domain=www.example.co.uk; secure; HttpOnly
  BerespHeader   Pragma: cache
  BerespHeader   Cache-Control: max-age=86400, public, s-maxage=86400
  BerespHeader   Expires: Thu, 25 Nov 2021 18:09:54 GMT
  BerespHeader   ALEX-TEST-HEADER: 37230
  BerespHeader   X-Magento-Tags:
  BerespHeader   X-Content-Type-Options: nosniff
  BerespHeader   X-XSS-Protection: 1; mode=block
  BerespHeader   X-Frame-Options: SAMEORIGIN
  BerespHeader   Content-Encoding: gzip
  BerespHeader   Vary: Accept-Encoding
  TTL            RFC 86400 10 0 1637777398 1637777398 1637777397 1637863794 86400 cacheable
  VCL_call       BACKEND_RESPONSE
  TTL            VCL 86400 259200 0 1637777398 cacheable
  BerespUnset    Set-Cookie: form_key=2qfq1dIqSyhkL8ho; expires=Wed, 24-Nov-2021 19:09:54 GMT; Max-Age=3600; path=/; domain=www.example.co.uk; secure
  BerespUnset    Set-Cookie: PHPSESSID=e9pbvu084k13675ka6vpkukdp2; expires=Wed, 24-Nov-2021 19:09:54 GMT; Max-Age=3600; path=/; domain=www.example.co.uk; secure; HttpOnly
  BerespUnset    Set-Cookie: PHPSESSID=e9pbvu084k13675ka6vpkukdp2; expires=Wed, 24-Nov-2021 19:09:54 GMT; Max-Age=3600; path=/; domain=www.example.co.uk; secure; HttpOnly
  VCL_return     deliver
  Filters         gunzip esi_gzip
  BerespUnset    Content-Encoding: gzip
  BerespHeader   Content-Encoding: gzip
  Storage        malloc s0
  Fetch_Body     2 chunked -
  Gzip           G F E 108872 22193 80 177464 177474
  Gzip           U F - 22188 108872 80 177424 177434
  BackendClose   60 default recycle
  Timestamp      BerespBody: 1637777397.649352 3.453894 0.010910
  Length         22193
  BereqAcct      1145 0 1145 916 0 916
  End

Any help please? I am at a dead-end here!

1 Answer 1

1

Before we make any assumptions about the cause of these HTTP 503 errors, let's look at the logs for these transactions.

Run the following command:

varnishlog -g request -q "ReqUrl eq '/'"

Please replace / with the URL you're trying to monitor and add the output to your question. Based on that, we'll try to figure out a solution, which I will then append to this reply.

8
  • I am sorry but with this command server load increases and I get no output....Like it hangs or something
    – G. G.
    Nov 24, 2021 at 10:33
  • @G.G. it shouldn't, try adding sudo in front of it. Please make sure you run this command on the server where the Varnish instance is running. As described on varnish-cache.org/docs/6.6/reference/varnishlog.html you can also use the -R parameter to perform rate limiting if you want to limit the number of transactions. Nov 24, 2021 at 11:57
  • Again this dont work....I am running command as root...No output at all. My command is: varnishlog -g request -q "ReqUrl eq 'https://www.example.com/masters-rain-wedge-bag-cover'"
    – G. G.
    Nov 24, 2021 at 12:05
  • Run varnishlog -g request -q "ReqUrl eq '/masters-rain-wedge-bag-cover'" instead. Nov 24, 2021 at 12:16
  • The same issue.....Keep in mind that my link in frontend returns Error 503 Backend fetch failed
    – G. G.
    Nov 24, 2021 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.