Magento 2.3.3 Varnish problems. Magento's VCL file is not working

Question

I have managed to set up Varnish for magento 2 using nginx to handle SSL. However if I try to use the vcl file provided by magento varnish stops working with the usual

"Error 503 Backend fetch failed Backend fetch failed"

error. There is nothing in the logs nothing shows up anywhere. There is something dodgy in the configuration file. Anyone else had the same issue?

Update log file:

-   ReqUnset       Accept-Encoding: gzip, deflate, br
-   ReqHeader      Accept-Encoding: gzip
-   VCL_return     hash
-   VCL_call       HASH
-   VCL_return     lookup
-   VCL_call       MISS
-   VCL_return     fetch
-   Link           bereq 6 fetch
-   Timestamp      Fetch: 1575437126.769898 0.000150 0.000150
-   RespProtocol   HTTP/1.1
-   RespStatus     503
-   RespReason     Backend fetch failed
-   RespHeader     Date: Wed, 04 Dec 2019 05:25:26 GMT
-   RespHeader     Server: Varnish
-   RespHeader     Content-Type: text/html; charset=utf-8
-   RespHeader     Retry-After: 5
-   RespHeader     X-Varnish: 5
-   RespHeader     Age: 0
-   RespHeader     Via: 1.1 varnish (Varnish/5.2)
-   VCL_call       DELIVER
-   RespUnset      Age: 0
-   RespHeader     Pragma: no-cache
-   RespHeader     Expires: -1
-   RespHeader     Cache-Control: no-store, no-cache, must-revalidate, max-age=0
-   RespUnset      Server: Varnish
-   RespUnset      X-Varnish: 5
-   RespUnset      Via: 1.1 varnish (Varnish/5.2)
-   VCL_return     deliver
-   Timestamp      Process: 1575437126.769912 0.000163 0.000014
-   RespHeader     Content-Length: 278
-   RespHeader     Connection: close
-   Timestamp      Resp: 1575437126.769948 0.000200 0.000036
-   ReqAcct        1158 0 1158 264 278 542
-   End

Other part of the log file:

-   BereqHeader    grace: none
-   BereqHeader    Accept-Encoding: gzip
-   BereqProtocol  HTTP/1.1
-   BereqHeader    X-Varnish: 9
-   VCL_call       BACKEND_FETCH
-   VCL_return     fetch
-   FetchError     no backend connection
-   Timestamp      Beresp: 1575437127.586058 0.000018 0.000018
-   Timestamp      Error: 1575437127.586061 0.000021 0.000003
-   BerespProtocol HTTP/1.1
-   BerespStatus   503
-   BerespReason   Service Unavailable
-   BerespReason   Backend fetch failed
-   BerespHeader   Date: Wed, 04 Dec 2019 05:25:27 GMT
-   BerespHeader   Server: Varnish
-   VCL_call       BACKEND_ERROR
-   BerespHeader   Content-Type: text/html; charset=utf-8
-   BerespHeader   Retry-After: 5
-   VCL_return     deliver
-   Storage        malloc Transient
-   ObjProtocol    HTTP/1.1
-   ObjStatus      503
-   ObjReason      Backend fetch failed
-   ObjHeader      Date: Wed, 04 Dec 2019 05:25:27 GMT
-   ObjHeader      Server: Varnish
-   ObjHeader      Content-Type: text/html; charset=utf-8
-   ObjHeader      Retry-After: 5
-   Length         278
-   BereqAcct      0 0 0 0 0 0
-   End

I managed to narrow it down to a 404 error in the apache access.log. This seems to be the issue:

127.0.0.1 - - [04/Dec/2019:06:28:59 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:29:09 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:29:19 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:29:29 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:29:39 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:31:15 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"
127.0.0.1 - - [04/Dec/2019:06:31:25 +0000] "GET /pub/health_check.php HTTP/1.1" 404 475 "-" "-"

varnishlog -g raw -i Backend_health shows

         0 Backend_health - boot.default Still sick 4--X-R- 4 5 10 0.000516 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 4 5 10 0.000516 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 4 5 10 0.000798 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 4 5 10 0.000393 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 4 5 10 0.000796 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 3 5 10 0.000514 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 2 5 10 0.000511 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 1 5 10 0.000609 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000646 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000650 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000580 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000661 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000842 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000931 0.000000 HTTP/1.1 404 Not Found
         0 Backend_health - boot.default Still sick 4--X-R- 0 5 10 0.000669 0.000000 HTTP/1.1 404 Not Found

the requested file is there and it is accessible. Unsure what causes the problem.

Answer 1

Based on the logs provided, a backend health check fails in Varnish for /pub/health_check.php.

If you have configured NGINX with root /path/to/magento/pub (which you should, as it's more secure), then you need to edit the health check VCL code in order to remove pub, e.g. here:

Change your generated VCL code (earlier link is just to point location in VCL code):

.url = "/pub/health_check.php";

to

.url = "/health_check.php";

Then restart Varnish.

Why: since NGINX's web root is pointed to pub directory, all the web-accessible files in that directory are accessible from webroot, e.g. http://example.com/foo and not http://example.com/pub/bar.

This is rather a bug with Magento 2 default VCL generation. Ideally, it should detect pub vs no-pub setup and generate the proper VCL.

If anything, they should have defaulted to pub as webroot setup as it is most correct. (makes most of the Magento source files non-accessible from the web browser, thus secure).