Assumptions
This entire answer is based on the assumption that you didn't perform any specific cache configuration in app/etc/env.php
If this is not the case for you, the answer might be irrelevant.
What is happening in Varnish
Varnish is returning an HTTP/503, not because it could not connect to the origin server, but because the health check fails. The backend became unhealthy because the URL it probes returns an HTTP/500 error.
You can actually verify this by running varnishlog -g raw -i Backend_health as illustrated below:
root@varnish:/etc/varnish# varnishlog -g raw -i Backend_health
0 Backend_health - boot.default Still sick 4---X-R- 1 5 10 0.122153 0.000000 HTTP/1.1 500 Internal Server Error
There's something wrong in /pub/health_check.php, and returns HTTP/500 as a consequence. We need to fix this in Magento.
What is happening in Magento
When you look in var/log/debug.log, you might see lines appearing that look as follows:
main.ERROR: Cache configuration is invalid
So /pub/health_check.php is failing because the cache configuration is faulty.
Looking at the source code of the health check
When we look at the source code of pub/health_check.php, you'll see that the following if-statement contains an error that matches what we see in the log files:
if (!isset($cacheConfig[ConfigOptionsListConstants::CONFIG_PATH_BACKEND]) ||
!isset($cacheConfig[ConfigOptionsListConstants::CONFIG_PATH_BACKEND_OPTIONS])) {
http_response_code(500);
$logger->error("Cache configuration is invalid");
exit(1);
}
FYI: the constant ConfigOptionsListConstants::CONFIG_PATH_BACKEND is actually translated into backend, and ConfigOptionsListConstants::CONFIG_PATH_BACKEND_OPTIONS is translated into backend_options.
So what the health check is expecting is that your cache configuration contains the backend and backend_options keys.
How to fix the issue
To fix the issue, simply update app/etc/env.php and add a caching backend, and potentially some backend options.
Here's how I configured it:
'cache' => [
'frontend' => [
'default' => [
'id_prefix' => '40d_',
'backend' => 'Cm_Cache_Backend_Redis',
'backend_options' => [
'server' => '127.0.0.1',
'port' => '6379',
]
],
]
],
FYI: I installed Redis to make sure the 127.0.0.1:6379 endpoint actually exists. After that, Magento will store its internal cache objects inside Redis.
Any other cache configuration in addition to the default key should also have a backend defined, and some backend_options.
If the page cache configuration gets in your way, simply delete it, because Varnish is going to be your page cache.
You're all set
After the cache config is finalized, /pub/health_check.php will no longer complain.
If you then run varnishlog -g raw -i Backend_health again, you'll get the following output:
0 Backend_health - boot.default Still sick 4---X-RH 4 5 10 0.189570 0.199402 HTTP/1.1 200 OK
0 Backend_health - boot.default Back healthy 4---X-RH 5 5 10 0.348480 0.236672 HTTP/1.1 200 OK
0 Backend_health - boot.default Still healthy 4---X-RH 6 5 10 0.202581 0.228149 HTTP/1.1 200 OK
You see that Varnish's health checks succeeded, and that the site is fully functional.
