I've followed the dev docs on setting up varnish, and increased the http_resp_size and http_resp_hdr_len when the daemon starts. Most of the time it's fine, but every once in a while I will get a backend fetch failed. (probably one every few thousand requests, very hard to replicate)
Checked using varnishlog -d -q "BerespStatus == 503"
There doesn't seem to be much indication of an actual problem other than the error I'm seeing:
FetchError HTC status -1
One such failure is as follows:
<< BeReq >> 426839
Begin bereq 426838 pass
Timestamp Start: 1534283980.206907 0.000000 0.000000
BereqMethod GET
BereqURL /
BereqProtocol HTTP/1.1
BereqHeader Host: <omitted>
BereqHeader Pragma: no-cache
BereqHeader Cache-Control: no-cache
BereqHeader Upgrade-Insecure-Requests: 1
BereqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36
BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
BereqHeader Accept-Encoding: gzip, deflate
BereqHeader Accept-Language: en-US,en;q=0.9
BereqHeader Cookie: mage-translation-storage=%7B%7D; mage-translation-file-version=%7B%7D; form_key=cm37N1pWTHh0CP1o; mage-cache-storage=%7B%7D; mage-cache-storage-section-invalidation=%7B%7D; mage-cache-sessid=true; mage-messages=; recently_viewed_product=%7B%7D; r
BereqHeader X-Forwarded-For: <omitted>
BereqHeader X-Varnish: 426839
VCL_call BACKEND_FETCH
VCL_return fetch
BackendOpen 25 boot.default 127.0.0.1 8080 127.0.0.1 55734
BackendStart 127.0.0.1 8080
Timestamp Bereq: 1534283980.207162 0.000255 0.000255
FetchError HTC status -1
BackendClose 25 boot.default
Timestamp Beresp: 1534283980.224352 0.017445 0.017190
Timestamp Error: 1534283980.224370 0.017463 0.000018
BerespProtocol HTTP/1.1
BerespStatus 503
BerespReason Service Unavailable
BerespReason Backend fetch failed
BerespHeader Date: Tue, 14 Aug 2018 21:59:40 GMT
BerespHeader Server: Varnish
VCL_call BACKEND_ERROR
BerespHeader Content-Type: text/html; charset=utf-8
BerespHeader Retry-After: 5
VCL_return deliver
Storage malloc Transient
ObjProtocol HTTP/1.1
ObjStatus 503
ObjReason Backend fetch failed
ObjHeader Date: Tue, 14 Aug 2018 21:59:40 GMT
ObjHeader Server: Varnish
ObjHeader Content-Type: text/html; charset=utf-8
ObjHeader Retry-After: 5
Length 283
BereqAcct 881 0 881 0 0 0
End