Nginx sometimes returns 502 because of varnish panic

Question

we are using Magento 2.3.5 with following setup:

User >> NGINX >> VARNISH (locale Docker) >> APACHE + PHP

Randomly some Requests returned by Nginx with 502. log file says:

recv() failed (104: Connection reset by peer) while reading response header from upstream

or

upstream prematurely closed connection while reading response header from upstream

The error comes from varnish:

Child (23669) Panic message:
Assert error in VEP_Init(), cache/cache_esi_parse.c line 1039:
  Condition((vep) != 0) not true.
thread = (cache-worker)
version = varnish-4.0.5 revision 07eff4c29
ident = Linux,3.10.0-1160.21.1.el7.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll
Backtrace:
  0x55ed00ab6dc3: varnishd(+0x3edc3) [0x55ed00ab6dc3]
  0x55ed00a9d16f: varnishd(VEP_Init+0x29f) [0x55ed00a9d16f]
  0x55ed00a9a59a: varnishd(vfp_esi_gzip_pull+0x1ea) [0x55ed00a9a59a]
  0x55ed00aa3a3e: varnishd(VFP_Fetch_Body+0x6e) [0x55ed00aa3a3e]
  0x55ed00aa0839: varnishd(+0x28839) [0x55ed00aa0839]
  0x55ed00aba07d: varnishd(Pool_Work_Thread+0x39d) [0x55ed00aba07d]
  0x55ed00acf372: varnishd(+0x57372) [0x55ed00acf372]
  0x7fd536c9ae25: /lib64/libpthread.so.0(+0x7e25) [0x7fd536c9ae25]
  0x7fd5369c834d: /lib64/libc.so.6(clone+0x6d) [0x7fd5369c834d]
  busyobj = 0x7fd524d00020 {
    ws = 0x7fd524d000e0 { OVERFLOW
      id = ""o",
      {s,f,r,e} = {0x7fd524d02008,+90136,(nil),+90136},
    },

The problem is: There is no way to reproduce this failure. It seems like it happens randomly. I know Varnish Version 4 is not the latest, but worked like a charme for month in our setup.

Is there anybody out there with a additional idea, where to search for the solution? Thanks in advanced

Answer 1

What the source code says

If we look at the error, we can see that an assert error is triggered in cache/cache_esi_parse.c on line `1039.

A little bit of context: this is the place where ESI tags get parsed and where the corresponding ESI endpoints are called.

According to the source code on https://github.com/varnishcache/varnish-cache/blob/4.0/bin/varnishd/cache/cache_esi_parse.c#L1038-L1039 this is what happens:

vep = (void*)WS_Alloc(bo->ws, sizeof *vep);
AN(vep);

I also added the line where a value is assigned to the vep variable. The AN(vep) is an assert that fails and that triggers the error.

What's causing this error?

I'm guessing the ESI parsing triggers an overflow of the workspace memory. According to your output vfp_esi_gzip_pull is also called, which implies that Gzip decompression has to take place, which can also consume some memory.

Workspace values

I'd like to see the values of your workspace parameters. You can see the values by running the following command:

varnishadm param.show | grep "workspace_"

Here's some output:

$ varnishadm param.show | grep "workspace_"
workspace_backend             64k [bytes] (default)
workspace_client              64k [bytes] (default)
workspace_session             0.50k [bytes] (default)
workspace_thread              2k [bytes] (default)

I'm guessing that ESI parsing is done using workspace_backend memory.

Measuring workspace overflows

As easy way to know whether or not your errors quality as workspace overflow, we can measure this by looking at some counters.

You can see them by running the following command:

varnishstat -f "MAIN.ws_*"

If the counter is not visible, you can press the d key to display counters with a zero value. When these counters increase, you know that the error you're getting is related.

If my assumption is correct, the MAIN.ws_backend_overflow should be increasing when your error is occuring.

Increasing workspace memory

Once you're certain that the error is related to workspace memory overflow, you can increase the value.

Imagine that the current workspace_backend value is set to 64k. When the overflow proves that this is not enough, you can add the following parameter to your varnishd command:

-p workspace_backend=128k

Usually this is done in the systemd unit file, but you mentioned that you're using a Docker container. You can change this in your Dockerfile.

Don't use Varnish 4

I would also like to emphasize that Varnish 4 is End Of Life. Bugs are issues are no longer fixed. I'd advise you to use Varnish 6, preferably the LTS version.

In case of Docker, you can use our official Varnish 6 image: https://hub.docker.com/_/varnish

If you want to install the packages yourself, please use the following packages: https://packagecloud.io/varnishcache/varnish60lts

Don't worry, VCL syntax remains stable between Varnish 4 & Varnish 6.