Magento 2.4 Varnish error STACK OVERFLOW

Question

I have a dockerised M2.4 installation. All worked fine with my theme and the installed extensions while working with Magento demo data. As soon as update the database with the live data I cannot get Varnish to work correctly. Contact page loads and non cached pages, but only the header will load for all other pages. Having checked the varnish container logs I see the following error below. It appears to be asking for more resources so I have updated /etc/default/varnish as follows:

DAEMON_OPTS="-a :6081
-T localhost:6082
-f /etc/varnish/default.vcl
-S /etc/varnish/secret
-p thread_pool_stack=4096k
-p http_resp_hdr_len=65536
-p http_resp_size=98304
-p thread_pool_min=50 -p thread_pool_max=1000 -s malloc,256m"

Doesn't seem to matter how high I set thread_pool_stack it still makes no difference. Any pointers or ideas much appreciated.

Debug: Child (8159) Started Info: Child (8159) said Child starts Error: Child (8159) died signal=6 (core dumped) Error: Child (8159) Panic at: Thu, 22 Oct 2020 15:51:27 GMT Wrong turn at cache/cache_main.c:284: Signal 11 (Segmentation fault) received at 0x7f5bc4c56ff8 si_code 2 THIS PROBABLY IS A STACK OVERFLOW - check thread_pool_stack parameter version = varnish-6.1.1 revision efc2f6c1536cf2272e471f5cff5f145239b19460, vrt api = 8.0 ident = Linux,5.4.0-26-generic,x86_64,-junix,-smalloc,-sdefault,-hcritbit,epoll now = 691226.238573 (mono), 1603381887.549008 (real) Backtrace: 0x55809884dd2a: varnishd(+0x4cd2a) [0x55809884dd2a] 0x5580988b5cc3: varnishd(VAS_Fail+0x13) [0x5580988b5cc3] 0x558098849736: varnishd(+0x48736) [0x558098849736] 0x7f5bd1505730: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f5bd1505730] 0x7f5bd16ea6d0: /lib/x86_64-linux-gnu/libpcre.so.3(+0x296d0) [0x7f5bd16ea6d0] 0x7f5bd17117d8: /lib/x86_64-linux-gnu/libpcre.so.3(+0x507d8) [0x7f5bd17117d8] 0x7f5bd16e84da: /lib/x86_64-linux-gnu/libpcre.so.3(pcre_exec+0x41a) [0x7f5bd16e84da] 0x5580988bceb0: varnishd(VRE_exec+0x80) [0x5580988bceb0] 0x558098866ac0: varnishd(VRT_regsub+0xd0) [0x558098866ac0] 0x7f5bc4d1cd91: vcl_boot.1603381053.170923471/vgc.so(VGC_function_vcl_recv+0x221) [0x7f5bc4d1cd91] thread = (cache-worker) thr.req = 0x7f5bc00aac20 { vxid = 32772, transport = ESI_INCLUDE step = R_STP_RECV, req_body = R_BODY_NONE, restarts = 0, esi_level = 1, sp = 0x7f5bbfc01220 { fd = 22, vxid = 32769, t_open = 1603381885.250061, t_idle = 1603381885.250061, ws = 0x7f5bbfc01260 { id = "ses", {s, f, r, e} = {0x7f5bbfc01298, +96, (nil), +352}, }, transport = HTTP/1 { state = HTTP1::Proc } client = 172.27.0.9 55530 :80, }, worker = 0x7f5bc4c605c0 { ws = 0x7f5bc4c60668 { id = "wrk", {s, f, r, e} = {0x7f5bc4c5f950, +264, (nil), +2040}, }, VCL::method = inside RECV, VCL::return = 0x0, VCL::methods = {RECV, HASH, MISS, DELIVER}, }, ws = 0x7f5bc00aad70 { id = "req", {s, f, r, e} = {0x7f5bc00acca8, +80, (nil), +57168}, }, http_conn = 0x7f5bc00acc48 { doclose = NULL, ws = (nil) { }, {rxbuf_b, rxbuf_e} = {(nil), (nil)}, {pipeline_b, pipeline_e} = {(nil), (nil)}, content_length = 0, body_status = none, first_byte_timeout = 0.000000, between_bytes_timeout = 0.000000, }, http[req] = 0x7f5bc00aae10 { ws = 0x7f5bc00aad70 { [Already dumped, see above] }, hdrs { "GET", "/page_cache/block/esi/blocks/%5B%22navpro.topnav%22%5D/handles/WyJkZWZhdWx0IiwiY21zX3BhZ2VfdmlldyJd/", "HTTP/1.1", "X-Real-IP: 172.27.0.1", "X-Forwarded-Proto: https", "X-Forwarded-Port: 443", "Ssl-Offloaded: 1", "DBG-SSL: 1", "Connection: close", "user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36", "accept: /", "cookie: __tawkuuid=e%3A%3Aukled.co.uk%3A%3AVsBD2bF%2BVnVdKnq%2Fkt5XC9Nf3eQVlGCHWWPn5xBRC3JdwddvER%2ByIcjbl9XKGQwz%3A%3A2;AMCVS_8F99160E571FC0427F000101@AdobeOrg=1;cookies-policy=1;__stripe_mid=9dbda864-ad86-4bd8-8913-4be3ea9f43b31ecab6;s_cc=true;s_sq=%5B%5BB%5D%5D;persistent_shopping_cart=gJG70kPdvatyWYhmkNXxQ50fy7qFSCN3W6SX8ZYkICUrqzfvLN;private_content_version=c815c46799d2bce89504143f7b959a06;STVID=023172da-283c-3872-1fb7-1d8d94cee485;mage-cache-storage=%7B%7D;mage-cache-storage-section-invalidation=%7B%7D;mage-messages=;recently_viewed_product=%7B%7D;recently_viewed_product_previous=%7B%7D;recently_compared_product=%7B%7D;recently_compared_product_previous=%7B%7D;product_data_storage=%7B%7D;easybanner=%7B%22banner-argento-stripes-newsletter%22%3A%7B%22display_count_per_day%22%3A1%2C%22display_count%22%3A1%2C%22display_count_per_day_time%22%3A1603274902742%2C%22display_count_per_week_time%22%3A1603274902742%2C%22display_count_per_week%22%3A1%2C%22display_count_per_month_time%22%3A1603274902743%2C%22display_count_per_month%22%3A1%7D%7D;AMCV_8F99160E571FC0427F000101@AdobeOrg=1585540135%7CMCIDTS%7C18557%7CMCMID%7C08450058324156366094370300402754205863%7CMCAAMLH-1603959063%7C6%7CMCAAMB-1603959063%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1603361463s%7CNONE%7CvVersion%7C4.4.0;form_key=QWw4EVvfHIs60b7X;mage-cache-sessid=true;__extfc=1;X-Magento-Vary=2b7eda7bf5649669a3b54e524557cfe015428b43;_gat=1;TawkConnectionTime=0;CacheWarmer=CacheWarmer5f6b1fdb02569:eyJQRVJTSVNURU5UIjoxLCJjdXN0b21lcl9ncm91cCI6IjEifQ==prod_id_begin_0_prod_id_endcat_id_begin_0_cat_id_end;mst-cache-warmer-track=1", "Host: dev.ukled.co.uk", "Accept-Encoding: gzip", "X-Forwarded-For: 172.27.0.1, 172.27.0.9", "grace: none", }, }, vcl = { name = "boot", busy = 3, discard = 0, state = auto, temp = warm, conf = { srcname = { "/etc/varnish/default.vcl", "Builtin", }, }, }, vmods = { std = {Varnish 6.1.1 efc2f6c1536cf2272e471f5cff5f145239b19460, 0.0}, }, flags = { }, privs = 0x7f5bc00aadf8 { }, }, thr.busyobj = (nil) { },

Debug: Child cleanup complete

Answer 1

I see that you set the value of thread_pool_stack to a very high number, and apparently it didn't make a difference.

Verifying the settings

The easiest way to figure out if Varnish took over these settings, is by running the following command:

varnishadm param.show thread_pool_stack

Figuring out what's going on

If increasing the thread_pool_stack doesn't stop the panic from happening, we could also try to figure out what is causing it.

The panic refers to the follow library:

/lib/x86_64-linux-gnu/libpcre.so.3

This means you have some sort of faulty regular expression in your VCL code.

Maybe have a look at your VCL code, look for potentially dodgy regexes, and try to fix them.

Enforcing parameters in the official Varnish Docker image

If your Docker image doesn't allow you to extend runtime parameters, you should consider using the official Varnish image for Docker. I'm one of the maintainers, and we've made sure you can just add parameters as an entrypoint.

Here's an example on how you can run this image, and extend the parameters you need:

docker run --name varnish -p "80:80" varnish:stable -p thread_pool_stack=4096k -p http_resp_hdr_len=65536 -p http_resp_size=98304 -p thread_pool_min=50 -p thread_pool_max=1000 -s malloc,256m

I've suggested varnish:stable as the image, which refers to the stable Varnish Cache 6.0 LTS version. If you want to use a fresh release, you can change this into varnish:latest, which will refer to Varnish Cache 6.5.

If you want to see for yourself which varnishd command we run inside this image, please have a look at https://github.com/varnish/docker-varnish/blob/master/docker-varnish-entrypoint