Varnish with brotli -> missing ESI handling/filtering

Question

I've compiled brotli modules for nginx and added following (from https://www.getpagespeed.com/server-setup/varnish/varnish-brotli-done-right) to varnish default.vcl:

sub vcl_recv {
    if(req.http.Accept-Encoding ~ "br" && req.url !~
            "\.(jpe?g|png|webp|gif|gz|mp3|mov|avi|mpg|mp4|swf|wmf)$") {
        set req.http.X-brotli = "true";
    }
}

# The data on which the hashing will take place
sub vcl_hash {
    if(req.http.X-brotli == "true" && req.http.X-brotli-unhash != "true") {
        hash_data("brotli");
    }
}

sub vcl_backend_fetch {
    if(bereq.http.X-brotli == "true") {
        set bereq.http.Accept-Encoding = "br";
        unset bereq.http.X-brotli;
    }
}

sub vcl_purge {
    # repeat purge for brotli or gzip object 
    # (force hash/no hash on "brotli" while doing another purge)
    # set Accept-Encoding: gzip so that we don't get brotli-encoded response upon purge
    if (req.url !~ "\.(jpg|png|gif|gz|mp3|mov|avi|mpg|mp4|swf|wmf)$" && 
            !req.http.X-brotli-unhash) {
        if (req.http.X-brotli == "true") {
            set req.http.X-brotli-unhash = "true";
            set req.http.Accept-Encoding = "gzip";
        } else {
            set req.http.X-brotli-unhash = "false";
            set req.http.Accept-Encoding = "br";
        }        
        return (restart);
    } 
}

All would be perfect (loading speed increased significantly) but top menu isn't displayed. Looks like ESI handling has been changed. ESI itself is enabled in Varnish and without brotli code there are no problems.

[Service]
ExecStart=
ExecStart=/usr/sbin/varnishd \
          -a localhost:6081 \
          -a localhost:8443,PROXY \
          -S /etc/varnish/secret \
          -f /etc/varnish/default.vcl \
          -s Cache=malloc,2G \
          -s Transient=malloc,512m \
          -p http_req_size=256k \
          -p http_resp_hdr_len=128k \
          -p http_resp_size=512k \
          -p workspace_backend=512k \
          -p workspace_client=256k \
          -p feature=+esi_ignore_other_elements \
          -p feature=+esi_disable_xml_check \
          -p feature=+esi_ignore_https \
          -p thread_pool_min=200 \
          -p thread_pool_max=2000 \
          -p thread_pool_add_delay=2

With brotli code, ESI is gone from varnish filter (and esi block is just sent over http and visible in browser dev tools)...

-   VCL_return     deliver
-   Timestamp      Process: 1667168951.732238 0.000115 0.000115
-   Filters        
-   RespHeader     Content-Length: 141996
-   RespHeader     Connection: close
-   Timestamp      Resp: 1667168951.732649 0.000526 0.000411
-   ReqAcct        2492 0 2492 5906 141996 147902
-   End            

Without brotli code:

--  VCL_return     deliver
--  Timestamp      Process: 1667090732.937994 1.547312 0.000076
--  Filters         esi
--  Storage        malloc Transient
--  Fetch_Body     2 chunked -
--  BackendClose   47 default recycle
--  Timestamp      BerespBody: 1667090733.003310 1.612627 0.065315
--  Length         124527
--  BereqAcct      1811 0 1811 6828 124527 131355
--  End            

Nginx proxy settings for SSL termination:

        proxy_pass http://127.0.0.1:6081;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Ssl-Offloaded "1";
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Forwarded-Port 443;
        proxy_http_version 1.1;
        proxy_buffer_size                   64k;
        proxy_buffers                       32 16k;
        proxy_busy_buffers_size             64k;

and Varnish backend:

server {
   server_name localhost example.com;
   listen 127.0.0.1:8080;
   set $MAGE_ROOT <magento_dir>;
   access_log off;
   error_log /var/log/nginx/error_mag_var.log warn;
   set_real_ip_from 127.0.0.1;
   real_ip_header X-Forwarded-For;
   real_ip_recursive on;
  include <magento_dir>/nginx.mag.conf;
}

I'm not very familiar with Varnish. Is it possible to restore ESI handling with brotli code in default.vcl?

Answer 1

As you mentioned yourself: Varnish Cache, the open source version of Varnish doesn't support Brotli natively.

Without this native support, Varnish cannot decode a Brotli response and look for ESI tags.

Varnish Enterprise, the commercial version of Varnish has a Brotli module that offers full support for Brotli-encoded responses. See https://docs.varnish-software.com/varnish-cache-plus/vmods/brotli/ for documentation.

To use this module, you either have to buy a Varnish Enterprise license with Varnish Software, or you can pay by the hour through the official Cloud market places images.

Answer 2

Solution or workaround is to exclude main document and let it be gzipped... All the rest (mainly js/css), what doesn't need to be processed by varnish, will remain compressed by brotli. So, add to vcl_backend_fetch "if (bereq.url !~ ".")":

sub vcl_backend_fetch {
    if(bereq.http.X-brotli == "true") {
        set bereq.http.Accept-Encoding = "br";
        unset bereq.http.X-brotli;
    }
    # unset accept-encoding for likely HTML request (no dot in URL),
    # so that the backend does not compress the response 
    if (bereq.url !~ "\.") {
        unset bereq.http.accept-encoding;
    }
}

Answer given with help of above config's author.