0

I'm running an Nginx sandwiched Varnish backed magento 2.3.6 shop in development mode, set up like:

Frontend nginx port 443, varnish port 80, backend nginx port 8080, plus redis for session cache. Nothing strange, fairly common setup.

Been developing my shop without varnish, but now after enabling varnish it seems that when I start my site with a clean magento cache, clean redis, clean varnish, and visit my site, add a product to the basket, after ~2 seconds I can see the cart being emptied.

After reading the entire internet, I'm at a loss. Disabling Varnish fixes this issue. Disabling Redis does not.

I've discovered that in my session storage ALL _session_validator_data have a remote_addr of 127.0.0.1, which seems wrong. Also the x-forwarded-for value is empty in all sessions.

After the first occurence, it does not happen again. When I enter production mode, it also doesn't happen.

Anyone has some pointers on the fix? Or some pointers on the troubleshooting? I've been looking for some 'advised' setup with Nginx SSL offloading in combination with Varnish, but the ones I can easily find on the interwebs seem to do the same as I'm doing....

thx in advance.

0

Hitch

In the Varnish ecosystem we have our own TLS terminator: Hitch

It's lightweight, it doesn't even speak HTTP, and just shuffles through the bytes. The main reason why Hitch is so useful, is because it supports PROXY protocol, just like Varnish.

With PROXY protocol enabled, both on Hitch & Varnish, the X-Forwarded-For header is automatically set to the client IP of the original client, regardless of the number of hops in between.

There are even official packages available on https://packagecloud.io/varnishcache/hitch

Enabling PROXY protocol on Varnish

The typical runtime configuration for Varnish usually only supports plain HTTP. Here's an example:

varnishd -a :80 -f /etc/varnish/default.vcl -s malloc,2G

If you want to enabling PROXY protocol, just add an extra listening port. Here's how you do that:

varnishd -a :80 -a :8443,PROXY -f /etc/varnish/default.vcl -s malloc,2G

It's important that Hitch uses port 8443 as a backend, instead of 80

TLS detection

Magento also needs to know what protocol was used in the original client request. The TLV properties of the PROXY protocol information contain these details. It's just a matter of extracting them and turning them into the right X-Forwarded-Proto value.

The proxy VMOD is a module in Varnish 6 that can grab the TLV attributes.

vcl 4.1;

import proxy;

sub vcl_recv {
    if (proxy.is_ssl()) {
        set req.http.X-Forwarded-Proto = "https";
    } else {
        set req.http.X-Forwarded-Proto = "http";
    }
}

The X-Forwarded-For header is automatically set

What if the problem is unrelated?

I'm pitching Hitch because it's simple, lightweight, it's an official TLS terminator, and it supports PROXY. I'm not saying Nginx is a bad choice, because a lot of people use it.

If it turns out that TLS termination and the X-Forwarded-For & X-Forwarded-Proto headers are not related to the problem, your best choice is to use varnishlog.

Just run the following command if you want to monitor the homepage:

varnishlog -g request -q "ReqUrl eq '/'"

If you want to monitor a different page, just modify the filter.

Then try to trigger the error, and add the varnishlog output here for me to anlyze. Maybe the logging will provide a lot more information about what's going on.

A couple of years ago I wrote an in-depth blog post about varnishlog: see https://feryn.eu/blog/varnishlog-measure-varnish-cache-performance/ for more information.

11
  • Thanks Thijs, very much appreciated, I will certainly look into the Hitch option. Can I deduce that you think this problem is in fact related to the remote_addr being 127.0.0.1? I will try and work on the varnishlog output while reproducing the event.
    – RGS
    Mar 1 at 13:01
  • @RGS Based on your information, I don't really know whether or not the 127.0.0.1 address is related. My Hitch suggestion is an option to no longer make this IP address a factor. However, varnishlog should be your priority right now, because we want to know why the error is occuring, so we can fix it later. Mar 1 at 14:43
  • Hi Thijs, here is a paste of the varnishlog from the command: varnishlog -g request -q 'ReqUrl !~ "/(static|media).*"' ybin.me/p/… I've altered the website name and anonymised my ip. The varnishlog was started right after a freshly generated magento deploy, cleaned cache, browser cookie cleaned, everything blank :-) So this is the reason for all the MISS returns.
    – RGS
    Mar 2 at 13:50
  • @RGS would it be possible to call /nl/pb/basis-modellen.html multiple times and report the logs of varnishlog -g request -q "ReqUrel eq '/nl/pb/basis-modellen.html'"? The first request is always a cache miss, so that doesn't tell use much. But if the same URL is called twice in quick succession, we can examine why Varnish isn't caching the page. Mar 2 at 16:21
  • Good morning, here is the log paste: ybin.me/p/…
    – RGS
    Mar 3 at 7:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.