We have a Magento 2.3.1 store that uses CloudFlare and has Varnish enabled. If we use wget or curl to load a page, that page is cached in Varnish if we try wget or curl again on that same page.

However, if we test that URL via a browser, the page is not cached. It is cached after the first browser visit, and then remains cached on reloads from a browser.

We added the CF+Varnish rules as listed at: https://support.cloudflare.com/hc/en-us/articles/200169376-Can-I-use-Cloudflare-and-Varnish-together-

but that did not seem to make a difference. We're using the recommended Magento default.vcl file that is generated.

Any ideas what could be stopping a wget/curl from loading a page into the Varnish cache such that any request from a browser would be served that cached page instead of causing Varnish to cache the page again?

Wget request:

/usr/bin/wget --no-cache --wait=0.3 --no-check-certificate --delete-after --header="user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" --header="Cache-control: max-age=0" --header="Accept-Language: en-US,en;q=0.5" --header="Accept-encoding: gzip, deflate" -S 'https://XXXX'
--2019-07-20 09:14:14--  https://XXXX
Resolving XXXX...
Connecting to XXXX||:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: nginx
  Date: Sat, 20 Jul 2019 13:14:16 GMT
  Content-Type: text/html; charset=UTF-8
  Content-Length: 51021
  Connection: keep-alive
  X-Content-Type-Options: nosniff
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Vary: Accept-Encoding
  X-Magento-Cache-Control: max-age=86400000, public, s-maxage=86400000
  Content-Encoding: gzip
  Age: 0
  X-Magento-Cache-Debug: MISS
  Pragma: no-cache
  Expires: -1
  Cache-Control: no-store, no-cache, must-revalidate, max-age=0
  Accept-Ranges: bytes
  Access-Control-Allow-Origin: *
Length: 51021 (50K) [text/html]

Browser response/header after wget runs:

accept-ranges: bytes
access-control-allow-origin: *
age: 0
cache-control: no-store, no-cache, must-revalidate, max-age=0
content-encoding: gzip
content-length: 51023
content-type: text/html; charset=UTF-8
date: Sat, 20 Jul 2019 13:14:29 GMT
expires: -1
pragma: no-cache
server: nginx
status: 200
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-magento-cache-control: max-age=86400000, public, s-maxage=86400000
x-magento-cache-debug: MISS
x-xss-protection: 1; mode=block
:authority: XXXX
:method: GET
:path: /YYYY
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cookie: mage-translation-storage=%7B%7D; mage-translation-file-version=%7B%7D; newsletter_popup=dontshowitagain; ku1-vid=e5cf0293-7c71-74d9-10df-ffd19dcd11bb; ku1-sid=UvdxKNYS6MKRhvKZFJN9a; _fbp=fb.1.1563628017878.1828386074; recordID=c7f615a9-d0c2-41d5-8ccc-7a11beee21b1; dmSessionID=c2397d68-85b5-4f04-ad24-0c5e92e69a57; __tawkuuid=e::XXXX::dr5R22jZlQVd2e0p7XTrs0xIAaO+8YriwgvRE8yBIalE8AkHbZFPjEheVHdRnQCI::2; form_key=WRPXsJkeLSDLVwoN; _gcl_au=1.1.997920531.1563628021; mage-cache-storage=%7B%7D; mage-cache-storage-section-invalidation=%7B%7D; mage-banners-cache-storage=%7B%7D; mage-messages=; _ga=GA1.3.129667006.1563628021; _gid=GA1.3.1515936369.1563628021; X-Magento-Vary=8bed0844a67603e244f2ca107da80466ff383efd; recently_viewed_product=%7B%7D; recently_viewed_product_previous=%7B%7D; recently_compared_product=%7B%7D; recently_compared_product_previous=%7B%7D; product_data_storage=%7B%7D; PHPSESSID=2197268281d0aedc4015f7be1f5974d7; mage-cache-sessid=true; section_data_ids=%7B%22cart%22%3A1563628090%7D; TawkConnectionTime=0
referer: https://XXXX/ZZZZ
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
  • Can you include headers as shown in browser? Use chrome dev tools for example. – Dominic Xigen Jul 20 '19 at 7:03
  • I have edited the question to include the headers for both wget and the browser test afterwards while the site was in developer mode to show the Varnish HIT/MISS line. Could it be the cookies that are causing the issue? – Rob Mangiafico Jul 20 '19 at 13:21
  • I think I understand now. So you're trying to warm the cache so that the browser fetches from varnish on first browser load (technically second load)? But at the moment nothing is hitting varnish? stackoverflow.com/a/52845031/10121426 – Dominic Xigen Jul 20 '19 at 14:09
  • The wget/curl request hits Varnish, and any subsequent wget/curl requests are served via cache. However the first browser hit as you can see above is also a "MISS" after wget hit of same page, then subsequent hits from browser are served by cache. Trying to determine why this is, and why first wget/curl hit of page does not cache the page for all users/browsers. I am using the Magento default.vcl file. – Rob Mangiafico Jul 20 '19 at 14:26
  • If I use the cookie removal code listed at varnish-software.com/wiki/content/tutorials/magento2/… that seems to make Varnish cache each page when hit by wget/curl. Is this safe to do for an M2 store? Anything else I should consider? – Rob Mangiafico Jul 20 '19 at 15:40

If I put this in the default.vcl file (along with the M2 rules) Varnish caches each page the first time it is hit, regardless if from wget/curl or from a web browser:

sub vcl_recv {
  if (req.http.cookie) {
    set req.http.cookie = ";" + req.http.cookie;
    set req.http.cookie = regsuball(req.http.cookie, "; +", ";");
    set req.http.cookie = regsuball(req.http.cookie, ";(COOKIE1|COOKIE2|COOKIE3)=", "; \1=");
    set req.http.cookie = regsuball(req.http.cookie, ";[^ ][^;]*", "");
    set req.http.cookie = regsuball(req.http.cookie, "^[; ]+|[; ]+$", "");

    if (req.http.cookie == "") {
      remove req.http.cookie;

This comes from the guide at https://www.varnish-software.com/wiki/content/tutorials/magento2/m2_step_by_step.html

I don't see any issues with doing this, but I find it odd that this is not mentioned in any docs when there are multiple cookies in play on pages...

  • Removing all cookies like this breaks the cart functionality... So it seems some cookies must be removed to allow caching of pages by wget/curl on the first load, but not all cookies or else Magento breaks... I'm at a loss as to what is the right vcl adjustment to get all of this to work as I'd expect... – Rob Mangiafico Jul 20 '19 at 18:04
  • there is something wrong with your request, not varnish. do not change default configs, you will get deeper in to the dark. – MagenX Jul 20 '19 at 18:28
  • That's what I'm trying to figure out is why a request for a page will not trigger a caching event from wget/curl the same way as from the browser. My guess is it is a different cookie that is blocking the cache hit, but I do not know how to adjust the config to account for it. – Rob Mangiafico Jul 20 '19 at 19:27
  • look at your headers, they are different for wget and browser. there is something in your url you request – MagenX Jul 20 '19 at 19:54
  • I've found that the "X-Magento-Vary" cookie gets set on this multistore site, which causes Varnish to only cache these unique items per browser (the hash). I haven't figured out why these sites set this cookie for a default page view... – Rob Mangiafico Jul 23 '19 at 3:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.