2

We have a Magento 2.3.1 store that uses CloudFlare and has Varnish enabled. If we use wget or curl to load a page, that page is cached in Varnish if we try wget or curl again on that same page.

However, if we test that URL via a browser, the page is not cached. It is cached after the first browser visit, and then remains cached on reloads from a browser.

We added the CF+Varnish rules as listed at: https://support.cloudflare.com/hc/en-us/articles/200169376-Can-I-use-Cloudflare-and-Varnish-together-

but that did not seem to make a difference. We're using the recommended Magento default.vcl file that is generated.

Any ideas what could be stopping a wget/curl from loading a page into the Varnish cache such that any request from a browser would be served that cached page instead of causing Varnish to cache the page again?

Wget request:

/usr/bin/wget --no-cache --wait=0.3 --no-check-certificate --delete-after --header="user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" --header="Cache-control: max-age=0" --header="Accept-Language: en-US,en;q=0.5" --header="Accept-encoding: gzip, deflate" -S 'https://XXXX'
--2019-07-20 09:14:14--  https://XXXX
Resolving XXXX... 1.2.3.4
Connecting to XXXX|1.2.3.4|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: nginx
  Date: Sat, 20 Jul 2019 13:14:16 GMT
  Content-Type: text/html; charset=UTF-8
  Content-Length: 51021
  Connection: keep-alive
  X-Content-Type-Options: nosniff
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
  Vary: Accept-Encoding
  X-Magento-Cache-Control: max-age=86400000, public, s-maxage=86400000
  Content-Encoding: gzip
  Age: 0
  X-Magento-Cache-Debug: MISS
  Pragma: no-cache
  Expires: -1
  Cache-Control: no-store, no-cache, must-revalidate, max-age=0
  Accept-Ranges: bytes
  Access-Control-Allow-Origin: *
Length: 51021 (50K) [text/html]

Browser response/header after wget runs:

accept-ranges: bytes
access-control-allow-origin: *
age: 0
cache-control: no-store, no-cache, must-revalidate, max-age=0
content-encoding: gzip
content-length: 51023
content-type: text/html; charset=UTF-8
date: Sat, 20 Jul 2019 13:14:29 GMT
expires: -1
pragma: no-cache
server: nginx
status: 200
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-magento-cache-control: max-age=86400000, public, s-maxage=86400000
x-magento-cache-debug: MISS
x-xss-protection: 1; mode=block
----
----
:authority: XXXX
:method: GET
:path: /YYYY
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
cookie: mage-translation-storage=%7B%7D; mage-translation-file-version=%7B%7D; newsletter_popup=dontshowitagain; ku1-vid=e5cf0293-7c71-74d9-10df-ffd19dcd11bb; ku1-sid=UvdxKNYS6MKRhvKZFJN9a; _fbp=fb.1.1563628017878.1828386074; recordID=c7f615a9-d0c2-41d5-8ccc-7a11beee21b1; dmSessionID=c2397d68-85b5-4f04-ad24-0c5e92e69a57; __tawkuuid=e::XXXX::dr5R22jZlQVd2e0p7XTrs0xIAaO+8YriwgvRE8yBIalE8AkHbZFPjEheVHdRnQCI::2; form_key=WRPXsJkeLSDLVwoN; _gcl_au=1.1.997920531.1563628021; mage-cache-storage=%7B%7D; mage-cache-storage-section-invalidation=%7B%7D; mage-banners-cache-storage=%7B%7D; mage-messages=; _ga=GA1.3.129667006.1563628021; _gid=GA1.3.1515936369.1563628021; X-Magento-Vary=8bed0844a67603e244f2ca107da80466ff383efd; recently_viewed_product=%7B%7D; recently_viewed_product_previous=%7B%7D; recently_compared_product=%7B%7D; recently_compared_product_previous=%7B%7D; product_data_storage=%7B%7D; PHPSESSID=2197268281d0aedc4015f7be1f5974d7; mage-cache-sessid=true; section_data_ids=%7B%22cart%22%3A1563628090%7D; TawkConnectionTime=0
referer: https://XXXX/ZZZZ
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36
----
8
  • Can you include headers as shown in browser? Use chrome dev tools for example. Jul 20, 2019 at 7:03
  • I have edited the question to include the headers for both wget and the browser test afterwards while the site was in developer mode to show the Varnish HIT/MISS line. Could it be the cookies that are causing the issue? Jul 20, 2019 at 13:21
  • I think I understand now. So you're trying to warm the cache so that the browser fetches from varnish on first browser load (technically second load)? But at the moment nothing is hitting varnish? stackoverflow.com/a/52845031/10121426 Jul 20, 2019 at 14:09
  • The wget/curl request hits Varnish, and any subsequent wget/curl requests are served via cache. However the first browser hit as you can see above is also a "MISS" after wget hit of same page, then subsequent hits from browser are served by cache. Trying to determine why this is, and why first wget/curl hit of page does not cache the page for all users/browsers. I am using the Magento default.vcl file. Jul 20, 2019 at 14:26
  • If I use the cookie removal code listed at varnish-software.com/wiki/content/tutorials/magento2/… that seems to make Varnish cache each page when hit by wget/curl. Is this safe to do for an M2 store? Anything else I should consider? Jul 20, 2019 at 15:40

1 Answer 1

0

If I put this in the default.vcl file (along with the M2 rules) Varnish caches each page the first time it is hit, regardless if from wget/curl or from a web browser:

sub vcl_recv {
  if (req.http.cookie) {
    set req.http.cookie = ";" + req.http.cookie;
    set req.http.cookie = regsuball(req.http.cookie, "; +", ";");
    set req.http.cookie = regsuball(req.http.cookie, ";(COOKIE1|COOKIE2|COOKIE3)=", "; \1=");
    set req.http.cookie = regsuball(req.http.cookie, ";[^ ][^;]*", "");
    set req.http.cookie = regsuball(req.http.cookie, "^[; ]+|[; ]+$", "");

    if (req.http.cookie == "") {
      remove req.http.cookie;
    }
  }

This comes from the guide at https://www.varnish-software.com/wiki/content/tutorials/magento2/m2_step_by_step.html

I don't see any issues with doing this, but I find it odd that this is not mentioned in any docs when there are multiple cookies in play on pages...

6
  • Removing all cookies like this breaks the cart functionality... So it seems some cookies must be removed to allow caching of pages by wget/curl on the first load, but not all cookies or else Magento breaks... I'm at a loss as to what is the right vcl adjustment to get all of this to work as I'd expect... Jul 20, 2019 at 18:04
  • there is something wrong with your request, not varnish. do not change default configs, you will get deeper in to the dark.
    – MagenX
    Jul 20, 2019 at 18:28
  • That's what I'm trying to figure out is why a request for a page will not trigger a caching event from wget/curl the same way as from the browser. My guess is it is a different cookie that is blocking the cache hit, but I do not know how to adjust the config to account for it. Jul 20, 2019 at 19:27
  • look at your headers, they are different for wget and browser. there is something in your url you request
    – MagenX
    Jul 20, 2019 at 19:54
  • I've found that the "X-Magento-Vary" cookie gets set on this multistore site, which causes Varnish to only cache these unique items per browser (the hash). I haven't figured out why these sites set this cookie for a default page view... Jul 23, 2019 at 3:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.