How to cache Magento 2 with Varnish

Question

Having setup Magento 2, after looking at response headers and the page itself i think that any page is not cacheable without a lot of work. I am not a Magento developer just a guy that was requested to look into Varnish and Magento 2.

On the teams live projects i see the responce cookie of PHPSESSID as well as Magento form key on every single page. On client requests there is always the PHPSESSID as well. Each Page has the anonymous users cart. On my clean Magento 2 i can also see that Pragma and Cache-Control do not allow caching of any page.

I have downloaded and read the whole Varnish Book, as well as doing all the exercises. So i do understand how Varnish basics work, as well as caching in general.

Even on a clean Magento2 with the default VCL for Varnish 4, every page is getting a MISS or Hit-for-Pass.

All Magento caches are enabled and full_page is configured to use Varnish. Sessions are stored on redis, webserver is Nginx and php-fpm7 for php code execution.

It's just feels strange because Magento and all people that post Tutorials and How To's claim that this should work fine just by setting it up(like fpc used to work).

I am looking into running Magenx's amazing setup script in case they have already made the needed changes somewhere but i doubt this will be the case.

Using Curl on the teams project(without varnish) on the landing page this is the response:

 HTTP/1.1 200 OK
Server: nginx
Date: Fri, 02 Mar 2018 16:33:26 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Trace: 2BCC3FA5AEE05BC675A023867155A0D3180B410BAF9A6B7D4ABF57BB2101
Set-Cookie: store=default; expires=Sat, 02-Mar-2019 16:33:20 GMT; Max-Age=31536000; path=/; HttpOnly
Set-Cookie: PHPSESSID=xxxxxxxxxx; expires=Fri, 02-Mar-2018 19:20:00 GMT; Max-Age=10000; path=/; domain=xxxxxxxxx; HttpOnly
Pragma: no-cache
Cache-Control: max-age=0, must-revalidate, no-cache, no-store
Expires: Thu, 02 Mar 2017 16:33:20 GMT
X-Magento-Debug: 1
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

Curl on the clean project(Note i made it work with some tweeking on the vcl that like /pub/check_health -> /check_health)

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 02 Mar 2018 16:24:39 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
X-Trace: 2BF11055A47A22D473F72419BD995CCC259228308C7C5011137B1003AC01
Set-Cookie: store=default; expires=Sat, 02-Mar-2019 16:24:39 GMT; Max-Age=31536000; path=/; HttpOnly
Set-Cookie: PHPSESSID=xxxxxx; expires=Fri, 02-Mar-2018 17:24:39 GMT; Max-Age=3600; path=/; domain=xxxxxxxx; HttpOnly
Pragma: cache
Cache-Control: max-age=86400, public, s-maxage=86400
Expires: Sat, 03 Mar 2018 16:24:39 GMT
X-Magento-Tags: store,cms_b,cms_p_2,cat_p
X-Magento-Debug: 1
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

So the clean Magento was my messup. But how can we handle the First one?The clean one has No Cache-control headers while the first one throws the VCL into this block:

   if (beresp.ttl <= 0s ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
       beresp.http.Cache-Control ~ "no-cache|no-store") ||
       beresp.http.Vary == "*") {
       # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
        set beresp.http.X-CantCache = true;
    }

Answer 1

It just feels strange because of Magento and all people that post Tutorials and How To's claim that this should work fine just by setting it up(like FPC used to work).

I will join them :).

Magento 2 and Varnish are compatible out of the box. If something is not working, there was a step missed:

On the teams live projects I see the response cookie of PHPSESSID as well as Magento form key on every single page

If you see it on every page, this likely means that the Magento 2 VCL file was not applied. For reference, you can look at this line.

To recap: Magento 2 does spit out PHPSESSID on every page but Magento 2's VCL unsets it and overrides the built-in VCL in a way that even with cookies, the page can be cached.

Troubleshoot

In proper configuration the Expires and Cache-Control header which is emitted by the Magento 2 app will ensure the TTL to be positive, e.g. 1-day expiration. That is the TTL which Varnish will use to cache the page.

You can verify the headers coming from your backend app via curl. Simply launch the curl against Magento 2 directly. This way you can see headers which Varnish sees.

Sample headers from Magento 2 (before Varnish):

HTTP/1.1 200 OK
Server: nginx
Date: Fri, 02 Mar 2018 14:40:55 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: store=default; expires=Sat, 02-Mar-2019 14:40:55 GMT; Max-Age=31536000; path=/; HttpOnly
Set-Cookie: PHPSESSID=hbtgntuutdaa460hjtfdobbjt3; expires=Fri, 02-Mar-2018 15:40:55 GMT; Max-Age=3600; path=/; domain=www.example.com; secure; HttpOnly
Pragma: cache
Cache-Control: max-age=86400, public, s-maxage=86400
Expires: Sat, 03 Mar 2018 14:40:55 GMT
X-Magento-Tags: store,cms_b,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_header_cms_content,cms_b_footer_cms_content,cms_p_58,cat_p
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN

There is Set-Cookie which is fine because M2 VCL will take care of it. The Expiress and Cache-Control specify 1 day of TTL for Varnish.

On my clean Magento 2 I can also see that Pragma and Cache-Control do not allow caching of any page.

This is to be expected. Those headers are meant for browsers and do not affect caching by Varnish. They are set by Magento 2 VCL.