Magento2: How to serve cookie-less cached version for crawlers on Varnish cache

Question

As far as i know; magento assign a X-Magento-Vary cookie to each visitor first you visit website. So your first request always be non-cached version. After you got a X-Magento-Vary you see cached version served by Varnish.

Is there anyone knows about how to define static X-Magento-Vary for only crawlers, in this way crawlers always get cached version on first attempt ?

Answer 1

Based on some VCL snippets I found on GitHub, this is probably the code that is responsible for creating the cache variation you want to get rid off:

sub vcl_hash {
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # For multi site configurations to not cache each other's content
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    if (req.url ~ "/graphql") {
        call process_graphql_headers;
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }

}

The trick is to modify the following if-statement, and add a clause for when the client was identified as a bot:

if (req.http.cookie ~ "X-Magento-Vary=") {
    hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
}

This would result in the following if-statement:

if (req.http.cookie ~ "X-Magento-Vary=" && !req.http.is-bot) {
    hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
}

In vcl_recv we can then write logic to identify the client as a bot. The easiest way is by adding a custom subroutine that contains this logic. That way we don't pollute the rest of the code.

sub check_bot {
    if(req.http.User-Agent ~ "(bot|googlebot|crawler|spider|robot|crawling)") {
        set req.http.is-bot = "true";
    }
    //Some other logic
}

The check_bot subroutine will be triggered inside vcl_recv:

sub vcl_recv {
    call check_bot;

    //The rest of the vcl_recv logic
}

Answer 2

As far as i know; magento assign a X-Magento-Vary cookie to each visitor first you visit website. So your first request always be non-cached version. After you got a X-Magento-Vary you see cached version served by Varnish.

No, that is not correct. This cookie is set only when changing HTTP context, e.g. upon user login, adding to cart, etc. For a regular visit to a page, this cookie is not set. Given a specific HTTP context (e.g. a user is not logged in and have not added to a cart), a page is uncached only for the very first visitor. Any subsequent users who fall into the same "HTTP context", will get the cached page.

Is there anyone knows about how to define static X-Magento-Vary for only crawlers, in this way crawlers always get cached version on first attempt ?

For the reason above, you don't need to do anything. Crawlers are presented with a cached page, provided it was visited before by a user who was not logged in and hasn't added to a cart, etc.