0

I use cache warmer (Mirasvit) with Varnish. Cache warmer doesn't work correctly - either doesn't warm up product pages (config from: https://www.varnish-software.com/developers/tutorials/configuring-varnish-magento/) or nothing at all when standard config is in use. What kind of changes does standard varnish config require to warm up cache properly?
UPDATE:
Above issue was a result of applying vcl file generated by Magento 2.4.4 Upon restoring previous vcl file, all came back to normal. Question is whether vcl file was/is generated with errors or cache warmer isn't compatible anymore...
UPDATE2:
So far, I've identified this code in sub vcl_deliver (Magento 2.4.4 generated):

if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
    set resp.http.Pragma = "no-cache";
    set resp.http.Expires = "-1";
    set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
}

Do you have it generated as well?

varnishlog ouput:

varnishlog -g request -q "ReqUrl eq '/'"
*   << Request  >> 197620    
-   Begin          req 197619 rxreq
-   Timestamp      Start: 1660390146.713244 0.000000 0.000000
-   Timestamp      Req: 1660390146.713244 0.000000 0.000000
-   VCL_use        boot
-   ReqStart       127.0.0.1 49184 a0
-   ReqMethod      GET
-   ReqURL         /
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Host: test.page.com
-   ReqHeader      X-Forwarded-Host: test.page.com
-   ReqHeader      X-Real-IP: X.X.X.X
-   ReqHeader      X-Forwarded-For: X.X.X.X
-   ReqHeader      Ssl-Offloaded: 1
-   ReqHeader      X-Forwarded-Proto: https
-   ReqHeader      X-Forwarded-Port: 443
-   ReqHeader      Connection: close
-   ReqHeader      accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
-   ReqHeader      accept-encoding: gzip, deflate, br
-   ReqHeader      user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6 Safari/605.1.15
-   ReqHeader      accept-language: pl-PL,pl;q=0.9
-   ReqHeader      referer: https://test.page.com/eko-curry-60g-dary-natury
-   ReqHeader      cookie: section_data_ids=%7B%22rewards%22%3A1660389966%2C%22recently_compared_product%22%3A1660389969%2C%22cart%22%3A1660151152%2C%22customer%22%3A1660151152%2C%22captcha%22%3A1660151152%2C%22recently_viewed_product%22%3A1660389776%2C%22directory-data%22
-   ReqUnset       X-Forwarded-For: X.X.X.X
-   ReqHeader      X-Forwarded-For: X.X.X.X, 127.0.0.1
-   VCL_call       RECV
-   ReqHeader      grace: none
-   ReqURL         /
-   ReqUnset       accept-encoding: gzip, deflate, br
-   ReqHeader      Accept-Encoding: gzip
-   VCL_return     hash
-   VCL_call       HASH
-   VCL_return     lookup
-   Hit            202 85836.119597 259200.000000 0.000000
-   VCL_call       HIT
-   VCL_return     deliver
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     OK
-   RespHeader     Server: nginx
-   RespHeader     Date: Sat, 13 Aug 2022 11:19:42 GMT
-   RespHeader     Content-Type: text/html; charset=UTF-8
-   RespHeader     Last-Modified: Fri, 05 Aug 2022 20:47:14 GMT
-   RespHeader     Pragma: cache
-   RespHeader     Cache-Control: max-age=86400, public, s-maxage=86400
-   RespHeader     Expires: Sun, 14 Aug 2022 11:19:42 GMT
-   RespHeader     X-Magento-Tags: store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_13,cms_b_8,cms_b_info_o_sklepie,cms_p_2,easyslide_slider_4,placeholder__1,banner__1,easycatalogimg_subcategori
-   RespHeader     Report-To: {"group":"report-endpoint","max_age":10886400,"endpoints":[{"url":"https:\/\/zakreconysloik.report-uri.com\/r\/d\/csp\/reportOnly\/"}]}
-   RespHeader     Content-Security-Policy-Report-Only: font-src *.fontawesome.com maxcdn.bootstrapcdn.com fonts.gstatic.com https://geowidget.easypack24.net *.cloudflare.com *.twitter.com *.gstatic.com *.googleapis.com *.bootstrapcdn.com 'self' data: test.zakreconysloik.c
-   RespHeader     Strict-Transport-Security: max-age=31536000
-   RespHeader     Content-Security-Policy: upgrade-insecure-requests;
-   RespHeader     X-Content-Type-Options: nosniff
-   RespHeader     X-XSS-Protection: 1; mode=block
-   RespHeader     X-Frame-Options: SAMEORIGIN
-   RespHeader     Vary: Accept-Encoding
-   RespHeader     Content-Encoding: gzip
-   RespHeader     X-Varnish: 197620 202
-   RespHeader     Age: 563
-   RespHeader     Via: 1.1 varnish (Varnish/7.0)
-   RespHeader     Accept-Ranges: bytes
-   VCL_call       DELIVER
-   RespUnset      Age: 563
-   RespUnset      Pragma: cache
-   RespHeader     Pragma: no-cache
-   RespUnset      Expires: Sun, 14 Aug 2022 11:19:42 GMT
-   RespHeader     Expires: -1
-   RespUnset      Cache-Control: max-age=86400, public, s-maxage=86400
-   RespHeader     Cache-Control: no-store, no-cache, must-revalidate, max-age=0
-   RespUnset      X-Magento-Tags: store,cms_b,gdpr_c,theme_editor_backend_css_block,cms_b_header_cms_links,cms_b_argento_scroll_up,cms_b_footer_cms_content,cms_b_13,cms_b_8,cms_b_info_o_sklepie,cms_p_2,easyslide_slider_4,placeholder__1,banner__1,easycatalogimg_subcategori
-   RespUnset      Server: nginx
-   RespUnset      X-Varnish: 197620 202
-   RespUnset      Via: 1.1 varnish (Varnish/7.0)
-   VCL_return     deliver
-   Timestamp      Process: 1660390146.713393 0.000149 0.000149
-   Filters         esi
-   RespHeader     Connection: close
-   RespHeader     Transfer-Encoding: chunked
-   Link           req 197621 esi
-   Timestamp      Resp: 1660390146.714287 0.001043 0.000894
-   ReqAcct        1650 0 1650 6363 185868 192231
-   End            
**  << Request  >> 197621    
--  Begin          req 197620 esi
--  Timestamp      Start: 1660390146.713558 0.000000 0.000000
--  ReqURL         /page_cache/block/esi/blocks/%5B%22navpro.topnav%22%5D/handles/WyJkZWZhdWx0IiwiY21zX2luZGV4X2luZGV4IiwiY21zX3BhZ2VfdmlldyIsImN1c3RvbWVyX2xvZ2dlZF9pbiJd/
--  ReqUnset       Host: test.page.com
--  ReqHeader      Host: test.page.com
--  ReqUnset       accept-encoding: gzip, deflate, br
--  ReqHeader      Accept-Encoding: gzip
--  ReqStart       127.0.0.1 49184 a0
--  ReqMethod      GET
--  ReqURL         /page_cache/block/esi/blocks/%5B%22navpro.topnav%22%5D/handles/WyJkZWZhdWx0IiwiY21zX2luZGV4X2luZGV4IiwiY21zX3BhZ2VfdmlldyIsImN1c3RvbWVyX2xvZ2dlZF9pbiJd/
--  ReqProtocol    HTTP/1.1
--  ReqHeader      X-Forwarded-Host: test.page.com
--  ReqHeader      X-Real-IP: X.X.X.X
--  ReqHeader      X-Forwarded-For: X.X.X.X
--  ReqHeader      Ssl-Offloaded: 1
--  ReqHeader      X-Forwarded-Proto: https
--  ReqHeader      X-Forwarded-Port: 443
--  ReqHeader      Connection: close
--  ReqHeader      accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
--  ReqHeader      user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.6 Safari/605.1.15
--  ReqHeader      accept-language: pl-PL,pl;q=0.9
--  ReqHeader      referer: https://test.page.com/eko-curry-60g-dary-natury
--  ReqHeader      cookie: section_data_ids=%7B%22rewards%22%3A1660389966%2C%22recently_compared_product%22%3A1660389969%2C%22cart%22%3A1660151152%2C%22customer%22%3A1660151152%2C%22captcha%22%3A1660151152%2C%22recently_viewed_product%22%3A1660389776%2C%22directory-data%22
--  ReqHeader      Host: test.page.com
--  ReqHeader      Accept-Encoding: gzip
--  ReqUnset       X-Forwarded-For: X.X.X.X
--  ReqHeader      X-Forwarded-For: X.X.X.X, 127.0.0.1
--  VCL_call       RECV
--  ReqHeader      grace: none
--  ReqURL         /page_cache/block/esi/blocks/%5B%22navpro.topnav%22%5D/handles/WyJkZWZhdWx0IiwiY21zX2luZGV4X2luZGV4IiwiY21zX3BhZ2VfdmlldyIsImN1c3RvbWVyX2xvZ2dlZF9pbiJd/
--  ReqUnset       Accept-Encoding: gzip
--  ReqHeader      Accept-Encoding: gzip
--  VCL_return     hash
--  VCL_call       HASH
--  VCL_return     lookup
--  Hit            204 3036.540065 259200.000000 0.000000
--  VCL_call       HIT
--  VCL_return     deliver
--  RespProtocol   HTTP/1.1
--  RespStatus     200
--  RespReason     OK
--  RespHeader     Server: nginx
--  RespHeader     Date: Sat, 13 Aug 2022 11:19:43 GMT
--  RespHeader     Content-Type: text/html; charset=UTF-8
--  RespHeader     Last-Modified: Fri, 05 Aug 2022 20:47:14 GMT
--  RespHeader     X-Magento-Tags: NAVIGATIONPRO_MENU_6
--  RespHeader     Pragma: cache
--  RespHeader     Cache-Control: max-age=3600, public, s-maxage=3600
--  RespHeader     Expires: Sat, 13 Aug 2022 12:19:43 GMT
--  RespHeader     Report-To: {"group":"report-endpoint","max_age":10886400,"endpoints":[{"url":"https:\/\/zakreconysloik.report-uri.com\/r\/d\/csp\/reportOnly\/"}]}
--  RespHeader     Content-Security-Policy-Report-Only: font-src *.fontawesome.com maxcdn.bootstrapcdn.com fonts.gstatic.com https://geowidget.easypack24.net *.cloudflare.com *.twitter.com *.gstatic.com *.googleapis.com *.bootstrapcdn.com 'self' data: test.zakreconysloik.c
--  RespHeader     Strict-Transport-Security: max-age=31536000
--  RespHeader     Content-Security-Policy: upgrade-insecure-requests;
--  RespHeader     X-Content-Type-Options: nosniff
--  RespHeader     X-XSS-Protection: 1; mode=block
--  RespHeader     X-Frame-Options: SAMEORIGIN
--  RespHeader     Vary: Accept-Encoding
--  RespHeader     Content-Encoding: gzip
--  RespHeader     X-Varnish: 197621 204
--  RespHeader     Age: 563
--  RespHeader     Via: 1.1 varnish (Varnish/7.0)
--  RespHeader     Accept-Ranges: bytes
--  VCL_call       DELIVER
--  RespUnset      Age: 563
--  RespUnset      Pragma: cache
--  RespHeader     Pragma: no-cache
--  RespUnset      Expires: Sat, 13 Aug 2022 12:19:43 GMT
--  RespHeader     Expires: -1
--  RespUnset      Cache-Control: max-age=3600, public, s-maxage=3600
--  RespHeader     Cache-Control: no-store, no-cache, must-revalidate, max-age=0
--  RespUnset      X-Magento-Tags: NAVIGATIONPRO_MENU_6
--  RespUnset      Server: nginx
--  RespUnset      X-Varnish: 197621 204
--  RespUnset      Via: 1.1 varnish (Varnish/7.0)
--  VCL_return     deliver
--  Timestamp      Process: 1660390146.713642 0.000084 0.000084
--  Filters        
--  RespHeader     Content-Length: 4917
--  Timestamp      Resp: 1660390146.713808 0.000250 0.000165
--  ReqAcct        0 0 0 0 4917 4917
--  End
Improve this question
6
  • Have u set up varnish like here? olegnax.com/… forget the magento2 documentation about varnish
    – Samuel Demir
    Aug 4 at 19:32
  • Varnish would normally not require any VCL changes to support a cache warmup tool. Check in your Magento or webserver logs which User-Agent this cache warmer uses and then we can filter the Varnish logs for this agent and figure out what's going on. If you let me know which User-Agent value to look for, I'll help you set up the varnishlog command to figure out if anything is coming in or if Varnish is not correctly caching these requests.
    – Thijs Feryn
    Aug 7 at 7:50
  • @SamuelDemir Yes, It's been set also as is described on that page. The issue did start when I regenerated varnish config in Magento 2.4.4. Can't find any reason for it...
    – Twisted_Jar
    Aug 7 at 20:28
  • @ThijsFeryn I couldn't find any user-agent so I asked developer and that's what I've got: "User-Agent. The extension uses those user agents with which pages were requested for the first time. But those user agents look like normal browsers' user agents so they should not affect the caching process (unless you have some blocking based on user agents and crawlers' user agents match those exceptions. But I believe you don't have such settings on your server)"
    – Twisted_Jar
    Aug 9 at 9:00
  • If there's no way to isolate requests made by the cache warmer, maybe you should run this on a machine that has no outside traffic. This way you're certain that incoming requests originate from the cache warmer. In that case you can run varnishlog -g request -q "ReqUrl eq '/'" to capture cache warming attempts on the homepage. Please set up a test case and try to capture caching warming attempts with the appropriate varnishlog command. Just add the output to your question and I'll look into it.
    – Thijs Feryn
    Aug 9 at 9:43

1 Answer 1

Reset to default
0

if u use https://olegnax.com/speed-up-magento-2-with-varnish-and-nginx-as-ssl-termination-on-ubuntu/

u need to configure magento2 and varnish like here:

sudo php bin/magento config:set system/full_page_cache/caching_application 2
sudo php bin/magento config:set system/full_page_cache/varnish/access_list localhost
sudo php bin/magento config:set system/full_page_cache/varnish/backend_port 8080
sudo php bin/magento setup:config:set --http-cache-hosts 127.0.0.1:6081



sudo mkdir -p /etc/varnish
sudo php bin/magento varnish:vcl:generate --export-version=6 | sudo tee default.vcl
sudo cp /var/www/ <<your_root_folder>> /default.vcl /etc/varnish/default.vcl
sudo sed -i 's/\/pub\/health_check.php/\/health_check.php/' /etc/varnish/default.vcl
sudo systemctl enable varnish
sudo systemctl start varnish

This is my varnish.service file in /etc/systemd/system/varnish.service

[Unit]
Description=Varnish Cache, a high-performance HTTP accelerator
After=network-online.target nss-lookup.target

[Service]
Type=forking
KillMode=process

# Maximum number of open files (for ulimit -n)
LimitNOFILE=131072

# Locked shared memory - should suffice to lock the shared memory log
# (varnishd -l argument)
# Default log size is 80MB vsl + 1M vsm + header -> 82MB
# unit is bytes
LimitMEMLOCK=85983232

# Enable this to avoid "fork failed" on reload.
TasksMax=infinity

# Maximum size of the corefile.
LimitCORE=infinity

ExecStart=varnishd \
      -a :6081 \
      -a localhost:8443,PROXY \
      -p feature=+http2 \
      -f /etc/varnish/default.vcl \
      -s malloc,4G \
      -p http_resp_hdr_len=90536 \
      -p http_resp_size=120536 \
      -p workspace_backend=256k \
      -p workspace_client=256k

# ExecReload=/usr/sbin/varnishreload

[Install]
WantedBy=multi-user.target
Improve this answer
1
  • Thanks. I have checked my current setting (config:show) and all is as you have it shown above. Have you customized (with exception of health check) varnish config?
    – Twisted_Jar
    Aug 12 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.