Apache proxy + Varnish + Magento (HTTPS not caching)

Question

I succefully passed all my magento store in full HTTPS (on all pages).

My server setup is as follow:

Apache proxy as a SSL offloader (443) -> Varnish (80) -> Apache backend (8080).

Using Magento CE 1.5.1 + Phonix-media varnish extension.

When I access my pages on http (8080), all html pages and static files are cached by Varnish. When I access my pages on https (443), all html pages and statics files are not cached by Varnish.

I suspect a wrong config file for varnish (.vcl)

Here is my Apache proxy SSL offloader.conf:

    <VirtualHost *:443>
  ServerName www.domain.com
  DocumentRoot /var/www/html/prod
  DirectoryIndex index.php index.html

  <Directory "/var/www/html/prod/">
    AllowOverride All
    Options -Indexes +FollowSymlinks
    Require all granted
  </Directory>

  # SSL config
  SSLEngine on
  ProxyPreserveHost On
  ProxyPass "/" http://127.0.0.1:80/ retry=1 acquire=3000 timeout=600 Keepalive=On
  ProxyPassReverse "/" http://127.0.0.1:80/

  # We set some request headers not sent by ProxyPass
  # NB: X-Forwarded-For, X-Forwarded-Host and X-Forwarded-Server are automatically sent with ProxyPass.
  RequestHeader set X-Forwarded-Port "443"
  RequestHeader set X-Forwarded-Proto "https"

  # SSL certificate 
  SSLCertificateFile /etc/apache2/certificates/www.domain.crt
  SSLCertificateKeyFile /etc/apache2/certificates/www.domain.key
  SSLCertificateChainFile /etc/apache2/certificates/chain_EV_intermediate_geotrust_2017_2019.crt
</VirtualHost>

# intermediate configuration, tweak to your needs
SSLProtocol             all -SSLv3
SSLCipherSuite          ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
SSLHonorCipherOrder     on
SSLCompression          off

And here is my varnish .vcl file:

# List of upstream proxies we trust to set X-Forwarded-For correctly.
acl upstream_proxy {
  "127.0.0.1";
}

# default backend definition.  Set this to point to your content server.
backend default {
  .host = "127.0.0.1";
  .port = "8080";
}

# admin backend with longer timeout values. Set this to the same IP & port as your default server.
backend admin {
  .host = "127.0.0.1";
  .port = "8080";
  .first_byte_timeout = 18000s;
  .between_bytes_timeout = 18000s;
}

# add your Magento server IP to allow purges from the backend
acl purge {
  "localhost";
  "127.0.0.1";
}

# Call the mobile detect vcl file to detect mobile and tablet ans set X-UA-DEVICE=MOBILE
include "mobile_detect.vcl";

sub vcl_recv {

    # redirect http to https
    if (client.ip !~ upstream_proxy && req.http.host ~ "(preprod)" ) {
        set req.http.x-Redir-Url = "https://" + req.http.host + req.url;
        error 750 req.http.x-Redir-Url;
    }

    # Add server['HTTPS']=on for preprod coming from proxy. So magento 1.5 is aware that connection is https
    if (req.http.host ~ "(domain)") {
        set req.http.Https = "on";
    }

    # Call devicetest function from mobile_detect.vcl
    call devicedetect;

   # Set the X-Forwarded-For header so the backend can see the original
    # IP address. If one is already set by an upstream proxy, we'll just re-use that.
    if (client.ip ~ upstream_proxy && req.http.X-Forwarded-For) {
        set req.http.X-Forwarded-For = req.http.X-Forwarded-For;
    } else {
        set req.http.X-Forwarded-For = regsub(client.ip, ":.*", "");
    }

    # Set another real ip header so that mod_remoteip deletes this one instead of X-Forwarded-for
    set req.http.X-RealFromVarnish-IP = req.http.X-Forwarded-For;

    if (req.url ~ ".*pdf.*|.*csv.*|.*xml.*|.*print*|.*export.*") { set req.http.connection = "close"; return (pipe); }
    if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE" &&
      req.request != "PURGE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }

    # purge request
    if (req.request == "PURGE") {
        if (!client.ip ~ purge) {
            error 405 "Not allowed.";
        }
        ban("obj.http.X-Purge-Host ~ " + req.http.X-Purge-Host + " && obj.http.X-Purge-URL ~ " + req.http.X-Purge-Regex + " && obj.http.Content-Type ~ " + req.http.X-Purge-Content-Type);
        error 200 "Purged.";
    }

    # switch to admin backend configuration
    if (req.http.cookie ~ "adminhtml=") {
        set req.backend = admin;
    }

    # we only deal with GET and HEAD by default
    if (req.request != "GET" && req.request != "HEAD") {
        return (pass);
    }

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://[^/]+", "");

    # static files are always cacheable. remove SSL flag and cookie
    if (req.url ~ "^[^?]*\.(css|jpg|js|gif|png)(\?.*)?$") {
        unset req.http.Https;
        unset req.http.Cookie;
    }

    # not cacheable by default
    if (req.http.Authorization) {
        return (pass);
    }

    # do not cache any page from
    # - index files
    # - ...
    if (req.url ~ "^/(index)") {
        return (pass);
    }

    if (req.http.host ~ "(preprod)") {
        return (pass);
    }

    # as soon as we have a NO_CACHE cookie pass request
    if (req.http.cookie ~ "NO_CACHE=") {
        return (pass);
    }

    # normalize Aceept-Encoding header
    # http://varnish.projects.linpro.no/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            remove req.http.Accept-Encoding;
        }
    }

    # remove Google gclid parameters
    #set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
    #set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
    #set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"

    # remove Google glid and utm parameters and also LENGOWCODE - new method
    if (req.url ~ "(\?|&)(LGWCODE|gclid|utm_[a-z]+)=") {
       set req.url = regsuball(req.url, "(LGWCODE|gclid|utm_[a-z]+)=[-_A-z0-9\+\;\%\.\(\)%]+&?", "");
       set req.url = regsub(req.url, "(\?|&)$", "");
    }

    return (lookup);
}

sub vcl_hash {
    # Default hash
    hash_data(req.url);
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # Include the X-Forward-Proto header, since we want to treat HTTPS
    # requests differently, and make sure this header is always passed
    # properly to the backend server.
    if (req.http.X-Forwarded-Proto ~ "https") {
      hash_data(req.http.X-Forwarded-Proto);
    }

    #Also hash based on device type
    if (req.http.X-UA-Device) {
        hash_data(req.http.X-UA-Device);
    }

    if (!(req.url ~ "^/(media|js|skin)/.*\.(png|jpg|jpeg|gif|css|js|swf|ico)$")) {
        call design_exception;
    }
    return (hash);
}

sub vcl_fetch {
    if (beresp.status == 500) {
       set beresp.saintmode = 10s;
       return (restart);
    }
    set beresp.grace = 5m;

    # add ban-lurker tags to object
    set beresp.http.X-Purge-URL = req.url;
    set beresp.http.X-Purge-Host = req.http.host;

    if (beresp.status == 200 || beresp.status == 301 || beresp.status == 404) {
        if (beresp.http.Content-Type ~ "text/html" || beresp.http.Content-Type ~ "text/xml") {

            if ((beresp.http.Set-Cookie ~ "NO_CACHE=") || (beresp.ttl < 1s)) {
                set beresp.ttl = 0s;
                return (hit_for_pass);
            }

            # marker for vcl_deliver to reset Age:
            set beresp.http.magicmarker = "1";

            # Don't cache cookies
            unset beresp.http.set-cookie;
        } else {
            # set default TTL value for static content
            set beresp.ttl = 4h;
        }
        return (deliver);
    }

    return (hit_for_pass);
}

sub vcl_deliver {
    # debug info
    if (resp.http.X-Cache-Debug) {
        if (obj.hits > 0) {
            set resp.http.X-Cache = "HIT";
            set resp.http.X-Cache-Hits = obj.hits;
        } else {
           set resp.http.X-Cache = "MISS";
        }
        set resp.http.X-Cache-Expires = resp.http.Expires;
        # Set UA-Device response in HTTP response from mobile_detect.vcl - mobile or desktop
        set resp.http.X-UA-Device = req.http.X-UA-Device;
    } else {
        set resp.http.X-UA-Device = req.http.X-UA-Device;
        set resp.http.X-CLIENTIP = client.ip;
        set resp.http.X-FORWARDED-FOR = req.http.X-Forwarded-For;
    }

    if (resp.http.magicmarker) {
        # Remove the magic marker
        unset resp.http.magicmarker;

        set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, post-check=0, pre-check=0";
        set resp.http.Pragma = "no-cache";
        set resp.http.Expires = "Mon, 31 Mar 2008 10:00:00 GMT";
        set resp.http.Age = "0";
    }
}

sub vcl_error {
    # redirect: part of recv routine for redirecting http to https
    if (obj.status == 750) {
        set obj.http.Location = obj.response;
        set obj.status = 301;
        return (deliver);
    }
}

sub design_exception {
}

I don't understand why varnish is caching files accessed via http and not via https. Can someone help me please?

Many thanks.

Answer 1

Problem solved.

The problem wasn't my .vcl file or apache config.

The problem was inside the module I use with my Magento sotre (CE 1.5.1). The module is Phoenix-media pagecache for MAgento (https://github.com/PHOENIX-MEDIA/Magento-PageCache-powered-by-Varnish) that has a function that disable all caching when Magento pages are secure.

To disable this behaviour, you have to comment these lines in App\code\community\Phoenix\VarnishCache\Helper\Cache.php:

// disable caching of secure pages



if (Mage::app()->getStore()->isCurrentlySecure()) 

{

return $this->setNoCacheHeader();

}