0

My store used to be very fast, but now I am getting >10 second load times. Recent system changes are, upgraded to 2.1.6, installed system patches, upped PHP FPM servers/children. Below are my configuration files and system specs. Any idea why I'm slow? I'm at my wits end. When I use curl I do see no-cache directives I don't think I should see that though, should I? I also have varnish logs for a main site load, not included here, its 900+ lines. Let me know if its useful and I will include.

Thanks for any help you can give.

Edit: Time to first byte is really high enter image description here

curl fetch

curl -I -v --location-trusted https://www.example.com/
*   Trying xxx.xxx.xxx.xxx...
* Connected to www.example.com (xxx.xxx.xxx.xxx) port 443 (#0)
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs/
* TLSv1.2, TLS handshake, Client hello (1):
* TLSv1.2, TLS handshake, Server hello (2):
* NPN, negotiated HTTP1.1
* TLSv1.2, TLS handshake, CERT (11):
* TLSv1.2, TLS handshake, Server key exchange (12):
* TLSv1.2, TLS handshake, Server finished (14):
* TLSv1.2, TLS handshake, Client key exchange (16):
* TLSv1.2, TLS change cipher, Client hello (1):
* TLSv1.2, TLS handshake, Unknown (67):
* TLSv1.2, TLS handshake, Finished (20):
* TLSv1.2, TLS change cipher, Client hello (1):
* TLSv1.2, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
*        subject: C=US; postalCode=***; ST=***; L=***; street=***; street=***; O=***; OU=InstantSSL Pro; CN=www.example.com
*        start date: 2015-04-29 00:00:00 GMT
*        expire date: 2018-05-06 23:59:59 GMT
*        subjectAltName: www.example.com matched
*        issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO RSA Organization Validation Secure Server CA
*        SSL certificate verify ok.
> HEAD / HTTP/1.1
> Host: www.example.User
> com-Agent: curl/7.42.1
> Accept: */*
>
HTTP/1.1 200 OK
Server: nginx
Date: Fri, 14 Apr 2017 14:45:58 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Vary: Accept-Encoding
Set-Cookie: PHPSESSID=99999999999999999999999999999999; expires=Fri, 14-Apr-2017 15:45:56 GMT; Max-Age=3600; path=/; domain=www.example.com; secure; HttpOnly
Set-Cookie: mage-messages=%5B%5D; expires=Sat, 14-Apr-2018 14:45:58 GMT; Max-Age=31536000; path=/
Pragma: no-cache
Cache-Control: max-age=0, must-revalidate, no-cache, no-store
Expires: Thu, 14 Apr 2016 14:45:56 GMT
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Accept-Ranges: bytes
Strict-Transport-Security: max-age=63072000; includeSubdomains;

System

  • openSUSE 13.2 (Harlequin) (x86_64)
  • Redis 3.2.3
  • nginx/1.11.4
  • varnish-4.1.3 revision 5e3b6d2
  • mysql Ver 15.1 Distrib 10.0.27-MariaDB, for Linux (x86_64) using readline 5.2

Magento Config

  • Merge JavaScript Files: Yes
  • Enable JavaScript Bundling: Yes
  • Minify JavaScript Files: Yes
  • Merge CSS Files: Yes
  • Minify CSS Files: Yes
  • Production mode

memory

# free -m
             total       used       free     shared    buffers     cached
Mem:          3958       3816        141         32          0       2931
-/+ buffers/cache:        885       3073
Swap:         2055          0       2055

CPU

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    1
Socket(s):             2
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            15
Model:                 4
Model name:            Intel(R) Xeon(TM) CPU 3.00GHz
Stepping:              1
CPU MHz:               3000.191
BogoMIPS:              5999.17
L1d cache:             16K
L2 cache:              1024K
NUMA node0 CPU(s):     0-3

Varnish

DAEMON_OPTS="-a :80 \
             -T 127.0.0.1:6082 \
             -f /usr/local/etc/varnish/default.vcl \
             -p cli_buffer=16384 \
             -P /var/run/varnishd.pid \
             -p http_resp_hdr_len=64000 \
             -S /usr/local/etc/varnish/secret \
             -s malloc,256m \
             -F"

vcl 4.0;

import std;

backend default {
    .host = "localhost";
    .port = "8080";
    .first_byte_timeout = 600s;
}

backend example {
    .host = "xxx.xxx.xxx.xxx";
    .port = "8080";
    .first_byte_timeout = 600s;
}

acl purge {
    "localhost";
    "192.168.0.0"/24;
}

sub vcl_recv {
   if (req.http.host ~ "example.com") {
     set req.backend_hint = example;
   } else {
     set req.backend_hint = default;
   }

    if (req.method == "PURGE") {
        if (client.ip !~ purge) {
            return (synth(405, "Method not allowed"));
        }
        if (!req.http.X-Magento-Tags-Pattern) {
            return (synth(400, "X-Magento-Tags-Pattern header required"));
        }
        ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
        return (synth(200, "Purged"));
    }

    if (req.method != "GET" &&
        req.method != "HEAD" &&
        req.method != "PUT" &&
        req.method != "POST" &&
        req.method != "TRACE" &&
        req.method != "OPTIONS" &&
        req.method != "DELETE") {
          /* Non-RFC2616 or CONNECT which is weird. */
          return (pipe);
    }

    # We only deal with GET and HEAD by default
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }

    # Bypass shopping cart, checkout and search requests
    if (req.url ~ "/checkout" || req.url ~ "/catalogsearch") {
        return (pass);
    }

    # normalize url in case of leading HTTP scheme and domain
    set req.url = regsub(req.url, "^http[s]?://", "");

    # collect all cookies
    std.collect(req.http.Cookie);

    # Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv)$") {
            # No point in compressing these
            unset req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";
        } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {
            # unkown algorithm
            unset req.http.Accept-Encoding;
        }
    }

    # Remove Google gclid parameters to minimize the cache objects
    set req.url = regsuball(req.url,"\?gclid=[^&]+$",""); # strips when QS = "?gclid=AAA"
    set req.url = regsuball(req.url,"\?gclid=[^&]+&","?"); # strips when QS = "?gclid=AAA&foo=bar"
    set req.url = regsuball(req.url,"&gclid=[^&]+",""); # strips when QS = "?foo=bar&gclid=AAA" or QS = "?foo=bar&gclid=AAA&bar=baz"

    # static files are always cacheable. remove SSL flag and cookie
        if (req.url ~ "^/(pub/)?(media|static)/.*\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)$") {
        unset req.http.Https;
        unset req.http.X-Forwarded-Proto;
        unset req.http.Cookie;
    }

    return (hash);
}

sub vcl_hash {
    if (req.http.cookie ~ "X-Magento-Vary=") {
        hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
    }

    # For multi site configurations to not cache each other's content
    if (req.http.host) {
        hash_data(req.http.host);
    } else {
        hash_data(server.ip);
    }

    # To make sure http users don't see ssl warning
    if (req.http.X-Forwarded-Proto) {
        hash_data(req.http.X-Forwarded-Proto);
    }
}

sub vcl_backend_response {
    if (beresp.http.content-type ~ "text") {
        set beresp.do_esi = true;
    }

    if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    # cache only successfully responses and 404s
    if (beresp.status != 200 && beresp.status != 404) {
        set beresp.ttl = 0s;
        set beresp.uncacheable = true;
        return (deliver);
    } elsif (beresp.http.Cache-Control ~ "private") {
        set beresp.uncacheable = true;
        set beresp.ttl = 86400s;
        return (deliver);
    }

    if (beresp.http.X-Magento-Debug) {
        set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
    }

    # validate if we need to cache it and prevent from setting cookie
    # images, css and js are cacheable by default so we have to remove cookie also
    if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
        unset beresp.http.set-cookie;
        if (bereq.url !~ "\.(ico|css|js|jpg|jpeg|png|gif|tiff|bmp|gz|tgz|bz2|tbz|mp3|ogg|svg|swf|woff|woff2|eot|ttf|otf)(\?|$)") {
            set beresp.http.Pragma = "no-cache";
            set beresp.http.Expires = "-1";
            set beresp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
            set beresp.grace = 1m;
        }
    }

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
        beresp.http.Surrogate-control ~ "no-store" ||
        (!beresp.http.Surrogate-Control && beresp.http.Vary == "*")) {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }
    return (deliver);
}

sub vcl_deliver {
    if (resp.http.X-Magento-Debug) {
        if (resp.http.x-varnish ~ " ") {
            set resp.http.X-Magento-Cache-Debug = "HIT";
        } else {
            set resp.http.X-Magento-Cache-Debug = "MISS";
        }
    } else {
        unset resp.http.Age;
    }

    unset resp.http.X-Magento-Debug;
    unset resp.http.X-Magento-Tags;
    unset resp.http.X-Powered-By;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;
    unset resp.http.Link;
}

nginx.conf

user wwwrun www;
worker_processes  auto;

events {
    worker_connections  1024;
    use epoll;
}

http {
    include mime.types;
    include /etc/nginx/fastcgi.conf;
    default_type  application/octet-stream;
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    server_tokens off;

    upstream fastcgi_backend {
        server unix:/var/run/php7-fpm.sock;
    }

    #example.com NON-SSL
    server {
        listen xxx.xxx.xxx.xxx:8080 default_server;
        server_name example.com www.example.com;
        set $MAGE_ROOT /srv/www/vhosts/example.com/htdocs;
        include magento.nginx_2.conf;
    }

    #example.com SSL
    server {
        listen xxx.xxx.xxx.xxx:443 ssl;
        server_name example.com  www.example.com;
        ssl_certificate      /etc/nginx/ssl.crt/www_example_com_2015-04-23-nginx-bundle.crt;
        ssl_certificate_key  /etc/nginx/ssl.key/www.example.com_2015-04-23.key;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:ECDHE-RSA-AES128-GCM-SHA256:AES256+EECDH:DHE-RSA-AES128-GCM-SHA256:AES256+EDH:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AE
        ssl_stapling on;
        ssl_stapling_verify on;
        add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; ";
        location / {
            proxy_pass http://127.0.0.1:80;
            proxy_read_timeout 600;
            proxy_set_header X-Real-IP  $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header X-Forwarded-Port 443;
            proxy_set_header Host $host;
        }
        set $MAGE_ROOT /srv/www/vhosts/example.com/htdocs;
        root $MAGE_ROOT/pub;
    }
}
  • I had similar problem locally with mergin js and css files enabled. I do not know why (as for local development I preferred this to have turned off) but every time magento was going through the process of merging those files via php code and didn't cache the result. Check how fast is the document response if you do not have all merging and minifying enabled. – Zefiryn Apr 14 '17 at 20:17
  • Turned off minifying and merging and turned on developer mode. Now time to first byte is 35 seconds with a 2 min total load time. – D. Hobaugh Apr 14 '17 at 20:57
  • So this is not it. You need to profile your application. You can start by using built in profiler. Docs on that are here: devdocs.magento.com/guides/v2.1/config-guide/bootstrap/… This will show you which part of the code takes the most time. – Zefiryn Apr 14 '17 at 21:09
  • Fixed it. I looked in Cache Management an all caches were disabled. I don't think that happened any other time up upgraded. Lesson learned check that the caches are enabled. lol – D. Hobaugh Apr 14 '17 at 21:21
0

It's a bug in all version

Disable all these options and enable the cache.

Merge JavaScript Files: No
Enable JavaScript Bundling: No
Minify JavaScript Files: No
Merge CSS Files: No
Minify CSS Files: No

And change to developer mode.

  • i dont think this will fix anything. what bug are you talking about? – MagenX Apr 15 '17 at 9:28
  • The high TTFB on Magento 2.1.6 – Rafael Corrêa Gomes Apr 16 '17 at 15:59
  • TTFB is not a bug... bug is what makes TTFB to be longer than 1-2 sec.... this is not the answer obviously. – MagenX Apr 17 '17 at 8:09
0

From my experience Using local development docker-debian with this stack nginx and varnish use same port 80

backend default {
.host = "127.0.0.1";
.port = "80";
.first_byte_timeout = 600s;
}

Developer mode enabled
No enabled minify and combine css
Copy varnish.vcl export from admin to your server. Varnish config stored in debian with location /etc/varnish/default.vcl

I got around 6-7 sec in local for loading all files

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.