2

I have the weirdest error...

I clear my Magento 2 cache and execute the following wget command to crawl the entire site:

wget --spider -o wget.log -e robots=off -r -l 5 -p -S -T3 --header="X-Bypass-Cache: 1" -H --domains=example.com http://example.com --delete-after

After I visit with my browser ALL pages, home page, category page, cms pages, product pages return a 404 error.

This does not happen when I clear cache and visit with my browser on the first try instead of using wget.

This is my Nginx config if it has anything to do with it:

server {

listen 80;
server_name example.com;
set $MAGE_ROOT /home/httpd/example.com;

root $MAGE_ROOT/pub;

index index.php;
autoindex off;
charset UTF-8;
error_page 404 403 = /errors/404.php;

# PHP entry point for setup application
location ~* ^/setup($|/) {
    root $MAGE_ROOT;
    location ~ ^/setup/index.php {
        fastcgi_pass   fastcgi_backend;

        fastcgi_param  PHP_FLAG  "session.auto_start=off \n suhosin.session.cryptua=off";
        fastcgi_param  PHP_VALUE "memory_limit=2G \n max_execution_time=600";
        fastcgi_read_timeout 600s;
        fastcgi_connect_timeout 600s;

        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        fastcgi_params;
    }

    location ~ ^/setup/(?!pub/). {
        deny all;
    }

    location ~ ^/setup/pub/ {
        add_header X-Frame-Options "SAMEORIGIN";
    }
}

# PHP entry point for update application
location ~* ^/update($|/) {
    root $MAGE_ROOT;

    location ~ ^/update/index.php {
        fastcgi_split_path_info ^(/update/index.php)(/.+)$;
        fastcgi_pass   fastcgi_backend;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        fastcgi_param  PATH_INFO        $fastcgi_path_info;
        include        fastcgi_params;
    }

    # Deny everything but index.php
    location ~ ^/update/(?!pub/). {
        deny all;
    }

    location ~ ^/update/pub/ {
        add_header X-Frame-Options "SAMEORIGIN";
    }
}

location / {
    try_files $uri $uri/ /index.php$is_args$args;
}

location /db/ {
    try_files $uri $uri/ /index.php$is_args$args;
}


location /pub/ {
    location ~ ^/pub/media/(downloadable|customer|import|theme_customization/.*\.xml) {
        deny all;
    }
    alias $MAGE_ROOT/pub/;
    add_header X-Frame-Options "SAMEORIGIN";
}

location /static/ {
    # Uncomment the following line in production mode
    # expires max;

    # Remove signature of the static files that is used to overcome the browser cache
    location ~ ^/static/version {
        rewrite ^/static/(version\d*/)?(.*)$ /static/$2 last;
    }

    location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ {
        add_header Cache-Control "public";
        add_header X-Frame-Options "SAMEORIGIN";
        expires +1y;

        if (!-f $request_filename) {
            rewrite ^/static/?(.*)$ /static.php?resource=$1 last;
        }
    }
    location ~* \.(zip|gz|gzip|bz2|csv|xml)$ {
        add_header Cache-Control "no-store";
        add_header X-Frame-Options "SAMEORIGIN";
        expires    off;

        if (!-f $request_filename) {
        rewrite ^/static/?(.*)$ /static.php?resource=$1 last;
        }
    }
    if (!-f $request_filename) {
        rewrite ^/static/?(.*)$ /static.php?resource=$1 last;
    }
    add_header X-Frame-Options "SAMEORIGIN";
}

location /media/ {
    try_files $uri $uri/ /get.php$is_args$args;

    location ~ ^/media/theme_customization/.*\.xml {
        deny all;
    }

    location ~* \.(ico|jpg|jpeg|png|gif|svg|js|css|swf|eot|ttf|otf|woff|woff2)$ {
        add_header Cache-Control "public";
        add_header X-Frame-Options "SAMEORIGIN";
        expires +1y;
        try_files $uri $uri/ /get.php$is_args$args;
    }
    location ~* \.(zip|gz|gzip|bz2|csv|xml)$ {
        add_header Cache-Control "no-store";
        add_header X-Frame-Options "SAMEORIGIN";
        expires    off;
        try_files $uri $uri/ /get.php$is_args$args;
    }
    add_header X-Frame-Options "SAMEORIGIN";
}

location /media/customer/ {
    deny all;
}

location /media/downloadable/ {
    deny all;
}

location /media/import/ {
    deny all;
}

# PHP entry point for main application
location ~ (index|get|static|report|404|503)\.php$ {
    try_files $uri =404;
    fastcgi_pass   fastcgi_backend;
    fastcgi_buffers 1024 4k;

    fastcgi_param  PHP_FLAG  "session.auto_start=off \n suhosin.session.cryptua=off";
    #fastcgi_param  PHP_VALUE "memory_limit=768M \n max_execution_time=18000";
    fastcgi_param  PHP_VALUE "memory_limit=2G \n max_execution_time=600";
    fastcgi_read_timeout 600s;
    fastcgi_connect_timeout 600s;

    fastcgi_index  index.php;
    fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    include        fastcgi_params;
}

gzip on;
gzip_disable "msie6";

gzip_comp_level 6;
gzip_min_length 1100;
gzip_buffers 16 8k;
gzip_proxied any;
gzip_types
    text/plain
    text/css
    text/js
    text/xml
    text/javascript
    application/javascript
    application/x-javascript
    application/json
    application/xml
    application/xml+rss
    image/svg+xml;
gzip_vary on;

# Banned locations (only reached if the earlier PHP entry point regexes don't match)
location ~* (\.php$|\.htaccess$|\.git) {
    deny all;
}

}

Here is the wget log file after trying to crawl and it shows 404 link:

$ cat wget.log
Spider mode enabled. Check if remote file exists.
--2019-03-22 16:26:52--  http://example.com/
Resolving example.com (example.com)... 111.222.333.444
Connecting to example.com (example.com)|111.222.333.444|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 404 Not Found
  Server: nginx
  Date: Fri, 22 Mar 2019 15:26:55 GMT
  Content-Type: text/html; charset=UTF-8
  Connection: keep-alive
  Vary: Accept-Encoding
  X-Magento-Cache-Control: max-age=86400, public, s-maxage=86400
  X-Magento-Cache-Debug: MISS
  Pragma: no-cache
  Cache-Control: max-age=0, must-revalidate, no-cache, no-store
  Expires: Thu, 22 Mar 2018 15:26:55 GMT
  X-Content-Type-Options: nosniff
  X-XSS-Protection: 1; mode=block
  X-Frame-Options: SAMEORIGIN
Remote file does not exist -- broken link!!!

Found 1 broken link.

http://example.com/

How is this technically possible?

How can wget, a client, create 404 cache pages for my entire Magento 2 store?

  • do you have any rules in place in .htaccess or somewhere else which blocks spiders/bots and redirects them to a 404 page across the board. I have seen this done on sites to block automated requests (bad bots) from overloading a site. – bjornredemption Mar 22 at 9:42
  • Not that I know of. I updated post with my Nginx, maybe there is something there but I can't see it. Even if I use -U flag with a custom browser user agent, I can't crawl anywhere. – John Mar 22 at 15:25
  • so im guessing that you've edited the logs and files u attached to replace your actual domain with example.com before posting or are you using some local setup ? – bjornredemption Mar 22 at 16:31
  • Yes of course, it is hosted on a dev server. – John Mar 22 at 17:17
0

Seems like this is a bug in Magento 2.3.0 with HEAD requests.

https://github.com/magento/magento2/issues/21299

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.