Upstream sent too big header while reading response header from upstream

Question

We're running Magento 2.3.1 on AWS EC2(t2.medium) Instance with an ELB. Webserver is nginx. Sometimes, we receive the "Upstream sent too big header while reading response header from upstream."-Error in our logs. After researching this issue, I found out that the most popular solution seems to be adjusting the fastcgi buffer size. I've done that without any effect.

To determine the current fastcgi response sizes, I did:

$ awk '($9 ~ /200/) { i++;sum+=$10;max=$10>max?$10:max; } END { printf("Maximum: %d\nAverage: %d\n",max,i?sum/i:0); }' /var/log/nginx/access.log

Maximum: 1613895
Average: 51433

Due to the huge amount of response sizes, I am not sure if the issue is nginx related. My guess is that the current buffer size rules are fine and Magento might has a bug with huge response sizes?

My current buffer sizes are:

fastcgi_pass   fastcgi_backend;
fastcgi_buffers 8 32k;

Answer 1

In my case fresh magento 2.4.1 (nginx on ubuntu) solution (by David Lambauer) worked:

In terminal:

• cd /etc/nginx/sites-available/
• ls

Open your nginx config file:

• code magento24.local.config

Find "fastcgi_buffers"

I change it like that:

        #fastcgi_buffers 1024 4k;
        fastcgi_buffers 16 256k;
        fastcgi_buffer_size 256k;

Save that config file. Restart nginx :

• sudo service nginx restart

Answer 2

Magento2 always have the issues with the response size. since you can solve it by enabling the cache which all recommended. Can you please try the below setup in nginx configuration to your location block

location / {
    proxy_pass       http://upstream;
    ...

    proxy_buffer_size          128k;
    proxy_buffers              4 256k;
    proxy_busy_buffers_size    256k;
  }

and restart the nginx service.

Answer 3

Coming back after a year, I don't have this issue anymore. I can't exactly explain the final solution but regarding the fastcgi parameters, I can tell that the following params work fine and are also proven by our hoster.

fastcgi_buffers 16 256k;
fastcgi_buffer_size 256k;