0

A recently launched a 2.3.2 Magento on-premise on Nexcess with Cloudflare over 2 months ago and have been experiencing sporadic, untraceable, non-reproducible 500 & 502 errors/timeouts that have no apparent trend. Need help figuring out what to try/where to look for debugging.

Tasks on Admin that have resulted in timeouts:

  • Update customer account information
  • Creating a customer account
  • Submitting a new order for an existing/new customer
  • Moving items to and from cart
  • Adding items to an order
  • Updating Sales Rep associated on customer account
  • Re-ording

Tasks on Frontend that have resulted in timeouts:

  • Viewing cart and subsequent cart summary pages
  • Adding product to cart from PDP
  • Querying product catalog using elastic-search bar

We see these errors, but don't know what they're specifically referring to:

  • upstream prematurely closed connection while reading response header from upstream
  • connect() to unix:/cf/socks/nginx-fl.sock failed (11: Resource temporarily unavailable) while connecting to upstream
  • upstream prematurely closed connection while reading response header from upstream
  • no live upstreams while connecting to upstream
  • upstream timed out (110: Connection timed out) while connecting to upstream

Some tasks/requests that result in a 502 error still complete, however for others it erases the update/create requests; for example, submitting an order, receives timeout error, but the order still processes or on the other hand, the whole new order is deleted.

Earlier on we were able to reproduce the 502 consistently by clicking Reorder on certain orders. In New Relic we saw that Magento\Sales\Model\AdminOrder\Create::initFromOrderItem was running extremely long (200,000+ ms)and got down to the part that validates the quantity of each item which is Magento\CatalogInventory\Model\Quote\Item\QuantityValidator::validate method which correlates with the reorder process where database table cataloginventory_stock_status.

We have since been able to prevent timeouts from happening during Reorders by bypassing the method to prevent it from validating the quantity of the item which took the shaved off 200,00ms from sales/order_create/reorder; but had no effect on all other occurring timeouts.

Now we have nothing to test with since none of the actions that result in timeouts are reproducible and aren't triggering any flags in New Relic or specific enough error codes in any logs.

We've:

  • confirmed there is plenty of server bandwidth/CPU
  • Confirmed normal traffic to site
  • Confirmed timeouts occur on all order quote types from orders with less than 10 line items to orders with greater than 100 line items
  • Confirmed timeouts occur on all order quote types from sub total less than $100 to totals greater than $3,000
  • Increased the max_input_vars to 75000 from 65000.
  • Changed the default /admin url to protect against DDOS/Brute Force attacks choking the server
  • Reviewing server logs
  • Reviewing New Relic
  • Asked Magento Support
  • Asked Nexcess Support
  • Asked Cloudflare Support

Any advice, tips, tricks, resource links, tools, people, or company are welcome.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.