0

Long time reader, first time asking...

Not sure if this a magento issue or other.

I use Firebear import/export tool to update the stock qty, approximately 105,000 SKU's.

Worked fantastic for about a month of testing, would complete in about 30 minutes. I was ok with that. After testing and putting the site live things went south fairly quickly. So before going live import would run in 28 minutes. First day live 41 minutes 1st update, 2nd, 3rd and 4th update 52 minutes, 5th update 56 minutes, Day two live 6th update 3 hours 25 minutes. After that things just got worse, eventually the update would just fail.

I've tried breaking the update into pieces just as a band aid but that soon stopped completing as well.

I've tried tweaking the mysql variables and no improvements noticed.

I've also noticed that deleting products with bulk actions if I do more than say 15 products at once it times out and takes awhile to recover.

The way the import tool is working is it seems to update 100 SKUs at a time, which it does quickly.

2020-01-08 10:50:56 : product with sku: COM7996-16 .... 0.00227s 2020-01-08 10:50:57 : Imported: 0 rows 2020-01-08 10:50:57 : Updated: 100 rows 2020-01-08 10:51:05 : The import was successful. 2020-01-08 10:51:06 : product with sku: COM7997-1 .... 0.00782s

So the first 100 gets updated it shows successful, you can see the time between the first 100 and the second 100 is like 9 seconds. Fast forward to the end of this chunk of update, 25% of the whole thing, 26,400 sku's

2020-01-08 12:13:14 : product with sku: EDE72285 .... 0.00226s 2020-01-08 12:13:14 : Imported: 0 rows 2020-01-08 12:13:14 : Updated: 100 rows 2020-01-08 12:13:34 : The import was successful. 2020-01-08 12:13:35 : product with sku: EDE7256 .... 0.00802s

now 21 seconds has elapsed between each 100 update.

So this update took 83 minutes and it's only 1/4 of the whole thing.

I am at a loss of why the full update of 105,000 sku's used to take 28 minutes now it's taking 12 to 13 times as long.

The only thing that changed from testing to live is the domain name and SSL, files did not move, database is the same, user is the same.

Thank you for any advice.

Improve this question
1
  • Additional information request. RAM size, # cores, any SSD or NVME devices on MySQL Host server? Post on pastebin.com and share the links. From your SSH login root, Text results of: B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) complete MySQLTuner report AND Optional helpful information, if available includes - htop OR top for most active apps, ulimit -a for a Linux/Unix list of limits, iostat -xm 5 3 for IOPS by device and core/cpu count, for server workload tuning analysis to provide suggestions.
    – Wilson Hauck
    Mar 4, 2020 at 12:11

2 Answers 2

Reset to default
0

We have the same issue with our shop. Since a while the import with firebear takes about 10-12 minutes where it was finished in 3 minutes before. It is a simple stock update for about 10.000 items with sku and quantity, so nothing complex. On every chunk of 100 items the time between the import lasts longer and longer. Any ideas?

Improve this answer
1
  • Please post to pastebin.com to avoid storage limit of Mage Stack Overflow A) the 'import query', B) EXPLAIN (your import query);, C) SHOW CREATE TABLE (each_tbl_name_used);
    – Wilson Hauck
    Mar 4, 2020 at 12:17
0

Have been having the same issue with Firebear too...

Import just slows more and more they further through it gets... 20,000+ products becomes a bit of an issue :-(

Felt like possibly an MYSQL config issue.. but everything look OK to me...

Still looking for a solution!

Improve this answer
2
  • You and Walter may have the same problem. Check-in occasionally. In the meantime, view my profile, Network profile for free downloadable Utility Scripts to assist with performance tuning, download, Checkout and use them.
    – Wilson Hauck
    Mar 4, 2020 at 12:19
  • Additional information request. RAM size, # cores, any SSD or NVME devices on MySQL Host server? Post on pastebin.com and share the links. From your SSH login root, Text results of: B) SHOW GLOBAL STATUS; after minimum 24 hours UPTIME C) SHOW GLOBAL VARIABLES; D) SHOW FULL PROCESSLIST; E) complete MySQLTuner report AND Optional helpful information, if available includes - htop OR top for most active apps, ulimit -a for a Linux/Unix list of limits, iostat -xm 5 3 for IOPS by device and core/cpu count, for server workload tuning analysis to provide suggestions.
    – Wilson Hauck
    Mar 10, 2020 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.