1

I'm using Magento 2.3.2 and a production site that has a decently large number of products. The site has been running fine for over a year but in the last week we suddenly have an issue where a runaway query will take up all of the site CPU and, over time, crash the site.

I can show processlist in MySQL and the query looks like this:

INSERT INTO `search_tmp_5f9b11f4bfb569_80230545` SELECT `main_select`.`entity_id`, SUM(score) AS `re`

This query will never finish, and then another one appears, and then more, and eventually the site crashes. I can restart the database, or I can kill the job as root, but they keep coming back.

This question has come up before on Stack Exchange, but I've tried the various suggestions to no luck, and also ours was not started by a specific action or change. We do not have Amasty Sorting Module installed.

One thing I did try was the patch described at this Github thread. This was a local edit: changing entity_id to source_id in the file CustomAttributeFilter. That did nothing, in fact it didn't even change the query that I can see, which surprised me.

I am also having issues getting Elastic Search to work properly with our current setup, which I think is unrelated but which has also been a suggested fix.

At this point I just want my site to function and not crash without regular manual intervention. Can someone advise where I should be looking, or what kind of further research I should do? I'd like to know for sure what pages are even causing this in the first place, and how to duplicate the issue on a different server so I can troubleshoot outside of production.

Edit 1: I have gotten around this issue by setting up a cron job running once a minute that kills every query starting with "INSERT INTO `search_tmp_". My site appears to be unaffected and does not run out of memory and crash. Of course this is not a fix. I'd still like to know the source of this error at the very least.

Edit 2: Full query using full processlist (of course the hex code after search_tmp_ is different each iteration)

INSERT INTO `search_tmp_5fc5942c05f071_72644368` SELECT `main_select`.`entity_id`, SUM(score) AS `relevance` FROM (SELECT DISTINCT  `search_index`.`entity_id`, (((0) + (0) + (0) + (0) + (0)) * 1) AS `score` FROM `catalog_product_index_eav` AS `search_index`
  INNER JOIN `catalog_product_index_eav` AS `brand_id_filter` ON `search_index`.`source_id` = `brand_id_filter`.`source_id` AND `brand_id_filter`.`attribute_id` = 161 AND `brand_id_filter`.`store_id` = 1
  INNER JOIN `catalog_product_index_eav` AS `group_filter` ON `search_index`.`source_id` = `group_filter`.`source_id` AND `group_filter`.`attribute_id` = 192 AND `group_filter`.`store_id` = 1
  INNER JOIN `cataloginventory_stock_status` AS `stock_index` ON stock_index.product_id = search_index.entity_id AND `stock_index`.`website_id` = 0 AND `stock_index`.`stock_status` = 1 AND `stock_index`.`stock_id` = 1
  INNER JOIN `cataloginventory_stock_status` AS `sub_products_stock_index` ON sub_products_stock_index.product_id = search_index.source_id AND `sub_products_stock_index`.`website_id` = 0 AND `sub_products_stock_index`.`stock_status` = 1 AND `sub_products_stock_index`.`stock_id` = 1
  INNER JOIN `catalog_product_index_eav` AS `visibility_filter` ON search_index.entity_id = visibility_filter.entity_id AND `visibility_filter`.`attribute_id` = 99 AND `visibility_filter`.`value` in ('2', '4') AND `visibility_filter`.`store_id` = '1'
  INNER JOIN `catalog_category_product_index_store1` AS `category_ids_index` ON search_index.entity_id = category_ids_index.product_id AND category_ids_index.store_id = '1'
  INNER JOIN `catalog_product_index_price` AS `price_index` ON search_index.entity_id = price_index.entity_id AND price_index.website_id = '1' WHERE (search_index.store_id = '1') AND (category_ids_index.category_id = 151) AND (`price_index`.`min_price` >= '3000' AND `price_index`.`min_price` <= '4000' AND `price_index`.`customer_group_id` = 0) AND (brand_id_filter.value  IN ('1')) AND (group_filter.value = '514')) AS `main_select` GROUP BY `entity_id` ORDER BY `relevance` DESC, `entity_id` DESC
  LIMIT 10000

Edit 3:

The select query, run by itself, takes no time at all to run alone and give results. It seems like the culprit table with index issues is the tmp table itself, which is of course transient.

Thank you folks for your interest in helping. These details were requested in comments, and are pasted in here without comment because I don't quite know what I should be looking for.

  CREATE TABLE `catalog_product_index_eav` (
    `entity_id` int(10) unsigned NOT NULL COMMENT 'Entity ID',
    `attribute_id` smallint(5) unsigned NOT NULL COMMENT 'Attribute ID',
    `store_id` smallint(5) unsigned NOT NULL COMMENT 'Store ID',
    `value` int(10) unsigned NOT NULL COMMENT 'Value',
    `source_id` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Original entity Id for attribute value',
    PRIMARY KEY (`entity_id`,`attribute_id`,`store_id`,`value`,`source_id`),
    KEY `CATALOG_PRODUCT_INDEX_EAV_ATTRIBUTE_ID` (`attribute_id`),
    KEY `CATALOG_PRODUCT_INDEX_EAV_STORE_ID` (`store_id`),
    KEY `CATALOG_PRODUCT_INDEX_EAV_VALUE` (`value`)
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Catalog Product EAV Index Table';

  CREATE TABLE `cataloginventory_stock_status` (
    `product_id` int(10) unsigned NOT NULL COMMENT 'Product Id',
    `website_id` smallint(5) unsigned NOT NULL COMMENT 'Website Id',
    `stock_id` smallint(5) unsigned NOT NULL COMMENT 'Stock Id',
    `qty` decimal(12,4) NOT NULL DEFAULT '0.0000' COMMENT 'Qty',
    `stock_status` smallint(5) unsigned NOT NULL COMMENT 'Stock Status',
    PRIMARY KEY (`product_id`,`website_id`,`stock_id`),
    KEY `CATALOGINVENTORY_STOCK_STATUS_STOCK_ID` (`stock_id`),
    KEY `CATALOGINVENTORY_STOCK_STATUS_WEBSITE_ID` (`website_id`),
    KEY `CATALOGINVENTORY_STOCK_STATUS_STOCK_STATUS` (`stock_status`)
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Cataloginventory Stock Status';

  CREATE TABLE `catalog_category_product_index_store1` (
    `category_id` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Category Id',
    `product_id` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'Product Id',
    `position` int(11) DEFAULT NULL COMMENT 'Position',
    `is_parent` smallint(5) unsigned NOT NULL DEFAULT '0' COMMENT 'Is Parent',
    `store_id` smallint(5) unsigned NOT NULL DEFAULT '0' COMMENT 'Store Id',
    `visibility` smallint(5) unsigned NOT NULL COMMENT 'Visibility',
    PRIMARY KEY (`category_id`,`product_id`,`store_id`),
    KEY `CAT_CTGR_PRD_IDX_STORE1_PRD_ID_STORE_ID_CTGR_ID_VISIBILITY` (`product_id`,`store_id`,`category_id`,`visibility`),
    KEY `IDX_216E521C8AD125E066D2B0BAB4A08412` (`store_id`,`category_id`,`visibility`,`is_parent`,`position`)
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Catalog Category Product Index Store1';

  CREATE TABLE `catalog_product_index_price` (
    `entity_id` int(10) unsigned NOT NULL COMMENT 'Entity ID',
    `customer_group_id` int(10) unsigned NOT NULL COMMENT 'Customer Group ID',
    `website_id` smallint(5) unsigned NOT NULL COMMENT 'Website ID',
    `tax_class_id` smallint(5) unsigned DEFAULT '0' COMMENT 'Tax Class ID',
    `price` decimal(12,4) DEFAULT NULL COMMENT 'Price',
    `final_price` decimal(12,4) DEFAULT NULL COMMENT 'Final Price',
    `min_price` decimal(12,4) DEFAULT NULL COMMENT 'Min Price',
    `max_price` decimal(12,4) DEFAULT NULL COMMENT 'Max Price',
    `tier_price` decimal(12,4) DEFAULT NULL COMMENT 'Tier Price',
    PRIMARY KEY (`entity_id`,`customer_group_id`,`website_id`),
    KEY `CATALOG_PRODUCT_INDEX_PRICE_CUSTOMER_GROUP_ID` (`customer_group_id`),
    KEY `CATALOG_PRODUCT_INDEX_PRICE_MIN_PRICE` (`min_price`),
    KEY `CAT_PRD_IDX_PRICE_WS_ID_CSTR_GROUP_ID_MIN_PRICE` (`website_id`,`customer_group_id`,`min_price`)
  ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Catalog Product Price Index Table';
8
  • Please post to pastebin.com TEXT results of SHOW FULL PROCESSLIST; when faced with this situation again. We need to see the complete Query, please. Nov 29, 2020 at 20:04
  • Could you take the INSERT query from Edit 2: and change it to EXPLAIN SELECT main_select........... LIMIT 100; (leaving out the insert into tblname` ? The result should be an EXPLAIN with clues of missing indexes. Dec 6, 2020 at 18:27
  • Please post SHOW CREATE TABLE (tbl_names); for your A) catalog_product_index_eav B) cataloginventory_stock_status C) catalog_category_product_index_store1 D) catalog_product_index_price for analysis of datatype consistency. Dec 6, 2020 at 20:00
  • Do you still have your Oct 29, 20 problems? Jan 29, 2021 at 15:33
  • 1
    I do still have this problem as of 3/11/21. It is mitigated by a job that runs once a minute that kills these queries. I don't even know how to replicate the issue, like for instance on a dev or staging server. It's just a thing that happens.
    – S Adamson
    Mar 12, 2021 at 0:02

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.