Magento 2.x Elasticsearch Category Products Not Showing

Question

I tried to upgrade my Magento 2.3.5 install to Magento 2.4.0, but am unable to get category pages to display products. Everything is in Docker, so I am using Elasticsearch 7.6 and have specified elasticsearch7 for catalog/search/engine. The test from the Magento admin is successful. The category pages were working with 2.3.5 so I am assuming this is something related to the switch to Elasticsearch. I've looked at various upgrade guides without much luck as reindexing isn't fixing it for me.

The search works and returns products, as does loading a product detail page directly, and viewing the products in the Admin, however viewing any category page just shows:

We can't find products matching the selection.

Things I've tried:

1. Updated to Magento 2.4 using composer (via Docker) and then ran magento setup:upgrade and cleared cache.

2. Verified that products are enabled, visible, in stock, have quantity, and are assigned to category

3. Disabled all non-Magento modules, deleted ./generated/code, ./var/view_preprocessed, changed theme to Magento Blank.

4. Cleared Cache / Reindexed

bash-5.0$ magento indexer:reindex
Design Config Grid index has been rebuilt successfully in 00:00:00
Customer Grid index has been rebuilt successfully in 00:00:00
Category Products index has been rebuilt successfully in 00:00:00
Product Categories index has been rebuilt successfully in 00:00:00
Catalog Rule Product index has been rebuilt successfully in 00:00:01
Product EAV index has been rebuilt successfully in 00:00:00
Stock index has been rebuilt successfully in 00:00:00
Product Price index has been rebuilt successfully in 00:00:00
Catalog Product Rule index has been rebuilt successfully in 00:00:00
Catalog Search index has been rebuilt successfully in 00:00:05

bash-5.0$ magento cache:clean && magento cache:flush
Cleaned cache types:
config
layout
block_html
collections
reflection
db_ddl
compiled_config
eav
customer_notification
config_integration
config_integration_api
full_page
config_webservice
translate
Flushed cache types:
config
layout
block_html
collections
reflection
db_ddl
compiled_config
eav
customer_notification
config_integration
config_integration_api
full_page
config_webservice
translate

5. Deleted Elasticsearch indices, reindexed, and checked using curl

bash-5.0$ curl elasticsearch:9200/_cat/indices

bash-5.0$ curl elasticsearch:9200/
{
  "name" : "18af19145c92",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "-95jfv_yQNqZWqXk1Z3nBg",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

bash-5.0$ magento indexer:reindex
Design Config Grid index has been rebuilt successfully in 00:00:00
Customer Grid index has been rebuilt successfully in 00:00:00
Category Products index has been rebuilt successfully in 00:00:00
Product Categories index has been rebuilt successfully in 00:00:00
Catalog Rule Product index has been rebuilt successfully in 00:00:00
Product EAV index has been rebuilt successfully in 00:00:00
Stock index has been rebuilt successfully in 00:00:00
Product Price index has been rebuilt successfully in 00:00:00
Catalog Product Rule index has been rebuilt successfully in 00:00:00
Catalog Search index has been rebuilt successfully in 00:00:04

bash-5.0$ curl elasticsearch:9200/_cat/indices
yellow open magento2_product_1_v2 J4HvEAGeTFmxlovr5hl7zw 1 1 694 0 1.6mb 1.6mb

I noticed that there is no "category" index being created but not sure if there should be...

6. Enabled and disabled Flat Categories

7. Deleted and re-imported all Products

After deleting and re-importing the products I reindexed, cleared cache, regenerated thumbnails, etc. but still seeing the same behavior.

Edit - I deleted the database and reinstalled from scratch, imported products/categories, cleared cache, etc and the category pages are still not working. Can someone with a working install let me know how many indices are created in Elasticsearch? Can someone provide a same product import CSV that is working correctly for me to test?

Answer 1

I located an exception in ./magento/var/log/exception.log that showed an issue with the configuration of a custom attribute. The attribute was set to a type of text for the attribute custom_attribute in the eav_attribute table, but is_filterable set to 1 in the catalog_eav_attribute table. Setting this value to to 0 and reindexing/clearing cache allowed the category pages to load as expected.

[2020-09-18 19:46:47] main.CRITICAL: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [custom_attribute] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"magento2_product_1_v10","node":"fqGZNT31RSOuoKWU0r5GXQ","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [custom_attribute] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [custom_attribute] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.","caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [custom_attribute] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}},"status":400} {"exception":"[object] (Elasticsearch\\Common\\Exceptions\\BadRequest400Exception(code: 400): {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [custom_attribute] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"magento2_product_1_v10\",\"node\":\"fqGZNT31RSOuoKWU0r5GXQ\",\"reason\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [custom_attribute] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\"}}],\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [custom_attribute] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Fielddata is disabled on text fields by default. Set fielddata=true on [custom_attribute] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.\"}}},\"status\":400} at /magento/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:632)"} []
[2020-09-19 17:58:37] main.CRITICAL: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_attribute] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"magento2_product_1_v2","node":"6KxafAupTSqGYWZUYyULWg","reason":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_attribute] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_attribute] in order to load field data by uninverting the inverted index. Note that this can use significant memory.","caused_by":{"type":"illegal_argument_exception","reason":"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_attribute] in order to load field data by uninverting the inverted index. Note that this can use significant memory."}}},"status":400} {"exception":"[object] (Elasticsearch\\Common\\Exceptions\\BadRequest400Exception(code: 400): {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_attribute] in order to load field data by uninverting the inverted index. Note that this can use significant memory.\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"magento2_product_1_v2\",\"node\":\"6KxafAupTSqGYWZUYyULWg\",\"reason\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_attribute] in order to load field data by uninverting the inverted index. Note that this can use significant memory.\"}}],\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_attribute] in order to load field data by uninverting the inverted index. Note that this can use significant memory.\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [custom_attribute] in order to load field data by uninverting the inverted index. Note that this can use significant memory.\"}}},\"status\":400} at /magento/vendor/elasticsearch/elasticsearch/src/Elasticsearch/Connections/Connection.php:632)"} []

Show misconfigured attributes:

SELECT *
FROM catalog_eav_attribute cea 
JOIN eav_attribute ea
  ON ea.attribute_id = cea.attribute_id 
  AND backend_type IN ('text')
WHERE 
  cea.is_filterable = 1 OR cea.is_filterable_in_search = 1

Set is_filterable to 0 to allow category pages to load.

UPDATE catalog_eav_attribute cea 
JOIN eav_attribute ea
  ON ea.attribute_id = cea.attribute_id 
  AND backend_type in ('text')
SET 
  cea.is_filterable = 0, cea.is_filterable_in_search = 0
WHERE 
  cea.is_filterable = 1 OR cea.is_filterable_in_search = 1

Reindex/Refresh Cache

magento indexer:reindex
magento cache:clean
magento cache:flush

Answer 2

Check the details and fix shared by Magento support - https://support.magento.com/hc/en-us/articles/360027356612-Elasticsearch-5-is-configured-but-search-page-does-not-load-with-Fielddata-is-disabled-error

Answer 3

Please run following command and then try:

service elasticsearch start

Answer 4

In my case,
The issue is happened because of 2 attribute “jajuma_product_latitude” and “jajuma_product_longitude“ don’t show in store view.
I’ve change 2 parameters of this 2 attributes and run this command:

php bin/magento c:f && php bin/magento indexer:reindex

And the products are show as expected in categories.

Hope it help.

Answer 5

For anyone still struggling with this, I did a clean install and compared the databases. I found that changing website_id in the table cataloginventory_stock to 0 fixed both search and categories on the frontend.

Answer 6

I got similar issue (no products listing) and solved by following command:

php bin/magento index:reindex