I have found the solution, it was a custom attribute created by an extension.
The problem - attribute type was text
and it was set to use in Layered Navigation.
The fix - tracked that attribute in catalog_eav_attribute
table and set the column is_filterable
to 0 (you won't be able to change this from the backend as the field is enabled for editing only for catalog input type Yes/No, Dropdown, Multiple Select and Price)
The problem occurs only when you have Elastic Search enabled.
The best way to identify the attribute is to execute the below SQL query -
select * from catalog_eav_attribute EAV1 join eav_attribute EAV2 on EAV2.attribute_id = EAV1.attribute_id and frontend_input = 'text' where EAV1.is_filterable = 1