Apply Magento\CatalogRule\Model\Rule\Condition\Combine to a collection FAST

Question

I am using Karliuka m2.SmartCategory module for dynamic categories. This seems great to do what I need as was having issues with Smile modules virtual categories and this seemed to offer the same functionality however just use an indexer to populate the categories based on the same conditions.

The rules are noted like below although a separate class for the new module (I am using a virtual type as had all the rules already set up from smile so my ones currently look like this until i save them):

{"type":"Smile\\ElasticsuiteVirtualCategory\\Model\\Rule\\Condition\\Combine","attribute":null,"operator":null,"value":"1","is_value_processed":null,"aggregator":"all","conditions":[{"type":"Smile\\ElasticsuiteVirtualCategory\\Model\\Rule\\Condition\\Product","attribute":"type_glassware","operator":"()","value":["1851"],"is_value_processed":false}]} 

https://github.com/karliuka/m2.SmartCategory

This uses code similar to the below question however it is awfully slow and indexing is taking over an hour...

How to get maching products based on rule conditions

...
        $this->resourceIterator->walk(
            $productCollection->getSelect(),
            [[$this, 'callbackValidateProduct']],
            [
                'attributes' => $this->getCollectedAttributes(),
                'product' => $this->productFactory->create()
            ]
        );
...


public function callbackValidateProduct($args)
{
    $product = clone $args['product'];
    $product->setData($args['row']);

    $websites = $this->getWebsitesMap();
    $results = [];

    foreach ($websites as $websiteId => $defaultStoreId) {
        $product->setStoreId($defaultStoreId);
        $results[$websiteId] = $this->getConditions()->validate($product);
        if (true === $results[$websiteId]) {
            $this->productIds[$product->getId()] = 1;
        }
    }
}

Everything works as expected however I just wondering if it can improve.

I have already dropped the time from 45 seconds to 15 seconds by adding a root category like Smile module however still too slow.

I have 300 and counting of these so index takes over an hour still and most time is spent in above checking of each product and feel a collection filter and getAllIds() on the final collection may be much more efficient and hopefully drop times down considerably if it can be done.

UPDATE

Think I basically want the below:

$conditions = $this->getConditions();
$conditions->collectValidatedAttributes($collection);
$this->sqlBuilder->attachConditionToCollection($collection, $conditions);

/**
 * Prevent retrieval of duplicate records. This may occur when multiselect product attribute matches
 * several allowed values from condition simultaneously
 */
$collection->distinct(true);

Seemed to work although unfortunately the time taken to do above was 30 seconds rather than the 15 seconds for the previous, so I am out of ideas...

UPDATE 2

public function callbackValidateProduct($args)
{
    $product = clone $args['product'];
    $product->setData($args['row']);
    $product->setStoreId($this->storeId);
    if($this->getConditions()->validate($product) === true) {
        $this->productIds[$product->getId()] = 1;
    }
}

As have set a root category I have tried to simplify the above function to just use the storeId in question which has considerably improved performance although not exactly how i intended to fix. 15 seconds is now anything from 0.2 to 3 seconds where before it was 15 to 60 seconds for each.

So wondering if anyone knows a quicker method for running these conditions as a filtered collection without walking through each product?

Basically generate a filter based on these conditions or RAW sql query that could accomplish this in milliseconds would be great. Not quite sure why that SQL builder is so slow as that is what it seems to do.

Just doesn't seem like it should take 15 seconds to return a few productIds based on a simple condition like brand is X for the majority of these categories?

UPDATE 3

Ok so went back to the SQL builder method today and that is now 3 times faster than the modified iterated function that I created.

So not sure if something strange was happening yesterday causing it to take 4 hours then 1 and a half hours with a root category. Then 7 mins suddenly today with modified function. Now under 2 mins suddenly with the sql builder method. I am happy now this should be OK.