Magento 2: how to use the collection walk iterator?

Question

Back in Magento 1, it was possible to use the collection iterator to walk through the results and avoid looping through them.

It was a huge improvement in terms of performance when dealing with massive collections.

Here is some sample code of what could be done.

$customers = Mage::getModel('customer/customer')->getCollection()->addAttributeToSelect(array('firstname'), 'inner');
// call iterator walk method with collection query string and callback method as parameters
Mage::getSingleton('core/resource_iterator')->walk($customers->getSelect(), array(array($this, 'customerCallback')));

Then you could define a callback function customerCallback to process the results one by one.

Is that still possible in Magento 2? If so how can I achieve that?

Answer 1

it still seems to be there: https://github.com/magento/magento2/blob/2.0/lib/internal/Magento/Framework/Model/ResourceModel/Iterator.php

Usage should be the same as in M1

Answer 2

Yes, it's possible! It's very similar to Magento 1, by the way.

\Magento\Framework\Model\ResourceModel\Iterator has the walk function which will call your callback function for every single item, rather than the entire collection at once.

public function iterator()
{
    // you can filter by attribute
    // $this
    //    ->collection // \Magento\Catalog\Model\ResourceModel\Product\Collection
    //    ->addAttributeToFilter('name', ['like' => '%samsung%']);

    $this
        ->iterator // \Magento\Framework\Model\ResourceModel\Iterator
        ->walk(
            $this->collection->getSelect(),
            [[$this, 'callback']]
        );
}

public function callback($args)
{
    // load of the product
    $product = $this
        ->productRepository // \Magento\Catalog\Api\ProductRepositoryInterface
        ->getById($args['row']['entity_id']);

    // do whatever you want to with the $product
    print_r($product->debug());
}

I also have written a blog post about working with large collections in Magento 2. And here are an example project.

Answer 3

<?php

use Magento\Framework\Model\ResourceModel\Iterator;
use Magento\Framework\Model\ResourceModel\IteratorFactory;
use Magento\Sales\Model\ResourceModel\Order\CollectionFactoryInterface;
use Zend_Db_Select;

class RunSomeLongTask
{
    public const ORDER_STATE_VALUE = 'example';

    /** @var CollectionFactoryInterface  */
    private $orderCollectionFactory;

    /** @var IteratorFactory */
    private $iteratorFactory;

    /** @var SomeTask */
    private $someTask;

    /**
     * constructor.
     * @param CollectionFactoryInterface $orderCollectionFactory
     * @param IteratorFactory $iteratorFactory
     * @param SomeTask $someTask
     */
    public function __construct(
        CollectionFactoryInterface $orderCollectionFactory,
        IteratorFactory $iteratorFactory,
        SomeTask $someTask
    ) {
        $this->orderCollectionFactory = $orderCollectionFactory;
        $this->iteratorFactory = $iteratorFactory;
        $this->someTask = $someTask; // "important: remember create this class"
    }

    public function process(): void
    {
        $orderCollection = $this->orderCollectionFactory->create();

        $orderCollection
            ->getSelect()
            ->reset(Zend_Db_Select::COLUMNS)
            ->columns(['entity_id', 'state', 'status'])
            ->where('state=?', self::ORDER_STATE_VALUE);

        /** @var Iterator $iterator */
        $iterator = $this->iteratorFactory->create();

        $iterator->walk($orderCollection->getSelect(), [[
            $this->someTask, // instance of class SomeTask
            'run' // the function in your class "SomeTask"
        ]]);
    }
}

class SomeTask
{
    public function run(array $args): void
    {
        // you will receive the row data here
        
        $orderId = $args['row']['entity_id'];

        // do your task here on individual row
    }
}

Explained well on http://www.rosenborgsolutions.com/collection-walk-iterator.php

Answer 4

Be careful with Iterator! According to my benchmarks it seems Iterator loads all data at same time! It loads in a flat array, not objects - so it takes less memory.

I prefer old-fashion way to loop the collection - page by page:

    $collection->setPageSize(50);
    $pages = $collection->getLastPageNumber();
    for ($pageNum = 1; $pageNum<=$pages; $pageNum++) {
        $collection->setCurPage($pageNum);
        foreach ($collection as $item) {
            //Do something.
        }
        $collection->clear();
    }

Full example is here: https://magento.stackexchange.com/a/273437/48222

Iterator.Benchmark:

Code

        print_mem();

        $iterator = $this->iteratorFactory->create();
        $iterator->walk($collection->getSelect(), [
            function ($batch) {
                print_mem();
                $this->batchProcessor->executeRow($batch);
            }
        ]);

Output

//Before `Iterator::walk`
^ "The script is now using: 87 403 KB of memory."
^ "Peak usage: 87 943KB of memory."

//Inside `Iterator::walk`
^ "The script is now using: 1 315 666 KB of memory."
^ "Peak usage: 1 315 666 KB of memory."

Collection page-by-page.Benchmark:

Code

        print_mem();

        $collection->setPageSize(10);

        $pages = $collection->getLastPageNumber();
        for ($pageNum = 1; $pageNum<=$pages; $pageNum++) {
            $collection->setCurPage($pageNum);
            foreach ($collection as $batch) {
                print_mem();
                $this->batchProcessor->execute($batch);
            }
            $collection->clear();
        }

Output

//Before
^ "The script is now using: 87 403 KB of memory."
^ "Peak usage: 87 943 KB of memory."

//Inside `foreach(collection)`
^ "The script is now using: 97 013 KB of memory."
^ "Peak usage: 106 350 KB of memory." //an order of magnitude less 

Answer 5

Probably this solution could be usefull in some cases for Magento 2

use Magento\Framework\DB\Query\Generator;
use Magento\Framework\DB\Query\BatchIteratorInterface;
use Magento\Framework\App\ResourceConnection;
use Psr\Log\LoggerInterface;

//class SomeClass ...

private ResourceConnection $resourceConnection;
private Generator $queryGenerator;
private LoggerInterface $logger;
public int $affectedRows = 0;

public function __construct(
    ResourceConnection $connection,
    Generator $queryGenerator,
    LoggerInterface $logger
) {
    $this->resourceConnection = $connection;
    $this->queryGenerator = $queryGenerator;
    $this->logger = $logger;
}

public function getBatches(): \Generator
{
    $connection = $this->resourceConnection->getConnection();
    $select = $connection->select()
        ->from($this->resourceConnection->getTableName('some_table'), ['entity_id'])
        ->where('field = ?', 'some value')
        ->where('some_field IS NOT NULL')
        ->where('some_another_condition', true);

    $field = 'entity_id'; //or could be non unique field of a table depends on a strategy
    $batchSize = 1000;
    $strategy = BatchIteratorInterface::UNIQUE_FIELD_ITERATOR; // or BatchIteratorInterface::NON_UNIQUE_FIELD_ITERATOR;
    
    $iterator = $this->queryGenerator->generate($field, $select, $batchSize, $strategy);

    foreach ($iterator as $query) {
        yield $connection->fetchCol($query); //could be fetchAll or any other method depends on your needs
    }
}

//Example of using

public function execute(): void
{
    $connection = $this->resourceConnection->getConnection();

    try {
        $connection->beginTransaction();
        
        // Here we use our iterator
 
        foreach ($this->getBatches() as $batch) {
  
            //Implementation of logic with our batches
            $this->affectedRows += $connection->update(
                $this->resourceConnection->getTableName('table_name'),
                ['some_column_to_be_clear' => new Zend_Db_Expr('NULL')],
                ['entity_id IN (?)' => $batch]
            );
        }
        
        $connection->commit();
    } catch (Exception $e) {
        $connection->rollBack();
        $this->logger->error($e->getMessage(), $e->getTrace());
        $this->affectedRows = 0;
        throw $e;
    }
}