Benefits of Mage::getSingleton('core/resource_iterator')

Question

I am dealing with huge collections of products and categories right now and am looking for the most efficient way of handling large collections.

I stumbled upon this: Mage_Core_Model_Resource_Iterator and have read that this should be used.

Can anyone point out why I would use this over setting the pageSize on a collection?

And also, what is the preferred way of dealing with a massive product collection.

The exact scenario is that I have 120,000 products + 10 websites and 37 stores and I need to get the urls and other bits of data from each ready for a custom csv export.

I am currently looping loading the store collecton, and the for each store loading the product collection and looping over each.

Answer 1

When you use a foreach to loop through the collection Magento first executes the query used for that collection, retrieves all results (storing it in memory) and then let's you iterate it. Like so:

$collection = Mage::getModel('catalog/product')->getCollection();
foreach ($collection as $_item)
{
   // your code
}

Setting the pageSize on the collection will limit the results retrieved but means the query will be executed multiple times (from what I understand) so that's still a performance issue.

When using the iterator the query is executed but the results are only fetched per row for each iteration using far less memory to store just the data for that iteration.

Also make sure you select only what you're really using in the CSV, so instead of doing ->addAttributeToSelect('*') specify ->addAttributeToSelect('name'), ->addAttributeToSelect('image'), etc.

And last but not least: please don't load the product model for each iteration. Doing something like the next code block will impact performance big time in a negative way

$collection = Mage::getModel('catalog/product')->getCollection();
foreach ($collection as $_item)
{
   $product = Mage::getModel('catalog/product')->load($_item->getId());
   // rest of code
}

If you need data not given in the original collection look into joining the table that contains that data to the collection you're already using.