How to get large collection of products?

Question

I have a store with over 100K products. I am using this code to get all my products.

$model = Mage::getModel('catalog/product')->getCollection();
foreach ($model  as $product) {
     $p = Mage::getModel('catalog/product')->load($product->getId());
     echo $p->getName();
     echo $p->getDescription();
     echo $p->shortDescription();
}

But it shows nothing. This code works where i have only less number of products in my store. So how to get all my 100K products? Thank you

Answer 1

$model = Mage::getModel('catalog/product')->getCollection()
    ->addAttributeToSelect(array('name', 'descryption','short_descryption'))
    ->setPageSize(1000);
$numberOfPages = $model->getLastPageNumber();

for ($i=1; $i<=$numberOfPages; $i++) {
    $products = $model->setCurPage($i)->load();
    foreach ($products as $product) {
        echo $product->getName() . '<br />'; 
        echo $product->getDescription() . '<br />'; 
        echo $product->shortDescription() . '<br />';
    }
}

Answer 2

There are several ways you can handle loading large collection, take a look at Magento: Working with large collection. Take a look at Mage::getSingleton('core/resource_iterator')->walk(), base on what your trying to accomplish I would mysql/query.

public function resourceIteratorMethod()
{
    $productCollections = Mage::getModel('catalog/product')->getCollection();

    Mage::getSingleton('core/resource_iterator')->walk(
    $productCollections->getSelect(),
    array(array($this,'productCallback'))
    );
}
public function productCallback($args)
{
    $cust = Mage::getModel('catalog/product');
    $cust->setData($args['row']);

    echo $cust->getName()."\n";
}

Answer 3

If by "shows nothing" you mean to say that the Web browser simply returns a blank page, you should know that PHP needs to hold all of $model in memory. You then proceed to loop through it 100k times and call load on every single one of them, which fires off more requests to the database. That requires a massive amount of memory. Increase your memory. Also, be aware that code like this will cripple your store the moment you get just a few people browsing concurrently, unless you plan to install 1TB memory.

Answer 4

If you really want to load all products with all information in one run without using up all memory you could do something like

$productIds = Mage::getModel('catalog/product')->getCollection()
        ->getAllIdsCache();
// Returns a array with 100k ids

foreach ($productIds as $productId) {
    // Load one product
    $product = Mage::getModel()->load($productId);
    /** @var $product \Mage_Catalog_Model_Product */

    echo $product->getName();
    echo $product->getDescription();
    echo $product->getShortDescription();

    $product->clearIntance();
}

If you know which attributes to get you could also just add those to your filter and iterate that.

$products = Mage::getModel('catalog/product')->getCollection()
        ->addAttributeToSelect('name')
        ->addAttributeToSelect('description')
        ->addAttributeToSelect('short_description');

foreach ($products as $product) {
    /** @var $product \Mage_Catalog_Model_Product */

    echo $product->getName();
    echo $product->getDescription();
    echo $product->getShortDescription();
}

The first one would take less memory than the second one because it wont hold old information in memory.