Bad performance looping through all products

Question

I need to loop through all products to build a xml - file but my script ceeps running for hours, and with every loop it seems getting slower. What can i do to speed it up ? First i loop through the category tree, when i find a "leaf" (endpoint) of a category, i try to get all of it's products with the following script:

$collection = Mage::getModel('catalog/category')
        ->load($catId)
        ->getProductCollection()
        ->addAttributeToSelect('*')
        ->addAttributeToFilter('category_id', $catId)
        ->addAttributeToFilter('type_id', 'simple')
        ->addAttributeToFilter('status', 1)->load();

Consideration Nr. 1:
Mage::getModel('catalog/product') in the foreach-loop:

      foreach($collection as $col) {
$prod = Mage::getModel('catalog/product')->load($col->getId());
    }

But the performance is horrible

Consideration Nr. 2: ceeping the Mage::getModel('...') out of the loop by just loading a new id inside the loop

$prod = Mage::getModel('catalog/product');
foreach($collection as $col) {
$prod->load($col->getId());
}

The performance is much better but some product values are alway the same so i think this solution wound't work ?

The best way would be using the $collection data itself but i'miss some important data in the $collection.

How can i get all attributest of a product in a collection when i try to get all attributes by using the $collection i get this:

foreach($collection as $col) {
    echo "<pre>";

    print_r(getAttributes($col));

    echo "</pre>";

    exit;
}

function getAttributes($prod) {
    $attributes = $prod->getAttributes();

    foreach ($attributes as $attribute) {
        if ($attribute->getIsVisibleOnFront()) {
            $value = $attribute->getFrontend()->getValue($prod);
            echo "The Value " . $value;
            if($value=='keine Angabe' || !$value)
                continue;
            if ($attribute->getFrontendInput() == 'price' && is_string($value)) {
                $value = Mage::app()->getStore()->convertPrice($value, true);
            }

            if (is_string($value) && strlen($value)) {
                $data[$attribute->getAttributeCode()] = array(
                    'label' => $attribute->getStoreLabel(),
                    'value' => $value,
                    'code'  => $attribute->getAttributeCode()
                );
            }
        }
    }
    return $data;
}

8 [color] => Array ( [label] => color [value] => No [code] => color )

But when i use the getModel i get the correct value

$_product = Mage::getModel('catalog/product')->load($col->getId());
print_r(getAttributes($_product));

[color] => Array ( [label] => color [value] => white [code] => color )

Is there any way to get all attributest by using the collection ?

Thx

Answer 1

Product Collection from Category

// Step 1
$category = Mage::getModel('catalog/category')->load($category_id);

// Step 2
$productCollection = Mage::getModel('catalog/product')->getCollection();

// Step 3
$productCollection->addCategoryFilter($category);

// Step 4
$productCollection
    ->addAttributeToSelect('sku')
    ->addAttributeToSelect('another_attribute')
    ->addAttributeToFilter('type_id', 'simple')
    ->addAttributeToFilter('status', 1);

// Now you can loop through your collection
foreach($productCollection as $product) {
    $product->getSku();
    $product->getAnotherAttribute();
    // and so on...
}

So what's going on?

1. We loaded a category model. We will use that in the product collection.
2. We instantiated the product collection model.
3. We added the category for filtering.
4. We added the appropriate attributes for selection and filtering.

NOTE: Do not use the asterisk unless you plan on using every attribute being returned. Just like you shouldn't use the asterisk in a MySQL query unless you plan on getting everything from that table.

Once I have all my attributes and filters in place, I can loop through my collection. I do not need to call the load() method because lazy loading does that for me.

Once I'm in the loop, I have access to the product object Mage_Catalog_Model_Product. I do not need to instantiate another model inside the loop. The collection does this for me.

Answer 2

Can't you just use the flat tables and read from those instead of loading every product. 'catalog/category_flat' rather than catalog/category is going to make the difference here since you are just reading.

Answer 3

To access all products limiting consumption of time and memory, the following piece of code can be used:

$products = Mage::getModel('catalog/product')
    ->getCollection()
    ->addAttributeToSelect('*')
;
while ($product = $products->fetchItem()) {
  // do something
}

Pay attention: with flat catalog enabled, the selected attributes will only be the ones which satisfy the conditions defined in Mage_Catalog_Model_Product_Indexer::matchEvent() method.

To avoid performance bottleneck you have to write scalable code; to get the point I suggest you to read the following article by Vinai Kopp: http://vinaikopp.com/2014/06/09/preloading-for-scalabiliy/

Hope it helps.