2

I'm having trouble with Magento's built-in collection paging skipping over many records. I am attempting to optimize a product feed generator that dumps a bunch of product data to a file, and have been banging my head against the wall for several days now trying to figure this one out. The code below works great if I set the page size to a value larger than the number of records returned. In other words, it works if there's only 1/1 pages to process. It does not work if it has multiple pages to process. By "does not work", I mean that many products are skipped or not returned by the Collection. Relevant code is below.

<?php
public function create()
{
    $productCollection = $this->_getProductCollection();
    $pages = $productCollection->getLastPageNumber();
    $pageNumber = 1;
    do {
        $productCollection->setCurPage($pageNumber);
        $productCollection->load();
        foreach ($productCollection as $product) {
            // Process product information and write it to file....
        }
        $pageNumber++;
        $productCollection->clear();

    } while ($pageNumber <= $pages);
}

protected function _getProductCollection()
{
    $adapter = $this->_getReadAdapter();
    $urlPathAttribute = $this->_getAttributeFromCode('url_path');
    $storeId = $adapter->quote($this->_getDefaultStoreId());

    /** @var Mage_Catalog_Model_Resource_Product_Collection $productCollection */
    $productCollection = Mage::getModel('catalog/product')->getResourceCollection();
    $productCollection->addFieldToFilter('status', Mage_Catalog_Model_Product_Status::STATUS_ENABLED)
        ->addAttributeToSort('name', Varien_Data_Collection::SORT_ORDER_ASC)
        ->addAttributeToSelect(array(
                'color', 'description', 'gender', 'height', 'image', 'manufacturer', 'manufacturerspartnumber',
                'manufacturersstylenumber', 'manufacturersupc', 'name', 'outer_material', 'oversize', 'price',
                'primary_color', 'short_description', 'size', 'special_from_date', 'special_price', 'special_to_date',
                'url_path', 'weight', 'width'
            )
        )
        ->joinField('parent_id', 'catalog/product_super_link', 'parent_id', 'product_id=entity_id', null, 'left')
        ->joinField('parent_sku', 'catalog/product', 'sku', 'entity_id=parent_id', null, 'left')
        ->setFlag('require_stock_items', 1)
        ->setPageSize($this->_page_size);

    // Add parent's url_path to child row, since simple child products are not directly visible.
    $select = $productCollection->getSelect();
    $select->joinLeft(array(
        'at_parent_url_path' => $urlPathAttribute->getBackendTable()),
        "(at_parent_id.parent_id = at_parent_url_path.entity_id) AND (at_parent_url_path.attribute_id = {$urlPathAttribute->getId()}) AND (at_parent_url_path.store_id = {$storeId})",
        array('parent_url_path' => 'at_parent_url_path.value')
    );

    return $productCollection;
}

I added a couple of debug lines to the log to illustrate what is happening:

2014-01-13T20:12:56+00:00 DEBUG (7): Started creating feed.
2014-01-13T20:12:56+00:00 DEBUG (7): START Processing product page #1/36. Current memory usage: 122.25 MB
2014-01-13T20:13:03+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:03+00:00 DEBUG (7): END Processing product page #1/36. Current memory usage: 126.75 MB
2014-01-13T20:13:03+00:00 DEBUG (7): START Processing product page #2/36. Current memory usage: 126.75 MB
2014-01-13T20:13:09+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:09+00:00 DEBUG (7): END Processing product page #2/36. Current memory usage: 126.75 MB
2014-01-13T20:13:09+00:00 DEBUG (7): START Processing product page #3/36. Current memory usage: 126.75 MB
2014-01-13T20:13:15+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:15+00:00 DEBUG (7): END Processing product page #3/36. Current memory usage: 126.75 MB
2014-01-13T20:13:15+00:00 DEBUG (7): START Processing product page #4/36. Current memory usage: 126.75 MB
2014-01-13T20:13:21+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:21+00:00 DEBUG (7): END Processing product page #4/36. Current memory usage: 126.75 MB
2014-01-13T20:13:21+00:00 DEBUG (7): START Processing product page #5/36. Current memory usage: 126.75 MB
2014-01-13T20:13:26+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:26+00:00 DEBUG (7): END Processing product page #5/36. Current memory usage: 126.75 MB
2014-01-13T20:13:26+00:00 DEBUG (7): START Processing product page #6/36. Current memory usage: 126.75 MB
2014-01-13T20:13:32+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:32+00:00 DEBUG (7): END Processing product page #6/36. Current memory usage: 126.75 MB
2014-01-13T20:13:32+00:00 DEBUG (7): START Processing product page #7/36. Current memory usage: 126.75 MB
2014-01-13T20:13:38+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:38+00:00 DEBUG (7): END Processing product page #7/36. Current memory usage: 126.75 MB
2014-01-13T20:13:38+00:00 DEBUG (7): START Processing product page #8/36. Current memory usage: 126.75 MB
2014-01-13T20:13:45+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:45+00:00 DEBUG (7): END Processing product page #8/36. Current memory usage: 126.75 MB
2014-01-13T20:13:45+00:00 DEBUG (7): START Processing product page #9/36. Current memory usage: 126.75 MB
2014-01-13T20:13:51+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:51+00:00 DEBUG (7): END Processing product page #9/36. Current memory usage: 126.75 MB
2014-01-13T20:13:51+00:00 DEBUG (7): START Processing product page #10/36. Current memory usage: 126.75 MB
2014-01-13T20:13:57+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:57+00:00 DEBUG (7): END Processing product page #10/36. Current memory usage: 126.75 MB
2014-01-13T20:13:57+00:00 DEBUG (7): START Processing product page #11/36. Current memory usage: 126.75 MB
2014-01-13T20:14:03+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:03+00:00 DEBUG (7): END Processing product page #11/36. Current memory usage: 126.75 MB
2014-01-13T20:14:03+00:00 DEBUG (7): START Processing product page #12/36. Current memory usage: 126.75 MB
2014-01-13T20:14:09+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:09+00:00 DEBUG (7): END Processing product page #12/36. Current memory usage: 126.75 MB
2014-01-13T20:14:09+00:00 DEBUG (7): START Processing product page #13/36. Current memory usage: 126.75 MB
2014-01-13T20:14:15+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:15+00:00 DEBUG (7): END Processing product page #13/36. Current memory usage: 126.75 MB
2014-01-13T20:14:15+00:00 DEBUG (7): START Processing product page #14/36. Current memory usage: 126.75 MB
2014-01-13T20:14:22+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:22+00:00 DEBUG (7): END Processing product page #14/36. Current memory usage: 126.75 MB
2014-01-13T20:14:22+00:00 DEBUG (7): START Processing product page #15/36. Current memory usage: 126.75 MB
2014-01-13T20:14:28+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:28+00:00 DEBUG (7): END Processing product page #15/36. Current memory usage: 126.75 MB
2014-01-13T20:14:28+00:00 DEBUG (7): START Processing product page #16/36. Current memory usage: 126.75 MB
2014-01-13T20:14:34+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:34+00:00 DEBUG (7): END Processing product page #16/36. Current memory usage: 126.75 MB
2014-01-13T20:14:34+00:00 DEBUG (7): START Processing product page #17/36. Current memory usage: 126.75 MB
2014-01-13T20:14:40+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:40+00:00 DEBUG (7): END Processing product page #17/36. Current memory usage: 126.75 MB
2014-01-13T20:14:40+00:00 DEBUG (7): START Processing product page #18/36. Current memory usage: 126.75 MB
2014-01-13T20:14:47+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:47+00:00 DEBUG (7): END Processing product page #18/36. Current memory usage: 126.75 MB
2014-01-13T20:14:47+00:00 DEBUG (7): START Processing product page #19/36. Current memory usage: 126.75 MB
2014-01-13T20:14:49+00:00 DEBUG (7): Processed 251 product(s).
2014-01-13T20:14:49+00:00 DEBUG (7): END Processing product page #19/36. Current memory usage: 126.75 MB
2014-01-13T20:14:49+00:00 DEBUG (7): START Processing product page #20/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #20/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #21/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #21/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #22/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #22/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #23/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #23/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #24/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #24/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #25/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #25/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #26/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #26/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #27/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #27/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #28/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #28/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #29/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #29/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #30/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #30/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #31/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #31/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #32/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #32/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #33/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #33/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #34/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #34/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #35/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #35/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #36/36. Current memory usage: 126.75 MB
2014-01-13T20:14:53+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:53+00:00 DEBUG (7): END Processing product page #36/36. Current memory usage: 126.75 MB
2014-01-13T20:14:53+00:00 DEBUG (7): Finished creating feed.

EDIT: I tried simplifying my collection with the same end result.

protected function _getProductCollection()
{
    /** @var Mage_Catalog_Model_Resource_Product_Collection $productCollection */
    $productCollection = Mage::getModel('catalog/product')->getResourceCollection();
    $productCollection->addFieldToFilter('status', Mage_Catalog_Model_Product_Status::STATUS_ENABLED)
        ->addAttributeToSort('name', Varien_Data_Collection::SORT_ORDER_ASC)
        ->addAttributeToSelect(array(
                'color', 'description', 'gender', 'height', 'image', 'manufacturer', 'manufacturerspartnumber',
                'manufacturersstylenumber', 'manufacturersupc', 'name', 'outer_material', 'oversize', 'price',
                'primary_color', 'short_description', 'size', 'special_from_date', 'special_price', 'special_to_date',
                'url_path', 'weight', 'width'
            )
        )
        ->setPageSize($this->_page_size);

    return $productCollection;
}
  • I've experienced similar issues when trying to page a collection in memory on ce 1.7.0.2. If I run into them again perhaps I'll dig in and see if I can find what's going on internally. – STW Dec 8 '14 at 15:45
2

I was able to workaround this issue by clearing the product collection and recreating my product collection object between each page, and then setting the current page on each new collection.

<?php
public function create()
{
    $productCollection = $this->_getProductCollection();
    $pages = $productCollection->getLastPageNumber();
    $pageNumber = 1;
    do {
        $productCollection->setCurPage($pageNumber);
        $productCollection->load();
        foreach ($productCollection as $product) {
            // Process product information and write it to file....
        }
        $pageNumber++;
        $productCollection->clear();
        $productCollection = $this->_getProductCollection();

    } while ($pageNumber <= $pages);
}

I added the line $productCollection = $this->_getProductCollection(); after clearing the original product collection, and now pagination works as expected. I'm still unsure as to why the original code does not work.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.