Collection pagination skipping records

Question

I'm having trouble with Magento's built-in collection paging skipping over many records. I am attempting to optimize a product feed generator that dumps a bunch of product data to a file, and have been banging my head against the wall for several days now trying to figure this one out. The code below works great if I set the page size to a value larger than the number of records returned. In other words, it works if there's only 1/1 pages to process. It does not work if it has multiple pages to process. By "does not work", I mean that many products are skipped or not returned by the Collection. Relevant code is below.

<?php
public function create()
{
    $productCollection = $this->_getProductCollection();
    $pages = $productCollection->getLastPageNumber();
    $pageNumber = 1;
    do {
        $productCollection->setCurPage($pageNumber);
        $productCollection->load();
        foreach ($productCollection as $product) {
            // Process product information and write it to file....
        }
        $pageNumber++;
        $productCollection->clear();

    } while ($pageNumber <= $pages);
}

protected function _getProductCollection()
{
    $adapter = $this->_getReadAdapter();
    $urlPathAttribute = $this->_getAttributeFromCode('url_path');
    $storeId = $adapter->quote($this->_getDefaultStoreId());

    /** @var Mage_Catalog_Model_Resource_Product_Collection $productCollection */
    $productCollection = Mage::getModel('catalog/product')->getResourceCollection();
    $productCollection->addFieldToFilter('status', Mage_Catalog_Model_Product_Status::STATUS_ENABLED)
        ->addAttributeToSort('name', Varien_Data_Collection::SORT_ORDER_ASC)
        ->addAttributeToSelect(array(
                'color', 'description', 'gender', 'height', 'image', 'manufacturer', 'manufacturerspartnumber',
                'manufacturersstylenumber', 'manufacturersupc', 'name', 'outer_material', 'oversize', 'price',
                'primary_color', 'short_description', 'size', 'special_from_date', 'special_price', 'special_to_date',
                'url_path', 'weight', 'width'
            )
        )
        ->joinField('parent_id', 'catalog/product_super_link', 'parent_id', 'product_id=entity_id', null, 'left')
        ->joinField('parent_sku', 'catalog/product', 'sku', 'entity_id=parent_id', null, 'left')
        ->setFlag('require_stock_items', 1)
        ->setPageSize($this->_page_size);

    // Add parent's url_path to child row, since simple child products are not directly visible.
    $select = $productCollection->getSelect();
    $select->joinLeft(array(
        'at_parent_url_path' => $urlPathAttribute->getBackendTable()),
        "(at_parent_id.parent_id = at_parent_url_path.entity_id) AND (at_parent_url_path.attribute_id = {$urlPathAttribute->getId()}) AND (at_parent_url_path.store_id = {$storeId})",
        array('parent_url_path' => 'at_parent_url_path.value')
    );

    return $productCollection;
}

I added a couple of debug lines to the log to illustrate what is happening:

2014-01-13T20:12:56+00:00 DEBUG (7): Started creating feed.
2014-01-13T20:12:56+00:00 DEBUG (7): START Processing product page #1/36. Current memory usage: 122.25 MB
2014-01-13T20:13:03+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:03+00:00 DEBUG (7): END Processing product page #1/36. Current memory usage: 126.75 MB
2014-01-13T20:13:03+00:00 DEBUG (7): START Processing product page #2/36. Current memory usage: 126.75 MB
2014-01-13T20:13:09+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:09+00:00 DEBUG (7): END Processing product page #2/36. Current memory usage: 126.75 MB
2014-01-13T20:13:09+00:00 DEBUG (7): START Processing product page #3/36. Current memory usage: 126.75 MB
2014-01-13T20:13:15+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:15+00:00 DEBUG (7): END Processing product page #3/36. Current memory usage: 126.75 MB
2014-01-13T20:13:15+00:00 DEBUG (7): START Processing product page #4/36. Current memory usage: 126.75 MB
2014-01-13T20:13:21+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:21+00:00 DEBUG (7): END Processing product page #4/36. Current memory usage: 126.75 MB
2014-01-13T20:13:21+00:00 DEBUG (7): START Processing product page #5/36. Current memory usage: 126.75 MB
2014-01-13T20:13:26+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:26+00:00 DEBUG (7): END Processing product page #5/36. Current memory usage: 126.75 MB
2014-01-13T20:13:26+00:00 DEBUG (7): START Processing product page #6/36. Current memory usage: 126.75 MB
2014-01-13T20:13:32+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:32+00:00 DEBUG (7): END Processing product page #6/36. Current memory usage: 126.75 MB
2014-01-13T20:13:32+00:00 DEBUG (7): START Processing product page #7/36. Current memory usage: 126.75 MB
2014-01-13T20:13:38+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:38+00:00 DEBUG (7): END Processing product page #7/36. Current memory usage: 126.75 MB
2014-01-13T20:13:38+00:00 DEBUG (7): START Processing product page #8/36. Current memory usage: 126.75 MB
2014-01-13T20:13:45+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:45+00:00 DEBUG (7): END Processing product page #8/36. Current memory usage: 126.75 MB
2014-01-13T20:13:45+00:00 DEBUG (7): START Processing product page #9/36. Current memory usage: 126.75 MB
2014-01-13T20:13:51+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:51+00:00 DEBUG (7): END Processing product page #9/36. Current memory usage: 126.75 MB
2014-01-13T20:13:51+00:00 DEBUG (7): START Processing product page #10/36. Current memory usage: 126.75 MB
2014-01-13T20:13:57+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:13:57+00:00 DEBUG (7): END Processing product page #10/36. Current memory usage: 126.75 MB
2014-01-13T20:13:57+00:00 DEBUG (7): START Processing product page #11/36. Current memory usage: 126.75 MB
2014-01-13T20:14:03+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:03+00:00 DEBUG (7): END Processing product page #11/36. Current memory usage: 126.75 MB
2014-01-13T20:14:03+00:00 DEBUG (7): START Processing product page #12/36. Current memory usage: 126.75 MB
2014-01-13T20:14:09+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:09+00:00 DEBUG (7): END Processing product page #12/36. Current memory usage: 126.75 MB
2014-01-13T20:14:09+00:00 DEBUG (7): START Processing product page #13/36. Current memory usage: 126.75 MB
2014-01-13T20:14:15+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:15+00:00 DEBUG (7): END Processing product page #13/36. Current memory usage: 126.75 MB
2014-01-13T20:14:15+00:00 DEBUG (7): START Processing product page #14/36. Current memory usage: 126.75 MB
2014-01-13T20:14:22+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:22+00:00 DEBUG (7): END Processing product page #14/36. Current memory usage: 126.75 MB
2014-01-13T20:14:22+00:00 DEBUG (7): START Processing product page #15/36. Current memory usage: 126.75 MB
2014-01-13T20:14:28+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:28+00:00 DEBUG (7): END Processing product page #15/36. Current memory usage: 126.75 MB
2014-01-13T20:14:28+00:00 DEBUG (7): START Processing product page #16/36. Current memory usage: 126.75 MB
2014-01-13T20:14:34+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:34+00:00 DEBUG (7): END Processing product page #16/36. Current memory usage: 126.75 MB
2014-01-13T20:14:34+00:00 DEBUG (7): START Processing product page #17/36. Current memory usage: 126.75 MB
2014-01-13T20:14:40+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:40+00:00 DEBUG (7): END Processing product page #17/36. Current memory usage: 126.75 MB
2014-01-13T20:14:40+00:00 DEBUG (7): START Processing product page #18/36. Current memory usage: 126.75 MB
2014-01-13T20:14:47+00:00 DEBUG (7): Processed 500 product(s).
2014-01-13T20:14:47+00:00 DEBUG (7): END Processing product page #18/36. Current memory usage: 126.75 MB
2014-01-13T20:14:47+00:00 DEBUG (7): START Processing product page #19/36. Current memory usage: 126.75 MB
2014-01-13T20:14:49+00:00 DEBUG (7): Processed 251 product(s).
2014-01-13T20:14:49+00:00 DEBUG (7): END Processing product page #19/36. Current memory usage: 126.75 MB
2014-01-13T20:14:49+00:00 DEBUG (7): START Processing product page #20/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #20/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #21/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #21/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #22/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #22/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #23/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #23/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #24/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:50+00:00 DEBUG (7): END Processing product page #24/36. Current memory usage: 126.75 MB
2014-01-13T20:14:50+00:00 DEBUG (7): START Processing product page #25/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #25/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #26/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #26/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #27/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #27/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #28/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #28/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #29/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:51+00:00 DEBUG (7): END Processing product page #29/36. Current memory usage: 126.75 MB
2014-01-13T20:14:51+00:00 DEBUG (7): START Processing product page #30/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #30/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #31/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #31/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #32/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #32/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #33/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #33/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #34/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #34/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #35/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:52+00:00 DEBUG (7): END Processing product page #35/36. Current memory usage: 126.75 MB
2014-01-13T20:14:52+00:00 DEBUG (7): START Processing product page #36/36. Current memory usage: 126.75 MB
2014-01-13T20:14:53+00:00 DEBUG (7): Processed 0 product(s).
2014-01-13T20:14:53+00:00 DEBUG (7): END Processing product page #36/36. Current memory usage: 126.75 MB
2014-01-13T20:14:53+00:00 DEBUG (7): Finished creating feed.

EDIT: I tried simplifying my collection with the same end result.

protected function _getProductCollection()
{
    /** @var Mage_Catalog_Model_Resource_Product_Collection $productCollection */
    $productCollection = Mage::getModel('catalog/product')->getResourceCollection();
    $productCollection->addFieldToFilter('status', Mage_Catalog_Model_Product_Status::STATUS_ENABLED)
        ->addAttributeToSort('name', Varien_Data_Collection::SORT_ORDER_ASC)
        ->addAttributeToSelect(array(
                'color', 'description', 'gender', 'height', 'image', 'manufacturer', 'manufacturerspartnumber',
                'manufacturersstylenumber', 'manufacturersupc', 'name', 'outer_material', 'oversize', 'price',
                'primary_color', 'short_description', 'size', 'special_from_date', 'special_price', 'special_to_date',
                'url_path', 'weight', 'width'
            )
        )
        ->setPageSize($this->_page_size);

    return $productCollection;
}

I've experienced similar issues when trying to page a collection in memory on ce 1.7.0.2. If I run into them again perhaps I'll dig in and see if I can find what's going on internally. — STW, Commented Dec 8, 2014 at 15:45

jarodh · Accepted Answer · 2014-01-14 17:48:31Z

I was able to workaround this issue by clearing the product collection and recreating my product collection object between each page, and then setting the current page on each new collection.

<?php
public function create()
{
    $productCollection = $this->_getProductCollection();
    $pages = $productCollection->getLastPageNumber();
    $pageNumber = 1;
    do {
        $productCollection->setCurPage($pageNumber);
        $productCollection->load();
        foreach ($productCollection as $product) {
            // Process product information and write it to file....
        }
        $pageNumber++;
        $productCollection->clear();
        $productCollection = $this->_getProductCollection();

    } while ($pageNumber <= $pages);
}

I added the line $productCollection = $this->_getProductCollection(); after clearing the original product collection, and now pagination works as expected. I'm still unsure as to why the original code does not work.

Stack Exchange Network

Collection pagination skipping records

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged
php
collection
ee-1.12
pagination
product-collection
or ask your own question.

Hot Network Questions

Collection pagination skipping records

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged phpcollectionee-1.12paginationproduct-collection or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
php
collection
ee-1.12
pagination
product-collection
or ask your own question.