0

I am writing an admin module. I created a collection and if the number of rows in the table are small it loads fine. Ultimately the table will have 250k rows in it and when I attempt to load the collection it exhausts memory. I was under the impression that Magento would lazy load collections. I also will probably need to know how to use the resource iterator in mage 2 as well. I can provide code samples but there is little more than a basic module skeleton at this point. I would greatly appreciate a code example of how to handle and iterate through a large collection.

Note: the collection is pulling from a custom table

Controller:

<?php
namespace Customer\Customerimportexport\Controller\Adminhtml\Import;

use Magento\Backend\App\Action;
use Magento\Backend\App\Action\Context;
use Customer\Customerimportexport\Model\WebcustomersFactory;

class Index extends \Magento\Backend\App\Action
{

    protected $_modelWebcustomersFactory;

    public function __construct(
        Context $context,
        WebcustomersFactory $modelWebcustomersFactory
    ) {
        parent::__construct($context);
        $this->_modelWebcustomersFactory = $modelWebcustomersFactory;
    }

    public function execute()
    {

        $webcustomersModel = $this->_modelWebcustomersFactory->create();

        // Get webcustomers collection
        $webcustomersCollection = $webcustomersModel->getCollection();

        // Load all data of collection
        foreach ($webcustomersModel->getCollection() as $key => $webcustomer) {
            var_dump($webcustomer->getData());
            die();
        }

        $this->_view->loadLayout();
        $this->_view->renderLayout();
    }

}

Model

<?php

namespace Customer\Customerimportexport\Model;

use \Magento\Framework\Model\AbstractModel;

class Webcustomers extends AbstractModel
{
    /**
     * Define resource model
     */
    protected function _construct()
    {
        $this->_init('Customer\Customerimportexport\Model\ResourceModel\Webcustomers');
    }
}

Resource Model

<?php

namespace Customer\Customerimportexport\Model\ResourceModel;

class Webcustomers extends \Magento\Framework\Model\ResourceModel\Db\AbstractDb
{
    /**
     * Define main table
     */
    protected function _construct()
    {
        $this->_init('web_customers', 'entity_id');
    }
}

Collection

<?php 

namespace Customer\Customerimportexport\Model\ResourceModel\Webcustomers;

use \Magento\Framework\Model\Resource\Db\Collection\AbstractCollection;

class Collection extends \Magento\Framework\Model\ResourceModel\Db\Collection\AbstractCollection
{
    /**
     * Define model & resource model
     */
    protected function _construct()
    {
        $this->_init(
            'Customer\Customerimportexport\Model\Webcustomers',
            'Customer\Customerimportexport\Model\ResourceModel\Webcustomers'
        );
    }
}
  • Did you already tried with repositories instead of collections? – Phoenix128_RiccardoT May 31 '16 at 17:21
  • Post a sample of the code and we can see if there is anything going on. – Ryan Street May 31 '16 at 17:21
  • @RyanStreet I posted the controller, model, resource model and collection. I get memory exhausted at the var_dump. – Michael Sole May 31 '16 at 17:26
  • @RiccardoT I am looking into it now, I never heard of it before. – Michael Sole May 31 '16 at 17:27
  • Well.. you updated your code. IMHO your problem is the var_dump itself – Phoenix128_RiccardoT May 31 '16 at 17:42
1

This is probably your culprit: var_dump($webcustomersCollection->count());

That loads the entire collection (of all records in the table) to get the count. The loop does the same thing.

Apply some filters to your collection before looping.

There is some more that needs to be cleaned up with this code, (dependency injection, rendering output, etc.), but that's an answer for a different question.

  • 1
    Also, if you need to simply get the count, at least in M1 (don't know about M2), there was a way to get special count SQL instead of loading the entire collection. – Kevin Schroeder May 31 '16 at 17:31
  • @KevinSchroeder, correct, (it is getSize()) and it is present in M2 :) – Ryan Street May 31 '16 at 17:33
  • Actually I only had the line there as I was trying to figure out if I could limit/paginate through the results but I had the problem before adding it – Michael Sole May 31 '16 at 17:33
  • Note: even if I limit the collection - ->setPageSize(20)->setCurPage(1); I still run out of memory. The only way it works is if I limit (e.g. delete) the number of rows in the table down. – Michael Sole May 31 '16 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.