How to handle Magento large customer records

Question

In my Magento customer have 4.0 laksh records,i want to list the all customer in custom admin form select field

Field

$fieldset->addField('user_id', 'select', array(
            'type' => 'int',
            'label'     => Mage::helper('catalog')->__('Lawyer'),
            'class'     => 'required-entry',
            'required'  => false,
            'name'      => 'user_id',
            'values'    =>$this->LawyerCollection(),
        ));

My Query:

public function LawyerCollection() {
    $collection = Mage::getModel('customer/customer')->getCollection()->addAttributeToSelect('firstname')->addAttributeToFilter('group_id',array('eq'=>5));
        $user = array();
        if($collection->getSize()) {
            foreach($collection as $_collection) {
               $user[$_collection->getId()] = $_collection->getFirstname();
            }
         }
       return $user
}

But the result is an empty page, i hope this is because of the 4.0 record, i tested with with 1000 records it was working fine,now it is not working.

My idea is, i like to do the field with "auto complete", but i don't have any idea to implement the auto suggestion in the admin custom field.

if you have any other idea means kindly let me know,thanks in advance for your answer.

Answer 1

You have run into a very common issue.

Basically your collection is to large, and you are most likely running out of memory. This is a hard fail by PHP, which results in the blank screen

You don't mention thst you looked at the logs. This should be your first point of call when debugging.

Google for help on how to deal with large collections. They usually happen in catalog collections, but the principal will be the same.

Also eliminate the conditional checking the collection size, before you iterate over it. It is pointless.

Good luck.

Answer 2

I did as per the below way to handle the large collection, but still the application is not much speed.

public function LawyerCollection() {
        $collection = Mage::getModel('customer/customer')->getCollection()->addAttributeToSelect(array('userid','firstname'));
        $collection
                ->joinAttribute('firstname', 'customer/firstname', 'entity_id', null, 'left');
        if($collection->getSize()) {
            Mage::getSingleton('core/resource_iterator')->walk(
                $collection->getSelect(),
                array(array($this,'customerCallback'))
            );
        }
        return $this->user;
    }


    public function customerCallback($args)
    {
        $_collection = Mage::getModel('customer/customer');
        $_collection->setData($args['row']);
        $this->user[$_collection->getId()] = $_collection->getFirstname();
    } 

then see the below query, it's only took 0.0013 sec in PHPMYADMIN . query :

SELECT `e`.*, `at_firstname`.`value` AS `firstname` FROM `customer_entity` AS `e` LEFT JOIN `customer_entity_varchar` AS `at_firstname` ON (`at_firstname`.`entity_id` = `e`.`entity_id`) AND (`at_firstname`.`attribute_id` = '5') WHERE (`e`.`entity_type_id` = '1')

so i want to improve the customer large collection load, how to optimize the query or have other way to optimize the magento customer tables , if any idea will help me much more.