Take the 2-minute tour ×
Magento Stack Exchange is a question and answer site for users of the Magento e-Commerce platform. It's 100% free, no registration required.

I'm trying to edit customer data in bulk (a couple of thousand records) and Magento keeps running out of available memory to run the script.

After some testing with memory_get_usage() the culprit seems to be the $customer->save() method which seems to take 5M of memory for each save but doesn't release it when done.

As a result, when looping through a couple of thousand records it is running of memory.

Here is what I've tried so far:

$customer->clearInstance()

unset($customer)

This doesn't seem to help though.

Below is my code:

public function createCustomerAddress($customerAddressData, $email){
     $customer = Mage::getModel('customer/customer');

     $customer->setWebsiteId(Mage::app()->getWebsite()->getId());
     $customer->loadByEmail($email);

     $address   = Mage::getModel('customer/address');

     $address->addData($customerAddressData);
     $customer->addAddress($address);
     unset($address);
     try{
         $customer->save();
     }catch (Exception $e){
         var_dump($customerAddressData);
         var_dump($e->getMessage());
     }
    echo "\n" . "Before unsetting \n" . memory_get_usage() . "\n";
    $customer->clearInstance();
    unset($customer);
    echo "\n" . "After  \n" . memory_get_usage() . "\n";  // no difference than before
}

Any help would be greatly appreciated.

p.s. I'm not sure if the clearInstance() function (in the Mage_Core_Model_Abstract) does anything, if someone has some insight into this it would be greatly appreciated if its shared :)

share|improve this question
    
What version of Magento are you using? –  JesseC Mar 6 at 15:37
    
Magento CE 1.7.0.2, but also testing it on Magento EE same issue –  pzirkind Mar 6 at 15:39
1  
Instead of the workarounds provided as answers, it might be possible to track down the actual memory leak if you have time. Please see the two articles I provide. The first can help you track down the issue, the second explains a bug in PHP itself. It's possible that the same issue is present in Magento. stackoverflow.com/questions/849549/… paul-m-jones.com/archives/262 –  JesseC Mar 6 at 16:09
    
what php version are you using? –  Flyingmana Mar 11 at 23:09
    
@Flyingmana 5.4.17 on dev machine and 5.3 on production –  pzirkind Mar 13 at 15:38
add comment

4 Answers 4

up vote 1 down vote accepted

I think that this is related to a core PHP issue, which is not easily resolved without extensive modifications to core Magento files.

The only way to get this done is using a workaround:

I propose:

a. If this is being called from the browser, I would do a separate ajax call, as @Julien Lacal recommended in his answer.

b. If this is all being done server side, I would break this down to two scripts, with 1. one that iterates over the existing customer list, and 2. one which calls the createCustomerAddress function, using post params. The second script should then be called using CURL from the first script, so that each instance of the second script runs in isolation, and is not affected by the memory leak.

share|improve this answer
add comment

Memory is only released once there are no references to that variable/object. I suspect -- and will edit my answer once confirmed, that the event system in Magento passes an instance of the model to the observer, and therefore there is a reference to it in memory.

But this is a well documented issue. (one which I commented on in 2011 http://www.magentocommerce.com/boards/viewthread/26561/).

share|improve this answer
    
Definitely makes sense, however the observer should release it at some point –  pzirkind Mar 6 at 18:50
    
afaik does the standard php garbage collector NOT find circles, therefore it might be, that if you have a customer and an address, both objects are not freed - just an idea, thanks for the answer joseph! –  Fabian Blechschmidt Mar 6 at 21:35
    
Sounds plausible - at a quick glance only the newsletter module seems to observe the customer save event. @pzirkind do you see the same memory consumption with the newsletter extension fully disabled? –  Fooman Mar 7 at 6:19
    
@Fooman yes it seems to have the same memory consumption when the newsletter module is disabled (via xml) –  pzirkind Mar 7 at 16:43
add comment

I believe this is a known issue across all platforms and versions of Magento. unset() or clearInstance() will not have any effect. Mage::getModel() leaked memory in all instances I've come across (customer, products, orders, etc.). The only way around that I have found is to use a collection instead, if possible (may not be applicable for you), when dealing with a large number of object creations. When your process is complete, the memory will be released.

Are you running out of memory? This is usually not an issue unless you are. If you are running out of memory, perhaps, you can try to break up the bulk process into separate runs in your extension. If this is in a shell script, then it would be easy to break it up. Just create two or more files and run them one after another with predefined ranges of customer entity_ids.

share|improve this answer
    
Thanks for the info will look into using collections for this, was wondering if there was some type of fix available for the model leak though –  pzirkind Mar 6 at 17:49
    
tried saving a collection using $collection->save() and seems to have the same memory leak, is there some code you can post where i could see the difference in memory? –  pzirkind Mar 6 at 19:24
add comment

A good workaround would be to build your own admin module for bulk customer import, and have it use Ajax calls to a controller to import data in smaller sets. This way you won't run out of memory because it'll launch a new process each time you make that Ajax call instead of constantly allocating memory because of the Mage::getModel('customer/address')

share|improve this answer
    
That is an interesting workaround (+1 for the idea), but i want something more consistent for the backend –  pzirkind Mar 6 at 17:48
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.