2

I'm using the following to run a direct insert query in a Magento Shell script.

Mage::getSingleton('core/resource')->getConnection('core_read')->query($q)

One of the string values goes into a text column, and when this string exceeds about 7000 bytes, PHP crashes and a Segmentation fault (core dumped) error occurs.

The 7000 bytes figure is an approximate string size as I was truncating the string in blocks to see at what size an error occurs.

Does anyone have a clue why this happens and how to fix it? The text column accepts much more than 7000 bytes and when I run $q in MySQLyog, it inserts just fine.

The string in question is something like... (array of image URLs imploded with a ,)

http://www.mysite.com/media/catalog/product/cache/0/small_image/135x/images/catalog/product/placeholder/small_image.jpg,http://www.mysite.com/media/catalog/product/cache/0/small_image/135x/images/catalog/product/placeholder/small_image.jpg,http://www.mysite.com/media/catalog/product/cache/0/small_image/135x/images/catalog/product/placeholder/small_image.jpg,http://www.mysite.com/media/catalog/product/cache/0/small_image/135x/images/catalog/product/placeholder/small_image.jpg,... (just repeated ~40 times)

And each of these image URL are obtained via (pesudo code)

foreach list of products {
    $product= Mage::getModel('catalog/product')->load($id);    
    $myImgUrl[] = (string)Mage::helper('catalog/image')->init($product, 'small_image')->resize(135);
}

EDIT: I'd like to add that segmentation fault error occurs whether that particular query is run on the x-th iteration or 1st iteration of the foreach loop.

I also see that at the end of my shell script, memory_get_usage() is over 670 MB.. I see the usage go up at every iteration starting from about 16MB before the first iteration. I am using the same variables, which are used to write to the DB, at every loop.

Update:

I managed to avoid this problem something like below instead of ...->load($id). Fooman provided some insight into this problem in his answer in this thread [Link];

Mage::getModel('catalog/product')->getCollection() ->addAttributeToSelect('my_atts') ->addAttributeToFilter('entity_id', $id) ->getFirstItem();

I understand. Your gist was very helpful. Although it's something I've done before, I didn't realize the memory impact. I replaced all of the ...->load() with your recommendation, ...->getCollection() and everything is well. Memory usage is constant around 16MB and noticeably faster.

  • Just an idea: Did you check your PHP/MySQL settings? Maybe you exceed a limit form there, for ex. max_allowed_packet or something like this? – Anna Völkl Sep 11 '13 at 20:05
  • Could you elaborate? max_allowed_packet is set to 943,718,400 right now. – musicliftsme Sep 11 '13 at 21:13
  • Are you calling INSERT from a core_read connection?? – philwinkle Sep 11 '13 at 22:05
  • Nope, I have a write connection (...->getConnection('core_write')) for INSERT and UPDATE – musicliftsme Sep 11 '13 at 22:10
  • I made another post regarding memory usage with more details, as I felt like it a little off topic. (let me know if it should be the same). In this example, I am not facing the seg. error as I'm keeping the problematic string shorter intentionally.magento.stackexchange.com/questions/7888/… – musicliftsme Sep 11 '13 at 22:11
3

As suggested by the below SO thread you may have run out of memory if you are (as I suspect) running this from the CLI.

To increase your PHP CLI memory_limit to the minimum recommended 256MB, you must first locate the correct INI file loaded for the CLI. To do so:

$ php -i | grep "Loaded Configuration"

Which will return something akin to:

Loaded Configuration File => /usr/local/etc/php/5.4/php.ini

Then you will edit the php.ini file at that location and increase memory_limit to a minimum 256MB.

After you save and close you can check that it's been updated with:

$ php -i | grep memory_limit

In all luck it should return:

memory_limit => 256M => 256M

Parting thoughts:

This isn't necessarily Magento's (or even PHP's) fault. Magento handles "direct" inserts of many many hundreds of kilobytes larger than your script. I suspect there are other factors at play, like how you're deriving the URLs in the first place. If you're doing Mage::getModel('catalog/product')->load() in a loop, for instance.

In short, you may have created some memory leaks, instantiated objects unnecessarily, loaded massive objects or collections from the db unnecessarily, and coded your way into this position.

Best of luck.


Sources:

https://stackoverflow.com/questions/12191996/php-in-commandline-segmentation-fault-core-dumped-debug-while-running-phpi

https://stackoverflow.com/questions/15714909/php-what-does-segmentation-fault-core-dumped-error-means

| improve this answer | |
  • For CLI, memory limit is -1 (test server).. I am doing Mage::getModel('catalog/product')->load() in a loop but but this error does occur with just one loop with that string in $q in the example. Right before it crashes, memory_get_usage() returns 18,567,280 bytes.. With memory limit at -1, am I reading this returned value wrong? I am also re-using same variable names in the loop.. hmmmm – musicliftsme Sep 11 '13 at 21:10
  • Your update provides some insight. First of all, you're causing it to make new images in a loop and then casting the object to a string. This seems heavy. Rather just get the image url from the helper in the gallery. In addition, you don't need to load a new product model, rather load the product collection by the Ids. See here gist.github.com/philwinkle/6530362 – philwinkle Sep 11 '13 at 22:00
  • Regarding 1st point, for some reason, the helper returns a string with EOL a few characters at the end. Reagrding 2nd point, I'll implement that. I will have to make some other changes as some functions were depent on the product model. – musicliftsme Sep 11 '13 at 22:26
  • You have the product model in the loop. $product IS the product model. If you see my update to the gist, you'll see that I have a test - if you create mytable and a url column it'll easily do a 7000 row insert. Try that as a barometer to see if as you suggest you cannot "insert over 700kb" – philwinkle Sep 11 '13 at 22:27
  • 1
    I understand. Your gist was very helpful. Although it's something I've done before, I didn't realize the memory impact. I replaced all of the ...->load() with your recommendation, ...->getCollection() and everything is well. Memory usage is constant around 16MB and noticeably faster. – musicliftsme Sep 11 '13 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.