2

Magento comes with the Mage_Dataflow_Model_Batch_Abstract class, which has this method:

/**
 * Set batch data
 * automatic convert to serialize data
 *
 * @param mixed $data
 * @return Mage_Dataflow_Model_Batch_Abstract
 */
public function setBatchData($data)
{
    if ('"libiconv"' == ICONV_IMPL) {
        foreach ($data as &$value) {
            $value = iconv('utf-8', 'utf-8//IGNORE', $value);
        }
    }

    $this->setData('batch_data', serialize($data));

    return $this;
}

This abstract class is extended by Mage_Dataflow_Model_Batch_Import (but this method is not changed in that class.)

The problem that I'm having, is that I'm importing from a CSV which is encoded as MS-ANSI. So I have extended Mage_Dataflow_Model_Batch_Import and replaced the setBatchData class to this:

/**
 * Set batch data
 * automatic convert to serialize data
 *
 * @param mixed $data
 * @return Mage_Dataflow_Model_Batch_Abstract
 */
public function setBatchData($data) {
    /*
     * This allows parser classes to explicitly set an input encoding
     * The default is utf-8
     */
    $inputEncoding = $this->getCustomInputEncoding();

    if (!empty($inputEncoding)) {
        if (ICONV_IMPL == '"libiconv"' || ICONV_IMPL == 'glibc') {
            foreach ($data as &$value) {
                $value = iconv($inputEncoding, 'utf-8//IGNORE', $value);
            }
        }

        $this->setData('batch_data', serialize($data));

        return $this;
    }

    return parent::setBatchData($data);
}

The custom input encoding is assigned by the parser class, like this:

$inputEncoding = Mage::getStoreConfig('erp/dataflow_general/input_encoding');    
$batchImportModel = $this->getBatchImportModel()
    ->setId(null)
    ->setBatchId($this->getBatchModel()->getId())
    ->setCustomInputEncoding($inputEncoding)
    ->setBatchData($itemData)
    ->setStatus(1)
    ->save();

This seems to do the trick. The problem is that I'm not sure what all is going on under the hood here. For instance, why is ICONV_IMPL set to '"libiconv"' on my Windows test machine, but set to 'glibc' on my Linux server?

The PHP documentation states that Writing implementation-dependent scripts with these constants is strongly discouraged. Should I not be checking ICONV_IMPL here at all then? Why does Magento check ICONV_IMPL?

Edit: I'm using the system's available encodings to select which encoding it can try to convert from, through a system config setting. The options for that config setting are as follows:

public function toOptionArray() {
        if (empty($this->_options)) {
            $this->_options = array('utf-8' => 'utf-8');

            $charSets = trim(shell_exec('iconv -l | grep -i ansi'));
            if (!empty($charSets)) {
                $charSets = explode("\n", $charSets);
                $charSets = str_replace("//", '', $charSets);

                foreach ($charSets as $charSet) {
                    $this->_options[ $charSet ] = $charSet;
                }
            }

            asort($this->_options);
        }

        return $this->_options;
    }
  • 1
    Short answer, because Windows and Linux use different encoding methods. – Tim Hallman May 20 '15 at 0:57
1

Easiest route: Re-encode your data file with something to change its encoding to UTF-8 and ISO-8859-1 before importing.

Reason(s):

Alternative route: Make sure you are using the right value for encoding and is relative to the host OS (obviously).

CLI example of input values for customInputEncoding:

iconv -l | grep -i ansi

https://stackoverflow.com/a/4595535/158325

If the PHP docs mention it's bad practice, I'd take heed and reconsider my implementation, regardless if I find it in Magento Core.

Related methods: https://php.net/manual/en/function.mb-convert-encoding.php

| improve this answer | |
  • Please see my edit. I'm already using that shell command to create the system config options for converting encodings. If MS-ANSI isn't in that list, it'll just default to utf-8 which is what Magento does by default anyway. - As for Writing implementation-dependent scripts which the docs warn about, I'm not sure that I am, or that I need to. I don't fully understand what all of the options for ICONV_IMPL are; only which options I've come across for this one site. – Tyler V. May 20 '15 at 1:46
  • PS: I looked into mb_convert_encoding and couldn't find a compelling reason why it was any better or worse than iconv, so I used iconv as it's what Magento is already using. – Tyler V. May 20 '15 at 1:51
  • Not sure if this helps but ran into some similar issues with the OSCommerce migration module: github.com/molotovbliss/magento-oscom-migration-tool/commit/… – B00MER May 20 '15 at 3:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.