1.9.0.1 dataflow import customers. Different results when ran from browser or shell

Question

I'm magrating to Magento, and I have a large amount of customers to import. I am using the dataflow profiles because more fields are available. The problem of importing dozens of thousands of customers by running the profile in a popup is that it will time out and fail.

I am then using a script to run a profile from the command line. It works fine and is MUCH faster than using the browser, however, I am facing a strange problem. Running the profile from the browser reads the "is_subscribed" fields and subscribes the customer to the newsletter is the value equals 1, but it won't subscribe the customer if I run the profile from the command line. Everything else imports fine from the command line. All the data is there, except the newsletter subscription which seems to be ignored.

I tried to change the file format of the import from CSV to XML, ran the script using root access, ... nothing, it will never subscribe the customer to the newsletter if running the script from the command line.

Here is a sample of the csv file with one customer that imports fine from a browser but not from the command line : website,email,group_id,firstname,lastname,billing_street1,billing_street2,billing_city,billing_region,billing_country,billing_postcode,billing_telephone,created_in,is_subscribed,group base,[email protected],group01,John,Doe,555 west street,,Los Angeles,CA,US,90001,5551237894,en,1,group01

Any input on what to do/look for ?

Thanks

Code:

<?php
/* ONLY WORK VIA CRON OR SSH: /usr/bin/php -f /var/www/magento/cron_import.php */
/* http://www.magentocommerce.com/boards/viewthread/35865/ */
if ($_SERVER["SERVER_ADDR"] == $_SERVER["REMOTE_ADDR"]) {

  $profileId = 7;
  $logFileName= 'import-profile-7.log';  
  $recordCount = 0;

  require_once 'app/Mage.php';

  Mage::app();

  Mage::log("Starting",null,$logFileName);  

  $profile = Mage::getModel('dataflow/profile');
  $userModel = Mage::getModel('admin/user');
  $userModel->setUserId(0);
  Mage::getSingleton('admin/session')->setUser($userModel);

  if ($profileId) {
    $profile->load($profileId);
    if (!$profile->getId()) {
      Mage::getSingleton('adminhtml/session')->addError('ERROR: Incorrect Profile');
    }
  }

  Mage::register('current_convert_profile', $profile);

  $profile->run();

  $batchModel = Mage::getSingleton('dataflow/batch');
  if ($batchModel->getId()) {
    if ($batchModel->getAdapter()) {
      $batchId = $batchModel->getId(); 
      $batchImportModel = $batchModel->getBatchImportModel();
      $importIds = $batchImportModel->getIdCollection();  
      $batchModel = Mage::getModel('dataflow/batch')->load($batchId);      
      $adapter = Mage::getModel($batchModel->getAdapter());
      foreach ($importIds as $importId) {
        $recordCount++;
        try{
          $batchImportModel->load($importId);
          if (!$batchImportModel->getId()) {
             $errors[] = Mage::helper('dataflow')->__('WARNING: Skip undefined row');
             continue;
          }
          $importData = $batchImportModel->getBatchData();
          try {
            $adapter->saveRow($importData);
          } catch (Exception $e) {
            Mage::log($e->getMessage(),null,$logFileName);          
            continue;
          }      
          if ($recordCount % 50 == 0) {
            Mage::log($recordCount . ' completados',null,$logFileName);
          }
        } catch(Exception $ex) {
          Mage::log('Fila ' . $recordCount . ', SKU ' . $importData['sku']. ' - ERROR: ' . $ex->getMessage(),null,$logFileName);        
        }
      }
      foreach ($profile->getExceptions() as $e) {
        Mage::log($e->getMessage(),null,$logFileName);          
      }
    }
  }
  printf("OK\n");
  Mage::log("Completed",null,$logFileName);
}
?> 

Answer 1

Ok, found the answer after discovering subscribeCustomer().

In the newsletter module we see the customer_save_after event defined for two scopes: frontend and adminhtml. The code tries to mimic this by instantiating an admin session, but from the looks of it, it's not enough for the event to fire.

Aside from some cleaning and type hints, see if the following works for you and carefully read the comment I put in:

<?php
/* ONLY WORK VIA CRON OR SSH: /usr/bin/php -f /var/www/magento/cron_import.php */
/* http://www.magentocommerce.com/boards/viewthread/35865/ */
$profileId = 7;
$logFileName= 'import-profile-7.log';
$recordCount = 0;

require_once '../app/Mage.php';

Mage::app('admin');
Mage::log("Starting",null,$logFileName);

/** @var Mage_Dataflow_Model_Profile $profile */
$profile = Mage::getModel('dataflow/profile');

if ($profileId) {
    $profile->load($profileId);
    if (!$profile->getId()) {
        Mage::throwException('ERROR: Incorrect Profile for id ' . $profileId);
    }
}

Mage::register('current_convert_profile', $profile);

$profile->run();

/** @var Mage_Dataflow_Model_Batch $batchModel */
$batchModel = Mage::getSingleton('dataflow/batch');
if ( ($batchId = $batchModel->getId()) ) {
    if ($batchModel->getAdapter()) {
        /** @var Mage_Dataflow_Model_Batch_Import $batchImportModel */
        $batchImportModel = $batchModel->getBatchImportModel();
        $importIds = $batchImportModel->getIdCollection();
        /** @var Mage_Dataflow_Model_Batch $batchModel */
        $batchModel = Mage::getModel('dataflow/batch')->load($batchId);
        /** @var Mage_Customer_Model_Convert_Adapter_Customer $adapter */
        $adapter = Mage::getModel($batchModel->getAdapter());
        foreach ($importIds as $importId) {
            $recordCount++;
            try{
                $batchImportModel->load($importId);
                if (!$batchImportModel->getId()) {
                    $errors[] = Mage::helper('dataflow')->__('WARNING: Skip undefined row');
                    continue;
                }
                $importData = $batchImportModel->getBatchData();
                try {
                    $adapter->saveRow($importData);
                    if( array_key_exists('is_subscribed', $importData)  )
                    {
                        /** @var Mage_Newsletter_Model_Subscriber $subscription */
                        $subscription = Mage::getModel('newsletter/subscriber');
                        /*
                         * Theoretically this should work, but the code sets
                         * "importMode()" but does not really do anything with
                         * it, so chances are email is sent.
                         * You may want to disable email communications
                         * while running mass imports.
                         */
                        $subscription->subscribeCustomer($adapter->getCustomer());
                    }
                } catch (Exception $e) {
                    Mage::log($e->getMessage(),null,$logFileName);
                    continue;
                }
                if ($recordCount % 50 == 0) {
                    Mage::log($recordCount . ' completados',null,$logFileName);
                }
            } catch(Exception $ex) {
                Mage::log('Fila ' . $recordCount . ', SKU ' . $importData['sku']. ' - ERROR: ' . $ex->getMessage(),null,$logFileName);
            }
        }
        foreach ($profile->getExceptions() as $e) {
            Mage::log($e->getMessage(),null,$logFileName);
        }
    }
}
printf("OK\n");
Mage::log("Completed",null,$logFileName);