I am trying to run a password hash to change user passwords from MD5 to SHA256 following a migration from Magento 1 to Magento 2.
The problem is, I have around 8 million customer records. I have ensured that the hardware is scaled right up, currently using 8 cores and 60gb RAM which is about the limit I can do.
The process starts off ok running about 1000 records in 10 seconds but over time it grinds down to about 1 min per 1000 records.
Based on rough calculations it will take weeks to run which isn't viable for us.
I have also tried to add some filters to the collection so that it runs in batches and checks the hash version before hashing so it doesn't redo previously hashed items.
I was wondering if there are any tips for optimising this core method to ensure it can run in a decent time?
See below for the modified method that I am using:
protected function configure()
{
$this->setName('customer:hash:upgrade')
->setDescription('Upgrade customer\'s hash according to the latest algorithm');
protected function execute(InputInterface $input, OutputInterface $output)
{
$this->collection = $this->customerCollectionFactory->create();
$this->collection->addAttributeToSelect('password_hash');
$this->collection->addAttributeToSelect('entity_id');
$this->collection->setPageSize(1000);
$this->collection->getSelect()->where('password_hash NOT LIKE "%0:1"');
$pages = $this->collection->getLastPageNumber();
$currentPage = 1;
echo $this->collection->getPasswordHash();
do {
$output->writeln("starting batch of 1000 customers");
$time_start = microtime(true);
$this->collection->setCurPage($currentPage);
$this->collection->load();
$this->collection->getSelect()->where('password_hash NOT LIKE "%0:1"');
$customerCollection = $this->collection->getItems();
/** @var $customer Customer */
foreach ($customerCollection as $customer) {
$customer->load($customer->getId());
if (!$this->encryptor->validateHashVersion($customer->getPasswordHash())) {
list($hash, $salt, $version) = explode(Encryptor::DELIMITER, $customer->getPasswordHash(), 3);
if ($version !== '0') {
$version .= Encryptor::DELIMITER . Encryptor::HASH_VERSION_LATEST;
$customer->setPasswordHash($this->encryptor->getHash($hash, $salt, $version));
$customer->save();
$output->write(".");
}
}
}
$output->writeln('last id: '.$customer->getId(). "");
$currentPage++;
//clear collection and free memory
$this->collection->clear();
$time_end = microtime(true);
//dividing with 60 will give the execution time in minutes otherwise seconds
$execution_time = ($time_end - $time_start) / 60;
$output->writeln("1000 customers comepleted in " . $execution_time . " min");
}
while ($currentPage <= $pages && ((1000*$currentPage) > $stopLimit || $stopLimit));
$output->writeln(".");
$output->writeln("<info>Finished</info>");
}
}
