Magento 2: Migrate changes made since last migration using non-live DB?

Question

When we started development of our M2 project, we migrated all data to the new M2 project from M1. We did this using the Data Migration Tool. Because we did not want to mess with the live database, we first made a copy of this database and used this copy to migrate our data to M2.

This has worked perfectly.

Now we want to migrate all changes made to the live M1 database since we started development on M2. But now I saw that in order to migrate the delta from the old M1 database Magento added some tables to that database when we migrated it (according to https://devdocs.magento.com/guides/v2.4/migration/migration-migrate-delta.html). However, since we migrated from a copy and not the live M1 database, no changes were tracked.

Is there a way to migrate changed data from M1 to M2 without Magento having kept track of changes in the M1 database?

If I get I right you just want to migrate things that were changed during the development time of M2 right, most likely orders and costumers that were added to M1? — Rui Silva, Commented Aug 31, 2020 at 16:27
@RuiSilva that is exactly right. However, I now figure out that this is not really possible with Magento out of the box, as I can not use the "delta" mode anymore. Here is another a bit hacky approach I am using right now: czettner.com/2018/09/13/magento-2-data-migration.html. — Louis B., Commented Aug 31, 2020 at 16:49
I had to do that, if you only need customers and orders, the way I did it was deleting all orders and customers from M2, with the migration tool ignore all the other info and then import just that, orders and customers. Was the best way I found, but you will lost all the orders and customers created in M2, idk if that's a problem, but yeah it's kind of that way in that post, I can send you the files map and config I used, but you might need to adjust it a bit for your needs — Rui Silva, Commented Aug 31, 2020 at 17:20
@RuiSilva: So you deleted all orders and customers (via the DB?) and then set up the map.xml to ignore everything but the order and customer related data, and then re-ran the full migration? That is kind of what I figured I would have to do as well. If you could post your map.xml and config.xml that'd be great. You can also add it as an answer and I'll accept it so you'll get internet points. Thanks! — Louis B., Commented Sep 1, 2020 at 6:44
They are big files, can't upload them here, I can upload the config but the maps are massive — Rui Silva, Commented Sep 1, 2020 at 9:59

Rui Silva · Accepted Answer · 2020-09-03 13:54:32Z

Kinda big, but I did it in two different imports

First deleted all orders and customers with the queries in this question https://github.com/magento/data-migration-tool/issues/467#issuecomment-627912182

Then do a map.xml, this might depend on your tables, I had to ignore extensions tables and other ones, was a trial error, backup your database and try, if something is wrong go back and check where it is on database then add that table to ignore, in that last answer on git a posted a bit but too much lines to post it here.

To import customers

<?xml version="1.0" encoding="UTF-8"?>

<steps mode="data">
    <!-- <step title="Data Integrity Step">
        <integrity>Migration\Step\DataIntegrity\Integrity</integrity>
    </step>
   <step title="EAV Step">
        <integrity>Migration\Step\Eav\Integrity</integrity>
        <data>Migration\Step\Eav\Data</data>
        <volume>Migration\Step\Eav\Volume</volume>
    </step>  -->
    <step title="Customer Attributes Step">
        <integrity>Migration\Step\Customer\Integrity</integrity>
        <data>Migration\Step\Customer\Data</data>
        <volume>Migration\Step\Customer\Volume</volume>
    </step>
    <!-- <step title="Map Step">
        <integrity>Migration\Step\Map\Integrity</integrity>
        <data>Migration\Step\Map\Data</data>
        <volume>Migration\Step\Map\Volume</volume>
    </step>
   <step title="Url Rewrite Step">
        <integrity>Migration\Step\UrlRewrite\Version191to2000</integrity>
        <data>Migration\Step\UrlRewrite\Version191to2000</data>
        <volume>Migration\Step\UrlRewrite\Version191to2000</volume>
    </step> --> 
 <!--   <step title="Log Step">
        <integrity>Migration\Step\Log\Integrity</integrity>
        <data>Migration\Step\Log\Data</data>
        <volume>Migration\Step\Log\Volume</volume>
    </step>  -->
  <!--  <step title="Ratings Step">
        <integrity>Migration\Step\Ratings\Integrity</integrity>
        <data>Migration\Step\Ratings\Data</data>
        <volume>Migration\Step\Ratings\Volume</volume>
    </step>
    <step title="ConfigurablePrices step">
        <integrity>Migration\Step\ConfigurablePrices\Integrity</integrity>
        <data>Migration\Step\ConfigurablePrices\Data</data>
        <volume>Migration\Step\ConfigurablePrices\Volume</volume>
    </step>  -->
   <!-- <step title="OrderGrids Step">
        <integrity>Migration\Step\OrderGrids\Integrity</integrity>
        <data>Migration\Step\OrderGrids\Data</data>
        <volume>Migration\Step\OrderGrids\Volume</volume>
    </step>
    <step title="Tier Price Step">
        <integrity>Migration\Step\TierPrice\Integrity</integrity>
        <data>Migration\Step\TierPrice\Data</data>
        <volume>Migration\Step\TierPrice\Volume</volume>
    </step>  
    <step title="SalesIncrement Step">
        <integrity>Migration\Step\SalesIncrement\Integrity</integrity>
        <data>Migration\Step\SalesIncrement\Data</data>
        <volume>Migration\Step\SalesIncrement\Volume</volume>
    </step>
    <step title="PostProcessing Step">
        <data>Migration\Step\PostProcessing\Data</data>
    </step>-->
</steps>
<steps mode="delta">
    <step title="Customer Attributes Step">
        <delta>Migration\Step\Customer\Delta</delta>
        <volume>Migration\Step\Customer\Volume</volume>
    </step>
</steps>

<source>
   <database host="xxxxxxx" name="database_name" user="user" password="password"/>
</source>
<destination>
   <database host="xxxxxxx" name="database_name" user="user" password="password"/>
</destination>
<options>
    <map_file>etc/opensource-to-opensource/1.9.0.1/map.xml.dist</map_file>
    <eav_map_file>etc/opensource-to-opensource/map-eav.xml.dist</eav_map_file>
    <eav_document_groups_file>etc/opensource-to-opensource/eav-document-groups.xml.dist</eav_document_groups_file>
    <eav_attribute_groups_file>etc/opensource-to-opensource/eav-attribute-groups.xml.dist</eav_attribute_groups_file>
    <log_map_file>etc/opensource-to-opensource/map-log.xml.dist</log_map_file>
    <log_document_groups_file>etc/opensource-to-opensource/log-document-groups.xml.dist</log_document_groups_file>
    <settings_map_file>etc/opensource-to-opensource/settings.xml.dist</settings_map_file>
    <customer_map_file>etc/opensource-to-opensource/map-customer.xml.dist</customer_map_file>
    <customer_document_groups_file>etc/opensource-to-opensource/customer-document-groups.xml.dist</customer_document_groups_file>
    <customer_attribute_groups_file>etc/opensource-to-opensource/customer-attribute-groups.xml.dist</customer_attribute_groups_file>
    <delta_document_groups_file>etc/opensource-to-opensource/deltalog.xml.dist</delta_document_groups_file>
    <order_grids_document_groups_file>etc/opensource-to-opensource/order-grids-document-groups.xml.dist</order_grids_document_groups_file>
    <map_document_groups>etc/opensource-to-opensource/map-document-groups.xml.dist</map_document_groups>
    <class_map>etc/opensource-to-opensource/class-map.xml.dist</class_map>
    <tier_price_map_file>etc/opensource-to-opensource/map-tier-price.xml.dist</tier_price_map_file>
    <stores_map_file>etc/opensource-to-opensource/map-stores.xml.dist</stores_map_file>
    <!--
    In case bulk_size=0 it will be auto-detected for every document.
    -->
    <bulk_size>0</bulk_size>
    <!--
    Set direct_document_copy = 1 for better performance.
    NOTE: 'source' and 'destination' databases MUST be placed on the same MySQL instance
    and 'destination' user MUST be granted with 'SELECT' permissions on 'source' database
     -->
    <direct_document_copy>0</direct_document_copy>
    <source_prefix />
    <dest_prefix>pe</dest_prefix>
    <auto_resolve_urlrewrite_duplicates>0</auto_resolve_urlrewrite_duplicates>
    <log_file>migration.log</log_file>
    <progress_bar_format>%percent%% [%bar%] Remaining Time: %remaining%</progress_bar_format>
    <upgrade_customer_password_hash>1</upgrade_customer_password_hash>
    <edition_migrate>opensource-to-opensource</edition_migrate>
    <edition_number>1.9.0.1</edition_number>
    <init_statements_source>SET NAMES utf8;</init_statements_source>
    <init_statements_destination>SET NAMES utf8;</init_statements_destination>
    <crypt_key>04a54072077cb6e32e54dee19b47cc3f</crypt_key>
</options>

After this, if you can't login with a customer account truncate customer_entity_varchar, it's caused by a change in hash I think so truncate so it's regenerated and all good.

then, orders

    <?xml version="1.0" encoding="UTF-8"?>
<!--
/**
 * Copyright © Magento, Inc. All rights reserved.
 * See COPYING.txt for license details.
 */
-->
<config xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:noNamespaceSchemaLocation="../../config.xsd">

    <steps mode="data">
            <!-- <step title="Data Integrity Step">
                <integrity>Migration\Step\DataIntegrity\Integrity</integrity>
            </step>
           <step title="EAV Step">
                <integrity>Migration\Step\Eav\Integrity</integrity>
                <data>Migration\Step\Eav\Data</data>
                <volume>Migration\Step\Eav\Volume</volume>
            </step>  
            <step title="Customer Attributes Step">
                <integrity>Migration\Step\Customer\Integrity</integrity>
                <data>Migration\Step\Customer\Data</data>
                <volume>Migration\Step\Customer\Volume</volume>
            </step>-->
            <step title="Map Step">
                <integrity>Migration\Step\Map\Integrity</integrity>
                <data>Migration\Step\Map\Data</data>
                <volume>Migration\Step\Map\Volume</volume>
            </step>
            <!--<step title="Url Rewrite Step">
                <integrity>Migration\Step\UrlRewrite\Version191to2000</integrity>
                <data>Migration\Step\UrlRewrite\Version191to2000</Imdata>
                <volume>Migration\Step\UrlRewrite\Version191to2000</volume>
            </step>
            <step title="Log Step">
                <integrity>Migration\Step\Log\Integrity</integrity>
                <data>Migration\Step\Log\Data</data>
                <volume>Migration\Step\Log\Volume</volume>
            </step>
            <step title="Ratings Step">
                <integrity>Migration\Step\Ratings\Integrity</integrity>
                <data>Migration\Step\Ratings\Data</data>
                <volume>Migration\Step\Ratings\Volume</volume>
            </step>
            <step title="ConfigurablePrices step">
                <integrity>Migration\Step\ConfigurablePrices\Integrity</integrity>
                <data>Migration\Step\ConfigurablePrices\Data</data>
                <volume>Migration\Step\ConfigurablePrices\Volume</volume>
            </step>  -->
            <step title="OrderGrids Step">
                <integrity>Migration\Step\OrderGrids\Integrity</integrity>
                <data>Migration\Step\OrderGrids\Data</data>
                <volume>Migration\Step\OrderGrids\Volume</volume>
            </step>
    <!--        <step title="Tier Price Step">
                <integrity>Migration\Step\TierPrice\Integrity</integrity>
                <data>Migration\Step\TierPrice\Data</data>
                <volume>Migration\Step\TierPrice\Volume</volume>
            </step>  -->
            <step title="SalesIncrement Step">
                <integrity>Migration\Step\SalesIncrement\Integrity</integrity>
                <data>Migration\Step\SalesIncrement\Data</data>
                <volume>Migration\Step\SalesIncrement\Volume</volume>
            </step>
            <!--<step title="PostProcessing Step">
                <data>Migration\Step\PostProcessing\Data</data>
            </step>-->
        </steps>
  <steps mode="delta">
        <step title="SalesIncrement Step">
            <delta>Migration\Step\SalesIncrement\Delta</delta>
            <volume>Migration\Step\SalesIncrement\Volume</volume>
        </step>
        <step title="OrderGrids Step">
            <delta>Migration\Step\OrderGrids\Delta</delta>
            <volume>Migration\Step\OrderGrids\Volume</volume>
        </step>
        
</steps>

    <source>
       <database host="xxxxxx" name="database" user="user" password="password"/>
    </source>
    <destination>
       <database host="xxxxxxx" name="database" user="user" password="password"/>
    </destination>
    <options>
        <map_file>etc/opensource-to-opensource/1.9.0.1/map.xml.dist</map_file>
        <eav_map_file>etc/opensource-to-opensource/map-eav.xml.dist</eav_map_file>
        <eav_document_groups_file>etc/opensource-to-opensource/eav-document-groups.xml.dist</eav_document_groups_file>
        <eav_attribute_groups_file>etc/opensource-to-opensource/eav-attribute-groups.xml.dist</eav_attribute_groups_file>
        <log_map_file>etc/opensource-to-opensource/map-log.xml.dist</log_map_file>
        <log_document_groups_file>etc/opensource-to-opensource/log-document-groups.xml.dist</log_document_groups_file>
        <settings_map_file>etc/opensource-to-opensource/settings.xml.dist</settings_map_file>
        <customer_map_file>etc/opensource-to-opensource/map-customer.xml.dist</customer_map_file>
        <customer_document_groups_file>etc/opensource-to-opensource/customer-document-groups.xml.dist</customer_document_groups_file>
        <customer_attribute_groups_file>etc/opensource-to-opensource/customer-attribute-groups.xml.dist</customer_attribute_groups_file>
        <delta_document_groups_file>etc/opensource-to-opensource/deltalog.xml.dist</delta_document_groups_file>
        <order_grids_document_groups_file>etc/opensource-to-opensource/order-grids-document-groups.xml.dist</order_grids_document_groups_file>
        <map_document_groups>etc/opensource-to-opensource/map-document-groups.xml.dist</map_document_groups>
        <class_map>etc/opensource-to-opensource/class-map.xml.dist</class_map>
        <tier_price_map_file>etc/opensource-to-opensource/map-tier-price.xml.dist</tier_price_map_file>
        <stores_map_file>etc/opensource-to-opensource/map-stores.xml.dist</stores_map_file>
        <!--
        In case bulk_size=0 it will be auto-detected for every document.
        -->
        <bulk_size>0</bulk_size>
        <!--
        Set direct_document_copy = 1 for better performance.
        NOTE: 'source' and 'destination' databases MUST be placed on the same MySQL instance
        and 'destination' user MUST be granted with 'SELECT' permissions on 'source' database
         -->
        <direct_document_copy>0</direct_document_copy>
        <source_prefix />
        <dest_prefix>pe</dest_prefix>
        <auto_resolve_urlrewrite_duplicates>0</auto_resolve_urlrewrite_duplicates>
        <log_file>migration.log</log_file>
        <progress_bar_format>%percent%% [%bar%] Remaining Time: %remaining%</progress_bar_format>
        <upgrade_customer_password_hash>1</upgrade_customer_password_hash>
        <edition_migrate>opensource-to-opensource</edition_migrate>
        <edition_number>1.9.0.1</edition_number>
        <init_statements_source>SET NAMES utf8;</init_statements_source>
        <init_statements_destination>SET NAMES utf8;</init_statements_destination>
        <crypt_key>04a54072077cb6e32e54dee19b47cc3f</crypt_key>
    </options>
</config>

You can try to join this, but i found it easier to do in two imports. If gallery tables give yu problems truncate them, don't worry, but remember to always have a copy of the original database, I got it wrong so many times, if you have any problem let me know I'll see if i can help!

Hope it helps :D

Question: How did you delete the customers and orders before starting the migration again? This blog post (czettner.com/2018/09/13/magento-2-data-migration.html) suggests truncating various customer & order related tables in the DB. Did this work for you? — Louis B., Commented Sep 3, 2020 at 9:11
Figured it out on my own, also check out my own answer that I added below. — Louis B., Commented Sep 3, 2020 at 12:45

Louis B. · Accepted Answer · 2020-09-03 13:05:27Z

After a lot of trial and error I was able to get this working.

Here are my quick notes. Ping me if more information is required:

Create a DB backup of the M2 db before the remigration
Ensure that the migration tool and your corresponding project specific extension are enabled as described in Magento's migration guide
Go to https://www.czettner.com/2018/09/13/magento-2-data-migration.html and run the truncation query as provided
From the same URL, copy the XML snippet provided and add it to your map.xml at the following path: map > destination > document_rules. I left all the other nodes that are already present at this path, and simply amended the XML snippet provided
Run the migration with the "--reset" flag as described in Magento's migration guide

When you encounter an issue, try to fix it and restart the whole process by resetting your DB with the backup you created.

Some issues that I encountered:

Document has ambiguous configuration: cms_page => Here I simply commented out everything related to "cms_page" from the map.xml
Missing required argument $options or similar => This seems to be an issue with EAV attributes that I already had during my initial migration. Here is what helped me identify the corresponding EAV attribute which I then deleted: “Missing required argument $options” After migration from magento 1.9.2.4 to 2.1.6
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '182-65' for key 'CAT_PRD_ENTT_MDA_GLR_VAL_TO_ENTT_VAL_ID_ENTT_ID' => Here I simply truncated the following tables: catalog_product_entity_media_gallery, catalog_product_entity_media_gallery_value, catalog_product_entity_media_gallery_value_to_entity, catalog_product_entity_media_gallery_value_video

Yes, that are the tables to truncate, in the link I referred to it's the same. Good you figured it out, hope I have helped, don't forget to test the logins with an imported user if it always says wrong password truncate customer_entity_varchar, I had some notes on the files I sent you, not much but I have this info there I think. But yeah, it's a painful trial an error this process — Rui Silva, Commented Sep 3, 2020 at 13:45

Stack Exchange Network

Magento 2: Migrate changes made since last migration using non-live DB?

2 Answers 2

Your Answer

Not the answer you're looking for? Browse other questions tagged
magento2
data-migration
data-migration-tool
or ask your own question.

Linked

Hot Network Questions

Magento 2: Migrate changes made since last migration using non-live DB?

2 Answers 2

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged magento2data-migrationdata-migration-tool or ask your own question.

Linked

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
magento2
data-migration
data-migration-tool
or ask your own question.