0

When we started development of our M2 project, we migrated all data to the new M2 project from M1. We did this using the Data Migration Tool. Because we did not want to mess with the live database, we first made a copy of this database and used this copy to migrate our data to M2.

This has worked perfectly.

Now we want to migrate all changes made to the live M1 database since we started development on M2. But now I saw that in order to migrate the delta from the old M1 database Magento added some tables to that database when we migrated it (according to https://devdocs.magento.com/guides/v2.4/migration/migration-migrate-delta.html). However, since we migrated from a copy and not the live M1 database, no changes were tracked.

Is there a way to migrate changed data from M1 to M2 without Magento having kept track of changes in the M1 database?

  • If I get I right you just want to migrate things that were changed during the development time of M2 right, most likely orders and costumers that were added to M1? – Rui Silva Aug 31 at 16:27
  • @RuiSilva that is exactly right. However, I now figure out that this is not really possible with Magento out of the box, as I can not use the "delta" mode anymore. Here is another a bit hacky approach I am using right now: czettner.com/2018/09/13/magento-2-data-migration.html. – Louis B. Aug 31 at 16:49
  • I had to do that, if you only need customers and orders, the way I did it was deleting all orders and customers from M2, with the migration tool ignore all the other info and then import just that, orders and customers. Was the best way I found, but you will lost all the orders and customers created in M2, idk if that's a problem, but yeah it's kind of that way in that post, I can send you the files map and config I used, but you might need to adjust it a bit for your needs – Rui Silva Aug 31 at 17:20
  • @RuiSilva: So you deleted all orders and customers (via the DB?) and then set up the map.xml to ignore everything but the order and customer related data, and then re-ran the full migration? That is kind of what I figured I would have to do as well. If you could post your map.xml and config.xml that'd be great. You can also add it as an answer and I'll accept it so you'll get internet points. Thanks! – Louis B. Sep 1 at 6:44
  • They are big files, can't upload them here, I can upload the config but the maps are massive – Rui Silva Sep 1 at 9:59
2

Kinda big, but I did it in two different imports

First deleted all orders and customers with the queries in this question https://github.com/magento/data-migration-tool/issues/467#issuecomment-627912182

Then do a map.xml, this might depend on your tables, I had to ignore extensions tables and other ones, was a trial error, backup your database and try, if something is wrong go back and check where it is on database then add that table to ignore, in that last answer on git a posted a bit but too much lines to post it here.

To import customers

<?xml version="1.0" encoding="UTF-8"?>
<steps mode="data">
    <!-- <step title="Data Integrity Step">
        <integrity>Migration\Step\DataIntegrity\Integrity</integrity>
    </step>
   <step title="EAV Step">
        <integrity>Migration\Step\Eav\Integrity</integrity>
        <data>Migration\Step\Eav\Data</data>
        <volume>Migration\Step\Eav\Volume</volume>
    </step>  -->
    <step title="Customer Attributes Step">
        <integrity>Migration\Step\Customer\Integrity</integrity>
        <data>Migration\Step\Customer\Data</data>
        <volume>Migration\Step\Customer\Volume</volume>
    </step>
    <!-- <step title="Map Step">
        <integrity>Migration\Step\Map\Integrity</integrity>
        <data>Migration\Step\Map\Data</data>
        <volume>Migration\Step\Map\Volume</volume>
    </step>
   <step title="Url Rewrite Step">
        <integrity>Migration\Step\UrlRewrite\Version191to2000</integrity>
        <data>Migration\Step\UrlRewrite\Version191to2000</data>
        <volume>Migration\Step\UrlRewrite\Version191to2000</volume>
    </step> --> 
 <!--   <step title="Log Step">
        <integrity>Migration\Step\Log\Integrity</integrity>
        <data>Migration\Step\Log\Data</data>
        <volume>Migration\Step\Log\Volume</volume>
    </step>  -->
  <!--  <step title="Ratings Step">
        <integrity>Migration\Step\Ratings\Integrity</integrity>
        <data>Migration\Step\Ratings\Data</data>
        <volume>Migration\Step\Ratings\Volume</volume>
    </step>
    <step title="ConfigurablePrices step">
        <integrity>Migration\Step\ConfigurablePrices\Integrity</integrity>
        <data>Migration\Step\ConfigurablePrices\Data</data>
        <volume>Migration\Step\ConfigurablePrices\Volume</volume>
    </step>  -->
   <!-- <step title="OrderGrids Step">
        <integrity>Migration\Step\OrderGrids\Integrity</integrity>
        <data>Migration\Step\OrderGrids\Data</data>
        <volume>Migration\Step\OrderGrids\Volume</volume>
    </step>
    <step title="Tier Price Step">
        <integrity>Migration\Step\TierPrice\Integrity</integrity>
        <data>Migration\Step\TierPrice\Data</data>
        <volume>Migration\Step\TierPrice\Volume</volume>
    </step>  
    <step title="SalesIncrement Step">
        <integrity>Migration\Step\SalesIncrement\Integrity</integrity>
        <data>Migration\Step\SalesIncrement\Data</data>
        <volume>Migration\Step\SalesIncrement\Volume</volume>
    </step>
    <step title="PostProcessing Step">
        <data>Migration\Step\PostProcessing\Data</data>
    </step>-->
</steps>
<steps mode="delta">
    <step title="Customer Attributes Step">
        <delta>Migration\Step\Customer\Delta</delta>
        <volume>Migration\Step\Customer\Volume</volume>
    </step>
</steps>

<source>
   <database host="xxxxxxx" name="database_name" user="user" password="password"/>
</source>
<destination>
   <database host="xxxxxxx" name="database_name" user="user" password="password"/>
</destination>
<options>
    <map_file>etc/opensource-to-opensource/1.9.0.1/map.xml.dist</map_file>
    <eav_map_file>etc/opensource-to-opensource/map-eav.xml.dist</eav_map_file>
    <eav_document_groups_file>etc/opensource-to-opensource/eav-document-groups.xml.dist</eav_document_groups_file>
    <eav_attribute_groups_file>etc/opensource-to-opensource/eav-attribute-groups.xml.dist</eav_attribute_groups_file>
    <log_map_file>etc/opensource-to-opensource/map-log.xml.dist</log_map_file>
    <log_document_groups_file>etc/opensource-to-opensource/log-document-groups.xml.dist</log_document_groups_file>
    <settings_map_file>etc/opensource-to-opensource/settings.xml.dist</settings_map_file>
    <customer_map_file>etc/opensource-to-opensource/map-customer.xml.dist</customer_map_file>
    <customer_document_groups_file>etc/opensource-to-opensource/customer-document-groups.xml.dist</customer_document_groups_file>
    <customer_attribute_groups_file>etc/opensource-to-opensource/customer-attribute-groups.xml.dist</customer_attribute_groups_file>
    <delta_document_groups_file>etc/opensource-to-opensource/deltalog.xml.dist</delta_document_groups_file>
    <order_grids_document_groups_file>etc/opensource-to-opensource/order-grids-document-groups.xml.dist</order_grids_document_groups_file>
    <map_document_groups>etc/opensource-to-opensource/map-document-groups.xml.dist</map_document_groups>
    <class_map>etc/opensource-to-opensource/class-map.xml.dist</class_map>
    <tier_price_map_file>etc/opensource-to-opensource/map-tier-price.xml.dist</tier_price_map_file>
    <stores_map_file>etc/opensource-to-opensource/map-stores.xml.dist</stores_map_file>
    <!--
    In case bulk_size=0 it will be auto-detected for every document.
    -->
    <bulk_size>0</bulk_size>
    <!--
    Set direct_document_copy = 1 for better performance.
    NOTE: 'source' and 'destination' databases MUST be placed on the same MySQL instance
    and 'destination' user MUST be granted with 'SELECT' permissions on 'source' database
     -->
    <direct_document_copy>0</direct_document_copy>
    <source_prefix />
    <dest_prefix>pe</dest_prefix>
    <auto_resolve_urlrewrite_duplicates>0</auto_resolve_urlrewrite_duplicates>
    <log_file>migration.log</log_file>
    <progress_bar_format>%percent%% [%bar%] Remaining Time: %remaining%</progress_bar_format>
    <upgrade_customer_password_hash>1</upgrade_customer_password_hash>
    <edition_migrate>opensource-to-opensource</edition_migrate>
    <edition_number>1.9.0.1</edition_number>
    <init_statements_source>SET NAMES utf8;</init_statements_source>
    <init_statements_destination>SET NAMES utf8;</init_statements_destination>
    <crypt_key>04a54072077cb6e32e54dee19b47cc3f</crypt_key>
</options>

After this, if you can't login with a customer account truncate customer_entity_varchar, it's caused by a change in hash I think so truncate so it's regenerated and all good.

then, orders

    <?xml version="1.0" encoding="UTF-8"?>
<!--
/**
 * Copyright © Magento, Inc. All rights reserved.
 * See COPYING.txt for license details.
 */
-->
<config xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:noNamespaceSchemaLocation="../../config.xsd">

    <steps mode="data">
            <!-- <step title="Data Integrity Step">
                <integrity>Migration\Step\DataIntegrity\Integrity</integrity>
            </step>
           <step title="EAV Step">
                <integrity>Migration\Step\Eav\Integrity</integrity>
                <data>Migration\Step\Eav\Data</data>
                <volume>Migration\Step\Eav\Volume</volume>
            </step>  
            <step title="Customer Attributes Step">
                <integrity>Migration\Step\Customer\Integrity</integrity>
                <data>Migration\Step\Customer\Data</data>
                <volume>Migration\Step\Customer\Volume</volume>
            </step>-->
            <step title="Map Step">
                <integrity>Migration\Step\Map\Integrity</integrity>
                <data>Migration\Step\Map\Data</data>
                <volume>Migration\Step\Map\Volume</volume>
            </step>
            <!--<step title="Url Rewrite Step">
                <integrity>Migration\Step\UrlRewrite\Version191to2000</integrity>
                <data>Migration\Step\UrlRewrite\Version191to2000</Imdata>
                <volume>Migration\Step\UrlRewrite\Version191to2000</volume>
            </step>
            <step title="Log Step">
                <integrity>Migration\Step\Log\Integrity</integrity>
                <data>Migration\Step\Log\Data</data>
                <volume>Migration\Step\Log\Volume</volume>
            </step>
            <step title="Ratings Step">
                <integrity>Migration\Step\Ratings\Integrity</integrity>
                <data>Migration\Step\Ratings\Data</data>
                <volume>Migration\Step\Ratings\Volume</volume>
            </step>
            <step title="ConfigurablePrices step">
                <integrity>Migration\Step\ConfigurablePrices\Integrity</integrity>
                <data>Migration\Step\ConfigurablePrices\Data</data>
                <volume>Migration\Step\ConfigurablePrices\Volume</volume>
            </step>  -->
            <step title="OrderGrids Step">
                <integrity>Migration\Step\OrderGrids\Integrity</integrity>
                <data>Migration\Step\OrderGrids\Data</data>
                <volume>Migration\Step\OrderGrids\Volume</volume>
            </step>
    <!--        <step title="Tier Price Step">
                <integrity>Migration\Step\TierPrice\Integrity</integrity>
                <data>Migration\Step\TierPrice\Data</data>
                <volume>Migration\Step\TierPrice\Volume</volume>
            </step>  -->
            <step title="SalesIncrement Step">
                <integrity>Migration\Step\SalesIncrement\Integrity</integrity>
                <data>Migration\Step\SalesIncrement\Data</data>
                <volume>Migration\Step\SalesIncrement\Volume</volume>
            </step>
            <!--<step title="PostProcessing Step">
                <data>Migration\Step\PostProcessing\Data</data>
            </step>-->
        </steps>
  <steps mode="delta">
        <step title="SalesIncrement Step">
            <delta>Migration\Step\SalesIncrement\Delta</delta>
            <volume>Migration\Step\SalesIncrement\Volume</volume>
        </step>
        <step title="OrderGrids Step">
            <delta>Migration\Step\OrderGrids\Delta</delta>
            <volume>Migration\Step\OrderGrids\Volume</volume>
        </step>
        
</steps>

    <source>
       <database host="xxxxxx" name="database" user="user" password="password"/>
    </source>
    <destination>
       <database host="xxxxxxx" name="database" user="user" password="password"/>
    </destination>
    <options>
        <map_file>etc/opensource-to-opensource/1.9.0.1/map.xml.dist</map_file>
        <eav_map_file>etc/opensource-to-opensource/map-eav.xml.dist</eav_map_file>
        <eav_document_groups_file>etc/opensource-to-opensource/eav-document-groups.xml.dist</eav_document_groups_file>
        <eav_attribute_groups_file>etc/opensource-to-opensource/eav-attribute-groups.xml.dist</eav_attribute_groups_file>
        <log_map_file>etc/opensource-to-opensource/map-log.xml.dist</log_map_file>
        <log_document_groups_file>etc/opensource-to-opensource/log-document-groups.xml.dist</log_document_groups_file>
        <settings_map_file>etc/opensource-to-opensource/settings.xml.dist</settings_map_file>
        <customer_map_file>etc/opensource-to-opensource/map-customer.xml.dist</customer_map_file>
        <customer_document_groups_file>etc/opensource-to-opensource/customer-document-groups.xml.dist</customer_document_groups_file>
        <customer_attribute_groups_file>etc/opensource-to-opensource/customer-attribute-groups.xml.dist</customer_attribute_groups_file>
        <delta_document_groups_file>etc/opensource-to-opensource/deltalog.xml.dist</delta_document_groups_file>
        <order_grids_document_groups_file>etc/opensource-to-opensource/order-grids-document-groups.xml.dist</order_grids_document_groups_file>
        <map_document_groups>etc/opensource-to-opensource/map-document-groups.xml.dist</map_document_groups>
        <class_map>etc/opensource-to-opensource/class-map.xml.dist</class_map>
        <tier_price_map_file>etc/opensource-to-opensource/map-tier-price.xml.dist</tier_price_map_file>
        <stores_map_file>etc/opensource-to-opensource/map-stores.xml.dist</stores_map_file>
        <!--
        In case bulk_size=0 it will be auto-detected for every document.
        -->
        <bulk_size>0</bulk_size>
        <!--
        Set direct_document_copy = 1 for better performance.
        NOTE: 'source' and 'destination' databases MUST be placed on the same MySQL instance
        and 'destination' user MUST be granted with 'SELECT' permissions on 'source' database
         -->
        <direct_document_copy>0</direct_document_copy>
        <source_prefix />
        <dest_prefix>pe</dest_prefix>
        <auto_resolve_urlrewrite_duplicates>0</auto_resolve_urlrewrite_duplicates>
        <log_file>migration.log</log_file>
        <progress_bar_format>%percent%% [%bar%] Remaining Time: %remaining%</progress_bar_format>
        <upgrade_customer_password_hash>1</upgrade_customer_password_hash>
        <edition_migrate>opensource-to-opensource</edition_migrate>
        <edition_number>1.9.0.1</edition_number>
        <init_statements_source>SET NAMES utf8;</init_statements_source>
        <init_statements_destination>SET NAMES utf8;</init_statements_destination>
        <crypt_key>04a54072077cb6e32e54dee19b47cc3f</crypt_key>
    </options>
</config>

You can try to join this, but i found it easier to do in two imports. If gallery tables give yu problems truncate them, don't worry, but remember to always have a copy of the original database, I got it wrong so many times, if you have any problem let me know I'll see if i can help!

Hope it helps :D

| improve this answer | |
  • Question: How did you delete the customers and orders before starting the migration again? This blog post (czettner.com/2018/09/13/magento-2-data-migration.html) suggests truncating various customer & order related tables in the DB. Did this work for you? – Louis B. Sep 3 at 9:11
  • Figured it out on my own, also check out my own answer that I added below. – Louis B. Sep 3 at 12:45
0

After a lot of trial and error I was able to get this working.

Here are my quick notes. Ping me if more information is required:

  • Create a DB backup of the M2 db before the remigration
  • Ensure that the migration tool and your corresponding project specific extension are enabled as described in Magento's migration guide
  • Go to https://www.czettner.com/2018/09/13/magento-2-data-migration.html and run the truncation query as provided
  • From the same URL, copy the XML snippet provided and add it to your map.xml at the following path: map > destination > document_rules. I left all the other nodes that are already present at this path, and simply amended the XML snippet provided
  • Run the migration with the "--reset" flag as described in Magento's migration guide

When you encounter an issue, try to fix it and restart the whole process by resetting your DB with the backup you created.

Some issues that I encountered:

  • Document has ambiguous configuration: cms_page => Here I simply commented out everything related to "cms_page" from the map.xml
  • Missing required argument $options or similar => This seems to be an issue with EAV attributes that I already had during my initial migration. Here is what helped me identify the corresponding EAV attribute which I then deleted: “Missing required argument $options” After migration from magento 1.9.2.4 to 2.1.6
  • SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '182-65' for key 'CAT_PRD_ENTT_MDA_GLR_VAL_TO_ENTT_VAL_ID_ENTT_ID' => Here I simply truncated the following tables: catalog_product_entity_media_gallery, catalog_product_entity_media_gallery_value, catalog_product_entity_media_gallery_value_to_entity, catalog_product_entity_media_gallery_value_video
| improve this answer | |
  • Yes, that are the tables to truncate, in the link I referred to it's the same. Good you figured it out, hope I have helped, don't forget to test the logins with an imported user if it always says wrong password truncate customer_entity_varchar, I had some notes on the files I sent you, not much but I have this info there I think. But yeah, it's a painful trial an error this process – Rui Silva Sep 3 at 13:45
  • Thanks man, your inputs definitely helped a lot! – Louis B. Sep 3 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.