Errors while using data migration tool from 1.9.0.1 -> 2.1.2

Question

I'm currently following through the Magento 2 Migration guide (http://devdocs.magento.com/guides/v2.0/migration/migration-migrate-data.html). I was able to successfully migrate settings, but while migrating data, I ran:

php bin/magento migrate:data /var/www/html/vendor/magento/data-migration-tool/etc/ce-to-ce/1.9.0.1/config.xml

The output I received from this said there was a "Integrity Check failed", i.e.:

[2016-10-29 04:27:05][INFO][mode: data][stage: integrity check][step: EAV Step]: started
100% [============================] Remaining Time: 1 sec
[2016-10-29 04:27:06][INFO][mode: data][stage: integrity check][step: Customer Attributes Step]: started
100% [============================] Remaining Time: 1 sec
[2016-10-29 04:27:06][INFO][mode: data][stage: integrity check][step: Map Step]: started
100% [============================] Remaining Time: 1 sec
[2016-10-29 04:27:08][ERROR]: Source documents are not mapped: advancedpopup,core_file_storage,easybanner_banner,easybanner_banner_placeholder,easybanner_banner_statistic,easybanner_banner_store,easybanner_layout_link,easybanner_layout_update,easybanner_placeholder,easyslide,easyslide_slides,rewardpoints_customer,rewardpoints_earning_catalog,rewardpoints_earning_product,rewardpoints_earning_sales,rewardpoints_rate,rewardpoints_refer_customer,rewardpoints_special_refer,rewardpoints_special_refer_value,rewardpoints_spending_catalog,rewardpoints_spending_sales,rewardpoints_transaction,shipnote_note,smtppro_email_log,tm_askit_item,tm_askit_vote,tm_attributepages_entity,tm_attributepages_entity_store,tm_core_module,tm_email_gateway_storage,tm_email_gateway_transport,tm_email_queue_message,tm_email_queue_queue,tm_navigationpro_column,tm_navigationpro_column_content,tm_navigationpro_menu,tm_navigationpro_menu_content,tm_navigationpro_sibling,tm_navigationpro_sibling_content,tm_prolabels_rules,tm_prolabels_rules_index,tm_prolabels_rules_store,tm_prolabels_rules_system,tm_prolabels_system_store,tm_soldtogether_customer,tm_soldtogether_order,tm_testimonials_data,tm_testimonials_store
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_creditmemo. Fields: rewardpoints_base_discount,rewardpoints_discount,rewardpoints_earn,rewardpoints_base_hidden_tax_amount,rewardpoints_hidden_tax_amount,rewardpoints_invited_discount,rewardpoints_invited_base_discount,zipmoney_txn_id,refund_reference
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_invoice. Fields: rewardpoints_base_discount,rewardpoints_discount,rewardpoints_earn,rewardpoints_base_hidden_tax_amount,rewardpoints_hidden_tax_amount,rewardpoints_invited_discount,rewardpoints_invited_base_discount
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_order. Fields: rewardpoints_earn,rewardpoints_spent,rewardpoints_base_discount,rewardpoints_discount,rewardpoints_base_amount,rewardpoints_amount,rewardpoints_base_hidden_tax_amount,rewardpoints_hidden_tax_amount,rewardpoints_base_shipping_hidden_tax_amount,rewardpoints_shipping_hidden_tax_amount,rewardpoints_referal_earn,rewardpoints_invited_discount,rewardpoints_invited_base_discount,rewardpoints_refer_customer_id,ship_note_id,codisto_orderid
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_order_item. Fields: rewardpoints_earn,rewardpoints_spent,rewardpoints_base_discount,rewardpoints_discount,rewardpoints_base_hidden_tax_amount,rewardpoints_hidden_tax_amount,rewardpoints_referal_earn,rewardpoints_invited_discount,rewardpoints_invited_base_discount
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_order_payment. Fields: is_zipmoney_express
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_quote. Fields: ship_note,zipmoney_token
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_quote_item. Fields: fooman_auto_added_qty,fooman_free_qty,fooman_applied_rule_ids
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: salesrule. Fields: fooman_discount_select,fooman_discount_groupsize,fooman_min_qty
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: Url Rewrite Step]: started
100% [============================] Remaining Time: 1 sec
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: Log Step]: started
100% [============================] Remaining Time: 1 sec
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: Ratings Step]: started
100% [============================] Remaining Time: 1 sec
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: ConfigurablePrices step]: started
100% [============================] Remaining Time: 1 sec
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: OrderGrids Step]: started
100% [============================] Remaining Time: 1 sec
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: Tier Price Step]: started
100% [============================] Remaining Time: 1 sec
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: SalesIncrement Step]: started
100% [============================] Remaining Time: 1 sec


  [Migration\Exception]
  Integrity Check failed



migrate:data [-r|--reset] config

In my "migration.log" file (/var/www/html/var/migration.log), the following log entries are present:

[2016-10-29 04:25:57][INFO][mode: settings][stage: integrity check][step: Settings Step]: started
[2016-10-29 04:26:36][INFO][mode: settings][stage: integrity check][step: Settings Step]: started
[2016-10-29 04:26:36][INFO][mode: settings][stage: integrity check][step: Stores Step]: started
[2016-10-29 04:26:36][INFO][mode: settings][stage: data migration][step: Settings Step]: started
[2016-10-29 04:26:38][INFO][mode: settings][stage: data migration][step: Stores Step]: started
[2016-10-29 04:26:38][INFO][mode: settings][stage: volume check][step: Stores Step]: started
[2016-10-29 04:26:38][INFO][mode: settings][stage: volume check][step: Stores Step]: Migration completed
[2016-10-29 04:27:05][INFO][mode: data][stage: integrity check][step: EAV Step]: started
[2016-10-29 04:27:06][INFO][mode: data][stage: integrity check][step: Customer Attributes Step]: started
[2016-10-29 04:27:06][INFO][mode: data][stage: integrity check][step: Map Step]: started
[2016-10-29 04:27:08][ERROR]: Source documents are not mapped: advancedpopup,core_file_storage,easybanner_banner,easybanner_banner_placeholder,easybanner_banner_statistic,easybanner_banner_store,easybanner_layout_link,easybanner_layout_update,easybanner_placeholder,easyslide,easyslide_slides,rewardpoints_customer,rewardpoints_earning_catalog,rewardpoints_earning_product,rewardpoints_earning_sales,rewardpoints_rate,rewardpoints_refer_customer,rewardpoints_special_refer,rewardpoints_special_refer_value,rewardpoints_spending_catalog,rewardpoints_spending_sales,rewardpoints_transaction,shipnote_note,smtppro_email_log,tm_askit_item,tm_askit_vote,tm_attributepages_entity,tm_attributepages_entity_store,tm_core_module,tm_email_gateway_storage,tm_email_gateway_transport,tm_email_queue_message,tm_email_queue_queue,tm_navigationpro_column,tm_navigationpro_column_content,tm_navigationpro_menu,tm_navigationpro_menu_content,tm_navigationpro_sibling,tm_navigationpro_sibling_content,tm_prolabels_rules,tm_prolabels_rules_index,tm_prolabels_rules_store,tm_prolabels_rules_system,tm_prolabels_system_store,tm_soldtogether_customer,tm_soldtogether_order,tm_testimonials_data,tm_testimonials_store
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_creditmemo. Fields: rewardpoints_base_discount,rewardpoints_discount,rewardpoints_earn,rewardpoints_base_hidden_tax_amount,rewardpoints_hidden_tax_amount,rewardpoints_invited_discount,rewardpoints_invited_base_discount,zipmoney_txn_id,refund_reference
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_invoice. Fields: rewardpoints_base_discount,rewardpoints_discount,rewardpoints_earn,rewardpoints_base_hidden_tax_amount,rewardpoints_hidden_tax_amount,rewardpoints_invited_discount,rewardpoints_invited_base_discount
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_order. Fields: rewardpoints_earn,rewardpoints_spent,rewardpoints_base_discount,rewardpoints_discount,rewardpoints_base_amount,rewardpoints_amount,rewardpoints_base_hidden_tax_amount,rewardpoints_hidden_tax_amount,rewardpoints_base_shipping_hidden_tax_amount,rewardpoints_shipping_hidden_tax_amount,rewardpoints_referal_earn,rewardpoints_invited_discount,rewardpoints_invited_base_discount,rewardpoints_refer_customer_id,ship_note_id,codisto_orderid
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_order_item. Fields: rewardpoints_earn,rewardpoints_spent,rewardpoints_base_discount,rewardpoints_discount,rewardpoints_base_hidden_tax_amount,rewardpoints_hidden_tax_amount,rewardpoints_referal_earn,rewardpoints_invited_discount,rewardpoints_invited_base_discount
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_order_payment. Fields: is_zipmoney_express
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_quote. Fields: ship_note,zipmoney_token
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: sales_flat_quote_item. Fields: fooman_auto_added_qty,fooman_free_qty,fooman_applied_rule_ids
[2016-10-29 04:27:08][ERROR]: Source fields are not mapped. Document: salesrule. Fields: fooman_discount_select,fooman_discount_groupsize,fooman_min_qty
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: Url Rewrite Step]: started
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: Log Step]: started
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: Ratings Step]: started
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: ConfigurablePrices step]: started
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: OrderGrids Step]: started
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: Tier Price Step]: started
[2016-10-29 04:27:08][INFO][mode: data][stage: integrity check][step: SalesIncrement Step]: started

Answer 1

All these table names and fields which are not "standard" need to be either ignored or mapped in your version of the map.xml.dist configuration files.

The first error line Source documents are not mapped: is about tables which exist in your m1 but m2 knows nothing about, created by extensions.

The next set [ERROR]: Source fields are not mapped. Document: sales_flat_creditmemo (etc) are about attributes (fields) which were created in sales_flat_creditmemo, sales_flat_invoice etc. by extensions, which again m2 knows nothing about.

It's a very long list (and it's only the first, linked to orders, there will be more, regarding customers, products etc.) so you might want to think whether you are using all of these extensions and whether you need it all migrated. The more you can ignore the better for you.

Things that you are not yet ready to migrate (eg:module not existing yet), or will not migrate go in '' tags

<document_rules>
    <ignore>
        <document>easybanner_*</document>
    </ignore>
</document_rules>

<field_rules>
    <ignore>
        <field>sales_flat_order_item.xero_rate</field>
    </ignore>
</field_rules>

Things that will migrate also need to be defined in the xml files too, with transformations if needed. Of course for tables and fields linked to an extension, you need to know if the extension has made it to m2 and if it has changed.

This is going to take you a while and trial and error! I suggest you start by ignoring everything then slowly "move" the things you need to migrate

For things like navigation, banners etc. you might decide to just "rebuild" on m2 rather than fight to migrate.

Ready? It all happens in map.xml.dist (applies across the board) and the more "specialised" map-eav.xml map-customer.xml etc. Each has a document_rules and a field_rules section, and yes, every single one of the names in these long error lists will need to be in there either defined or ignored.

<document_rules>
    <ignore>
        <document>easybanner_*</document>
    </ignore>
</document_rules>

<field_rules>
    <ignore>
        <field>sales_flat_order_item.xero_rate</field>
    </ignore>
</field_rules>