Magento 2 Data Migration

Take on topic of Data migration from Magento 1.x to Magento 2.0. Available tools, valuable insights and gothas. YT video is here - https://www.youtube.com/watch?v=UBK1rNjMauo

Transcript

1. MAGE1 >> MAGE2 NOT EVERYTHING THAT CAN BE COUNTED COUNTS, AND NOT EVERYTHING THAT COUNTS CAN BE COUNTED. ALBERT EINSTEIN, PHYSICIST (16.03.2016) LONDON MAGENTO MEETUP MARCIN SZTERLING REDBOX DIGITAL @php4u
2. Image: http://fishki.pl/fishka,Syzyfowa-praca,31405.html MARCIN SZTERLING REDBOX DIGITAL @php4u
3. ▸ Data migration is the process of importing legacy data, from one or more old systems, to a new system or systems. ▸ Data migration is typically part of a larger programme ▸ “Big bang” and “trickle” migrations WHAT IS DATA MIGRATION? MARCIN SZTERLING REDBOX DIGITAL @php4u
4. MIGRATE DATA ▸ Should we bring all or part of the data? ▸ If just part, which parts? - based on creation date? based on status of product, customer? or some combination of criteria? ▸ If we choose to bring over data on or after a certain date, which should it be - last 3 months, last 6 months, last year?...should it be all the data? ▸ Should we clean data at source or destination? BEFORE ALL STARTS MARCIN SZTERLING REDBOX DIGITAL @php4u
5. CHANGING TIRES AT 60 MPH THIS IS NOT A 1 DAY TASK ▸ CE -> CE, CE -> EE, EE -> EE ▸ There is no silver-bullet solution ▸ There will be issues with source data ▸ There will be issues with migrated data ▸ Amount of data matters ▸ Reserve time for it and prepare for battle! ▸ http://alankent.me/2015/01/31/ magento-1-to-2-migration-tool-plans/ MARCIN SZTERLING REDBOX DIGITAL @php4u
6. WHAT DATA DO WE NEED? MAIN DATA TO BE MIGRATED (WHAT THE HELL IS EAV??) ▸ Products ▸ Categories (and its products relations) ▸ Customers ▸ Orders ▸ Configurations ▸ CMS elements ▸ Websites, Store groups, Stores ▸ Media assets for Products, Categories, CMS MARCIN SZTERLING REDBOX DIGITAL @php4u
7. WHAT EXACTLY HAS CHANGED FROM M1 TO M2? ▸ Tables can be deleted ▸ Tables can be renamed ▸ New tables can exist in Magento 2 but not in Magento 1 ▸ Table columns can be deleted ▸ Table columns can be renamed ▸ New table columns can exist in Magento 2 but not in Magento 1 MARCIN SZTERLING REDBOX DIGITAL @php4u
8. SOME OF THE DATA MUST BE TRANSFORMED ▸ Values must be changed to match Magento 2 new classes namespaces (models) ▸ Values must be nulled even if are empty ▸ Names must be prefixed to avoid naming issues WHAT EXACTLY HAS CHANGED FROM M1 TO M2? MARCIN SZTERLING REDBOX DIGITAL @php4u
9. DO YOUR HOMEWORK! RIGHT TOOLS TO DO RIGHT JOB MARCIN SZTERLING REDBOX DIGITAL @php4u
10. WASHING AND IRONING AS WELL?? CRITERIA ▸ Ease of use ▸ Speed ▸ Security ▸ Accuracy ▸ Repeatable ▸ Multi-project MARCIN SZTERLING REDBOX DIGITAL @php4u
11. IN THE WILD WHAT IS AVAILABLE? ▸ Ubertheme Magento 2 data migration (1.0.4)
 https://github.com/ubertheme/module-ubdatamigration ▸ Magento data migration tool (2.0.2)
 https://github.com/magento/data-migration-tool ▸ CSV based import ▸ Custom own written solution (API?) ▸ https://github.com/SchumacherFM/Magento2-Data- Migration MARCIN SZTERLING REDBOX DIGITAL @php4u
12. UBERTHEME MIGRATION TOOL MARCIN SZTERLING REDBOX DIGITAL @php4u
13. JUST WORKS? UBERTHEME MIGRATION TOOL ▸ Yii framework based browser tool ▸ Not secured at all so you must not use it on production unless secured ▸ Almost zero configuration required ▸ Slow and dependant on server settings ▸ Supports delta changes and can be migrated on top of existing data ▸ Does the job! ▸ More info how to use it here
 https://github.com/ubertheme/module-ubdatamigration/blob/master/ README.md MARCIN SZTERLING REDBOX DIGITAL @php4u
14. SHOW ME SHOW ME… UBERTHEME MIGRATION TOOL MARCIN SZTERLING REDBOX DIGITAL @php4u
15. MAGENTO DATA MIGRATION TOOL MARCIN SZTERLING REDBOX DIGITAL @php4u
16. CARTE BLANCHE MAGENTO DATA MIGRATION TOOL ▸ Now one tool (2.0.2) for both EE and CE ▸ Preconfigured for all magento versions and editions ▸ Works with magento sample data (!) ▸ Supports delta changes ▸ Cannot be migrated on top of existing data ▸ Requires “perfect” source data integrity (ex. eav_entity_type) ▸ Blazing fast (with exceptions) ▸ xml based configuration - which can be stored in VCS, can have multiple profiles ▸ No way to specify database connection details in CLI ▸ Very flexible configuration to adjust to needs, easy to extend ▸ Progress bar with estimated time per task (but not for overall task) ▸ No automated data verification after import MARCIN SZTERLING REDBOX DIGITAL @php4u
17. COMPOSER WHAT? INSTALLATION ▸ http://devdocs.magento.com/guides/v2.0/migration/ migration-tool-install.html ▸ composer config repositories.data-migration-tool git https://github.com/magento/data-migration-tool ▸ composer require magento/data-migration-tool:2.0.2 ▸ Installed in vendor/magento/data-migration ▸ bin/magento migrate:data [-r|--reset] {<path to config.xml>} MARCIN SZTERLING REDBOX DIGITAL @php4u
18. WHERE TO START? MAGENTO DATA MIGRATION TOOL ▸ 3 main types of run modes in Migration/Console 
 Data, 
 Delta, 
 Settings ▸ Data is read, transformed and then saved in batches ▸ CPU and memory heavy MARCIN SZTERLING REDBOX DIGITAL @php4u
19. STEPS <?xml version="1.0" encoding="UTF-8"?> <config xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:noNamespaceSchemaLocation="../../config.xsd"> <steps mode="settings"> <step title="Settings Step"> <integrity>Migration\Step\Settings\Integrity</integrity> <data>Migration\Step\Settings\Data</data> </step> <step title="Stores Step"> <integrity>Migration\Step\Stores\Integrity</integrity> <data>Migration\Step\Stores\Data</data> <volume>Migration\Step\Stores\Volume</volume> </step> </steps> </config> MARCIN SZTERLING REDBOX DIGITAL @php4u
20. WHERE, FROM AND MORE SETTINGS… <?xml version="1.0" encoding="UTF-8"?> <config xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:noNamespaceSchemaLocation="../../config.xsd"> <source> <database host="localhost" name="magento1" user="root"/> </source> <destination> <database host="localhost" name="magento2" user="root"/> </destination> <options> <map_file>etc/ce-to-ee/1.7.0.2/map.xml.dist</map_file> <eav_map_file>etc/ce-to-ee/map-eav.xml.dist</eav_map_file> <eav_document_groups_file>etc/ce-to-ee/eav-document-groups.xml.dist</eav_document_groups_file> <eav_attribute_groups_file>etc/ce-to-ee/eav-attribute-groups.xml.dist</eav_attribute_groups_file> <log_map_file>etc/ce-to-ee/map-log.xml.dist</log_map_file> <log_document_groups_file>etc/ce-to-ee/log-document-groups.xml.dist</log_document_groups_file> <settings_map_file>etc/ce-to-ee/settings.xml.dist</settings_map_file> <customer_map_file>etc/ce-to-ee/map-customer.xml.dist</customer_map_file> <customer_document_groups_file>etc/ce-to-ee/customer-document-groups.xml.dist</customer_document_groups_file> <customer_attribute_groups_file>etc/ce-to-ee/customer-attribute-groups.xml.dist</customer_attribute_groups_file> <delta_document_groups_file>etc/ce-to-ee/deltalog.xml.dist</delta_document_groups_file> <order_grids_document_groups_file>etc/ce-to-ee/order-grids-document-groups.xml.dist</order_grids_document_groups_file> <map_document_groups>etc/ce-to-ee/map-document-groups.xml.dist</map_document_groups> <class_map>etc/ce-to-ee/class-map.xml.dist</class_map> <bulk_size>0</bulk_size> <direct_document_copy>0</direct_document_copy> <source_prefix /> <dest_prefix /> <auto_resolve_urlrewrite_duplicates>0</auto_resolve_urlrewrite_duplicates> <log_file>migration.log</log_file> <progress_bar_format>%percent%% [%bar%] Remaining Time: %remaining%</progress_bar_format> <upgrade_customer_password_hash>1</upgrade_customer_password_hash> <crypt_key /> </options> </config> http://devdocs.magento.com/guides/v2.0/migration/migration-tool-internal-spec.html MARCIN SZTERLING REDBOX DIGITAL @php4u
21. WHEN DOCUMENT IS NOT DOCUMENT ANYMORE DEFINITIONS ▸ Source - your old Magento database ▸ Destination - your new Magento 2 database ▸ Document - mysql table ▸ Field - mysql column ▸ https://github.com/magento/data-migration-tool/blob/ master/etc/ce-to-ee/map-eav.xml.dist MARCIN SZTERLING REDBOX DIGITAL @php4u
22. ETC/CE-TO-EE/MAP-EAV.XML.DIST MARCIN SZTERLING REDBOX DIGITAL @php4u
23. RUN, NOTE ERRORS, FIX, REPEAT… ERRORS ERRORS ERRORS ▸ [ERROR]: Source fields not mapped. Document: sales_flat_order. Fields: field1, field2 ▸ [ERROR]: Destination fields not mapped. Document: customer_entity. Fields: field1, field2 ▸ Interesting script - http://oyenetwork.com/articles/data-migration- to-magento2-case-study/ ▸ Problem with EAV relationships after migration ▸ Error messages are not detailed enough (Mismatch of entities in the document) MARCIN SZTERLING REDBOX DIGITAL @php4u
24. LIFE SAVERS! TIPS ▸ Divide and Conquer - prepare config.xml per task group ▸ Try with smaller data samples - it is easier to debug it ▸ Use -vvv for extended debugging ▸ Use -r to restart process ▸ Source and Destination resource classes are disabling foreign keys check ▸ Password rehash is performance killer ▸ Create your own steps, create your own handlers ▸ Some of the attributes might be required in M2 and are not in M1 (telephone) MARCIN SZTERLING REDBOX DIGITAL @php4u
25. LIFE SAVERS! TIPS ▸ Data migration should be your first task - copy data on top of clean Magento 2 installation ▸ Ideally try to have two databases on same server ▸ Magento 1 instance can be live during migration ▸ Docs - http://devdocs.magento.com/guides/v2.0/migration/bk-migration- guide.html ▸ Logging can be extended ▸ The versions of the Data Migration Tool and your Magento software must exactly match. For example, if you’re using version 2.0.2 of the Magento software, you must use version 2.0.2 of the Data Migration Tool. MARCIN SZTERLING REDBOX DIGITAL @php4u
26. LIFE SAVERS! TIPS ▸ Use scripting to have repeatable process ▸ Do many mysql dumps in the process, so data can be reverted to previous state ▸ DROP DATABASE …, CREATE DATABASE … ▸ Keep your files separate from core data-migration files MARCIN SZTERLING REDBOX DIGITAL @php4u
27. HOW QUICK IS QUICK? RESULTS ▸ 20k customers - minutes ▸ 300k products - 1 hour MARCIN SZTERLING REDBOX DIGITAL @php4u
28. HOPE YOU LIKED IT! THANKS! @php4u Image: http://justsomething.co/ MARCIN SZTERLING REDBOX DIGITAL