Share
Embed
Magento 2 Data Migration
Take on topic of Data migration from Magento 1.x to Magento 2.0. Available tools, valuable insights and gothas. YT video is here - https://www.youtube.com/watch?v=UBK1rNjMauo
php4u
March 16, 2016
Transcript
28 slides
-
MAGE1 >> MAGE2 NOT EVERYTHING THAT CAN BE COUNTED COUNTS,
AND NOT EVERYTHING THAT COUNTS CAN BE COUNTED. ALBERT EINSTEIN, PHYSICIST (16.03.2016) LONDON MAGENTO MEETUP MARCIN SZTERLING REDBOX DIGITAL @php4u -
▸ Data migration is the process of importing legacy data,
from one or more old systems, to a new system or systems. ▸ Data migration is typically part of a larger programme ▸ “Big bang” and “trickle” migrations WHAT IS DATA MIGRATION? MARCIN SZTERLING REDBOX DIGITAL @php4u -
MIGRATE DATA ▸ Should we bring all or part of
the data? ▸ If just part, which parts? - based on creation date? based on status of product, customer? or some combination of criteria? ▸ If we choose to bring over data on or after a certain date, which should it be - last 3 months, last 6 months, last year?...should it be all the data? ▸ Should we clean data at source or destination? BEFORE ALL STARTS MARCIN SZTERLING REDBOX DIGITAL @php4u -
CHANGING TIRES AT 60 MPH THIS IS NOT A 1
DAY TASK ▸ CE -> CE, CE -> EE, EE -> EE ▸ There is no silver-bullet solution ▸ There will be issues with source data ▸ There will be issues with migrated data ▸ Amount of data matters ▸ Reserve time for it and prepare for battle! ▸ http://alankent.me/2015/01/31/ magento-1-to-2-migration-tool-plans/ MARCIN SZTERLING REDBOX DIGITAL @php4u -
WHAT DATA DO WE NEED? MAIN DATA TO BE MIGRATED
(WHAT THE HELL IS EAV??) ▸ Products ▸ Categories (and its products relations) ▸ Customers ▸ Orders ▸ Configurations ▸ CMS elements ▸ Websites, Store groups, Stores ▸ Media assets for Products, Categories, CMS MARCIN SZTERLING REDBOX DIGITAL @php4u -
WHAT EXACTLY HAS CHANGED FROM M1 TO M2? ▸ Tables
can be deleted ▸ Tables can be renamed ▸ New tables can exist in Magento 2 but not in Magento 1 ▸ Table columns can be deleted ▸ Table columns can be renamed ▸ New table columns can exist in Magento 2 but not in Magento 1 MARCIN SZTERLING REDBOX DIGITAL @php4u -
SOME OF THE DATA MUST BE TRANSFORMED ▸ Values must
be changed to match Magento 2 new classes namespaces (models) ▸ Values must be nulled even if are empty ▸ Names must be prefixed to avoid naming issues WHAT EXACTLY HAS CHANGED FROM M1 TO M2? MARCIN SZTERLING REDBOX DIGITAL @php4u -
DO YOUR HOMEWORK! RIGHT TOOLS TO DO RIGHT JOB MARCIN
SZTERLING REDBOX DIGITAL @php4u -
WASHING AND IRONING AS WELL?? CRITERIA ▸ Ease of use
▸ Speed ▸ Security ▸ Accuracy ▸ Repeatable ▸ Multi-project MARCIN SZTERLING REDBOX DIGITAL @php4u -
IN THE WILD WHAT IS AVAILABLE? ▸ Ubertheme Magento 2
data migration (1.0.4) https://github.com/ubertheme/module-ubdatamigration ▸ Magento data migration tool (2.0.2) https://github.com/magento/data-migration-tool ▸ CSV based import ▸ Custom own written solution (API?) ▸ https://github.com/SchumacherFM/Magento2-Data- Migration MARCIN SZTERLING REDBOX DIGITAL @php4u -
JUST WORKS? UBERTHEME MIGRATION TOOL ▸ Yii framework based browser
tool ▸ Not secured at all so you must not use it on production unless secured ▸ Almost zero configuration required ▸ Slow and dependant on server settings ▸ Supports delta changes and can be migrated on top of existing data ▸ Does the job! ▸ More info how to use it here https://github.com/ubertheme/module-ubdatamigration/blob/master/ README.md MARCIN SZTERLING REDBOX DIGITAL @php4u -
CARTE BLANCHE MAGENTO DATA MIGRATION TOOL ▸ Now one tool
(2.0.2) for both EE and CE ▸ Preconfigured for all magento versions and editions ▸ Works with magento sample data (!) ▸ Supports delta changes ▸ Cannot be migrated on top of existing data ▸ Requires “perfect” source data integrity (ex. eav_entity_type) ▸ Blazing fast (with exceptions) ▸ xml based configuration - which can be stored in VCS, can have multiple profiles ▸ No way to specify database connection details in CLI ▸ Very flexible configuration to adjust to needs, easy to extend ▸ Progress bar with estimated time per task (but not for overall task) ▸ No automated data verification after import MARCIN SZTERLING REDBOX DIGITAL @php4u -
COMPOSER WHAT? INSTALLATION ▸ http://devdocs.magento.com/guides/v2.0/migration/ migration-tool-install.html ▸ composer config repositories.data-migration-tool
git https://github.com/magento/data-migration-tool ▸ composer require magento/data-migration-tool:2.0.2 ▸ Installed in vendor/magento/data-migration ▸ bin/magento migrate:data [-r|--reset] {<path to config.xml>} MARCIN SZTERLING REDBOX DIGITAL @php4u -
WHERE TO START? MAGENTO DATA MIGRATION TOOL ▸ 3 main
types of run modes in Migration/Console Data, Delta, Settings ▸ Data is read, transformed and then saved in batches ▸ CPU and memory heavy MARCIN SZTERLING REDBOX DIGITAL @php4u -
STEPS <?xml version="1.0" encoding="UTF-8"?> <config xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xs:noNamespaceSchemaLocation="../../config.xsd"> <steps mode="settings"> <step
title="Settings Step"> <integrity>Migration\Step\Settings\Integrity</integrity> <data>Migration\Step\Settings\Data</data> </step> <step title="Stores Step"> <integrity>Migration\Step\Stores\Integrity</integrity> <data>Migration\Step\Stores\Data</data> <volume>Migration\Step\Stores\Volume</volume> </step> </steps> </config> MARCIN SZTERLING REDBOX DIGITAL @php4u -
WHERE, FROM AND MORE SETTINGS… <?xml version="1.0" encoding="UTF-8"?> <config xmlns:xs="http://www.w3.org/2001/XMLSchema-instance"
xs:noNamespaceSchemaLocation="../../config.xsd"> <source> <database host="localhost" name="magento1" user="root"/> </source> <destination> <database host="localhost" name="magento2" user="root"/> </destination> <options> <map_file>etc/ce-to-ee/1.7.0.2/map.xml.dist</map_file> <eav_map_file>etc/ce-to-ee/map-eav.xml.dist</eav_map_file> <eav_document_groups_file>etc/ce-to-ee/eav-document-groups.xml.dist</eav_document_groups_file> <eav_attribute_groups_file>etc/ce-to-ee/eav-attribute-groups.xml.dist</eav_attribute_groups_file> <log_map_file>etc/ce-to-ee/map-log.xml.dist</log_map_file> <log_document_groups_file>etc/ce-to-ee/log-document-groups.xml.dist</log_document_groups_file> <settings_map_file>etc/ce-to-ee/settings.xml.dist</settings_map_file> <customer_map_file>etc/ce-to-ee/map-customer.xml.dist</customer_map_file> <customer_document_groups_file>etc/ce-to-ee/customer-document-groups.xml.dist</customer_document_groups_file> <customer_attribute_groups_file>etc/ce-to-ee/customer-attribute-groups.xml.dist</customer_attribute_groups_file> <delta_document_groups_file>etc/ce-to-ee/deltalog.xml.dist</delta_document_groups_file> <order_grids_document_groups_file>etc/ce-to-ee/order-grids-document-groups.xml.dist</order_grids_document_groups_file> <map_document_groups>etc/ce-to-ee/map-document-groups.xml.dist</map_document_groups> <class_map>etc/ce-to-ee/class-map.xml.dist</class_map> <bulk_size>0</bulk_size> <direct_document_copy>0</direct_document_copy> <source_prefix /> <dest_prefix /> <auto_resolve_urlrewrite_duplicates>0</auto_resolve_urlrewrite_duplicates> <log_file>migration.log</log_file> <progress_bar_format>%percent%% [%bar%] Remaining Time: %remaining%</progress_bar_format> <upgrade_customer_password_hash>1</upgrade_customer_password_hash> <crypt_key /> </options> </config> http://devdocs.magento.com/guides/v2.0/migration/migration-tool-internal-spec.html MARCIN SZTERLING REDBOX DIGITAL @php4u -
WHEN DOCUMENT IS NOT DOCUMENT ANYMORE DEFINITIONS ▸ Source -
your old Magento database ▸ Destination - your new Magento 2 database ▸ Document - mysql table ▸ Field - mysql column ▸ https://github.com/magento/data-migration-tool/blob/ master/etc/ce-to-ee/map-eav.xml.dist MARCIN SZTERLING REDBOX DIGITAL @php4u -
RUN, NOTE ERRORS, FIX, REPEAT… ERRORS ERRORS ERRORS ▸ [ERROR]:
Source fields not mapped. Document: sales_flat_order. Fields: field1, field2 ▸ [ERROR]: Destination fields not mapped. Document: customer_entity. Fields: field1, field2 ▸ Interesting script - http://oyenetwork.com/articles/data-migration- to-magento2-case-study/ ▸ Problem with EAV relationships after migration ▸ Error messages are not detailed enough (Mismatch of entities in the document) MARCIN SZTERLING REDBOX DIGITAL @php4u -
LIFE SAVERS! TIPS ▸ Divide and Conquer - prepare config.xml
per task group ▸ Try with smaller data samples - it is easier to debug it ▸ Use -vvv for extended debugging ▸ Use -r to restart process ▸ Source and Destination resource classes are disabling foreign keys check ▸ Password rehash is performance killer ▸ Create your own steps, create your own handlers ▸ Some of the attributes might be required in M2 and are not in M1 (telephone) MARCIN SZTERLING REDBOX DIGITAL @php4u -
LIFE SAVERS! TIPS ▸ Data migration should be your first
task - copy data on top of clean Magento 2 installation ▸ Ideally try to have two databases on same server ▸ Magento 1 instance can be live during migration ▸ Docs - http://devdocs.magento.com/guides/v2.0/migration/bk-migration- guide.html ▸ Logging can be extended ▸ The versions of the Data Migration Tool and your Magento software must exactly match. For example, if you’re using version 2.0.2 of the Magento software, you must use version 2.0.2 of the Data Migration Tool. MARCIN SZTERLING REDBOX DIGITAL @php4u -
LIFE SAVERS! TIPS ▸ Use scripting to have repeatable process
▸ Do many mysql dumps in the process, so data can be reverted to previous state ▸ DROP DATABASE …, CREATE DATABASE … ▸ Keep your files separate from core data-migration files MARCIN SZTERLING REDBOX DIGITAL @php4u -
HOW QUICK IS QUICK? RESULTS ▸ 20k customers - minutes
▸ 300k products - 1 hour MARCIN SZTERLING REDBOX DIGITAL @php4u