Tweet
Tweet

Slide 1

Slide 1 text

DrupalCampLA 2013 Move into Drupal with Migrate Move into Drupal with Migrate

Slide 2

Slide 2 text

DrupalCampLA 2013 Move into Drupal with Migrate Agenda Introduction to Migrate Theory Implementation hooks classes Migration Handlers Alterations • • • • • • • •

Slide 3

Slide 3 text

DrupalCampLA 2013 Move into Drupal with Migrate About the Presenter Ashok Modi BTMash (http://drupal.org/user/60422) Very interested in content upgrades/migrations. Contributed modules since 2006. Contributed to core since 2010. Upgrade path maintainer: 2012 - 2013. • • • • • •

Slide 4

Slide 4 text

DrupalCampLA 2013 Move into Drupal with Migrate Thanks Mike Ryan (mikeryan) http://drupal.org/user/4420 Moshe Weitzman (moshe weitzman) http://drupal.org/user/23 Frank Carey http://drupal.org/user/112063 • • • • • •

Slide 5

Slide 5 text

DrupalCampLA 2013 Move into Drupal with Migrate Disclaimers Ask questions. Have something to show? Come on up! Try to walk through demos if we have time. Primarily focusing on Migrate 2.6 since that should be out shortly. • • • •

Slide 6

Slide 6 text

DrupalCampLA 2013 Move into Drupal with Migrate Migration Options By hand Great unless you value your time. Custom scripts Flexible Ok as a ‘one-off’ solution. Need to do / plan a lot on your own. • • • • • •

Slide 7

Slide 7 text

DrupalCampLA 2013 Move into Drupal with Migrate Migration Options - Feeds Pros Easy(ish) to setup. Map fields from source -> destination Can import from various sources. JSON, XML, CSV, DB, even LDAP Well Documented • • • • • •

Slide 8

Slide 8 text

DrupalCampLA 2013 Move into Drupal with Migrate Migration Options - Feeds Cons Performance issues. Content update handling. Doesn’t work particularly well with references. Feeds Tamper will only take you so far. • • • • •

Slide 9

Slide 9 text

DrupalCampLA 2013 Move into Drupal with Migrate Migrate Powerful Framework to bring content into Drupal. Already defined many input sources. CSV, JSON, XML, DB Support to migrate into various types of content. All core entities out of the box. Can define own handler. Can import into a particular table! • • • • • • •

Slide 10

Slide 10 text

DrupalCampLA 2013 Move into Drupal with Migrate

Slide 11

Slide 11 text

DrupalCampLA 2013 Move into Drupal with Migrate Migrate D6 version requires autoload and dbtng modules. Code looks exactly the same :) Migrate Extras provides support for many contrib modules. EntityAPI for supporting non-core entities. More field modules implementing field handlers. Most comprehensive example to look at would be migrate examples (specifically beer.inc and • • • • •

Slide 12

Slide 12 text

DrupalCampLA 2013 Move into Drupal with Migrate Goals Source sid title user field1 field2 ... fieldN Destination content_id(auto) title uid field1 field2 ... fieldN

Slide 13

Slide 13 text

DrupalCampLA 2013 Move into Drupal with Migrate Source Interface to your current list of data. Provides a list of fields. Responsible over iterating through rows of data. • • •

Slide 14

Slide 14 text

DrupalCampLA 2013 Move into Drupal with Migrate Destination Responsible for specific type of content in Drupal node, comment, row in a table **Each source content row correlates to one destination record. • • •

Slide 15

Slide 15 text

DrupalCampLA 2013 Move into Drupal with Migrate Field Mappings Links a source field to a destination field. Basic function such as splitting into an array based on separators, etc. Can pass additional arguments (as defined by field handler). • • •

Slide 16

Slide 16 text

DrupalCampLA 2013 Move into Drupal with Migrate Mappings (goal) Source sid title user field1 field2 field3 field4 ... fieldN Destination content_id(auto) title uid field1 field2 ... fieldN Mapping (alter and reference) Mapping

Slide 17

Slide 17 text

DrupalCampLA 2013 Move into Drupal with Migrate Map Connects the Source and Destination IDs allowing for translations between them. Track key schema format. Allows migrate to re-run and update existing records. Allows imported records to be removed. Allows referencing in another migration. • • • • •

Slide 18

Slide 18 text

DrupalCampLA 2013 Move into Drupal with Migrate Migration Map Source sid title user field1 field2 field3 field4 ... fieldN Destination content_id(auto) title uid field1 field2 ... fieldN Mapping (alter and reference) Map Table

Slide 19

Slide 19 text

DrupalCampLA 2013 Move into Drupal with Migrate Migration Sets up all necessary pieces Source Destination Map Field Mappings May provide logic for skipping over rows during migration. May alter source data during migration. • • • • • • •

Slide 20

Slide 20 text

DrupalCampLA 2013 Move into Drupal with Migrate Field Handler Converts source data into a format Drupal understands. $row->bar = array('foo', 'bar') into$entity_field_bar = array( 'und' => array( 0 => array('value' => 'foo'), 1 => array('value' => 'bar'), ), ); • •

Slide 21

Slide 21 text

DrupalCampLA 2013 Move into Drupal with Migrate Destination Handler Extends destinations and adds additional functionality. MigrateCommentNodeHandler allows for allowing comments on a given node. Flag module would be a candidate to get handled this way. • • •

Slide 22

Slide 22 text

DrupalCampLA 2013 Move into Drupal with Migrate Migration Map Source sid title user field1 field2 field3 field4 ... fieldN Destination content_id(auto) title (text) uid field1 (image) field2 ... fieldN (tags) Mapping (alter and reference) Map Table

Slide 23

Slide 23 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: Getting Started Let migrate know about your module (hook) Build a migration class (constructor) Provide a description Provide information where content is coming from (Source) Provide information where content is going to get saved (Destination) Map fields from Source to Destination • • • • • •

Slide 24

Slide 24 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: Getting Started Optional Components Massage data / add fields you were not able to get in initial mapping Add / massage any data that does not have field handlers before it gets saved • • •

Slide 25

Slide 25 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: Hooks Just one (though rather large) Provide api version number along a few other details: function my_module_migrate_api() { $api = array( 'api' => 2, 'groups' => array( 'courses' => array( 'title' => t('Course Related Migrations')), 'policies' => array( 'title' => t('Intranet Related Migrations')), ), • • •

Slide 26

Slide 26 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation Class 1 (required) function and 3 (optional) functions class MyBundleMigration extends Migration { public function __construct() { ... } # REQ'D. public function prepareRow($row) public function prepare($entity, $row) public function complete($entity, $row) } • •

Slide 27

Slide 27 text

DrupalCampLA 2013 Move into Drupal with Migrate Import Flow Source iterates until it finds an appropriate record Calls prepareRow(row) letting you modify or reject the data in a row. Migration applies the mappings and field handlers to convert $row into $entity. Calls prepare($entity, $row) to modify the entity before it gets saved. Entity is finally saved! • • • • •

Slide 28

Slide 28 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Set up source, destination, field mappings in constructor. class MyBundleMigration extends Migration { public function __construct($arguments) { parent::__construct(); $this->source = ; $this->destination = ; $this->map = ; $this->addFieldMapping( $my_dest_fld, $my_src_fld); } } • •

Slide 29

Slide 29 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Source fields Lets migration class know a little about the fields that are coming in (like compound fields) Can set it to an array if nothing complex. $source_fields = array( ‘mtid’ => t(‘Source row ID’), ‘compound_field_1’ => t(‘Field not from query’) ); • • • •

Slide 30

Slide 30 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Source (Current Database) // Required $query = db_select('my_table', 'mt'); $query->fields('mt', array('mtid', 'style', 'details', 'updated', 'style_parent', 'style_image')); $query->join('mt_extras', 'mte', 'mt.mtid = mte.mtid'); $query->orderBy('mt.updated', 'ASC'); // Implement a count_query if it is different. Or set to NULL. • •

Slide 31

Slide 31 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() External Database // Using another db connection called 'for_migration'. $connection = Database::getConnection('for_migration'); $query = $connection->select('my_table', 'mt'); $query->fields('mt', array('mtid', 'style', 'details', 'updated', 'style_parent', 'style_image')); $query->orderBy('mt.updated', 'ASC'); // Implement a count_query if it is different. Or set to NULL. • •

Slide 32

Slide 32 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() CSV File // The definition of the columns. Keys are integers // values are an array of: field name then description. $columns = array( 0 => array('cvs_uid', 'Id'), 1 => array('email', 'Email'), 2 => array('name', 'Name'), 3 => array('date', 'Date'), ); • •

Slide 33

Slide 33 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Other Sources Comes with base source migration classes to migrate from JSON, XML, File Directories. Expect to make some changes depending on the migration format. • • •

Slide 34

Slide 34 text

DrupalCampLA 2013 Move into Drupal with Migrate Source Base Classes If you have source IDs referenced separately from your values Use MigrateSourceList as a source Implement MigrateList for fetching counts and IDs, and MigrateItem for fetching values. If everything is in a single file with IDs mixed in use MigrateSourceMultiItems as a source. Implement MigrateItems for extracting IDs and values. • • • • • •

Slide 35

Slide 35 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Migration Map $this->map = new MigrateSQLMap( $this->machineName, // Describe your primary ID schema array( 'mtid' => array( 'type' => 'integer', 'unsigned' => TRUE, 'not null' => TRUE, 'alias' => 'mt' ), ), MigrateDestinationNode::getKeySchema() • •

Slide 36

Slide 36 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Highwater Migrate feature to figure out if a piece of content can be updated rather than inserted just once. Need to let migrate know which column contains highwater data. $this->highwaterField =array( ‘name’ => ‘updated’, ‘alias’ => ‘mt’ • • • • •

Slide 37

Slide 37 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Destination // Terms $this->destination = new MigrateDestinationTerm(‘site_vocabulary’); // Nodes $this->destination = new MigrateDestinationNode(‘articles’) //Users $this->destination = new MigrateDestinationUser(); // Entity; Commerce Prouducts $this->destination = new • •

Slide 38

Slide 38 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Field Mapping $this->addFieldMapping(‘dt_name’, ‘src_nm’); $this->addFieldMapping(‘uid’)- >defaultValue(1); $this->addFieldMapping(‘path’)- >issueGroup(‘DNM’); $this->addFieldMapping(‘tags’, ‘src_tags’)- >separator(‘,’); $this->addFieldMapping(‘field_body’, ‘desc’)- • • • • •

Slide 39

Slide 39 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Field Mapping Most meta settings for fields now also field mappings. More things to map. Most no longer require arguments any more! • • • •

Slide 40

Slide 40 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Field Mapping: Source Migrations Your have a value from another migration and need to set up the new ID from the migration map. Content Author References $this->addFieldMapping(‘uid’, ‘author_id’- >sourceMigration(‘MyUserMigration’); • • • • •

Slide 41

Slide 41 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: prepareRow($row) Passes row from the current source (after some of the mapping data has been processed) as an object so you can make modifications. Indicate a row should be skipped by returning FALSE; Add or change field values $row->field3 = $row->field4 . ‘ ‘ . $row->field5; $row->images = array($image1, $image2); • • • •

Slide 42

Slide 42 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: prepare($entity, $row) Work directly with the entity object that has been populated with field mappings. Final opportunity for changes before $entity is saved. Must save fields in entity field format. Use prepare to populate fields that do not have a field handler (relation, location, maybe others?) $entity->field_relation[‘und’][0][‘value’] = 100; • • • • •

Slide 43

Slide 43 text

DrupalCampLA 2013 Move into Drupal with Migrate Implementation: complete($entity, $row) Called after entity is saved - chance to update any *other* records that reference the current entity or perform other functions. Do *NOT* save same record again. Its bad. • • •

Slide 44

Slide 44 text

DrupalCampLA 2013 Move into Drupal with Migrate Circular Dependencies Implement stubs (http://drupal.org/node/1013506) Specify a source migration(‘NodeBundleMigration’) on the IDs field mapping. Add createStub($migration, $source_key) to NodeBundleMigration which creates an empty record and returns an ID. When the NodeBundleMigration runs, it will • • • •

Slide 45

Slide 45 text

DrupalCampLA 2013 Move into Drupal with Migrate Dealing with Dynamic Migrations Some projects like wordpress migrate / commerce migrate will migrate most but not all of your content. Extend by creating a destination migration. Same as regular migration but you provide type of record and map on id. $this->systemOfRecord = Migration::DESTINATION; $this->addFieldMapping(‘nid’, ‘nid’)- • • • •

Slide 46

Slide 46 text

DrupalCampLA 2013 Move into Drupal with Migrate Suggestions Separate your file migrations Migrate has a class to migrate your files separately. Can retain structure of source file directory Or specify in prepareRow/prepare where you want the file to move. Have your content then use the file migration(s) as a dependency source migration. • • • • •

Slide 47

Slide 47 text

DrupalCampLA 2013 Move into Drupal with Migrate References Projects http://drupal.org/project/migrate http://drupal.org/project/migrate_extras https://drupal.org/project/migrate_d2d http://drupal.org/sandbox/btmash/1092900 http://drupal.org/sandbox/btmash/1492598 • • • • • •

Slide 48

Slide 48 text

Move into Drupal with Migrate DrupalCampLA 2013 Demo / Questions / Notes Thank you :)