Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DCLA 2013: Move into Drupal using Migrate

btmash
July 13, 2013

DCLA 2013: Move into Drupal using Migrate

Bring your content into Drupal by using the powerful Migrate module.

btmash

July 13, 2013
Tweet

More Decks by btmash

Other Decks in Technology

Transcript

  1. DrupalCampLA 2013 Move into Drupal with Migrate Agenda Introduction to

    Migrate Theory Implementation hooks classes Migration Handlers Alterations • • • • • • • •
  2. DrupalCampLA 2013 Move into Drupal with Migrate About the Presenter

    Ashok Modi BTMash (http://drupal.org/user/60422) Very interested in content upgrades/migrations. Contributed modules since 2006. Contributed to core since 2010. Upgrade path maintainer: 2012 - 2013. • • • • • •
  3. DrupalCampLA 2013 Move into Drupal with Migrate Thanks Mike Ryan

    (mikeryan) http://drupal.org/user/4420 Moshe Weitzman (moshe weitzman) http://drupal.org/user/23 Frank Carey http://drupal.org/user/112063 • • • • • •
  4. DrupalCampLA 2013 Move into Drupal with Migrate Disclaimers Ask questions.

    Have something to show? Come on up! Try to walk through demos if we have time. Primarily focusing on Migrate 2.6 since that should be out shortly. • • • •
  5. DrupalCampLA 2013 Move into Drupal with Migrate Migration Options By

    hand Great unless you value your time. Custom scripts Flexible Ok as a ‘one-off’ solution. Need to do / plan a lot on your own. • • • • • •
  6. DrupalCampLA 2013 Move into Drupal with Migrate Migration Options -

    Feeds Pros Easy(ish) to setup. Map fields from source -> destination Can import from various sources. JSON, XML, CSV, DB, even LDAP Well Documented • • • • • •
  7. DrupalCampLA 2013 Move into Drupal with Migrate Migration Options -

    Feeds Cons Performance issues. Content update handling. Doesn’t work particularly well with references. Feeds Tamper will only take you so far. • • • • •
  8. DrupalCampLA 2013 Move into Drupal with Migrate Migrate Powerful Framework

    to bring content into Drupal. Already defined many input sources. CSV, JSON, XML, DB Support to migrate into various types of content. All core entities out of the box. Can define own handler. Can import into a particular table! • • • • • • •
  9. DrupalCampLA 2013 Move into Drupal with Migrate Migrate D6 version

    requires autoload and dbtng modules. Code looks exactly the same :) Migrate Extras provides support for many contrib modules. EntityAPI for supporting non-core entities. More field modules implementing field handlers. Most comprehensive example to look at would be migrate examples (specifically beer.inc and • • • • •
  10. DrupalCampLA 2013 Move into Drupal with Migrate Goals Source sid

    title user field1 field2 ... fieldN Destination content_id(auto) title uid field1 field2 ... fieldN
  11. DrupalCampLA 2013 Move into Drupal with Migrate Source Interface to

    your current list of data. Provides a list of fields. Responsible over iterating through rows of data. • • •
  12. DrupalCampLA 2013 Move into Drupal with Migrate Destination Responsible for

    specific type of content in Drupal node, comment, row in a table **Each source content row correlates to one destination record. • • •
  13. DrupalCampLA 2013 Move into Drupal with Migrate Field Mappings Links

    a source field to a destination field. Basic function such as splitting into an array based on separators, etc. Can pass additional arguments (as defined by field handler). • • •
  14. DrupalCampLA 2013 Move into Drupal with Migrate Mappings (goal) Source

    sid title user field1 field2 field3 field4 ... fieldN Destination content_id(auto) title uid field1 field2 ... fieldN Mapping (alter and reference) Mapping
  15. DrupalCampLA 2013 Move into Drupal with Migrate Map Connects the

    Source and Destination IDs allowing for translations between them. Track key schema format. Allows migrate to re-run and update existing records. Allows imported records to be removed. Allows referencing in another migration. • • • • •
  16. DrupalCampLA 2013 Move into Drupal with Migrate Migration Map Source

    sid title user field1 field2 field3 field4 ... fieldN Destination content_id(auto) title uid field1 field2 ... fieldN Mapping (alter and reference) Map Table
  17. DrupalCampLA 2013 Move into Drupal with Migrate Migration Sets up

    all necessary pieces Source Destination Map Field Mappings May provide logic for skipping over rows during migration. May alter source data during migration. • • • • • • •
  18. DrupalCampLA 2013 Move into Drupal with Migrate Field Handler Converts

    source data into a format Drupal understands. $row->bar = array('foo', 'bar') into$entity_field_bar = array( 'und' => array( 0 => array('value' => 'foo'), 1 => array('value' => 'bar'), ), ); • •
  19. DrupalCampLA 2013 Move into Drupal with Migrate Destination Handler Extends

    destinations and adds additional functionality. MigrateCommentNodeHandler allows for allowing comments on a given node. Flag module would be a candidate to get handled this way. • • •
  20. DrupalCampLA 2013 Move into Drupal with Migrate Migration Map Source

    sid title user field1 field2 field3 field4 ... fieldN Destination content_id(auto) title (text) uid field1 (image) field2 ... fieldN (tags) Mapping (alter and reference) Map Table
  21. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: Getting Started

    Let migrate know about your module (hook) Build a migration class (constructor) Provide a description Provide information where content is coming from (Source) Provide information where content is going to get saved (Destination) Map fields from Source to Destination • • • • • •
  22. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: Getting Started

    Optional Components Massage data / add fields you were not able to get in initial mapping Add / massage any data that does not have field handlers before it gets saved • • •
  23. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: Hooks Just

    one (though rather large) Provide api version number along a few other details: function my_module_migrate_api() { $api = array( 'api' => 2, 'groups' => array( 'courses' => array( 'title' => t('Course Related Migrations')), 'policies' => array( 'title' => t('Intranet Related Migrations')), ), • • •
  24. DrupalCampLA 2013 Move into Drupal with Migrate Implementation Class 1

    (required) function and 3 (optional) functions class MyBundleMigration extends Migration { public function __construct() { ... } # REQ'D. public function prepareRow($row) public function prepare($entity, $row) public function complete($entity, $row) } • •
  25. DrupalCampLA 2013 Move into Drupal with Migrate Import Flow Source

    iterates until it finds an appropriate record Calls prepareRow(row) letting you modify or reject the data in a row. Migration applies the mappings and field handlers to convert $row into $entity. Calls prepare($entity, $row) to modify the entity before it gets saved. Entity is finally saved! • • • • •
  26. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Set

    up source, destination, field mappings in constructor. class MyBundleMigration extends Migration { public function __construct($arguments) { parent::__construct(); $this->source = <my_source>; $this->destination = <my_destination>; $this->map = <my_map>; $this->addFieldMapping( $my_dest_fld, $my_src_fld); } } • •
  27. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Source

    fields Lets migration class know a little about the fields that are coming in (like compound fields) Can set it to an array if nothing complex. $source_fields = array( ‘mtid’ => t(‘Source row ID’), ‘compound_field_1’ => t(‘Field not from query’) ); • • • •
  28. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Source

    (Current Database) // Required $query = db_select('my_table', 'mt'); $query->fields('mt', array('mtid', 'style', 'details', 'updated', 'style_parent', 'style_image')); $query->join('mt_extras', 'mte', 'mt.mtid = mte.mtid'); $query->orderBy('mt.updated', 'ASC'); // Implement a count_query if it is different. Or set to NULL. • •
  29. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() External

    Database // Using another db connection called 'for_migration'. $connection = Database::getConnection('for_migration'); $query = $connection->select('my_table', 'mt'); $query->fields('mt', array('mtid', 'style', 'details', 'updated', 'style_parent', 'style_image')); $query->orderBy('mt.updated', 'ASC'); // Implement a count_query if it is different. Or set to NULL. • •
  30. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() CSV

    File // The definition of the columns. Keys are integers // values are an array of: field name then description. $columns = array( 0 => array('cvs_uid', 'Id'), 1 => array('email', 'Email'), 2 => array('name', 'Name'), 3 => array('date', 'Date'), ); • •
  31. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Other

    Sources Comes with base source migration classes to migrate from JSON, XML, File Directories. Expect to make some changes depending on the migration format. • • •
  32. DrupalCampLA 2013 Move into Drupal with Migrate Source Base Classes

    If you have source IDs referenced separately from your values Use MigrateSourceList as a source Implement MigrateList for fetching counts and IDs, and MigrateItem for fetching values. If everything is in a single file with IDs mixed in use MigrateSourceMultiItems as a source. Implement MigrateItems for extracting IDs and values. • • • • • •
  33. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Migration

    Map $this->map = new MigrateSQLMap( $this->machineName, // Describe your primary ID schema array( 'mtid' => array( 'type' => 'integer', 'unsigned' => TRUE, 'not null' => TRUE, 'alias' => 'mt' ), ), MigrateDestinationNode::getKeySchema() • •
  34. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Highwater

    Migrate feature to figure out if a piece of content can be updated rather than inserted just once. Need to let migrate know which column contains highwater data. $this->highwaterField =array( ‘name’ => ‘updated’, ‘alias’ => ‘mt’ • • • • •
  35. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Destination

    // Terms $this->destination = new MigrateDestinationTerm(‘site_vocabulary’); // Nodes $this->destination = new MigrateDestinationNode(‘articles’) //Users $this->destination = new MigrateDestinationUser(); // Entity; Commerce Prouducts $this->destination = new • •
  36. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Field

    Mapping $this->addFieldMapping(‘dt_name’, ‘src_nm’); $this->addFieldMapping(‘uid’)- >defaultValue(1); $this->addFieldMapping(‘path’)- >issueGroup(‘DNM’); $this->addFieldMapping(‘tags’, ‘src_tags’)- >separator(‘,’); $this->addFieldMapping(‘field_body’, ‘desc’)- • • • • •
  37. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Field

    Mapping Most meta settings for fields now also field mappings. More things to map. Most no longer require arguments any more! • • • •
  38. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: __construct() Field

    Mapping: Source Migrations Your have a value from another migration and need to set up the new ID from the migration map. Content Author References $this->addFieldMapping(‘uid’, ‘author_id’- >sourceMigration(‘MyUserMigration’); • • • • •
  39. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: prepareRow($row) Passes

    row from the current source (after some of the mapping data has been processed) as an object so you can make modifications. Indicate a row should be skipped by returning FALSE; Add or change field values $row->field3 = $row->field4 . ‘ ‘ . $row->field5; $row->images = array($image1, $image2); • • • •
  40. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: prepare($entity, $row)

    Work directly with the entity object that has been populated with field mappings. Final opportunity for changes before $entity is saved. Must save fields in entity field format. Use prepare to populate fields that do not have a field handler (relation, location, maybe others?) $entity->field_relation[‘und’][0][‘value’] = 100; • • • • •
  41. DrupalCampLA 2013 Move into Drupal with Migrate Implementation: complete($entity, $row)

    Called after entity is saved - chance to update any *other* records that reference the current entity or perform other functions. Do *NOT* save same record again. Its bad. • • •
  42. DrupalCampLA 2013 Move into Drupal with Migrate Circular Dependencies Implement

    stubs (http://drupal.org/node/1013506) Specify a source migration(‘NodeBundleMigration’) on the IDs field mapping. Add createStub($migration, $source_key) to NodeBundleMigration which creates an empty record and returns an ID. When the NodeBundleMigration runs, it will • • • •
  43. DrupalCampLA 2013 Move into Drupal with Migrate Dealing with Dynamic

    Migrations Some projects like wordpress migrate / commerce migrate will migrate most but not all of your content. Extend by creating a destination migration. Same as regular migration but you provide type of record and map on id. $this->systemOfRecord = Migration::DESTINATION; $this->addFieldMapping(‘nid’, ‘nid’)- • • • •
  44. DrupalCampLA 2013 Move into Drupal with Migrate Suggestions Separate your

    file migrations Migrate has a class to migrate your files separately. Can retain structure of source file directory Or specify in prepareRow/prepare where you want the file to move. Have your content then use the file migration(s) as a dependency source migration. • • • • •
  45. DrupalCampLA 2013 Move into Drupal with Migrate References Projects http://drupal.org/project/migrate

    http://drupal.org/project/migrate_extras https://drupal.org/project/migrate_d2d http://drupal.org/sandbox/btmash/1092900 http://drupal.org/sandbox/btmash/1492598 • • • • • •