轉移百萬筆資料 + 使用者無縫接合

2013年7月6日 Drupal Camp Taipei 2013

Transcript

1. 轉移百萬筆資料 + 使用者無縫接合 2013年7月6日 Drupal Camp Taipei 2013
2. About me • Joe Tsui，香港人 • 5年 Drupal 開發經驗 • Drupal Camp Taipei 2011 講者 • Imagefield_crop maintainer • 成立了 ，接洽 Drupal 專案 • 為香港最大的英文報紙(scmp) 和信報負責 Drupal 開發 • 香港最活躍 Drupal blogger 之一 • Hong Kong Drupal User group 成員
3. Warning Programming ahead
4. 轉移百萬筆資料 • 1,200,000+ entity in total • Data from 2008 • 12G images • do not exists user table!
5. 轉移百萬筆資料 • Source • Destination • Key Mapping • Field Mapping • Migration flow • PrepareRow • Prepare • Complete, CompleteRollback • Test run • Rollback • Tricks
6. 轉移百萬筆資料 Legacy system Drupal Migrate module (v2.5)
7. 轉移百萬筆資料 Legacy system • Article • Comment • User • • Banned user • Image, File Drupal • Node • Comment • User • User Picture • Banned user • Image, File Source Destination
8. 轉移百萬筆資料 - Source • MigrateSourceSQL • MigrateSourceCSV • MigrateSourceList (XML) • MigrateSourceMSSQL • MigrateSourceOracle
9. 轉移百萬筆資料 - Destination • MigrateDestinationNode • MigrateDestinationComment • MigrateDestinationUser • MigrateDestinationRoles • MigrateDestinationTable
10. 轉移百萬筆資料 • Source • Destination • Key Mapping • Field Mapping • Migration flow • PrepareRow • Prepare • Complete, CompleteRollback • Test run • Rollback • Tricks Mappings
11. Key Mappings Legacy: Article id: 15368 Node: Nid : 15368 Unique identifier as Key • 一對一的關係 • 唯一的 source entity • 對應一個 destination entity
12. Key Mappings Example: {node}
13. Field Mappings /admin/content/migrate/ForumArticle
14. Migration flow 1. Migrate::__construct() 2. Migrate::prepareRow() 3. Migrate::prepare() 4. Migrate - DB writes 5. Migrate::complete()
15. 轉移百萬筆資料 • Source • Destination • Key Mapping • Field Mapping • Migration flow • PrepareRow • Prepare • Complete, CompleteRollback • Test run • Rollback • Tricks Extra customization
16. prepareRow($row) Entity (destination) Row(source)
17. prepare($entity, $row) • Connect to other migration (user, user picture)
18. complete($entity, $row) Missing parts in other tables
19. 轉移百萬筆資料 • Source • Destination • Key Mapping • Field Mapping • Migration flow • PrepareRow • Prepare • Complete, CompleteRollback • Test run • Rollback • Tricks Run time
20. Test run • drush mi ForumArticle --limit="10 items" • Tip: Add order by • Tip: Add condition to MigrateSourceSQL to pin point specific data Conditions
21. Rollback • Rollback 比 import 慢得多 • Rollback on 1M+ rows • 在 import 之前用 sql-dump 建立原始狀態 snapshot • mysql < backup.sql to mass “rollback”
22. tips 1 • Test 策略: 找出 data 的所有組合 ◦ post with 1 image: nid: 123, 456 ◦ post with 2 images ◦ deleted post ◦ //deleted post, with 1 image ◦ "spammed" post ◦ "comment closed" post • 不移動 image ◦ preserve_files')->defaultValue(TRUE); ◦ file_replace')->defaultValue(MigrateFile:: FILE_EXISTS_REUSE); • 預先將 files 放到 files/legacy
23. tips 2 • user picture -> fid ◦ 用戶頭像是 {user} 內的一個 fid ◦ 先建立 files, 再 import User 之後在 prepare() 插入 fid • field_image[] for multiple values $this->addFieldMapping('field_image', 'field_image_value'); //prepareRow($row) $row->field_image_value[] = substr(trim($image_path), 1);
24. 使用者無縫接合 例子 • auth.joe.com • forum.joe.com • news.joe.com
25. 使用者無縫接合 • Active action ◦ Login API ◦ Provider side ◦ Active logout ◦ Provider side • Passive action ◦ cookie API ◦ passive logout ◦ Drupal side
26. Active Action - login API forum.joe.com/user auth.joe.com/auth/login Auth? New user? Register set cookie user_login_submit() submit No Yes Yes No
27. Active Action - login API forum.joe.com/user auth.joe.com/auth/login Auth? New user? Register set cookie user_login_submit() submit No Yes Yes No
28. Active Action - login API forum.joe.com/user auth.joe.com/auth/login submit
29. Active Action - login API forum.joe.com/user auth.joe.com/auth/login Auth? New user? Register set cookie user_login_submit() submit No Yes Yes No
30. Active Action - login API New user? Register set cookie Yes No Register
31. Cookie • auth.joe.com • forum.joe.com • news.joe.com • Set cookie • $cookie_domain = '.joe.com'
32. auth.joe.com (Provider) Active Action - login API forum.joe.com/user auth.joe.com/auth/login Auth? New user? Register set cookie user_login_submit() submit No Yes Yes No Save Session Cookie value agreement news.joe.com
33. Active action - logout • clear cookie • Other sites should check cookie on page loads hook_init()
34. Passive Action - cookie API forum.joe.com/node/123 auth.joe.com/auth/authenticate Auth? New user? Register set cookie user_login_submit() found a cookie No Yes Yes No
35. Passive Action - cookie API forum.joe.com/node/123 auth.joe.com/auth/authenticate Auth? New user? Register set cookie user_login_submit() found a cookie No Yes Yes No
36. Passive Action - cookie API forum.joe.com/node/123 auth.joe.com/auth/authenticate found a cookie
37. Passive action - logout • Drupal logged in • on page loads, cookie value not found • logout Drupal as well
38. 使用者無縫接合 tips • Browser private mode • Chrome develop resources tab to check cookie • login tester form
39. 現場示範－網站改造馬拉松 • Ubuntu 正體中文站 • http://www.ubuntu-tw.org/