Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Our Automation Tool for Migrating 1,800 MySQL I...

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers.

Our Automation Tool for Migrating 1,800 MySQL Instances in Only Six Months

Avatar for Tech-Verse2022

Tech-Verse2022

November 17, 2022

More Decks by Tech-Verse2022

Other Decks in Technology

Transcript

  1. Agenda - Self Introduction - MySQL Migration in LINE -

    MySQL Upgrade Helper (MUH) - Internal Operation for Safety MySQL Migration - Summary
  2. Yoshinobu Harachi @yy_harachi - LINE Mid-Career (2020~) - DBA, ITSC

    DB dept MySQL1 team - MySQL Operation - Develop Operation Tools (PHP/Go) - Funny Experience - Won 2t Truck Driving Competition in my school days Hello, I'm …
  3. 6,000+ MySQL Instances in LINE Number of MySQL Instances 0

    1000 2000 3000 4000 5000 6000 7000 2016 2017 2018 2019 2020 2021 2022
  4. HW Scale Up New MySQL Platform in Verda MySQL Operation

    ⎯ 5.6 → 5.7 ⎯ 5.7 → 8.0 ⎯ Need more performance ⎯ Support for data growth ⎯ Load distribution ⎯ Commonize operations ⎯ Need more utilities MySQL Version Up Database Sharding
  5. MySQL 5.7 EOL CentOS6 EOS Closing Old MySQL Platform Responsibility

    with Deadline ⎯ Feb. 2021 ⎯ Oct. 2023 ⎯ Instances will be power-off ⎯ Not supported any more MySQL 5.6 EOL IDC Room Renovation ⎯ Nov. 2020
  6. Migration Tasks - Create New MySQL Instances - Add User

    ACL - Set MySQL Variables - Export/Import Data - Start Replication - Compare Data - Inspect Query Performance - Switch to New MySQL (in Midnight...?) - Shutdown Old MySQL
  7. !

  8. My Boss said … • Manual operation is dangerous •

    Operate MySQL migration via web browser • Everyone can do MySQL migration safely I want to do migration without SSH login!
  9. MUH API MySQL Query Replayer (MQR) Simple-Switchover API Role ⎯

    `CREATE USER` for MUH ⎯ Inspect Configuration ⎯ Set Variables ⎯ Export/Import Data ⎯ Start Replication ⎯ Check Metadata ⎯ Compare Data ⎯ etc. ⎯ Get/Set Variables ⎯ Get MySQL Status ⎯ etc. ⎯ Inspect Query Performance in Target MySQL ⎯ Switchover to Target MySQL Grant User API MyInfo API
  10. MUH History Too many MySQL migrations… ⎯ 2020/10~ (900 migrations)

    ⎯ DBA can manage MySQL migration. ⎯ Increasing MySQL instances ⎯ CentOS6 EOS / IDC room renovation / Closing old Verda platform DBA-MUH
  11. MUH History Too many MySQL migrations… MUH-Prod ⎯ 2020/10~ (900

    migrations) ⎯ DBA can manage MySQL migration. ⎯ Increasing MySQL instances ⎯ CentOS6 EOS / IDC room renovation / Closing old Verda platform ⎯ 2022/11~ ⎯ Developers can migrate MySQL themselves. DBA-MUH
  12. Migration Steps - Create New MySQL Instances - Spec Selection

    API return optimized instance spec (CPU/Memory/Disk)
  13. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  14. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  15. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  16. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  17. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  18. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  19. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  20. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  21. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  22. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  23. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  24. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  25. Inspection Purpose - Inspect both MySQL configuration - Request to

    DBA if there are any issues - Can the Source and Target MySQL be migrated?
  26. Non-Primary Key Table i_s.TABLES i_s.COLUMNS WHERE COLUMN_KEY='PRI' Non-InnoDB Table i_s.TABLES

    WHERE ENGINE != 'InnoDB' DEFINER ACL Exists TRIGGER/PROCEDURE/FUNCTION/VIEW/EVENT DEFINER ACL must exist in Target MySQL Inspection
  27. DROP SUPER_priv User mysql.user WHERE Super_priv = 'Y' lower_case_table_names !=

    0 https://dev.mysql.com/doc/refman/8.0/ja/identifier- case-sensitivity.html FTS Parameters innodb_ft_enable_stopword innodb_ft_min_token_size innodb_ft_total_cache_size innodb_ft_result_cache_limit mecab_rc_file ngram_token_size Inspection
  28. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  29. Set Variables Purpose - Same parameters after migration - Same

    SQL behavior after migration - Set Target MySQL Variables like Source MySQL
  30. sql_mode Affects behavior of SQL character_set_server Affects result set collation_server

    Affects result set explicit_defaults_for_timestamp Affects TIMESTAMP type max_connections Target should be rather than Source MySQL lock_wait_timeout Metadata Lock innodb_lock_wait_timeout InnoDB row lock Set Variables
  31. sql_mode Affects behavior of SQL character_set_server Affects result set collation_server

    Affects result set explicit_defaults_for_timestamp Affects TIMESTAMP type max_connections Target should be rather than Source MySQL lock_wait_timeout Metadata Lock innodb_lock_wait_timeout InnoDB row lock Set Variables
  32. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  33. Data Export/Import How to Execute Safely - Avoid Source/Target MySQL

    loads - Avoid Target MySQL disk full - Use logical backup for MySQL upgrade
  34. Data Export/Import Reasons for Adoption ⎯ https://github.com/mydumper/mydumper ⎯ Faster than

    mysqldump because it can be perform in parallel. ⎯ MySQL Shell's dump loading utility did not support MySQL 5.6 at that time. MyDumper
  35. Data Export/Import Reasons for Adoption Ingenuity ⎯ https://github.com/mydumper/mydumper ⎯ Faster

    than mysqldump because it can be perform in parallel. ⎯ MySQL Shell's dump loading utility did not support MySQL 5.6 at that time. ⎯ Dynamically change the number of threads with the number of CPUs. ⎯ Purging binlogs and turn off slow query log to free up space in Target MySQL. MyDumper
  36. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  37. SCHEMA SELECT TABLE_SCHEMA, TABLE_NAME, IFNULL(TABLE_COLLATION,'none') FROM TABLES WHERE TABLE_TYPE='VIEW' AND

    TABLE_SCHEMA=? TABLE SELECT TABLE_SCHEMA, TABLE_NAME, IFNULL(TABLE_COLLATION, 'none') FROM TABLES WHERE TABLE_TYPE='BASE TABLE' AND TABLE_SCHEMA=? VIEW SELECT SCHEMA_NAME, SCHEMA_NAME, IFNULL(DEFAULT_COLLATION_NAME, 'none') FROM SCHEMATA WHERE SCHEMA_NAME=? TRIGGER SELECT TRIGGER_SCHEMA, TRIGGER_NAME, DATABASE_COLLATION FROM TRIGGERS WHERE TRIGGER_SCHEMA=? ROUTINE SELECT ROUTINE_SCHEMA, ROUTINE_NAME, DATABASE_COLLATION FROM ROUTINES WHERE ROUTINE_SCHEMA=? EVENT SELECT EVENT_SCHEMA, EVENT_NAME, DATABASE_COLLATION FROM EVENTS WHERE EVENT_SCHEMA=? Metadata Check Get Metadata from information_schema
  38. TABLE SHOW CREATE TABLE INDEX SELECT INDEX_NAME, GROUP_CONCAT(COLUMN_NAME ORDER BY

    SEQ_IN_INDEX), MAX(INDEX_TYPE) columns, MAX(NON_UNIQUE) FROM statistics WHERE TABLE_SCHEMA=? AND TABLE_NAME=? GROUP BY INDEX_NAME PARTITION SELECT CONCAT(PARTITION_NAME,':',IFNULL(SUBPARTITION_NAME,'none')), PARTITION_NAME, IFNULL(SUBPARTITION_NAME,’none’), IFNULL(PARTITION_METHOD,'none’), UPPER(IFNULL(PARTITION_EXPRESSION,'none’)) FROM partitions WHERE table_schema=? and table_name=? and PARTITION_NAME is not null COLUMN SELECT COLUMN_NAME, IFNULL(COLUMN_DEFAULT,'no_defaults'), IS_NULLABLE, IFNULL(CHARACTER_SET_NAME,'none'), IFNULL(COLLATION_NAME,'none'), DATA_TYPE, ORDINAL_POSITION, IFNULL(CHARACTER_MAXIMUM_LENGTH,0) FROM COLUMNS WHERE TABLE_SCHEMA=? AND TABLE_NAME=? ORDER BY ORDINAL_POSITION FOREIGN KEY SELECT CONSTRAINT_NAME, CONSTRAINT_TYPE FROM TABLE_CONSTRAINTS WHERE TABLE_SCHEMA=? AND TABLE_NAME=? Metadata Check About Table
  39. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  40. Data Comparison Purpose - Notice different rows between Source and

    Target - Compare Source and Target MySQL data.
  41. Comparing Comparing Rows ⎯ Error if table doesn't have PK

    ⎯ SELECT COUNT(*) FROM table; Primary Key
  42. Comparing Comparing Rows Comparing Hash Value ⎯ Error if table

    doesn't have PK ⎯ SELECT COUNT(*) FROM table; ⎯ SELECT SHA2(CONCAT_WS('|', col1, col2…)) FROM table ORDER BY PK ASC/DESC LIMIT 500000; Primary Key
  43. Migration Steps - Add User ACL - Inspection - Set

    Variables - Prepare Data Import - Data Import - Check Metadata - Start Replication - Data Comparison - Migration - Enable HA
  44. Constraint of Online Migration Same User ACL ⎯ Source and

    Target MySQL must be same Port number ⎯ Source and Target MySQL's User ACL must be same User/Host/Password/Privileges Same Port
  45. Constraint of Online Migration Same User ACL Disable DNS Cache

    ⎯ Source and Target MySQL must be same Port number ⎯ Source and Target MySQL's User ACL must be same User/Host/Password/Privileges ⎯ Apps will continue to query Source MySQL after Switchover Same Port
  46. Summary - DBA reduces operation and communication costs! - And…

    - Developers can operate MySQL Migration safely without DBA!