Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Utilization of data of RDS aurora

Kazuki Numazawa
January 13, 2018
52

Utilization of data of RDS aurora

Kazuki Numazawa

January 13, 2018
Tweet

Transcript

  1. Aurora Multi-Master • re:Invent 2017 Ͱൃද͞ΕͨɺAurora ͷ৽ػೳ(ϓϨϏϡʔ) • ཁ͸ Aurora

    ͷ Writer ΠϯελϯεΛεέʔϧΞ΢τ͢Δ΋ͷ • ैདྷ͸ Writer Πϯελϯε͸Ϋϥελʔ಺ʹ1୆ͷΈͰɺWriter Πϯελ ϯεͷো֐࣌ʹ͸ Reader ΠϯελϯεʹϑΣΠϧΦʔόʔ͢Δ͜ͱͰՄ༻ ੑΛอ͍ͬͯͨ • ࢀߟ • Amazon Aurora Multi-Master ͷϓϨϏϡʔਃ͠ࠐΈ։࢝ • https://aws.amazon.com/jp/blogs/news/sign-up-for-the-preview-of-amazon-aurora-multi- master/ • ʲ଎ใʳAmazon AuroraͷϚϧνϚελػೳ͕ൃද͞Ε·ͨ͠ʂ • https://dev.classmethod.jp/cloud/aws/reinvent-2017-aurora-multi-master/
  2. Aurora Multi-Master ͸ • Aurora Ϋϥελʔ಺ʹ Master(Writer) ͕ෳ਺࡞੒ Մೳ •

    ॻ͖ࠐΈੑೳͷେ෯ͳ޲্͕ݟࠐΊΔ • Writer ͕1ͭࢮΜͰ΋ϑΣΠϧΦʔόʔ͕࣮࣭ແ͍ • θϩμ΢ϯλΠϜͷ࣮ݱ • ͋͘·Ͱ1Ϋϥελʔ಺ͷ࿩
  3. ૉ੖Β͠΍ެࣜυΩϡϝϯτ Aurora ͱ MySQL ͱͷؒɺ·ͨ͸ Aurora ͱผͷ Aurora DB Ϋ

    ϥελʔͱͷؒͷϨϓϦέʔγϣϯ https://docs.aws.amazon.com/ja_jp/AmazonRDS/latest/ UserGuide/AuroraMySQL.Replication.MySQL.html
  4. ༨ஊɿϨϓϦέʔγϣϯํࣜ ࠓճͷ Aurora ͸ MySQL 5.6 ޓ׵ͷ Aurora Λର৅ͱ͍ͯ͠Δ MySQL

    5.6 Ͱ͸ɺGTID (Global Transaction ID) ʹΑΔϨϓϦέʔ γϣϯํ͕ࣜ࠾༻͞ΕɺࠓޙҰൠԽ͍ͯ͘͠Α͏ʹݟ͑Δ͕ɺRDS Ͱ͸ MariaDB Ҏ֎͸ GTID Λ࢖͑ͳ͍ͬΆ͍(※ͪΌΜͱௐ΂ͯͳ ͍) υΩϡϝϯτʹ΋͋Δ௨Γɺࠓճ͸ binlog ํࣜͰͷϨϓϦέʔγϣ ϯΛ͢Δ͚ͩͳͷͰɺMySQL ʹ໌Δ͍ํʑʹ͸ɺͳΜͩͦΜͳ͜ ͱ͔ͱམ୾ͤͯ͞͠·ͬͨ͜ͱͰ͠ΐ͏ ਃ͠༁ͳ͍͆
  5. લఏ৚݅ • ҎԼ͸ׂѪ • VPC ࡞੒ͱ VPC Peering ͷઃఆ •

    DB ઀ଓ༻ͷ EC2 ͷ࡞੒ • ηΩϡϦςΟάϧʔϓͷ࡞੒ • ຊ൪༻ͷ Aurora Ϋϥελʔ͸طʹ͋Γɺσʔλ΋ೖ͍ͬͯΔ΋ͷͱ͢Δ • ຊ൪༻ͷ Aurora Ϋϥελʔ͸࣍ทͷΑ͏ͳঢ়ଶͱ͢Δ • ࢦఆͷͳ͍߲໨ͳͲ͸σϑΥϧτ஋Λ૝ఆ(΋͘͠͸೚ҙͰઃఆ)
  6. લఏ৚݅ • VPC: “master-vpc” • DB Ϋϥελʔࣝผࢠ: “master-cluster” • Writer:

    “master1”(AZ-a) • Reader: “master2”(AZ-c), “master3”(AZ-a) • Writer 1୆ɺReader 2୆ͷ Multi-AZ ߏ੒ • σϑΥϧτ஋͔Βมߋͷͳ͍ “cluster-param-group” ͱ͍͏Ϋϥελʔ ύϥϝʔλάϧʔϓ • ύϒϦοΫΞΫηε: ͍͍͑
  7. • σϑΥϧτ͕ "OFF" ͷͨΊඞͣมߋ͢Δ • ROW, STATEMENT Ͱ΋ྑ͍Α͏͕ͩɺެࣜυΩϡϝϯτͰ͸ɺಛʹඞཁੑ ͕ͳ͍৔߹͸ MIXED

    Λਪ঑͍ͯ͠Δ • ͜ͷύϥϝʔλͷ൓өʹ͸࠶ىಈ͕ඞཁ • ՔಇதͷΫϥελʔͷ৔߹͸ཁ஫ҙ • Multi-AZ ߏ੒Ͱ΋ 1 ෼ະຬͷμ΢ϯλΠϜ • ݸਓతͳ࣮੷Ͱ͸20ඵߦ͔ͳ͍͜ͱ͕΄ͱΜͲ(ࢀߟ·Ͱ) binlog_format Λ MIXED ʹมߋ "cluster-param-group" Λฤू
  8. binlog ͷอ࣋ظؒΛઃఆ • “master-cluster” ͷΫϥελʔΤϯυϙΠϯτ΁ MySQL Ϋϥ ΠΞϯτͰ઀ଓ • ࣍ทͷίϚϯυͰ

    “binlog retention hours” ͷ஋Λ೚ҙʹࢦ ఆ͢Δ • ࠓճ͸1िؒ(168࣌ؒ)Λࢦఆ • ͜ͷઃఆมߋͰ͸μ΢ϯλΠϜ͸ൃੜ͠ͳ͍
  9. mysql> CALL mysql.rds_set_configuration('binlog retention hours', 168); Query OK, 0 rows

    affected (0.05 sec) mysql> mysql> CALL mysql.rds_show_configuration()\G *************************** 1. row *************************** name: binlog retention hours value: 168 description: binlog retention hours specifies the duration in hours before binary logs are automatically deleted. 1 row in set (0.00 sec) Query OK, 0 rows affected (0.00 sec) binlog ͷอ࣋ظؒΛઃఆ ࣌ؒʹઃఆ
  10. mysql> GRANT REPLICATION Slave ON *.* TO 'repl'@'%' IDENTIFIED BY

    “hogehoge"; Query OK, 0 rows affected (0.03 sec) mysql> mysql> SELECT user,host FROM mysql.user where user = "repl"; +------+------+ | user | host | +------+------+ | repl | % | +------+------+ 1 row in set (0.00 sec) ϨϓϦέʔγϣϯ༻Ϣʔβ࡞੒ ࠓճ͸lSFQMzϢʔβΛ࡞੒
  11. Slave ΫϥελʔΛ༻ҙ • લఏ৚݅ͷΫϥελʔ͸ Master ͱͳΔΫϥελʔ • ͔͜͜Β͸ϨϓϦέʔγϣϯઌͱͳΔ Slave ΫϥελʔΛ࡞੒

    • طଘͷσʔλ͕͋Δͱ͍͏લఏͳͷͰɺʮಛఆ࣌఺΁ͷ෮ݩʯ Λ࢖ͬͯۃྗ࠷৽ঢ়ଶͷσʔλΛ࣋ͬͨΫϥελʔΛ࡞੒
  12. • DB Πϯελϯεࣝผࢠ: “slave1" • ͜͜Ͱࢦఆͨ͠΋ͷ͕ Writer Πϯελϯε໊ͱΫϥελʔ໊ʹͳΔ • ޙͰΫϥελʔ໊͸ϦωʔϜ͢Δ(೚ҙ)(ޙड़)

    • ʮ෮ݩՄೳͳ࠷৽࣌ࠁΛ࢖༻͢ΔʯΛબ୒ • VPC: “slave-vpc” • ΞϕΠϥϏϦςΟʔκʔϯ: ap-northeast-1a • AZ ͸ “master-cluster” ͷΠϯελϯεͱ߹Θͤͱ͘ Slave ΫϥελʔΛ༻ҙ “master1” Ͱʮಛఆ࣌఺΁ͷ෮ݩʯΛ࣮ࢪ
  13. Slave ΫϥελʔΛ༻ҙ Reader ΠϯελϯεΛ௥Ճ (slave2, slave3) • "slave1" ΠϯελϯεͰʮAurora ϨϓϦΧͷ࡞੒ʯΛબ୒

    • DB Πϯελϯεࣝผࢠ: “slave2” • ΞϕΠϥϏϦςΟʔκʔϯ: ap-northeast-1c • ༏ઌ౓: ൣғ -1 • DB Πϯελϯεࣝผࢠ:“slave3” • ΞϕΠϥϏϦςΟʔκʔϯ: ap-northeast-1a • ༏ઌ౓: ൣғ -2
  14. Slave ΫϥελʔΛ༻ҙ Slave Ϋϥελʔ໊ΛϦωʔϜ(೚ҙ) • ͜ͷ࣌఺ͰɺSlave Ϋϥελʔ໊͸ “slave1-cluster” ͱͳ͍ͬͯΔ •

    ඞਢͷ࡞ۀͰ͸ͳ͍͕ɺΫϥελʔ໊ʹ “1” ͱ෇͍͍ͯΔͷ͕ؾʹͳΔͷͰ ௚͓͖͍ͯͨ͠(޷Έͷ໰୊) • "slave1-cluster" ͰʮΫϥελʔͷมߋʯΛબ୒ • DB Ϋϥελʔࣝผࢠ: “slave-cluster” • ʮ͙͢ʹద༻ʯʹνΣοΫ
  15. ϨϓϦέʔγϣϯͷઃఆ • ྆ΫϥελʔͷΫϥελʔΤϯυϙΠϯτʹ MySQL ઀ଓ • "show binary logs;" Λ࣮ߦ

    • ్த·Ͱͷϩάঢ়ଶ͕Ұக͍ͯ͠Δ͜ͱΛ֬ೝ • binlog ͷঢ়ଶʹ͕ࠩग़͍ͯΔͱ͜ΖΛ֬ೝ • Slave ͷ Log_name, File_size ͷ஋ΛϝϞ binlog ͷঢ়ଶΛ֬ೝ
  16. ϨϓϦέʔγϣϯͷઃఆ binlog ͷঢ়ଶΛ֬ೝ(Master) mysql> show binary logs; +----------------------------+-----------+ | Log_name

    | File_size | +----------------------------+-----------+ | mysql-bin-changelog.000001 | 120 | | mysql-bin-changelog.000002 | 21660 | +----------------------------+-----------+ 2 rows in set (0.00 sec)
  17. ϨϓϦέʔγϣϯͷઃఆ binlog ͷঢ়ଶΛ֬ೝ(Slave) mysql> show binary logs; +----------------------------+-----------+ | Log_name

    | File_size | +----------------------------+-----------+ | mysql-bin-changelog.000001 | 120 | | mysql-bin-changelog.000002 | 2484 | | mysql-bin-changelog.000003 | 120 | | mysql-bin-changelog.000004 | 120 | | mysql-bin-changelog.000005 | 704 | +----------------------------+-----------+ 5 rows in set (0.01 sec) ͜͜Ͱࠩ෼͕ग़ͯΔͷͰɺ ͜ͷߦͷ஋ΛϝϞ
  18. ϨϓϦέʔγϣϯͷઃఆ Slave ଆͰϨϓϦέʔγϣϯઃఆίϚϯυΛ࣮ߦ mysql> CALL mysql.rds_set_external_master ( mysql> 'master-cluster.cluster-xxxxxxxxxxxx.ap-northeast-1.rds.amazonaws.com', mysql>

    3306, mysql> 'repl', mysql> 'xxxxxxxx', mysql> 'mysql-bin-changelog.000002', mysql> 2484, mysql> 0 mysql> ); Query OK, 0 rows affected (0.19 sec) NZTRMSET@TFU@FYUFSOBM@NBTUFSͰϨϓϦέʔγϣϯઃఆ Ҿ਺͸্͔Βɺ ɾNBTUFSDMVTUFSͷΫϥελʔΤϯυϙΠϯτ ɾNBTUFSDMVTUFS΁ͷ઀ଓϙʔτ ɾNBTUFSDMVTUFSʹ࡞੒ͨ͠ϨϓϦέʔγϣϯ༻Ϣʔβ ɾϨϓϦέʔγϣϯ༻Ϣʔβͷύεϫʔυ ɾϝϞͨ͠CJOMPHͷϑΝΠϧ໊ ɾϝϞͨ͠CJOMPHͷϑΝΠϧαΠζ ϙδγϣϯ  ɾ44-҉߸Խ ݱ࣌఺Ͱະ࣮૷ͱͷ͜ͱͳͷͰݻఆ ࢀߟ: https://docs.aws.amazon.com/ja_jp/AmazonRDS/latest/UserGuide/mysql_rds_set_external_master.html
  19. ϨϓϦέʔγϣϯͷঢ়ଶ֬ೝ Master ଆͷঢ়ଶΛ֬ೝ mysql> show master status\G *************************** 1. row

    *************************** File: mysql-bin-changelog.000002 Position: 21660 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) CJOMPHϑΝΠϧ໊ CJOMPHϙδγϣϯ
  20. ϨϓϦέʔγϣϯͷঢ়ଶ֬ೝ Slave ଆͷঢ়ଶΛ֬ೝ mysql> show slave status\G *************************** 1. row

    *************************** Slave_IO_State: Waiting for Master to send event Master_Host: master-cluster.cluster-xxxxxxxxxxxx.ap-northeast-1.rds.amazonaws.com Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin-changelog.000002 Read_Master_Log_Pos: 21660 Relay_Log_File: relaylog.000002 Relay_Log_Pos: 19469 Relay_Master_Log_File: mysql-bin-changelog.000002 Slave_IO_Running: Yes Slave_SQL_Running: Yes .... .BTUFSଆͷCJOMPHϑΝΠϧ໊ͱҰக .BTUFSଆͷCJOMPHϙδγϣϯͱҰக ͜Ε͕:FTʹͳͬͯΕ͹0,
  21. ϨϓϦέʔγϣϯͷঢ়ଶΛ֬ೝ(Master) master-cluster F/O mysql> show master status\G *************************** 1. row

    *************************** File: mysql-bin-changelog.000003 Position: 38257 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) CJOMPHϑΝΠϧ໊ CJOMPHϙδγϣϯ
  22. mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting

    for master to send event Master_Host: master-cluster.cluster-xxxxxxxxxxxx.ap-northeast-1.rds.amazonaws.com Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin-changelog.000003 Read_Master_Log_Pos: 38257 Relay_Log_File: relaylog.000008 Relay_Log_Pos: 38430 Relay_Master_Log_File: mysql-bin-changelog.000003 Slave_IO_Running: Yes Slave_SQL_Running: Yes .... .BTUFSଆͷCJOMPHϑΝΠϧ໊ͱҰக .BTUFSଆͷCJOMPHϙδγϣϯͱҰக ͜Ε͕:FTͷ··ͳͷͰ0, master-cluster F/O ϨϓϦέʔγϣϯͷঢ়ଶΛ֬ೝ(Slave)
  23. ҎԼͷखॱͰ slave-cluster ͷ F/O ޙͷঢ়ଶΛ֬ೝ͢Δ • slave-cluster ΛखಈͰϑΣΠϧΦʔόʔ • master-cluster

    ʹద౰ʹॻ͖ࠐΉ(ׂѪ) • MasterɺSlave ͦΕͧΕͰϨϓϦέʔγϣϯͷঢ়ଶ Λ֬ೝ slave-cluster F/O
  24. ϨϓϦέʔγϣϯͷঢ়ଶΛ֬ೝ(Master) slave-cluster F/O mysql> show master status\G *************************** 1. row

    *************************** File: mysql-bin-changelog.000013 Position: 27917068 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 1 row in set (0.00 sec) CJOMPHϑΝΠϧ໊ CJOMPHϙδγϣϯ
  25. mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting

    for master to send event Master_Host: master-cluster.cluster-xxxxxxxxxxxx.ap-northeast-1.rds.amazonaws.com Master_User: repl Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin-changelog.000013 Read_Master_Log_Pos: 27917068 Relay_Log_File: relaylog.000099 Relay_Log_Pos: 4730917 Relay_Master_Log_File: mysql-bin-changelog.000013 Slave_IO_Running: Yes Slave_SQL_Running: Yes .... .BTUFSଆͷCJOMPHϑΝΠϧ໊ͱҰக .BTUFSଆͷCJOMPHϙδγϣϯͱҰக ͜Ε͕:FTͷ··ͳͷͰ0, slave-cluster F/O ϨϓϦέʔγϣϯͷঢ়ଶΛ֬ೝ(Slave)