MySQL Failover with Consul vol.2

Slide 1

Slide 1 text

MYSQL FAILOVER WITH CONSUL VOL. 2 12/12/2014 @ijin MySQL Casual Talks vol.7

Slide 2

Slide 2 text

ABOUT ME (@IJIN) • Michael H. Oshita • Japanese American • 10+ Years in Japan • Software/Infra/Cloud Engineer • http://ijin.github.io

Slide 3

Slide 3 text

ABOUT ME (@IJIN) τϥϒϧˑ͠Ύʔͨʔͣ

Slide 4

Slide 4 text

https://walti.io WALTI

Slide 5

Slide 5 text

BLOG https://ijin.github.io

Slide 6

Slide 6 text

MYSQL FAILOVER • MySQL • MHA (MySQL Master HA) • Consul INGREDIENTES:

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

CONSUL

Slide 9

Slide 9 text

CONSUL HTTP APIͱDNSͰૢ࡞͕Ͱ͖ΔKV಺ଂͳ෼ࢄܕΫϥελ

Slide 10

Slide 10 text

CONSUL AGENT HTTP API /v1/kv/ key/value endpoint: RESTful: GET, PUT, DELETE

Slide 11

Slide 11 text

CONSUL AGENT HTTP API key/value: base64 raw

Slide 12

Slide 12 text

CONSUL AGENT DNS Interface .node.. Node Lookup:

Slide 13

Slide 13 text

CONSUL AGENT DNS Interface

Slide 14

Slide 14 text

CONSISTENCY

Slide 15

Slide 15 text

consensus protocol using raft server server (leader) server CONSISTENCY server server

Slide 16

Slide 16 text

consensus protocol using raft server server (leader) server write CONSISTENCY server server

Slide 17

Slide 17 text

consensus protocol using raft server server (leader) server write CONSISTENCY server server write

Slide 18

Slide 18 text

consensus protocol using raft server server (leader) server write CONSISTENCY server server write write

Slide 19

Slide 19 text

consensus protocol using raft server server (leader) server write CONSISTENCY server server write write quorum[(n/2)+1] reached, commited!

Slide 20

Slide 20 text

CONSISTENCY writes: ! "?state" or "?consistent" parameter ɾdefault ɾstrongly consistent ɾstale ! consensus protocol using raft reads: ! Always Strongly Consistent

Slide 21

Slide 21 text

MHA Master Slave Slave Manager MasterͰো֐ൃੜ

Slide 22

Slide 22 text

Master Master Slave Manager ֤slaveʹ࠷৽binlogΛద༻͠ɺMasterঢ֨ binlog binlog binlog MHA

Slide 23

Slide 23 text

MHA master_ip_failover

Slide 24

Slide 24 text

Slave masterdb.node.consul Slave MHA

Slide 25

Slide 25 text

Master Slave masterdb.node.consul MHA

Slide 26

Slide 26 text

"Starting master failover."! "* Phase 1: Configuration Check Phase..\n"! "* Phase 2: Dead Master Shutdown Phase..\n"! ==> ͜͜ͰچMaster৘ใ࡟আ (master_ip_failover)! "* Phase 3: Master Recovery Phase..\n"! "* Phase 3.1: Getting Latest Slaves Phase..\n"! "* Phase 3.2: Saving Dead Master's Binlog Phase..\n"! "* Phase 3.3: Determining New Master Phase..\n"! ==> ͜͜Ͱ৽Master৘ใొ࿥ (master_ip_failover)! "* Phase 4: Slaves Recovery Phase..\n" FAILOVERγʔέϯε MHA

Slide 27

Slide 27 text

MHA & CONSUL master_ip_failover چmaster IPΛແޮԽ ৽master IPΛొ࿥

Slide 28

Slide 28 text

IN ACTION https://www.youtube.com/watch?v=rA4hyJ-pccU

Slide 29

Slide 29 text

ALTERNATIVES ɾconsul event & consul watch ! ɾconsul-template

Slide 30

Slide 30 text

events propagated over gossip protocol event CONSUL EVENT watch watch

Slide 31

Slide 31 text

CONSUL EVENT

Slide 32

Slide 32 text

CONSUL-TEMPLATE haproxy.cfg

Slide 33

Slide 33 text

CONSUL-TEMPLATE

Slide 34

Slide 34 text

IN CLOSING • Use consul DNS catalog for MHA failover • Consul-Template over consul event