Yohei Yoshimuta 500ສਓ͕ར༻͢Δʮ༑ୡͱ༡΂Δͨ·Γ৔ ΞϓϦ ύϥϨϧʯʹ͓͚Δσʔλϕʔε ج൫ͷܧଓతվળ SRE NEXT 2024
 4 August 2024 ϋογϡλά: #srenext #srenext_c

• X/GitHub: @yoheimuta • ॴଐ: גࣜձࣾύϥϨϧ • ओʹ࢖͍ͬͯΔٕज़: GKE,MySQL,Vitess,Flink • ۀ຿֎Ͱ: Go ࣗݾ঺հ

No content

޷͖ͳਓͨͪͱա࣌ؒ͢͝Λ࠷େԽ͢ΔSNSΛ࡞Δ • Ϣʔβʔͷ7ׂ͕Zੈ୅ʢʙ20୅ʣ • 1೔͋ͨΓͷฏۉར༻࣌ؒ͸3࣌ؒ௒ • ि5Ҏ্ͰΞϓϦΛ্ཱͪ͛ΔϢʔ βʔ͕໿7ׂ • ௨࿩΍νϟοτΛ͠ͳ͕ΒήʔϜ΍ ಈըɺԻָΛָ͠Ή͜ͱ͕Ͱ͖Δɺ ΦϯϥΠϯͷͨ·Γ৔Λఏڙ ༑ୡͱ༡΂Δͨ·Γ৔ΞϓϦ ύϥϨϧ ※data.aiௐ΂ : 2021೥6݄ʙ2024೥4݄ɺiOSˍAndroidͷ߹ࢉ஋ɺ೔ຊʹ͓͚ΔMAUͷάϥϑ

ύϥϨϧ͕௚໘͖ͯͨ͠ӡ༻՝୊ MySQL ͷύϑΥʔϚϯε௿ԼΛൃ୺ʹαʔϏεશମ͕μ΢ϯ͢Δ͜ ͱ͕աڈʹԿ౓΋ൃੜ • αʔϏεͷ੒௕ʹΑΔෛՙ૿ • قઅΠϕϯτʢ1/1 ͷ͚͓͋Ί͕୅දྫʣ • ΞϓϦͷ࣮૷ߟྀ࿙ΕʹΑΔաෛՙ • Ϋϥ΢υϕϯμʔʹΑΔෆఆظϝϯςφϯεʢ= ࠶ىಈɺωοτϫʔΫॠஅʣ

ࠓ೔࿩͢͜ͱ • എܠͱ໰୊ͷ֓ཁ • λΠϜΞ΢τઃఆͷϕετϓϥΫςΟε • αʔΩοτϒϨʔΧʔͷ࣮૷ͱςετ • ίωΫγϣϯϓʔϦϯάϓϩΩγͷಋೖ • ·ͱΊ

ύϥϨϧͷγεςϜߏ੒ʢ౰࣌ʣ

λΠϜΞ΢τઃఆͷϕετϓϥΫ ςΟε

ࣄྫ1: ෛՙͰ MySQL ͕Ԡ౴͠ͳ͘ͳΔ ૝ఆ֎ͷϦΫΤετूத • ໰୊: ਓ਺͕ۃ୺ʹଟ͍άϧʔϓͰಉظΛऔΖ͏ ͱͯ͠ɺ૝ఆҎ্ʹϦΫΤετൃੜ • Өڹ: ΫΤϦॲཧ͕٧·ͬͯɺCPU࢖༻཰͕ߴࢭ ·Γɺ௕࣌ؒԠ౴ෆೳʹͳͬͨ • ୹ظతରࡦ: ΫϥΠΞϯτଆʹthrottlerΛಋೖ͠ɺ ॏෳ͢ΔಉظϦΫΤετΛল͘Α͏ʹͯܰ͠ݮ • ݕ౼ࣄ߲: σʔλϕʔεଆͰ΋Կ͔ରࡦΛଧͯͳ ͍͔ʁ

λΠϜΞ΢τઃఆ ͕͔͔͍࣌ؒͬͯΔΫΤϦͷதஅ • σϑΥϧτͷλΠϜΞ΢τઃఆ͸ OLTPʢΦϯϥΠϯτϥϯβΫγϣϯ؀ڥʣ ʹ͸ෆे෼ • ௕͔͔࣌ؒΔΫΤϦͱ͍͏͚ͩͰ͸தஅ͞Εͳ͍ • ઀ଓཱ֬ʹ͔͔Δ࣌ؒͷ੍ݶ͸Railsͷmysql2Ͱ120ඵͱ௕Ί • ΫΤϦͷॲཧ٧·Γ͕ى͖ͯશମʹ೾ٴ͢ΔΑΓ΋ɺ Ұ࣌తͳλΠϜΞ΢τΤϥʔͷํ͕ ڐ༰Ͱ͖Δ • ͨͩɺMySQL ͷλΠϜΞ΢τ͸छྨ͕ଟ͍͠ɺͦΕͧΕͰαʔϏεʹద੾ͳ஋Λݟ͚ͭ Δͷ͸೉͍͠

λΠϜΞ΢τઃఆͷϕετϓϥΫςΟε ઃఆύϥϝʔλʔ ઃఆର৅ ΞΫγϣϯ σϑΥϧτ஋ ௐ੔͠ ͨ஋ max_execution_time MySQLαʔόʔ αʔόʔ͕͕͔͔͍࣌ؒͬͯΔΫΤϦ ΛΤϥʔʹ͢Δɻॏ͍ΫΤϦʹ༗ޮɻ 0ʢແ੍ݶʣ 2ඵ read_timeout (MYSQL_OPT_READ_TIMEOUT) MySQLΫϥΠΞϯτ ΫϥΠΞϯτ͕͕͔͔͍࣌ؒͬͯΔΫ ΤϦΛΤϥʔʹ͢ΔɻωοτϫʔΫॠ அʹ༗ޮɻ ϥΠϒϥϦґ ଘʢmysql2 ͸0ʣ 3ඵ connect_timeout MySQLΫϥΠΞϯτ ΫϥΠΞϯτ͕͕͔͔͍࣌ؒͬͯΔ઀ ଓཱ֬ॲཧΛΤϥʔʹ͢Δ ϥΠϒϥϦґ ଘʢmysql2 ͸120ඵʣ 2ඵ ॳظ஋Λେ͖͘ઃఆ͠ɺঃʑʹ஋ΛԼ͛ͳ͕Βௐ੔

• max_execution_time ͱread_timeout ͸͢΂ͯͷΫΤϦʹద༻͞ΕΔͷͰྫ֎ʹରԠ͢Δඞཁ͕͋Δ • max_execution_time: MySQL ͷΫΤϦώϯτΛॻ͚͹ɺಛఆͷΫΤϦͷ࣮ߦ࣌ؒΛࢦఆͰ͖ Δɻྫ: SELECT /*+ MAX_EXECUTION_TIME(2000) */ * FROM table_name; • read_timeout: ΫϥΠΞϯτϥΠϒϥϦʹґΔɻRails ͷ mysql2 ϥΠϒϥϦͰ͸ϋοΫ͕ඞཁ c o n n = A c t i v e R e c o r d : : B a s e . c o n n e c t i o n . i n s t a n c e _ v a r i a b l e _ g e t ( : @ r a w _ c o n n e c t i o n ) o r i g = c o n n . i n s t a n c e _ v a r i a b l e _ g e t ( : @ r e a d _ t i m e o u t ) b e g i n c o n n . i n s t a n c e _ v a r i a b l e _ s e t ( : @ r e a d _ t i m e o u t , t i m e o u t ) # d o s o m e t h i n g e n s u r e c o n n . i n s t a n c e _ v a r i a b l e _ s e t ( : @ r e a d _ t i m e o u t , o r i g ) e n d ΫΤϦλΠϜΞ΢τઃఆͷ஫ҙ఺

αʔΩοτϒϨʔΧʔͷ࣮૷ͱ ςετ

ࣄྫ2: Ϋϥ΢υϝϯςφϯεதͷ API Τϥʔ ෆఆظʹൃੜ͢Δɺ࠶ىಈͱωοτϫʔΫͷҰ࣌੾அ • ໰୊: Ұ෦ͷMySQLαʔόʔΠϯελϯεͰ࠶ىಈ΍ωοτϫʔΫॠஅ͕ൃੜ • Өڹ: ઀ଓཱ֬΍ΫΤϦ଴ͪͰAPIαʔόʔͷεϨου͕઎༗͞Εɺ৽نϦΫ ΤετΛड͚෇͚ΒΕͳ͘ͳͬͨ • ՝୊: λΠϜΞ΢τͷௐ੔ͰӨڹ͸؇࿨͞Ε͕ͨɺAPIͷΤϥʔϨεϙϯε͕ ਺෼ଓ͘ঢ়گ • ໨ඪ: ͜ͷΤϥʔϨεϙϯεͷൃੜظؒΛ਺ඵʹ཈͑Δʹ͸ʁ

ґଘઌͷෆௐݕ஌ͱ੾Γସ͑ • ໨త: ໰୊ͷ͋Δґଘઌ΁ͷϦΫΤετྔΛ੍ݶ͠ɺγεςϜશ ମͷ҆ఆੑΛอͭ • ಈ࡞ͷྲྀΕ: • ґଘઌΛݺͼग़͢લʹαʔΩοτ͕։͍͍ͯΔ͔νΣοΫ͢Δ • αʔΩοτ͕ด͍ͯ͡Ε͹ґଘઌΛݺͼग़͢ • ґଘઌ͔ΒΤϥʔ͕ฦ͖ͬͯͨΒαʔΩοτΛ։͘ • αʔΩοτ͕։͍͍ͯΔ৔߹: ґଘઌΛݺ͹ͳ͍ɻ଴ػ͍ͯ͠Δ γεςϜ͕͋Ε͹ͦΕΛɺͳ͚Ε͹ૣظʹΤϥʔΛฦ͢ɻͦͷؒ ΋গྔݺͼग़ͯ͠Τϥʔ͕ղফ͍ͯ͠Δ͔֬ೝ͢Δ αʔΩοτϒϨʔΧʔ https://github.com/Shopify/semian?tab=readme-ov- fi le#circuit-breaker

αʔΩοτϒϨʔΧʔͷٕज़બఆ ҰͭͷRailsΞϓϦ΁ͷಋೖͷͨΊɺΞϓϦ૊ΈࠐΈϞσϧΛ࠾༻ Ϟσϧ ಛ௃ ۩ମྫ αΠυΧʔϞσϧ ωοτϫʔΫ૚ͰͷॲཧΛߦ͏ͷͰෳ਺ΞϓϦͰ౷Ұతͳ࣮ ૷͕Մೳ Envoy ΞϓϦ૊ΈࠐΈϞσϧ ֤ΞϓϦέʔγϣϯʹ௚઀૊ΈࠐΊΔͨΊɺಋೖ͕؆୯Ͱॳ ظઃఆͷख͕ؒগͳ͍ ݴޠ͝ͱͷϥΠϒϥϦ ʢRuby ͳΒ semianɺ Java ͳΒ Resilience4j΍ Hystrixʣ

• ࡞੒ݩ: Shopify ͕࡞ͬͯϝϯςφϯε͍ͯ͠Δ • Πϯετʔϧ: λʔήοτͷґଘઌ͝ͱʹAdapter ͕ඞཁ • MySQL, Postgres, Redis, HTTP͸ެࣜͰαϙʔτ • ઃఆ: ActiveRecordͰ͸database.yamlʹॻ͘ • error_timeoutඵҎ಺ʹΤϥʔ͕ error_thresholdݸ Λ௒͑ΔͱαʔΩοτ͕։͘ • success_thresholdݸ͚ͩ੒ޭͨ͠ΒαʔΩοτ ͕ด͡Δ semian ʹΑΔαʔΩοτϒϨʔΧʔ࣮૷ s e m i a n : & s e m i a n e r r o r _ t h r e s h o l d : 3 e r r o r _ t i m e o u t : 4 s u c c e s s _ t h r e s h o l d : 2 n a m e : s e m i a n _ i d e n t i f i e r _ n a m e d e f a u l t : & d e f a u l t a d a p t e r : m y s q l 2 u s e r n a m e : r o o t p a s s w o r d : h o s t : l o c a l h o s t s e m i a n : < < : * s e m i a n

ΦϖϛεͰઃఆ͕֎Εͨഎܠ͔ΒςετΛ௥Ճ • ࡞੒ݩ: Semian ͱͱ΋ʹ Shopify ͕࡞ ੒͠ɺsemian ࣗ਎ͷςετʹ΋࢖༻ • ػೳ: ೚ҙͷTCP઀ଓ͢Δ֎෦Ϧιʔ εͱͷؒʹϓϩΩγͱͯ͠ڬ·Γɺͦ ͷϦιʔε͕μ΢ϯ΍஗Ԇ͍ͯ͠Δঢ় گΛγϛϡϨʔγϣϯ Toxiproxy ʹΑΔαʔΩοτϒϨʔΧʔͷςετ # R u b y クライアントを使った例 # # 特定の M y S Q L 接続先をダウン T o x i p r o x y [ : r e p l i c a 1 ] . d o w n d o # これは例外をスローする U s e r . f i r s t # 接続先が r e p l i c a 2 に切り替わって成功 U s e r . f i r s t e n d

ίωΫγϣϯϓʔϦϯάϓϩΩγ ͷಋೖ

ࣄྫ3: ৽ن઀ଓͷཱࣦ֬ഊ قઅΠϕϯτʢ2023೥1݄1೔ ͚͓͋Ίʣ • ໰୊: ಉ࣌઀ଓ਺͕ٸܹʹ૿Ճͨ͜͠ͱͰɺγεςϜ্ݶʢmax_connections ύϥϝʔλʣͷ 4096 ʹୡ͠ɺ৽نͷ઀ଓΛड͚෇͚ͳ͘ͳͬͨ • Ұ࣌తͳରԠ: γεςϜ্ݶΛσϑΥϧτ஋͔ΒҾ্͖͛ͯରԠɻ·ͨɺRails ϓϩηε͝ͱͷίωΫγϣϯϓʔϦϯάύϥϝʔλʔΛௐ੔ɻ • কདྷͷݒ೦: ಉ࣌઀ଓ਺͕͜ͷ··૿͑ଓ͚ΔͱɺΫΤϦύϑΥʔϚϯεʹ΋ ӨڹΛ༩͑ΔՄೳੑ͕͋Δ

ࣄྫ4: εϩʔΫΤϦͷൃੜ • ݱ৅: PKͰ1Ϩίʔυऔಘ͢Δ͚ͩͷΫΤϦ͕εϩʔΫΤϦʹͳΔɻൃੜ͢Δ ςʔϒϧ͸༷ʑͰɺΞΫηε͕ଟ͍Ҏ֎ʹڞ௨఺͕ͳ͍ɻ • ྫ: SELECT * FROM chat_group_rooms WHERE id = 1 • ௐࠪํ๏: SHOW ENGINE INNODB STATUSΛ܁Γฦ࣮͠ߦͯ͠semaphore waitΛ֬ೝ αʔϏεͷ੒௕ʹΑΔෛՙ૿ʢ2023೥Նʣ

ಉ࣮࣌ߦεϨου਺ͷεύΠΫ • ݁Ռ: ڞ༗Ϧιʔε΁ͷڝ߹ঢ়ଶͷه࿥ͱε ϩʔΫΤϦͷλΠϛϯά͕Ұக • ڝ߹ঢ়ଶ͕ղফ͞ΕΔ·Ͱͷ଴ػ͕࣌ؒ௕ Ҿ͘ͱɺεϩʔΫΤϦ͕ൃੜ͢Δ • ͞ΒʹMySQLͷಉ࣮࣌ߦεϨου਺͕ڝ߹ ঢ়ଶͷλΠϛϯάͰͨ·ʹେ͖͘௓Ͷ͍ͯ Δ͜ͱ΋Θ͔ͬͨ • ՝୊: ಉ࣌઀ଓ਺ͷ্ݶରࡦͱ߹ΘͤͯɺҰॹ ʹಉ࣮࣌ߦεϨου਺ͷεύΠΫΛղফ͢Δ ํ๏͸ͳ͍ͩΖ͏͔ʁ 20:00 ~ 23:59:59 ·Ͱͷ Threads_running Λຖඵूܭ

• ޮՌ: ઀ଓͷ࠶ར༻ʹΑͬͯɺಉ࣌઀ଓ਺ͱಉ࣮࣌ߦε Ϩου਺ΛԼ͛Δ͜ͱ͕Ͱ͖Δ • ֬ೝϙΠϯτ: ϓϩΩγಋೖʹ͋ͨͬͯ͸࣍ͷ؍఺͔Β ໰୊͕ͳ͍͔֬ೝ͕ඞཁ • ΫΤϦͷޓ׵ੑ: ϓϩΩγ͕͢΂ͯͷΫΤϦΛਖ਼͘͠ ॲཧͰ͖Δ͔ • ΫΤϦͷϨΠςϯγʔ: ϓϩΩγಋೖޙͷΫΤϦͷϨ Πςϯγʔ͕ڐ༰ൣғʹऩ·͍ͬͯΔ͔ • SET จͷ࢖༻ঢ়گ: SET จ͸ηογϣϯ͝ͱʹઃఆΛ มߋ͢ΔͨΊɺίωΫγϣϯϓʔϧͷ࢓૊Έͱ߹Θͳ ͍͜ͱ͕͋ΔɻϓϩΩγ͕͜ΕΒͷ SET จΛਖ਼͘͠ ॲཧͰ͖Δ͔ ίωΫγϣϯϓʔϦϯάϓϩΩγ

ίωΫγϣϯϓʔϦϯάϓϩΩγͷٕज़બఆ কདྷͷεέʔϥϏϦςΟΛݟਾ͑ͯ Vitess Λબఆ • Vitess: MySQLͷεέʔϥϏϦςΟͱ؅ཧͷͨΊʹઃܭ͞ΕͨσʔλϕʔεΫ ϥελϦϯάγεςϜ • ར఺: γϟʔσΟϯάͳ͠Ͱ΋ಋೖՄೳͰɺίωΫγϣϯϓʔϦϯάϓϩΩγ ͷԸܙΛड͚ΒΕΔ • CNCF Graduated ϓϩδΣΫτ: ։ൃ͸׆ൃʹߦΘΕ͍ͯΔɻGitHub, Shopify, Slack, Activision ͳͲ͕ಋೖͯ͠ɺੵۃతʹίϯτϦϏϡʔγϣϯ͍ͯ͠Δ • ࣮ߦ؀ڥ: ެࣜͷ vitess-operator Λ࢖༻ͯ͠ Kubernetes ্ͰߏஙՄೳ

Vitess ͷΞʔΩςΫνϟ VTTablet ͕ίωΫγϣϯϓʔϦϯάϓϩΩγͷ໾ׂΛՌͨ͢ https://vitess.io/docs/20.0/overview/architecture/

Vitess ʹΑΔίωΫγϣϯϓʔϧϓϩΩγ࣮૷ • ΫΤϦޓ׵ੑͷରԠ: GitHub, Shopify͕RailsͰ΋εϜʔζʹಋೖͰ͖ΔΑ͏ʹ ޓ׵ੑͷରԠΛͯ͘͠ΕͨͨΊɺΫΤϦޓ׵ੑͷ໰୊͸΄΅ͳ͠ • ΫΤϦͷϨΠςϯγʔ: ϦΫΤετதͰ10ճҎ্ΫΤϦΛൃߦ͢Δ͜ͱ΋௝͠ ͘ͳ͍͕ɺ௥ՃͷϨΠςϯγʔ͸2ʙ3ϛϦඵͰࡁΉͨΊ໰୊ͳ͠ͱ൑அ • SETจͷ໰୊: SETจ͕࢖ΘΕΔͱͦͷ઀ଓ͸ϓʔϦϯάର৅֎ʹͳΓԸܙΛ ड͚ΒΕͳ͘ͳΔʢ࠷৽όʔδϣϯͰ͸վળʣɻಉ͡஋ʹηοτ͢ΔSETจ͸ ໰୊ͳ͍ͷͰɺαʔόʔม਺ʹ͋Β͔͡Ί୅දతͳ஋Λઃఆ

Vitess ΁ͷίϯτϦϏϡʔγϣϯ ύϥϨϧͷϢʔεέʔεʹ߹Θͤͨमਖ਼ ϦϙδτϦ ֓ཁ Issue/PR਺ औΓ૊Έྫ vitessio/vitess Vitess ຊମ (Go) 8/10 ߴτϥϑΟοΫԼͰ MySQL ͕ҰॠԠ౴ෆೳʹͳΔ ͱ VTTablet ͕ϋϯάͯ͠࠶ىಈ͢Δ໰୊Λमਖ਼ɻ Vitess ͷςετʹ Toxiproxy Λಋೖ planetscale/vitess- operator Vitess ͷͨΊͷ Kubernetes ΦϖϨʔλʔ (Go) 3/4 Vitess Λ CloudSQL ͳͲͷϚωʔδυ MySQL Ͱ࢖ ͏৔߹ͷ E2E ςετΛ CI ʹ௥Ճ debezium/debezium- connector-vitess Vitess ͷͨΊͷ Debezium CDC ίωΫλʔ (Java) 1/1 εφοϓγϣοτͷऔΓࠐΈʢόοΫϑΟϧʣػೳ ͕ະ࣮૷ͩͬͨͷͰɺ࣮૷

ύϥϨϧͷγεςϜߏ੒ʢVitess ಋೖޙʣ

Vitess ಋೖͷ݁Ռ1 ಉ࣌઀ଓ਺͸ܹݮɻ҆৺ͯ͠ 2024 ೥ͷ͚͓͋ΊΛܴ͑ΒΕͨ 😎 • ઀ଓ਺ͷ҆ఆ: ઀ଓ਺͸ৗʹҰఆʹͳͬͨʢԼͷάϥϑ͸ϦϦʔεલޙͷϝτϦΫεʣ • ΠϕϯτରԠ: 2024೥ͷ͚͓͋Ί͸2.5ഒͷAPIϦΫΤετɺ3ഒͷδϣϒىಈ͕͕͋ͬͨΤϥʔͳ͘ࡹ͚ͨ ϦϦʔεͨ͠λΠϛϯά

Vitess ಋೖͷ݁Ռ2 ಉ࣮࣌ߦεϨου਺΋ݮগɻεϩʔΫΤϦ΋ܹݮͨ͠ 😆 20:00 ~ 23:59:59 ·Ͱͷ Threads_running Λຖඵूܭ

·ͱΊ

σʔλϕʔεج൫վળͷऔΓ૊ΈҰཡ ໰୊ ରࡦ ࣮૷ ΫΤϦॲཧ͕٧·ͬͯԠ౴͕஗͘ͳΔ λΠϜΞ΢τઃఆΛߦ͍ɺ௕࣮࣌ؒߦ ͞ΕΔΫΤϦΛ੍ݶ͢Δ read_timeout/max_execution_time/ connect_timeout ͷઃఆ αʔόʔ࠶ىಈ΍ωοτϫʔΫෆௐͰ େྔͷ଴͕ͪൃੜ αʔΩοτϒϨʔΧʔΛಋೖ͠ɺҟৗ ͳґଘઌΛःஅ semian/toxiproxy ͷಋೖ ಉ࣌઀ଓ਺͕γεςϜͷ্ݶʹୡ͢Δ ίωΫγϣϯϓʔϦϯάϓϩΩγΛಋ ೖ͠ɺ઀ଓ؅ཧΛޮ཰Խ Vitess ͷಋೖ ಉ࣌εϨου਺͕૿͑ଓ͚ͯεϩʔΫ ΤϦ͕සൃ͢Δ

Q&A ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ ϝʔϧΞυϨε: [email protected]