https://connpass.com/event/282120/
ʰελσΟαϓϦʱʹ͓͚ΔSLI/SLO ͷܧଓతվળTakeshi Kondo / @chaspy2023/05/13SLOconf Tokyo 2023
View Slide
Who am Ichaspy chaspy_Engineering ManagerSite Reliability and Web Application Developmentat Recruit Co., Ltd.Takeshi Kondohttps://chaspy.me
SRE NEXT 2020 & 2022• 2020• SLI/SLO ͱ͍͏ݴ༿͕ͳ͍ঢ়ଶͰ৫ಋೖΛࢼΈͨࣄྫ• 2022• SLI/SLO Λಋೖͨ͠ޙͷ• ৫શମͰ Site Reliability EngineeringΛਐΊΔͨΊʹඞཁͳ͜ͱΛߟ͑ͨ
SRE & Web Application Development2018 2020 2021 20232019 20222VJQQFSೖࣾ43&/&954-0Λ৫ʹಋೖ͠Α͏ͱؤுΔ&OHJOFFSJOH.BOBHFSͱͯ͠8FC։ൃνʔϜʹࢀՃ43&/&95&OHJOFFSJOH.BOBHFSʹͳΔ4-0DPOG5PLZP✨
SRE & Web Application Development2018 2020 2021 20232019 20222VJQQFSೖࣾ43&/&954-0Λ৫ʹಋೖ͠Α͏ͱؤுΔ43&/&95&OHJOFFSJOH.BOBHFSʹͳΔ&OHJOFFSJOH.BOBHFSͱͯ͠8FC։ൃνʔϜʹࢀՃࠓ։ൃऀઢͰ͠·͢ʂ4-0DPOG5PLZP✨
ࠓ͍͑ͨ͜ͱҰܾΊͨ SLI/SLO ܧଓతʹݟ͠·͠ΐ͏
Ұઃఆͯ͠ݟ͞ͳ͔ͬͨΒͲ͏ͳ͔ͬͨͷΛ͠·͢😅
ʰελσΟαϓϦʱʹ͓͚ΔSLI/SLO ͷܧଓతվળΛ͜Ε͔Β͍ͬͯͧ͘ͱ͍͏Takeshi Kondo / @chaspy2023/05/13SLOconf Tokyo 2023
Outline• ࣗݾհ• ʰελσΟαϓϦ தֶߨ࠲ʱʹ͍ͭͯ• SLI/SLO ͳΜͷͨΊʹ͋Δͷ͔• αʔϏεӡ༻ͷݱঢ়ͱ՝• ՝ʹ࣮͋ͨͬͯࡍʹऔΓΜͩ͜ͱ• ·ͱΊ
લఏɿϓϩμΫτհ - ελσΟαϓϦ
20222݄ʹϦχϡʔΞϧ• Ϣʔβج൫Ҏ֎ͷ෦Λ৽نϚΠΫϩαʔϏεͱͯ͠2ʹΓ։ൃ• ϦϦʔε͔Β1ܦաɻݱࡏܧଓతʹΤϯϋϯε͍ͯ͠·͢https://www.recruit.co.jp/newsroom/pressrelease/2022/0131_9881.htmlϦχϡʔΞϧͷϙΠϯτʂ ࠓिͷϛογϣϯͱ෮ԋशػೳʹΑΔݸผֶशࢧԉ ԋशྔɾқΛେ෯֦ॆ ʮఆظςετରࡦߨ࠲ʯΛؚΉ৽ߨ࠲͕ଓʑొ ֶशը໘ͷσβΠϯΛҰ৽
උߟ: tara ͱ͍͏ͷ͜ͷϦχϡʔΞϧϓϩδΣΫτͷίʔυωʔϜͰɺ࠷ۙΠϯλϏϡʔͰύϒϦοΫʹͳͬͨ https://brand.studysapuri.jp/career/interview/article/Saori_Suzuki/ݩʑ͋ͬͨϢʔβج൫ΛؚΉαʔϏε
Why SLI/SLO?• ػೳ։ൃorඇػೳ։ൃɺͲͪΒʹ࣌ؒΛ͏ͷ͔ΛFact-BasedͰܾఆ͢ΔͨΊ• Error Budget ͕͋Δ͏ͪ1ͭ1ͭͷΤϥʔʹରॲ͠ͳ͍• Burn Out Λආ͚ΒΕΔ• Budget ͕͋Δ͏ͪϦεΫ͕ͱΕΔ
උߟ: tara ͱ͍͏ͷ͜ͷϦχϡʔΞϧϓϩδΣΫτͷίʔυωʔϜͰɺ࠷ۙΠϯλϏϡʔͰύϒϦοΫʹͳͬͨ https://brand.studysapuri.jp/career/interview/article/Saori_Suzuki/ݩʑ͋ͬͨϢʔβج൫ΛؚΉαʔϏε3FWFSTF1SPYZ /HJOY• SLI/SLO શ෦Ͱ8ͭ• (a)Availability ͱ (b)Latency• http ͷ metrics Λ͏• ҎԼͷ4Օॴʹ(a/b)2छྨͣͭ• ᶃ api-gateway• ᶄ api-gateway -> main• ᶅ api-gawatey -> content• ᶆ main -> content• SLO• Availability: 99.9%• Latency: 95 percentile < 1000msecᶃᶄ ᶅᶆ
Why Envoy?• ࣌ϚΠΫϩαʔϏεؒͷ metrics Λऔಘ͢Δํ๏͕ͳ͔ͬͨ• Control Plane ΛؚΜͩService Mesh Ͱͳ͘ɺSide-car container ͱͯ͠୯ʹૉͷEnvoy ΛࡌͤΔͷΈ
DevSupport: ସΘΓ൪Ͱఆৗӡ༻ۀΛߦ͏• Slack ͷ௨Λ֬ೝͯ͠ݪҼௐࠪ• Sentry Exception, SLO Alert, GCP Pub/Sub Dead Letter• खಈରԠ͕ඞཁͳͷ֤νʔϜʹΤεΧϨʔγϣϯ• CS(Customer Support)͍߹ΘͤͷҰ࣍ड͚• શମ͚ϝϯγϣϯͷ1࣍ड͚
ى͖͍ͯͨ՝: No SLO Alert• ϦϦʔε͔Βࠓ·ͰҰ SLO Alert ͕໐ͬͨ͜ͱͳ͍• Sentry ͷ Exception ྔ͕ SLI ʹө͞Ε͍ͯͳ͍ؾ͕͢Δ• Կ͕ى͖͍ͯΔͷͩΖ͏͔ʁ• গͳ͘ͱ Sentry Exception Λ1݅ͣͭݟ͍ͯΔ࣌Ͱ Error Budget ͱ͍͏֓೦ར༻Ͱ͖ͯͳ͍• SLO ͕ࣗͨͪͷظΑΓ؇͗͢Δʁ• SLI ͷઃఆ͕ޡ͍ͬͯΔʁ• ௐࠪͨ͠
Ծઆ: Envoy ͷ metrics (SLIᶄᶅᶆ) ͕͓͔͍͠ͷͰʁ• Yes• Exception ͷҰ෦ DNS ໊લղܾͰࣦഊ͍ͯͨ͠• ͭ·Γɺhttp request ʹࢸ͍ͬͯͳ͍• envoy.cluster.upstream_rq_2xx ʹܭ্͞Εͳ͍ͷͦΕͦ͏• ᶄͷ௨৴࣌ɺ໊લղܾʹࣦഊύλʔϯ• ᶃ ͷ SLI Ͱܭଌ͞Ε͍ͯΕྑ͍͕…?௨ৗͷ௨৴ UBSBBQJHBUFXBZDPOUBJOFS͕IUUQUBSBNBJOΛ໊લղܾ͢Δ͕ࣦ͜͜ഊͨ͠ IUUQUBSBNBJOͰ௨৴͢Δ
Ծઆ2: Reverse Proxy ͷ metrics (SLIᶃ) ͕͓͔͍͠ͷͰʁ• Yes• GraphQL ϦΫΤετ్͕தͰࣦഊͨ͠߹ɺhttp Ͱ 200 Λฦ͍ͯͨ͠😱• ϦϦʔε࣌ɺ෦ࣦഊ 500 Ͱฦ͢͜ͱΛܾΊ͕ͨɺͦ͏͞Ε͍ͯͳ͔ͬͨ௨ৗͷ௨৴ $MJFOU͔ΒIUUQTKVOJPSMFBSOTUVEZTBQVSJKQʹΞΫηε͢Δͱ3FWFSTF1SPYZʹ౸ୡ 3FWFSTF1SPYZ͔ΒțBSBBQJHBUFXBZQSPYZᶄ UBSBBQJHBUFXBZ͔ΒțBSBNBJO௨৴ᶄ͜͜ͰΤϥʔ͕ൃੜ
ରॲ1ɿGraphQL Error ͷ߹ http 500 Λฦ͢• ݩʑ GraphQL http ͷ͜ͱΛؾʹ͍ͯ͠ͳ͍• ڍಈ GraphQL server library ͷڍಈʹґଘ͢Δ• Response status 200 ʹ౷Ұ͢ΔϓϥΫςΟε͋Δ• Client Error Response ͷ errors ΛݟΔͷͰͳ͍ಉ྅͕γϡοͱͯ͘͠Ε·ͨ͠🙏4QFDJBM5IBOLT!2VSBNZ
ରॲ2ɿ Envoy ΛΊͯ Datadog APM metrics Λར༻• ෳࡶੑʹΑΔτϥϒϧγϡʔτͷ͠͞ΛݮΒͨ͢Ί• Envoy ͷ metrics ʹ͕͋ͬͨΘ͚Ͱͳ͍• ӡ༻ͷ՝ଟ͘ metrics औಘҎ֎ͷϝϦοτಘΒΕ͍ͯͳ͔ͬͨ• Curcuit Breaker ೖΕ͍ͯͨͷͷൃಈͨ͠έʔε΄ͱΜͲͳ͍• Envoy ͷ version up ରԠʢग़དྷ͍ͯͳ͍ʣ• Pod side-car container ͷىಈɾऴྃॱ੍ޚʢenvoy Λͨͳ͍ͱΤϥʔʹͳΔʣ• Rollouts Λ͍ͬͯΔ߹ͷ Patch ํ๏ʢResource ٯసͯ͠োʹͳͬͨ͜ͱʣ
খωλ: Datadog APM ݁ߏบ͕͋Δ(1)• http client ͷ APM Plugin ͷ resource tag default Ͱhttp method Ͱ͋Δ• Ѽઌ͝ͱͷ SLI ͱͯ͠࠾༻͢Δʹ hostname ͕ඞཁ• Node, Ruby ͰͦΕͧΕରԠ• ৫Ͱ http-client ͷ resource tag ͷ໋໊نΛ߹ҙ
খωλ: Datadog APM ݁ߏบ͕͋Δ(2)• trace.http.request.errors Ͱ http 5xx ֘͠ͳ͍• ٯʹ 4xx ֘͢Δ• trace.http.request.hits.by_http_status Λར༻͢Δඞཁ͕͋Δ
උߟ: tara ͱ͍͏ͷ͜ͷϦχϡʔΞϧϓϩδΣΫτͷίʔυωʔϜͰɺ࠷ۙΠϯλϏϡʔͰύϒϦοΫʹͳͬͨ https://brand.studysapuri.jp/career/interview/article/Saori_Suzuki/ݩʑ͋ͬͨϢʔβج൫ΛؚΉαʔϏε3FWFSTF1SPYZ /HJOY• SLO Λݟͨ͠• (a)Availability ͱ (b)Latency• http ͷ metrics Λ͏• ҎԼͷ4Օॴʹ(a/b)2छྨͣͭ• ᶃ api-gateway• ᶄ api-gateway -> main• ᶅ api-gawatey -> content• ᶆ main -> content•🆕ᶇ api-gateway -> Ϣʔβج൫ͷ request• SLO• Availability: 99.9%• Latency: 95 percentile < 1000msec• -> αʔϏε͝ͱʹݱঢ়ΛՃຯ͠ɺ100~500msecᶃᶄ ᶅᶆᶇϚΠΫϩαʔϏε͝ͱͷ4-*4-0Λഇࢭ4-*Λ͚ΔϝϦοτ͕ෳ4-*4-0Λཧ͢Δίετʹݟ߹͍ͬͯͳ͍ͨΊϢʔβج൫͚4-*4-0ՃϢʔβج൫͚ͷڞ௨4-*͜Ε·ͰFOWPZNFUSJDTΛར༻͍ͯͨ͠ɻFOWPZΛ֎ͨͨ͠Ί%BUBEPH"1.NFUSJDTΛར༻ͨ͠4-*4-0ΛՃ
DevSupport ݟ͠• Sentry Exception ͰΞϓϦέʔγϣϯίʔυىҼͷͷҎ֎શͯ Ignore ͢Δ• SLO Alert ͕དྷͨ࣌ͷجຊతͳରॲํΛυΩϡϝϯτԽ• ରԠͰ͖ͳ͔ͬͨͷΛ2िؒʹ1ճνʔϜͰରԠ
Կ͕ى͖͍ͯͨͷ͔• ϦϦʔε࣌ʹҰઃఆ͞Εͨ SLI/SLO 1ݟ͞Εͯͳ͔ͬͨ• SLO ͕ԿͷՁൃش͍ͯ͠ͳ͔ͬͨ• SLI/SLO ྆ํΛݟ͠ɺࠓޙܧଓతʹݟ͢͜ͱʹͨ͠
Ͳ͏͖ͩͬͨ͢ͷ͔• ։ൃऀࢹ• SLO ͕ຊʹՁΛͨΒ͍ͯ͠Δͷ͔Λఆظతʹݕࠪ͢Δ• ൪ྑ͍͕ɺͨ·ʹશһͰݟΔ࣌ؒΛऔΔ͜ͱॏཁ• 1ަͩͱਂ͘ௐΔΠϯηϯςΟϒ͕ಇ͔ͳ͍• SRE ࢹ• ։ൃνʔϜ͕ SLI/SLO Λఆظతʹݟ͢ΈΛ࡞Δ• ϫʔΫϩʔυ͝ͱʹ SLI/SLO Λࣗಈੜ͢ΔΈΛ࡞Δ
Thank you!chaspy chaspy_Engineering ManagerSite Reliability and Web Application Developmentat Recruit Co., Ltd.Takeshi Kondohttps://chaspy.me
chaspy
kawagoi
junki
ewolff
bengo4com
non97
soracom
grimoh
pistatium
miyashino
kobayang
colopl
hirosatogamo
pedronauck
eileencodes
myddelton
aki_iinuma
nonsquared
searls
addyosmani
bluesmoon
kevinliebholz
destraynor
jeffersonlam
bryan