ʰελσΟαϓϦʱʹ͓͚Δ SLI/SLO ͷܧଓతվળ Takeshi Kondo / @chaspy 2023/05/13 SLOconf Tokyo 2023 

Who am I chaspy chaspy_ Engineering Manager Site Reliability and Web Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me 

SRE NEXT 2020 & 2022 • 2020 • SLI/SLO ͱ͍͏ݴ༿͕ͳ͍ঢ়ଶͰ૊৫ ΁ಋೖΛࢼΈͨࣄྫ • 2022 • SLI/SLO Λಋೖͨ͠ޙͷ࿩ • ૊৫શମͰ Site Reliability Engineering ΛਐΊΔͨΊʹඞཁͳ͜ͱΛߟ͑ͨ 

SRE & Web Application Development 2018 2020 2021 2023 2019 2022 2VJQQFS
ೖࣾ 43&
/&95
 4-0
Λ૊৫ʹಋೖ ͠Α͏ͱؤுΔ &OHJOFFSJOH
.BOBHFS
ͱͯ͠
8FC։ൃνʔϜʹ΋ࢀՃ
43&
/&95
 &OHJOFFSJOH
.BOBHFS
ʹͳΔ 4-0DPOG
5PLZP
✨ 

SRE & Web Application Development 2018 2020 2021 2023 2019 2022 2VJQQFS
ೖࣾ 43&
/&95
 4-0
Λ૊৫ʹಋೖ ͠Α͏ͱؤுΔ 43&
/&95
 &OHJOFFSJOH
.BOBHFS
ʹͳΔ &OHJOFFSJOH
.BOBHFS
ͱͯ͠
8FC։ൃνʔϜʹ΋ࢀՃ
ࠓ೔͸։ൃऀ໨ઢͰ࿩͠·͢ʂ
4-0DPOG
5PLZP
✨ 

ࠓ೔఻͍͑ͨ͜ͱ Ұ౓ܾΊͨ SLI/SLO ͸ܧଓతʹݟ௚͠·͠ΐ͏ 

Ұ౓ઃఆͯ͠ݟ௚͞ͳ͔ͬͨΒͲ͏ͳ͔ͬͨͷ࿩Λ͠·͢😅 

ʰελσΟαϓϦʱʹ͓͚Δ SLI/SLO ͷܧଓతվળ Λ͜Ε͔Β΍͍ͬͯͧ͘ͱ͍͏࿩ Takeshi Kondo / @chaspy 2023/05/13 SLOconf Tokyo 2023 

Outline • ࣗݾ঺հ • ʰελσΟαϓϦ தֶߨ࠲ʱʹ͍ͭͯ • SLI/SLO ͸ͳΜͷͨΊʹ͋Δͷ͔ • αʔϏεӡ༻ͷݱঢ়ͱ՝୊ • ՝୊ʹ࣮͋ͨͬͯࡍʹऔΓ૊Μͩ͜ͱ • ·ͱΊ 

Outline • ࣗݾ঺հ • ʰελσΟαϓϦ தֶߨ࠲ʱʹ͍ͭͯ • SLI/SLO ͸ͳΜͷͨΊʹ͋Δͷ͔ • αʔϏεӡ༻ͷݱঢ়ͱ՝୊ • ՝୊ʹ࣮͋ͨͬͯࡍʹऔΓ૊Μͩ͜ͱ • ·ͱΊ 

લఏɿϓϩμΫτ঺հ - ελσΟαϓϦ 

No content

2022೥2݄ʹϦχϡʔΞϧ • Ϣʔβج൫Ҏ֎ͷ෦෼Λ৽نϚΠΫϩ αʔϏεͱͯ͠2೥ʹ౉Γ։ൃ • ϦϦʔε͔Β1೥ܦաɻݱࡏ΋ܧଓత ʹΤϯϋϯε͍ͯ͠·͢ https://www.recruit.co.jp/newsroom/pressrelease/2022/0131_9881.html ϦχϡʔΞϧͷϙΠϯτʂ
 ࠓिͷϛογϣϯͱ൓෮ԋशػೳʹΑΔݸผֶशࢧԉ
 ԋशྔɾ೉қ౓Λେ෯֦ॆ
 ʮఆظςετରࡦߨ࠲ʯΛؚΉ৽ߨ࠲͕ଓʑొ৔
 ֶशը໘ͷσβΠϯΛҰ৽ 

උߟ: tara ͱ͍͏ͷ͸͜ͷϦχϡʔΞϧϓϩδΣΫτͷίʔυωʔϜͰɺ࠷ۙΠϯλϏϡʔͰύϒϦοΫʹͳͬͨ https://brand.studysapuri.jp/career/interview/article/Saori_Suzuki/ ݩʑ͋ͬͨ Ϣʔβج൫Λ ؚΉαʔϏε 

Outline • ࣗݾ঺հ • ʰελσΟαϓϦ தֶߨ࠲ʱʹ͍ͭͯ • SLI/SLO ͸ͳΜͷͨΊʹ͋Δͷ͔ • αʔϏεӡ༻ͷݱঢ়ͱ՝୊ • ՝୊ʹ࣮͋ͨͬͯࡍʹऔΓ૊Μͩ͜ͱ • ·ͱΊ 

Why SLI/SLO? • ػೳ։ൃorඇػೳ։ൃɺͲͪΒʹ࣌ؒΛ࢖͏ͷ͔Λ Fact-BasedͰܾఆ͢ΔͨΊ • Error Budget ͕͋ Δ͏ͪ͸1ͭ1ͭͷ Τϥʔʹରॲ͠ͳ͍ • Burn Out Λආ͚ΒΕΔ • Budget ͕͋Δ͏ͪ͸Ϧε Ϋ͕ͱΕΔ 

Outline • ࣗݾ঺հ • ʰελσΟαϓϦ தֶߨ࠲ʱʹ͍ͭͯ • SLI/SLO ͸ͳΜͷͨΊʹ͋Δͷ͔ • αʔϏεӡ༻ͷݱঢ়ͱ՝୊ • ՝୊ʹ࣮͋ͨͬͯࡍʹऔΓ૊Μͩ͜ͱ • ·ͱΊ 

උߟ: tara ͱ͍͏ͷ͸͜ͷϦχϡʔΞϧϓϩδΣΫτͷίʔυωʔϜͰɺ࠷ۙΠϯλϏϡʔͰύϒϦοΫʹͳͬͨ https://brand.studysapuri.jp/career/interview/article/Saori_Suzuki/ ݩʑ͋ͬͨ Ϣʔβج൫Λ ؚΉαʔϏε 3FWFSTF
1SPYZ /HJOY • SLI/SLO ͸શ෦Ͱ8ͭ • (a)Availability ͱ (b)Latency • http ͷ metrics Λ࢖͏ • ҎԼͷ4Օॴʹ(a/b)2छྨͣͭ • ᶃ api-gateway • ᶄ api-gateway -> main • ᶅ api-gawatey -> content • ᶆ main -> content • SLO • Availability: 99.9% • Latency: 95 percentile < 1000msec ᶃ ᶄ ᶅ ᶆ 

Why Envoy? • ౰࣌͸ϚΠΫϩαʔϏεؒͷ metrics Λऔಘ͢Δํ๏͕ ͳ͔ͬͨ • Control Plane ΛؚΜͩ Service Mesh Ͱ͸ͳ͘ɺSide- car container ͱͯ͠୯ʹૉͷ Envoy ΛࡌͤΔͷΈ 

DevSupport: ೔ସΘΓ౰൪Ͱఆৗӡ༻ۀ຿Λߦ͏ • Slack ͷ௨஌Λ֬ೝͯ͠ݪҼௐࠪ • Sentry Exception, SLO Alert, GCP Pub/Sub Dead Letter • खಈରԠ͕ඞཁͳ΋ͷ͸֤νʔϜʹΤεΧϨʔγϣϯ • CS(Customer Support)໰͍߹ΘͤͷҰ࣍ड͚ • શମ޲͚ϝϯγϣϯͷ1࣍ड͚ 

ى͖͍ͯͨ՝୊: No SLO Alert • ϦϦʔε͔Βࠓ·ͰҰ౓΋ SLO Alert ͕໐ͬͨ͜ͱ͸ͳ͍ • Sentry ͷ Exception ྔ͕ SLI ʹ൓ө͞Ε͍ͯͳ͍ؾ͕͢Δ • Կ͕ى͖͍ͯΔͷͩΖ͏͔ʁ • গͳ͘ͱ΋ Sentry Exception Λ1݅ͣͭݟ͍ͯΔ࣌఺Ͱ Error Budget ͱ͍͏֓೦ ͸ར༻Ͱ͖ͯͳ͍ • SLO ͕ࣗ෼ͨͪͷظ଴஋ΑΓ΋؇͗͢Δʁ • SLI ͷઃఆ͕ޡ͍ͬͯΔʁ • ௐࠪͨ͠ 

Outline • ࣗݾ঺հ • ʰελσΟαϓϦ தֶߨ࠲ʱʹ͍ͭͯ • SLI/SLO ͸ͳΜͷͨΊʹ͋Δͷ͔ • αʔϏεӡ༻ͷݱঢ়ͱ՝୊ • ՝୊ʹ࣮͋ͨͬͯࡍʹऔΓ૊Μͩ͜ͱ • ·ͱΊ 

Ծઆ: Envoy ͷ metrics (SLIᶄᶅᶆ) ͕͓͔͍͠ͷͰ͸ʁ • Yes • Exception ͷҰ෦͸ DNS ໊લղܾͰࣦഊ͍ͯͨ͠ • ͭ·Γɺhttp request ʹࢸ͍ͬͯͳ͍ • envoy.cluster.upstream_rq_2xx ʹܭ্͞Εͳ͍ͷ͸ͦΕ͸ͦ͏ • ᶄͷ௨৴࣌ɺ໊લղܾʹࣦഊύλʔϯ • ᶃ ͷ SLI Ͱܭଌ͞Ε͍ͯΕ͹ྑ͍͕…? ௨ৗͷ௨৴
 UBSBBQJHBUFXBZ
DPOUBJOFS
͕
IUUQ UBSBNBJO
Λ໊લղܾ͢Δ
Š
͕͜͜ ࣦഊͨ͠
 IUUQUBSBNBJO
Ͱ௨৴͢Δ


Ծઆ2: Reverse Proxy ͷ metrics (SLIᶃ) ͕͓͔͍͠ͷͰ͸ʁ • Yes • GraphQL ϦΫΤετ్͕தͰࣦഊͨ͠৔߹ɺhttp Ͱ͸ 200 Λฦ͍ͯͨ͠😱 • ϦϦʔε࣌ɺ෦෼ࣦഊ͸ 500 Ͱฦ͢͜ͱΛܾΊ͕ͨɺͦ͏͞Ε͍ͯͳ͔ͬͨ ௨ৗͷ௨৴
 $MJFOU
͔Β
IUUQTKVOJPSMFBSOTUVEZTBQVSJKQ
ʹΞΫηε͢Δͱ
3FWFSTF
1SPYZ
ʹ౸ୡ
 3FWFSTF
1SPYZ
͔Β
țBSBBQJHBUFXBZ
΁
QSPYZᶄ
 UBSBBQJHBUFXBZ
͔Β
țBSBNBJO
΁௨৴ᶄ
Š͜͜ͰΤϥʔ͕ൃੜ


ରॲ1ɿGraphQL Error ͷ৔߹ http 500 Λฦ͢ • ݩʑ GraphQL ͸ http ͷ͜ͱΛؾʹ͍ͯ͠ͳ͍ • ڍಈ͸ GraphQL server library ͷڍಈʹґଘ͢Δ • Response status ͸ 200 ʹ౷Ұ͢ΔϓϥΫςΟε΋͋Δ • Client ΋ Error ͸ Response ͷ errors ΛݟΔͷͰ໰୊͸ͳ͍ ಉ྅͕γϡοͱ௚ͯ͘͠Ε·ͨ͠🙏
4QFDJBM
5IBOLT
!2VSBNZ


ରॲ2ɿ Envoy Λ΍Ίͯ Datadog APM metrics Λར༻ • ෳࡶੑʹΑΔτϥϒϧγϡʔτͷ೉͠͞ΛݮΒͨ͢Ί • Envoy ͷ metrics ʹ໰୊͕͋ͬͨΘ͚Ͱ͸ͳ͍ • ӡ༻ͷ՝୊΋ଟ͘ metrics औಘҎ֎ͷϝϦοτ͸ಘΒΕ͍ͯͳ͔ͬͨ • Curcuit Breaker ೖΕ͍ͯͨ΋ͷͷൃಈͨ͠έʔε͸΄ͱΜͲͳ͍ • Envoy ͷ version up ରԠʢग़དྷ͍ͯͳ͍ʣ • Pod ಺ side-car container ͷىಈɾऴྃॱ੍ޚʢenvoy Λ଴ͨͳ͍ͱΤϥʔʹͳΔʣ • Rollouts Λ࢖͍ͬͯΔ৔߹ͷ Patch ํ๏ʢResource ٯసͯ͠ো֐ʹͳͬͨ͜ͱ΋ʣ 

খωλ: Datadog APM ݁ߏบ͕͋Δ(1) • http client ͷ APM Plugin ͷ resource tag ͸ default Ͱ͸ http method Ͱ͋Δ • Ѽઌ͝ͱͷ SLI ͱͯ͠࠾༻͢Δʹ͸ hostname ͕ඞཁ • Node, Ruby ͰͦΕͧΕରԠ • ૊৫಺Ͱ http-client ͷ resource tag ͷ໋໊ن໿Λ߹ҙ 

খωλ: Datadog APM ݁ߏบ͕͋Δ(2) • trace.http.request.errors Ͱ͸ http 5xx ͸֘౰͠ͳ͍ • ٯʹ 4xx ͸֘౰͢Δ • trace.http.request.hits.by_http_status Λར༻͢Δඞཁ͕͋Δ 

උߟ: tara ͱ͍͏ͷ͸͜ͷϦχϡʔΞϧϓϩδΣΫτͷίʔυωʔϜͰɺ࠷ۙΠϯλϏϡʔͰύϒϦοΫʹͳͬͨ https://brand.studysapuri.jp/career/interview/article/Saori_Suzuki/ ݩʑ͋ͬͨ Ϣʔβج൫Λ ؚΉαʔϏε 3FWFSTF
1SPYZ /HJOY • SLO Λݟ௚ͨ͠ • (a)Availability ͱ (b)Latency • http ͷ metrics Λ࢖͏ • ҎԼͷ4Օॴʹ(a/b)2छྨͣͭ • ᶃ api-gateway • ᶄ api-gateway -> main • ᶅ api-gawatey -> content • ᶆ main -> content • 🆕ᶇ api-gateway -> Ϣʔβج൫΁ͷ request • SLO • Availability: 99.9% • Latency: 95 percentile < 1000msec • -> αʔϏε͝ͱʹݱঢ়ΛՃຯ͠ɺ 100~500msec ᶃ ᶄ ᶅ ᶆ ᶇ ϚΠΫϩαʔϏε͝ͱͷ
4-*4-0
Λഇࢭ
4-*
Λ෼͚ΔϝϦοτ͕ෳ਺
4-*4-0
Λ؅ཧ͢Δίετʹ ݟ߹͍ͬͯͳ͍ͨΊ Ϣʔβج൫޲͚
4-*4-0
௥Ճ
Ϣʔβج൫޲͚ͷڞ௨
4-*
͜Ε·Ͱ
FOWPZ
NFUSJDT
Λར༻͍ͯͨ͠ɻFOWPZ
Λ֎ͨͨ͠Ί
%BUBEPH
"1.
NFUSJDT
Λར༻ͨ͠
4-*4-0
Λ௥Ճ 

DevSupport ݟ௚͠ • Sentry Exception Ͱ͸ΞϓϦέʔγϣϯίʔυىҼͷ΋ͷҎ ֎͸શͯ Ignore ͢Δ • SLO Alert ͕དྷͨ࣌ͷجຊతͳରॲํ਑ΛυΩϡϝϯτԽ • ౰೔ରԠͰ͖ͳ͔ͬͨ΋ͷΛ2िؒʹ1ճνʔϜͰରԠ 

Outline • ࣗݾ঺հ • ʰελσΟαϓϦ தֶߨ࠲ʱʹ͍ͭͯ • SLI/SLO ͸ͳΜͷͨΊʹ͋Δͷ͔ • αʔϏεӡ༻ͷݱঢ়ͱ՝୊ • ՝୊ʹ࣮͋ͨͬͯࡍʹऔΓ૊Μͩ͜ͱ • ·ͱΊ 

Կ͕ى͖͍ͯͨͷ͔ • ϦϦʔε࣌ʹҰ౓ઃఆ͞Εͨ SLI/SLO ͸1౓΋ݟ௚͞Εͯͳ ͔ͬͨ • SLO ͕ԿͷՁ஋΋ൃش͍ͯ͠ͳ͔ͬͨ • SLI/SLO ྆ํΛݟ௚͠ɺࠓޙ΋ܧଓతʹݟ௚͢͜ͱʹͨ͠ 

Ͳ͏͢΂͖ͩͬͨͷ͔ • ։ൃऀࢹ఺ • SLO ͕ຊ౰ʹՁ஋Λ΋ͨΒ͍ͯ͠Δͷ͔Λఆظతʹݕࠪ͢Δ • ౰൪΋ྑ͍͕ɺͨ·ʹ͸શһͰݟΔ࣌ؒΛऔΔ͜ͱ΋ॏཁ • 1೔ަ୅ͩͱਂ͘ௐ΂ΔΠϯηϯςΟϒ͕ಇ͔ͳ͍ • SRE ࢹ఺ • ։ൃνʔϜ͕ SLI/SLO Λఆظతʹݟ௚͢࢓૊ΈΛ࡞Δ • ϫʔΫϩʔυ͝ͱʹ SLI/SLO Λࣗಈੜ੒͢Δ࢓૊ΈΛ࡞Δ 

ࠓ೔఻͍͑ͨ͜ͱ Ұ౓ܾΊͨ SLI/SLO ͸ܧଓతʹݟ௚͠·͠ΐ͏ 

Thank you! chaspy chaspy_ Engineering Manager Site Reliability and Web Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me