プロダクション環境の信頼性を損ねず観測する技術

Slide 1

Slide 1 text

ϓϩμΫγϣϯ؀ڥͷ৴པੑΛଛͶͣ ؍ଌ͢Δٕज़ 2022/05/15 SRE NEXT 2022 Sohei Iwahori (GREE, Inc.)

Slide 2

Slide 2 text

Agenda » Introduction ؍ଌ͸௿ίετʁ » case1 ؂ࢹSaaSαʔϏεো֐͕ϓϩμΫγϣϯαʔϏεো֐ʹͭͳ͕ͬͨ࿩ » case2 ϩάΤʔδΣϯτෛՙ͕ϓϩμΫγϣϯো֐ΛҾ͖ى͜͢࿩ » case3 Ұ෦ͷ؀ڥͷ໰୊͕γεςϜશମʹӨڹͯ͠͠·͏࿩ » Recap

Slide 3

Slide 3 text

Introduction: ؍ଌ͸௿ίετʁ

Slide 4

Slide 4 text

؍ଌίετ͸ແࢹͯ͠Α͍ʁ ؍࡯ऀޮՌ͸ؾʹ͠ͳ͍ ෛՙ͸ඇৗʹখ͍͞ʢ͸ͣʣ » ͱʮೖ໳؂ࢹʯͷίϥϜʹ΋ॻ͔Ε͓ͯΓ » ओʹϝτϦΫεͷจ຺ͰಛʹΤʔδΣϯτΛಋೖ͢Δ͜ͱΛؾʹ͢Δ΂͖Ͱ ͸ͳ͍ͱड़΂ΒΕ͍ͯ·͢ » ͜Ε͸ଟ͘ͷ৔߹࣮ࡍʹͦͷͱ͓Γ

Slide 5

Slide 5 text

؍ଌͷίετ » ͬ͘͟ΓͱΠϕϯτͷස౓ɺॲཧ͢Δσʔλͷྔʹґଘ͢Δ » Metrics » ҰൠʹҰఆͷίϯτϩʔϧ͞ΕִͨؒͰऩू͞ΕΔ » ྔ͕໰୊ʹͳΔ͜ͱ͸͋·Γͳ͍ʢΧʔσΟφϦςΟ͕ߴ͗͢Δ৔߹͸ແࢹͰ͖ͳ͍͜ͱ΋͋Δʣ » Logging » Πϕϯτͷස౓ʹґଘ͢ΔʢΠϕϯτࣗମ͸ίϯτϩʔϥϒϧͰ͸ͳ͍͜ͱ͕ଟ͍ʣ » ڊେͳߏ଄ԽϩάɺελοΫτϨʔεͳͲ͸΍΍ίετ͕ߴ͘ͳΔ » Tracing » ֬཰ʹΑΔස౓ͷίϯτϩʔϧ

Slide 6

Slide 6 text

؍ଌίετ͕ແࢹͰ͖ͳ͘ͳΔ࣌ » ऩूͷʮաఔʯͰ໰୊͕ൃੜ͢Δέʔε » Πϕϯτස౓͕૝ఆ͍ͯ͠ΔॲཧྔΛΦʔόʔͯ͠͠·͏έʔε » ର৅γεςϜҎ֎͔ΒͷӨڹΛड͚ͯ͠·͏έʔε » Ҏ্ͷέʔεʹ͍ͭͯͷ໰୊ͷ࣮ྫͱߟྀ͢΂͖͜ͱʹ͍͓ͭͯ࿩͠·͢

Slide 7

Slide 7 text

case1 ؂ࢹSaaSαʔϏεো֐͕ ϓϩμΫγϣϯαʔϏεো֐ʹ ͭͳ͕ͬͨ࿩

Slide 8

Slide 8 text

ൃੜͨ͠ࣄ৅ » 1000୆Ҏ্ͷαʔόͰGrafana Agent͕Քಇ͍ͯ͠ΔΦϯϓϨ؀ڥ » Grafana CloudͰো֐͕ൃੜʢ2hఔ౓ͷμ΢ϯʣ » ͠͹Βͯ͘͠GrafanaCloudΛར༻͍ͯ͠ΔΦϯϓϨͷϓϩμΫγϣϯ؀ڥͰ ো֐͕ൃੜ » ֎෦΁ͷ௨৴Ͱ࠶ૹɺλΠϜΞ΢τ͕ൃੜ » Կ͕͓͖ͨͷ͔ʁ

Slide 9

Slide 9 text

Կ͕͓͖ͨͷ͔

Slide 10

Slide 10 text

Կ͕͓͖ͨͷ͔

Slide 11

Slide 11 text

Կ͕͓͖ͨͷ͔

Slide 12

Slide 12 text

Կ͕͓͖ͨͷ͔

Slide 13

Slide 13 text

Կ͕͓͖ͨͷ͔

Slide 14

Slide 14 text

Կ͕͓͖ͨͷ͔

Slide 15

Slide 15 text

Grafana AgentͷRemoteWrite » σϑΥϧτͷϦτϥΠڍಈ͕͔ͳΓϔϏʔ » ͍ΘΏΔexponential backoff্͕ͩݶ͕100msʢPrometheus͸ݱࡏ͸5secʹʣ » https://github.com/prometheus/prometheus/pull/9634 » γϟʔυ͸࠷େͰ200·Ͱ૿͑Δ » 1/4ͱͯ͠΋ো֐ͷى͖͍ͯΔRemoteWriteઌʹඵؒ50 * 10 * 1000ϦΫΤετ͢ Δ͜ͱʹ » ্هissueͰ͸DDOSͱݴΘΕ͍ͯΔɺෛՙࢼݧฒͷϦΫΤετ͕ඈΜͰ͠·͏

Slide 16

Slide 16 text

ઃఆ஋ͷݟ௚͠ͷͨΊͷݕূ » ঢ়گ͸Θ͔ͬͨͷͰϦτϥΠΛ཈͑ΔͨΊͷઃఆ஋Λﬁxͤ͞ΔͨΊ࠶ݱࢼݧΛ࣮ࢪ » ͔ͭͯ͸Retry্ݶͷઃఆ͕͕͋ͬͨ࡟আ͞Εͯ͠·ͬͨ » backoffͱshard਺ͷదਖ਼஋Λݕ౼ » ݕূ؀ڥʹPrometheusΛߏங࣮ͯ͠ࡍͷΦϯϓϨ؀ڥ͔ΒϦϞʔτϥΠτͰ࠶ݱݕূ » ΦϯϓϨαʔό -> ALB -> ݕূEC2Πϯελϯε » --enable-feature=remote-write-receiver ͕ͪΐ͏Ͳ࢖͑ͨͷͰศར » ઃఆΛม͑ɺALBϨΠϠʔͰݻఆͰ503Λฦ͢ɺ໭͢Λ܁Γฦ͢͜ͱͰঢ়گΛ࠶ݱ͠ςετ

Slide 17

Slide 17 text

ઃఆ஋ͷݟ௚͠ͷͨΊͷݕূ » ύϥϝʔλΛม͑ͯো֐࣌ͷ1ϊʔυͷڍಈΛ࠶ݱ » ϦτϥΠ࣌ʹτϥϑΟοΫΛ཈͑ͭͭαʔϏε෮چʹͦΕͳΓʹૣ͘ରԠͰ ͖Δͱ͍͏͜ͱͰҎԼͷઃఆʹ min_shards: 1 max_shards: 5 (default 200) min_backo : 100ms (default 30ms) max_backo : 1m (default 100ms)

Slide 18

Slide 18 text

Lesson Learned » ϝτϦΫεͷऔಘ͸Ұൠʹ௿ෛՙ͕ͩpushڍಈʹ͸ؾΛ͚ͭͨํ͕ྑ͍ » ಛʹ࠶ૹ·ΘΓͷڍಈ͸೺Ѳ͓ͯ͘͠ͱྑ͍ » backoffͷΞϧΰϦζϜɺlimitͳͲ » ͜ΕΒͷڍಈʹؔΘΔϝτϦΫε΋औಘ͓ͯ͘͠ » ՄೳͰ͋Ε͹࣮ࡍʹμ΢ϯ࣌ͷڍಈΛࣄલʹςετ͓ͯ͘͠΂͖ » ݱ୅͸͜ͷ͋ͨΓ΍Γ΍͘͢ͳ͍ͬͯΔͷͰ݁ߏ΍ΓΑ͏͕͋Δ » ALBͰͷݻఆͷΤϥʔϨεϙϯεͷ੾Γସ͑ͳͲ

Slide 19

Slide 19 text

case2 ϩάΤʔδΣϯτෛՙ͕ ϓϩμΫγϣϯো֐ΛҾ͖ى͜͢࿩

Slide 20

Slide 20 text

ൃੜͨ͠ࣄ৅ » AWSͰՔಇ͍ͯ͠ΔKVSͷΠϯελϯε͕ߴෛՙͰ઀ଓΤϥʔ » CPUෛՙ͕ߴෛՙʹͳ͓ͬͯΓ઀ଓ͕ॲཧͰ͖͍ͯͳ͍ » ର৅αʔό಺Ͱ͸cronͰىಈ͍ͯ͠ΔϩάνΣοΫΤʔδΣϯτ͕଺ཹͯ͠ େྔͷϓϩηε͕ىಈ͍ͯͨ͠

Slide 21

Slide 21 text

ͳʹ͕ى͖ͨͷ͔ʁ » ผىҼʹΑΓ·ͱ·ͬͨΤϥʔϩά͕ు͔ΕΔ » 5෼୯ҐͰىಈ͍ͯ͠ΔcronͷϩάνΣοΫ͕ऴΘΒͣϓϩηε͕଺ཹ » ϩάνΣοΫʹΑΔIO/CPUෛՙͱϚονߦͷऔΓࠐΈʹΑΔϝϞϦ࢖༻্ঢ » αʔόϦιʔεΛṧഭ͠Քಇ͍ͯ͠ΔKVSαʔϏεͷύϑΥʔϚϯε͕ѱԽ ͠ɺো֐ʹ

Slide 22

Slide 22 text

ରԠ » Ұ࣍ରԠ » ϩάͷϙδγϣϯϑΝΠϧΛඈ͹ͯ͠εΩοϓ » ελοΫ͍ͯ͠ΔϓϩηεͷkillͰҰ࣍ରԠ » ೋ࣍ରԠͱͯ͠ΤʔδΣϯτमਖ਼ » จࣈྻ݁߹͕஗͘Ϛονߦ͕ଟ͍৔߹ʹ໰୊ͱͳΔͨΊमਖ਼ » εΩοϓॲཧͷ࣮૷

Slide 23

Slide 23 text

Lesson Learned » Τϥʔϩά͸جຊతʹ௿ස౓Πϕϯτ͕ͩΠϯϑϥʹ໰୊͕͋Δ৔߹ͳͲ͸ਖ਼ৗܥΑΓେ ྔʹग़ྗ͞ΕΔ » Ξϥʔτͷ௨஌ͱ͍͏໨తʹ͓͍ͯ͸ϩάΛ࢒Βͣಧ͚ΔΑΓՄ༻ੑɺ଎ใੑΛ༏ઌ͢Δ » େྔͷ௨஌͸ͦ΋ͦ΋ਓؒ΋ಡΉ͜ͱ͕Ͱ͖ͳ͍ » ෳ਺݅ى͖͍ͯΔͷ͔ʁ1ϊʔυͰى͖͍ͯΔͷ͔ʁͳͲͷ৘ใ͕෼͔Ε͹े෼ » cgroupͳͲʹΑΔϦιʔε੍ݶ͸༗ޮ͕ͩॲཧ͸ελοΫͯ͠͠·͏ͷͰ෮چखॱ΋ runbookʹ༻ҙ͓ͯ͘͠ʢಡΈඈ͹ͯ͠࠶ىಈ͢ΔखॱͳͲʣ

Slide 24

Slide 24 text

+ΤʔδΣϯτͷߟྀࣄ߲ » xxx-agent͕ϝϞϦΛେྔʹফඅͯ͠͠·͍ͬͯΔέʔε͸͋Δ » ྫ͑͹td-agent͸pluginଆͰϦʔΫ͕ൃੜͯ͠͠·͏έʔεͳͲ͕Α͋͘Δ » ఆظతʹ࠶ىಈ͕͔͔ΔɺೖΕସΘΔΑ͏ͳ࡞ΓͳΒͦΕ·Ͱ࣋ͯ͹͍͍ͷͰϚΠφʔͳ΋ͷ͸ແࢹͰ͖Δ » ϦʔΫͷ܏޲͸ྫ͑͹process-exporterͳͲͰϢʔβʔ͝ͱͷϝϞϦ࢖༻ྔΛτϥοΫ͓ͯ͘͠ͱݕ஌Ͱ ͖ͯศར » ϩάΤʔδΣϯτʹؔͯ͠͸ѻ͏ϩάͷछྨʹԠͯ͡ΤʔδΣϯτΛ෼͚͓ͯ͘ͱྑ͍ » Ϗδωε෼ੳ༻ͷϩάͱγεςϜϩάͷΤʔδΣϯτ͸ཁٻ͕ҟͳΔ » ͦΕͧΕ͕ӨڹΛ༩͑ͣʹૢ࡞Ͱ͖Δͱѻ͍΍͍͢

Slide 25

Slide 25 text

ϦʔΫՄࢹԽͷྫ

Slide 26

Slide 26 text

case3 Ұ෦ͷ؀ڥͷ໰୊͕γεςϜશମʹӨ ڹͯ͠͠·͏࿩

Slide 27

Slide 27 text

ൃੜͨ͠ࣄ৅ » େྔͷϩάΞϥʔτ͕ಛఆͷϓϩμΫτͰൃੜ » ϓϩμΫτϩʔϯνͷλΠϛϯά » ΞϥʔτΛ༗ޮԽͯ͠ͷෛՙࢼݧͳͲ » ΞϥʔτΛॲཧ͢ΔͨΊͷΩϡʔ͕٧·ΓϓϩμΫγϣϯ؀ڥͷΞϥʔτʹ ஗Ԇ͕ൃੜ » ഑ૹ஗ԆͷSLO͸10෼·Ͱͱ͍ͯ͠Δ͕͜ΕΛ௒͑ͯ͠·͏

Slide 28

Slide 28 text

Ωϡʔͷͭ·Γͱ͸ » AWS/GCPͰ͸಺੡ͷyusuraͱ͍͏ruby੡ͷαʔϏε͕ΞϥʔτΛॲཧ » ໾ׂతʹ͸Alertmanagerʹ͍ۙ΋ͷ » Ωϡʔ͸୯ҰͷAmazon SQSΩϡʔʹੵ·ΕΔ » Ωϡʔ͕ੵ·ΕΔͱॲཧ͢ΔϫʔΧʔ΋εέʔϧ͢ΔΑ͏ʹͳ͍ͬͯΔ͕ɺ େྔͷϩάϕʔεͷΞϥʔτͳͲʹΑΓ٧·ͬͯ͠·͏͜ͱ͕͋Δ » Ωϡʔ͕େྔʹੵ·ΕΞϥʔτ͕஗Ԇ͢Δ͜ͱ͕ෳ਺ճൃੜ

Slide 29

Slide 29 text

Կ͕͓͖ͨͷ͔

Slide 30

Slide 30 text

Կ͕͓͖ͨͷ͔

Slide 31

Slide 31 text

Կ͕͓͖ͨͷ͔

Slide 32

Slide 32 text

Կ͕͓͖ͨͷ͔

Slide 33

Slide 33 text

࣮૷໘ͰͷվળରԠ » ֤छηʔϑςΟͷ࣮૷ » ֤छ੍ݶػೳͷ࣮૷ » rate limitͷ࣮૷ » ಉҰΞϥʔτͷsuppressɺsummarize » ϩδοΫ͕୯७Ͱෛՙ͕௿͘ػೳ͢Δ΋ͷ͔Βॱʹࢪߦ » ۓٸ࣌ͷϓϩμΫτ͝ͱͷ࡟আػೳΛఏڙ͢ΔlambdaϑΝϯΫγϣϯ

Slide 34

Slide 34 text

ঢ়گͷՄࢹԽ » Ωϡʔͷॲཧɺ஗Ԇঢ়گΛՄࢹԽ » ֤ϓϩμΫτ͝ͱͷॲཧ਺ͳͲ͸ϫʔ Χʔࣗ਎͕exportͯ͠ՄࢹԽ » ಛఆͷϓϩμΫτىҼͰ͋Δ͜ͱ͕Θ ͔ͬͨΒରԠґཔ

Slide 35

Slide 35 text

Lesson Learned » جຊతʹ͸Ωϡʔ͸٧·ΒͤͨΒΞ΢τ » ڞ༻γεςϜ͸ಛఆͷ؀ڥ͔Βͷෛՙ͕શମʹӨڹΛٴ΅͞ͳ͍Α͏ઃܭ͢ Δ » ਓ͕ؒΈΔ͜ͱͷͰ͖Δ௨஌͸ݶΒΕΔͷͰɺಉҰɺࣅ͍ͯΔ௨஌͸ੵۃత ʹdrop͍ͯͬͯ͠Α͍ » શମͷՄ༻ੑɺ଎ใੑΛ༏ઌ͢Δ

Slide 36

Slide 36 text

Recap

Slide 37

Slide 37 text

Recap » ؍ଌίετ͸جຊతʹ͸ແࢹͰ͖Δ͕ɺ૝ఆ֎ͷίετ͕͔͔Δ͜ͱ͕͋Δ » γεςϜͷڥք෦෼͸ಛʹؾʹ͓ͯ͘͠ͱྑ͍ » ૹ৴ઌͷμ΢ϯͳͲ͸ࣄલʹ૝ఆɺݕূͰ͖͍ͯΔͱΑ͍ » ࠶ૹ͕ͲͷΑ͏ͳස౓ɺΞϧΰϦζϜͰߦΘΕΔ͔ » ਓؒ΁ͷ௨஌Λ໨తͱͨ͠ΤʔδΣϯτ͸଎ใੑɺՄ༻ੑΛ༏ઌ͢Δ » ݹ͘ͳͬͨ৘ใΛ࣌ؒΛ͔͚ͯ௨஌͢Δ͜ͱʹ͸͋·Γҙຯ͸ͳ͍ » ڞ༻ج൫Ͱ͸ಛఆ؀ڥͷ໰୊͕શମʹӨڹΛٴ΅͠ʹ͍͘Α͏ʹઃܭ͢Δ » ؍ଌͷͨΊͷ࢓૊Έͦͷ΋ͷ΋؍ଌ͢Δ

Slide 38

Slide 38 text

thank you for listening

Slide 39

Slide 39 text

who? » Sohei Iwahori (@egmc) » GREE, Inc. » Πϯϑϥ / Monitoring Unit Leader » ήʔϜͷΠϯϑϥͱαʔό؂ࢹɺओʹ AWS

Slide 40

Slide 40 text

Appendix » How to troubleshoot remote write issues in Prometheus » https://grafana.com/blog/2021/04/12/how-to-troubleshoot-remotewrite-issues-in-prometheus/ » REMOTE WRITE TUNING » https://prometheus.io/docs/practices/remote_write/ » Remote write not honoring max_retries setting #5636 » https://github.com/prometheus/prometheus/issues/5636

Slide 41

Slide 41 text

Appendix » 10೥΋ͷͷϝτϦΫεऩूػߏΛϦϓϨʔεͨ͠࿩ » https://labs.gree.jp/blog/2021/10/21408/