SLOをゼロからつくる

Slide 1

Slide 1 text

4-0Λθϩ͔Βͭ͘Δ 10X, Inc. 2023/7/5 SRE meetup (@babarot) Brand new SLO at 10X, inc.

Slide 2

Slide 2 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE $POUFOUT 43&νʔϜͷ͜Ε·Ͱ 4-0Λͭ͘ΔϑΣʔζʹ θϩ͔Βಋೖ͢Δ ࣮૷ͱӡ༻ ͜Ε͔Β

Slide 3

Slide 3 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE @babarot SRE / Reliability & Security Team, 10X

Slide 4

Slide 4 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE IUUQTTQFBLFSEFDLDPNYJOD[IVTIJIVJTIFYDVMUVSFEFDL

Slide 5

Slide 5 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE IUUQTMQTUBJMFSKQ

Slide 6

Slide 6 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE 43&νʔϜͷ͜Ε·Ͱ

Slide 7

Slide 7 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE IUUQTQSPEVDUYDPKQFOUSZ

Slide 8

Slide 8 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE νʔϜ νʔϜ͕Ͱ͖ΔɺϩʔυϚοϓ࡞੒ +PJO!CBCBSPU 43& ,VCFSOFUFT/BNFTQBDF੔උ %BUBEPHಋೖ 5FSSBGPSNಋೖ 0O$BMM੍౓੔උ 3FMJBCJMJUZ4FDVSJUZνʔϜʹ43& 48& 43& 4FDVSJUZ JOGSBTUSVDUVSFϦϙδτϦ ৽نύʔτφʔͷΠϯϑϥϦιʔε࡞੒ͷࣗಈԽ શࣾ'PDVTͰ4-0Λ͔͔͛ɺ4-0νʔϜ૊੒ ֤։ൃνʔϜͰ4-0Λ௥͍͸͡ΊΔ IUUQTUXJUUFSDPNCCSTUBUVT

Slide 9

Slide 9 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE 4-0Λͭ͘ΔϑΣʔζʹ

Slide 10

Slide 10 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE 4-0Λͭ͘ΔϑΣʔζʹ w 43&νʔϜ͕ͳ͍ঢ়ଶ͔Β͸͡·ͬͨ w αʔϏεϨϕϧʹ޲͖߹͑Δͱ͜Ζ·Ͱ͖ͨ w طଘͷϫʔΫϑϩʔͷݟ௚͠΍*B$ɺϞχλϦϯά؀ڥΛ੔͑ͨΓ w αʔϏεͷٸ੒௕͕߇͍͑ͯΔ w ଟ͘ͷϩʔϯν༧ఆ΍4UBJMFSͷ֦େ༧ఆ͕͋ΓɺΑΓ৴པੑʹ   ϑΥʔΧε͢ΔϑΣʔζ΁ w UP$΋͋Δ͕UP#΋͋Δ w UP#΋͋Δˠ4-"ΛͪΌΜͱ݁ͼ͍ͨ 4-0Λͭ͘Δ

Slide 11

Slide 11 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE ͜Ε·Ͱͷ4-" w ͜Ε·Ͱͷ9ʹ΋4-"͸͋ͬͨ w ͕ɺͪΌΜͱ4-0ͷઃఆ͕Ͱ͖͍ͯͳ͔ͬͨ w ʮՔಇ཰ʯΛهࡌ͍͕ͯͨ͠ఆ͕ٛᐆດʢܖ໿ऀؒʣ w ܧଓతͳܭଌ͕Ͱ͖͍ͯͳ͍ w ݄຤ͰͷखಈͷՔಇ཰ϨϙʔςΟϯά w ͖ͪΜͱ4-"ɺ4-0Λӡ༻͍ͨ͠ w ܧଓతͳܭଌ͕Մೳͳ4-*Λ΋ͭ w $6+ʢαʔϏεͷίΞͱͳΔମݧՁ஋ʣΛఆٛ͢Δ

Slide 12

Slide 12 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE ͳͥ4-0Λͭ͘Δ͔ w ͲΕ͘Β͍ͷ৴པੑΛ໨ࢦ͢ͷ͔Λશһͷ΋ͱͰ͸͖ͬΓͤ͞Δ w ʮߴ͍৴པੑΛ໨ࢦ͠·͠ΐ͏ʯ͸ਓͦΕͧΕ w αʔϏε͕໨ࢦ͢৴པੑͷ߹ҙϥΠϯ 4-0 Λ࡞Δ w ϦεΫςΠΫ &SSPSCVEHFUͷൣғ಺Ͱ߈ΊΔ w ϦεΫϔοδ ඼࣭Λᆝଛͦ͠͏ʹͳͬͨΒϦϦʔεΛࢭΊΔ൑அ w ա౓ͳཁٻ͔ΒνʔϜΛकΔ w ʮμ΢ϯͤ͞ͳ͍γεςϜʯ͸ͭ͘Εͳ͍

Slide 13

Slide 13 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE 4-"4-04-*ͱ$6+ 4FSWJDF-FWFM"HSFFFNFOU 4FSWJDF-FWFM0CKFDU 4FSWJDF-FWFM*OEJDBUPS $SJUJDBM6TFS+PVSOFZ ҙຯ ΫϥΠΞϯτ΍Ϣʔβʹ   ର͢Δ໿ଋ͝ͱɺܖ໿ࣄ߲ 4-"Λᆝଛ͠ͳ͍Α͏ʹ΋ͭ   ໨ඪ஋ 4-0ͰΈΔࢦඪ   ԿͰ4-0ΛଌΔ͔ αʔϏεͷఏڙʹ͓͍ͯ   ॏཁͱͳΔར༻ମݧɺҰ࿈ ͷػೳ ྫ ϲ݄Ͱ෼Ҏ্ɺαʔϏε μ΢ϯͨ͠Β࣌ؒ୯Ґ͋ͨ Γ̋ԁฦۚ͠·͢ μ΢ϯλΠϜ͸ϲ݄Ͱ෼ ະຬͱ͢Δ   ʢ4-"ͷ਺஋ҎԼͰόο ϑΝΛ࣋ͭʣ αΫηεϨʔτ ϨΠςϯγ ঎඼ΛߪೖͰ͖Δʁ ߪೖͨ͠঎඼͕ಧ͘ʁ όφʔ͕දࣔ͞ΕΔʁ ͓͢͢Ί͕ਪન͞ΕΔʁ

Slide 14

Slide 14 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE θϩ͔Βಋೖ͢Δ

Slide 15

Slide 15 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE IUUQTMQTUBJMFSKQ

Slide 16

Slide 16 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE 4UBJMFSͱ͸ εʔύʔ΍υϥοάετΞͳͲͷখചࣄۀऀ͕&$ΛӡӦ͢ΔͨΊͷϓϥοτϑΥʔϜ 4UBJMFS "ࣾ #ࣾ $ࣾ ࡏݿ   ੜ੒ ΞϓϦ ϐοΫ ύοΫ ഑ୡ ஫จ   ؅ཧ ച্   ܭ্

Slide 17

Slide 17 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE 4UBJMFSͱ͸ εʔύʔ΍υϥοάετΞͳͲͷখചࣄۀऀ͕&$ΛӡӦ͢ΔͨΊͷϓϥοτϑΥʔϜ 4UBJMFS "ࣾ #ࣾ $ࣾ ࡏݿ   ੜ੒ ΞϓϦ ϐοΫ ύοΫ ഑ୡ ஫จ   ؅ཧ ച্   ܭ্ 5FBN" 5FBN# 5FBN$

Slide 18

Slide 18 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE Ͳ͏΍͔ͬͨ ܦӦϝϯόʔ΍1E.ΛؚΉ1+νʔϜΛ૊੒ αʔϏεͷίΞମݧΛચ͍ग़͠$6+ΛܾΊΔ C 4-0μογϡϘʔυΛͭ͘Δ B 4-0λʔήοτΛܾΊᮢ஋Λௐ੔͢Δ $6+Λߏ੒͢Δ"1*ͷ4-* ϨΠςϯγ౳ ΛܾΊΔ τοϓμ΢ϯ ϘτϜΞοϓ

Slide 19

Slide 19 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE ܦӦϝϯόʔ΍1E.ΛؚΉ1+νʔϜΛ૊੒ 1E. &OHJOFFST 43& 2" #PBSET 5FBN" 5FBN# 5FBN$

Slide 20

Slide 20 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE ܦӦϝϯόʔ΍1E.ΛؚΉ1+νʔϜΛ૊੒ 1E. &OHJOFFST 43& 2" #PBSET 5FBN" 5FBN# 5FBN$ 5FBN4-0

Slide 21

Slide 21 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE αʔϏεͷίΞମݧΛચ͍ग़͠$6+ΛܾΊΔ w 4UBJMFSͷίΞମݧ w ঎඼ΛߪೖͰ͖Δ w ߪೖͨ͠঎඼Λड͚औΕΔ w ച্ʹؔ͢Δσʔλ͕ద੾ʹॲཧ͞Ε Δɺ࿈ܞ͞ΕΔ w $SJUJDBM6TFS+PVSOFZ w $6+5BSHFU͓٬͞· w ച৔Ͱ঎඼͕දࣔ͞Ε͍ͯΔ w ഑ୡ࿮ͷӾཡɺ֬อ͕Ͱ͖Δ w ஫จ಺༰Λ֬ೝͰ͖Δ w $6+5BSHFU഑ୡελοϑ

Slide 22

Slide 22 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE $6+Λߏ੒͢Δ"1*ͷ4-* ϨΠςϯγ౳ ΛܾΊΔ ʮച৔Ͱ঎඼͕දࣔ͞Ε͍ͯΔʯͷ$6+Λߏ੒͢Δ"1*ʹରͯ͠ҎԼͭͷ4-*ΛՄࢹԽ w αΫηεϨʔτ w QFSDFOUJMFϨΠςϯγ ਺஋͸ΠϝʔδͰ͢

Slide 23

Slide 23 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE C 4-0μογϡϘʔυΛͭ͘Δ શࣾ4-0μογϡϘʔυ ਺஋͸ΠϝʔδͰ͢

Slide 24

Slide 24 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE C 4-0μογϡϘʔυΛͭ͘Δ $6+͝ͱͷ4-0μογϡϘʔυ ਺஋͸ΠϝʔδͰ͢

Slide 25

Slide 25 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE B 4-0λʔήοτΛܾΊᮢ஋Λௐ੔͢Δ w %BUBEPHϞχλʔΛந৅Խͨ͠ઃఆ ϑΝΠϧ 5FSSBGPSNϞδϡʔϧ Λఏڙ w ֤νʔϜ͕ࣗ෼ͨͪͰɺ ύεͷ௥Ճɺ࡟আ͕Ͱ͖Δ ᮢ஋Λௐ੔Ͱ͖Δ มߋ಺༰ΛϨϏϡʔͰ͖Δ ύϑΥʔϚϯεվળͰ͖Δ w 1E.͸มߋ಺༰ΛঝೝͰ͖Δ

Slide 26

Slide 26 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE B 4-0λʔήοτΛܾΊᮢ஋Λௐ੔͢Δ 200ms 400ms 600ms 800ms 1000ms ᮢ஋͕Ξϥʔτʹୡ͢Δ৔߹ w %BUBEPHϞχλʔΛந৅Խͨ͠ઃఆ ϑΝΠϧ 5FSSBGPSNϞδϡʔϧ Λఏڙ w ֤νʔϜ͕ࣗ෼ͨͪͰɺ ύεͷ௥Ճɺ࡟আ͕Ͱ͖Δ ᮢ஋Λௐ੔Ͱ͖Δ มߋ಺༰ΛϨϏϡʔͰ͖Δ ύϑΥʔϚϯεվળͰ͖Δ w 1E.͸มߋ಺༰ΛঝೝͰ͖Δ w ᮢ஋Λௐ੔͢Δ͔ɺύϑΥʔϚϯεΛվ ળ͢Δ͔

Slide 27

Slide 27 text

Slide 28

Slide 28 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE B 4-0λʔήοτΛܾΊᮢ஋Λௐ੔͢Δ 200ms 400ms 600ms 800ms 1000ms ύϑΥʔϚϯεΛվળ͢Δ w %BUBEPHϞχλʔΛந৅Խͨ͠ઃఆ ϑΝΠϧ 5FSSBGPSNϞδϡʔϧ Λఏڙ w ֤νʔϜ͕ࣗ෼ͨͪͰɺ ύεͷ௥Ճɺ࡟আ͕Ͱ͖Δ ᮢ஋Λௐ੔Ͱ͖Δ มߋ಺༰ΛϨϏϡʔͰ͖Δ ύϑΥʔϚϯεվળͰ͖Δ w 1E.͸มߋ಺༰ΛঝೝͰ͖Δ w ᮢ஋Λௐ੔͢Δ͔ɺύϑΥʔϚϯεΛվ ળ͢Δ͔

Slide 29

Slide 29 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE αϚϦ τοϓμ΢ϯʹਐΊͯେ࿮Λͭ͘Δ $&0$50ɺ2"ɺ43&ɺ1E.ͳͲΛத৺ͱͨ͠1+νʔϜΛͭ͘Γτοϓμ΢ϯͳҙࢥܾఆͰେ࿮ΛܾΊΔ 4-*4-04-"͓Αͼ$6+ $SJUJDBM6TFS+PVSOFZ ͳͲͷҙຯΛΈΜͳ͕ཧղ͢Δ αʔϏε 4UBJMFS ͷίΞͱͳΔϢʔβମݧΛચ͍ग़͠$6+ΛܾΊΔ $6+Λ΋ͱʹ4-*4-0ΛԾஔ͖͢Δ μογϡϘʔυΛ࡞Δ ͩΕ΋͕؆୯ʹμογϡϘʔυΛ࡞ΕΔΑ͏ʹ͢Δ ΦʔφʔγοϓΛ࣋ͨͤͯϘτϜΞοϓʹӡӦ͢Δ ֤$6+ͷΦʔφʔγοϓΛ࣋ͭνʔϜʹ4-0ؔ࿈ϨΫΛೖΕͨ͋ͱ ֤νʔϜ͕੹೚Λ࣋ͭ΂͖$6+Λࣗೝ͢Δ ֤νʔϜ͕ࣗ෼ͷνʔϜͷ$6+ʹ֘౰͢Δ4-0ΛܾΊΔʢᮢ஋ͷௐ੔ʣ 13Ͱௐ੔͓ͯ͠ޓ͍ʹঝೝ͠߹͏

Slide 30

Slide 30 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE ࣮૷ͱӡ༻

Slide 31

Slide 31 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE Ͳ͏࣮૷͢Δ͔ʁ w ϞχλϦϯάج൫ʹ%BUBEPHΛར༻͍ͯ͠Δ w %BUBEPHͷ4-0ػೳΛ࢖͍͍ͨ w %BUBEPHͷ4-0ػೳ͸ͭҎ্ͷ%BUBEPHϞχλʔ "MFSUϧʔϧͷΑ͏ͳ΋ͷ   ΛBHHSFHBUJPOͨ͠΢ΟδΣοτ w ෳ਺ͷ4-0ΛτϥοΩϯά͢Δ৔߹ɺ<4-0ͷ਺>Y<Ϟχλʔͷ਺>ʹͳΔ

Slide 32

Slide 32 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE Ͳ͏࣮૷͢Δ͔ʁ %BUBEPH%BTICPBSE

Slide 33

Slide 33 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE Ͳ͏࣮૷͢Δ͔ʁ %BUBEPH4-0T %BUBEPH%BTICPBSE $6+ʮച৔Ͱ঎඼͕දࣔ͞Ε͍ͯΔʯ

Slide 34

Slide 34 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE Ͳ͏࣮૷͢Δ͔ʁ %BUBEPH4-0T %BUBEPH%BTICPBSE $6+ʮച৔Ͱ঎඼͕දࣔ͞Ε͍ͯΔʯ $6+͝ͱͷ4-0μογϡϘʔυ

Slide 35

Slide 35 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE Ͳ͏࣮૷͢Δ͔ʁ BQJDVKY 4VDDFTT3BUF BQJDVKY -BUFODZ BQJDVKY 4VDDFTT3BUF BQJDVKY -BUFODZ BQJDVKZ 4VDDFTT3BUF BQJDVKZ -BUFODZ BQJDVKZ 4VDDFTT3BUF BQJDVKZ -BUFODZ BQJDVK[ 4VDDFTT3BUF BQJDVK[ -BUFODZ BQJDVK[ 4VDDFTT3BUF BQJDVK[ -BUFODZ %BUBEPH.POJUPST %BUBEPH4-0T %BUBEPH%BTICPBSE $6+ʮച৔Ͱ঎඼͕දࣔ͞Ε͍ͯΔʯ

Slide 36

Slide 36 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE Ͳ͏࣮૷͢Δ͔ʁ BQJDVKY 4VDDFTT3BUF BQJDVKY -BUFODZ BQJDVKY 4VDDFTT3BUF BQJDVKY -BUFODZ BQJDVKZ 4VDDFTT3BUF BQJDVKZ -BUFODZ BQJDVKZ 4VDDFTT3BUF BQJDVKZ -BUFODZ BQJDVK[ 4VDDFTT3BUF BQJDVK[ -BUFODZ BQJDVK[ 4VDDFTT3BUF BQJDVK[ -BUFODZ $6+ʮച৔Ͱ঎඼͕දࣔ͞Ε͍ͯΔʯ %BUBEPH.POJUPST %BUBEPH4-0T %BUBEPH%BTICPBSE

Slide 37

Slide 37 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE 5FSSBGPSNϞδϡʔϧ w %BUBEPH.POJUPSͱ%BUBEPH4-0Λ ϥοϓͨ͠5FSSBGPSNϞδϡʔϧΛ࡞੒ w EBUBEPH@NPOJUPSUG w EBUBEPH@TFSWJDF@MFWFM@PCKFDUJWFUG

Slide 38

Slide 38 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE 5FSSBGPSNϞδϡʔϧ w %BUBEPH.POJUPSͱ%BUBEPH4-0Λ ϥοϓͨ͠5FSSBGPSNϞδϡʔϧΛ࡞੒ w EBUBEPH@NPOJUPSUG w EBUBEPH@TFSWJDF@MFWFM@PCKFDUJWFUG w ධՁର৅ͷ"1*ύεΛ*OQVUʹ΋Β͏ w 4VDDFTTSBUF w -BUFODZ w ͳͲͷ4-*ΛબΜͰઃఆͰ͖Δ w Ξϥʔτ͢Δᮢ஋΋ݸผʹઃఆ

Slide 39

Slide 39 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE ӡ༻ 4-0Λͳͥ΍Δ͔ͷɺશࣾΨΠμϯεΛ௨ͯ͠ཧղͯ͠΋Β͍ɺ)PXUPʹ͍ͭͯ͸ΨΠυΛॻ͍ͯ   ֤νʔϜʹల։͢Δ ͳͥ4-0ΛܾΊΔඞཁ͕͋Δͷ͔ʁͦΕ͸Ͳͷ͘Β ͍ͷ৴པੑ 3FMJBCJMJUZ Λ໨ࢦ͢ͷ͔Λશһͷ΋ͱͰ ͸͖ͬΓͤ͞ΔͨΊͰ͢ɻαʔϏεʹ͓͍ͯʮߴ͍৴ པੑʯΛ௥ٻ͠·͠ΐ͏ͱ͍͏ͷ͸ͦͷͱ͓ΓͰ͕͢ ʮͲΕ͘Β͍͕ߴ͍৴པੑͳͷ͔ʯ͸ਓͦΕͧΕʹͳ Γ͕ͪͰ૊৫ʹ͓͍ͯ΋ະఆٛͰ͢ɻͦΕΛ਺஋໨ඪ ʹ͠·͠ΐ͏ͱ͍͏ͷ͕4-0Ͱ͢ɻ4-0Λఆٛ͢Δ ͜ͱͰɺࠓ͸໨ඪΛຬ͍ͨͯ͠Δ͔Β։ൃΛՃ଎ͤ͞ ·͠ΐ͏ʢϦεΫςΠΫʣɺ໨ඪΛׂΓͦ͏ͳͷͰ։ ൃ଎౓Λམͱ͠·͠ΐ͏ɾ໨ඪୡ੒Ͱ͖ͳ͔ͬͨͷͰ ։ൃΛࢭΊ·͠ΐ͏ʢϦεΫϔοδʣͳͲͷҙࢥܾఆ ͕ՄೳʹͳΓ·͢ɻ

Slide 40

Slide 40 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE ͜Ε͔Β

Slide 41

Slide 41 text

$PQZSJHIU9 *OD"MMSJHIUSFTFSWFE ͜Ε͔Β w ֤νʔϜͰ4-0Λத৺ͱͨ͠։ൃɾӡ༻ମ੍Λ੔͑ͨ w ͜Ε͸43&νʔϜ͚ͩͰ΍ΔͷͰ͸ͳ͘τοϓμ΢ϯʹ߹ҙ͢Δ΄͏͕   ͏·͍͖͘΍͍͢ w ΤϥʔόδΣοτ΍4-0Ξϥʔτʹର͢Δ࣮ӡ༻͸͜Ε͔Β w 943&Ͱ͸ҰॹʹਐΊͯ͘ΕΔ஥ؒΛืू͍ͯ͠·͢