Slide 1

Slide 1 text

© 2018 Nulab Inc. #BDLMPHʹ͓͚Δ
 43&ͷࣄྫ ʙϓϩμΫτͷ੒௕ͷͨΊʹ43&͸ͳʹΛ͢΂͖͔ʙ גࣜձࣾψʔϥϘ #BDLMPHνʔϜ4JUF3FMJBCJMJUZ&OHJOFFS ࢁ࡚ؽ

Slide 2

Slide 2 text

© 2018 Nulab Inc. ࣗݾ঺հ • ࢁ࡚ ؽ (@revsystem) • BacklogνʔϜͷSite Reliability Engineer (SRE) • 2017೥3݄ʹψʔϥϘʹೖࣾʢ౦ژࣄ຿ॴˠ2018೥4݄͔Β෱Ԭຊࣾʣ • Ҏલ͸ɺSIۈ຿ɻWebΞϓϦͷઃܭ΍ओʹΦϯϓϨ؀ڥͰͷΠϯϑϥઃܭ/ߏங/ӡ༻  

Slide 3

Slide 3 text

© 2018 Nulab Inc. ͜ͷൃදʹ͍ͭͯ • ౦ژࣄ຿ॴͷ Backlog SRE ٢ᖒʢMuziʣ͕
 SRE Lounge #5 Ͱൃදͨ͠಺༰Λݩʹ͍ͯ͠·͢ • ʮSRE Lounge #5 ٢ᖒʯͰݕࡧ͢Δͱ
 ࢿྉͱΠϕϯτϨϙʔτ͕ग़͖ͯ·͢ • SRE Lounge ͸౦ژͰఆظతʹ։࠵͞Ε͍ͯΔ
 SRE ಉ࢜ͷަྲྀͷ৔Ͱ͢ʢ౦ژͷಉ྅ʹͥͻ͝঺հΛʂʣ
 https://sre-lounge.connpass.com/   ࠷ۙͷ౦ژࣄ຿ॴ ʢϑΟϯΨʔϘʔυങ͍·ͨ͠ʣ

Slide 4

Slide 4 text

© 2018 Nulab Inc. ࠓ೔࿩͍ͨ͜͠ͱ • ૊৫͕େ͖͘ͳΔͱɺશһ͕։ൃͱӡ༻Λ݉຿͢Δͷ͸೉͍͠ • SREͷۀ຿ൣғ͸ɺ์͓ͬͯ͘ͱࡍݶͳ͘޿͕͍ͬͯ͘ • ψʔϥϘͰ͸ͲͷΑ͏ͳ͜ͱ͕ى͖ͯɺͲ͏ରॲ͔ͨ͠ʁ  

Slide 5

Slide 5 text

© 2018 Nulab Inc. BacklogͱSRE

Slide 6

Slide 6 text

© 2018 Nulab Inc.  

Slide 7

Slide 7 text

© 2018 Nulab Inc. ਌͠Έ΍͍͢ϢʔβʔΠϯλʔϑΣΠεͰ   ϝχϡʔΛΫϦοΫ ಺༰Λॻ͍ͯ ௥ՃϘλϯͰ࡞੒   

Slide 8

Slide 8 text

© 2018 Nulab Inc. ϓϩδΣΫτ؅ཧͷத৺ʹ • ϓϩδΣΫτ؅ཧͷඞਢػೳΛඋ͑ɺଞαʔϏεͷ࿈ܞ΋   λεΫ؅ཧ • λεΫͷ؅ཧ͕Ͱ͖Δ͔ʁ • λεΫ͔ΒࣗಈతʹΨϯτνϟʔ τ͕࡞੒Ͱ͖Δ͔ʁ จॻ؅ཧ • จॻΛΦϯϥΠϯ্Ͱ࡞੒ͯ͠ڞ ༗Ͱ͖Δ͔ʁ ϑΝΠϧ؅ཧ • ϑΝΠϧΛڞ༗Ͱ͖Δ͔ʁ • 8FC%"7Λ࢖ͬͯ1$͔Β௚઀ ϑΝΠϧʹΞΫηεͰ͖Δ͔ʁ ιʔείʔυ؅ཧ • (JUͰιʔείʔυ͕؅ཧͰ͖Δ ͔ʁ • 4VCWFSTJPOͰιʔείʔυ͕؅ ཧͰ͖Δ͔ʁ ϞόΠϧରԠ • ެࣜͷϞόΠϧΞϓϦ͕ఏڙ͞Ε ͍ͯΔ͔ʁ ηΩϡϦςΟ • άϩʔόϧ*1ΞυϨεʹΑΔαʔ Ϗε΁ͷΞΫηε੍ݶ౳ͷઃఆ͕ Ͱ͖Δ͔ʁ • ΞΫηεϩάͷఏڙαʔϏε͕͋ Δ͔ʁ ֦ுੑ • "1*͕ఏڙ͞Ε͍ͯΔ͔ʁ • ৘ใͷߋ৽࣌ʹ֎෦γεςϜʹ௨ ஌͢Δ࢓૊Έ 8FCIPPLͳͲ ͕ ఏڙ͞Ε͍ͯΔ͔ʁ • ֎෦γεςϜͳͲ͔Βϝʔϧܦ༝ ͰλεΫΛ࡞੒Ͱ͖Δ͔ʁ

Slide 9

Slide 9 text

© 2018 Nulab Inc. τϥΠΞϧͷํ๏ • https://backlog.com/ja/ ͷʮແྉτϥΠΞϧʯΛΫϦοΫ  

Slide 10

Slide 10 text

© 2018 Nulab Inc. ψʔϥϘ BacklogϓϩδΣΫτ ࠓճͷ࿩ͷഎܠɿBacklogͷ։ൃɾӡ༻ମ੍ • ࣗࣾͰ։ൃͨ͠αʔϏεΛࣗࣾͰӡ༻ • ίϯϙʔωϯτผʹෳ਺ͷ։ൃνʔϜ • ։ൃνʔϜͷ1ͭʹɺվળدΓͷ։ൃνʔϜ΋͋Δ   … ։ൃνʔϜ ։ൃνʔϜ վળدΓͷ ։ൃνʔϜ ɹSREνʔϜ αϙʔτνʔϜ

Slide 11

Slide 11 text

© 2018 Nulab Inc. ࠓճͷ࿩ͷഎܠɿγεςϜߏ੒ͷ֓ཁ • ࠃ֎ɾւ֎ʹෳ਺ͷ؀ڥʢαʔϏεΫϥελʣΛߏங • 2018೥࣌఺Ͱͷαʔό૯਺͸200୆ऑ • ৄ͘͠͸Backlog SREদӜʹΑΔաڈͷϓϨθϯΛࢀর • Backlogͷӡ༻؂ࢹ
 https://speakerdeck.com/nulabinc/operational-monitoring-of-backlog  

Slide 12

Slide 12 text

© 2018 Nulab Inc. SRE͸ͳʹΛ͢Δਓ͔ʁ

Slide 13

Slide 13 text

© 2018 Nulab Inc. SREͷఆٛ • جຊతʹ͸Opsͷ৬छ • େ͖͘෼͚ͯɺӡ༻ۀ຿ͱվળͷΤϯδχΞϦϯά • αʔϏε͝ͱʹ୲౰͸͋Δ΋ͷͷɺSREࣗମ͸શࣾ ԣஅతͳ૊৫ • SREͷۀ຿ʹؔ͢Δ໌֬ͳఆٛ͸ͳ͍   https://www.oreilly.co.jp/books/9784873117911/

Slide 14

Slide 14 text

© 2018 Nulab Inc. SREຊͷୈII෦ʮݪଇʯ • ΤϥʔόδΣοτ • αʔϏεϨϕϧ໨ඪʢSLOʣ • τΠϧ • ϞχλϦϯά • ࣗಈԽ • ϦϦʔεΤϯδχΞϦϯά   https://www.oreilly.co.jp/books/9784873117911/

Slide 15

Slide 15 text

© 2018 Nulab Inc. SRE͸ͳʹΛ͢Δਓ͔ʁ • ιϑτ΢ΣΞΤϯδχΞͱͯ͠ͷεΩϧΛ΋ͬͯɺϓϩμΫτͷ৴པੑΛҰఆҎ্ʹอͭ • ৴པੑΛอͭͨΊʹɺैདྷ͸ΦϖϨʔλ΍ΠϯϑϥΤϯδχΞͳͲ͕୲౰͍ͯͨ͠ • ۀ຿ʴαΛظ଴͞ΕΔ͜ͱ͕ଟ͍ • ΠϯϑϥͱΞϓϦ྆ํͷཧղ͕ඞཁ  

Slide 16

Slide 16 text

© 2018 Nulab Inc. Πϯϑϥͷཧղ͕ඞཁͳλεΫ • Πϯϑϥઃܭɾߏங • Πϯϑϥ؂ࢹɺো֐ରԠɺ͓Αͼ؂ࢹγεςϜߏங • ηΩϡϦςΟΞοϓσʔτରԠʢAWSΠϯελϯεɺOSʣ • ͦͷଞͷηΩϡϦςΟରࡦશൠ • AWSͷίετ؅ཧ • Backlog SREࢁ࡚ʹΑΔϒϩάهࣄʮre:dashͰAWSͷίετΛ෼ੳͯ͠Έͨʯ • ISMSରԠɺ؂ࠪରԠͷख఻͍ • WebαΠτߏஙʢWordPressʣ • etc...  

Slide 17

Slide 17 text

© 2018 Nulab Inc. ΞϓϦͷཧղ͕ඞཁͳλεΫ • γεςϜઃܭɾߏங • σʔλϕʔεɺϛυϧ΢ΣΞͳͲ • ΞϓϦ؂ࢹɺΤεΧϨʔγϣϯ͓Αͼ؂ࢹγεςϜߏங • ఆৗۀ຿ʢτΠϧʣ͓ΑͼͦͷࣗಈԽ • CI؀ڥͷߏஙɾӡ༻ʢCI؀ڥࣗମͷෆ۩߹ରԠؚΉʣ • CI͕ࣦഊͨ͠ͱ͖ͷݪҼ੾Γ෼͚ͷԆ௕Ͱɺςετͷमਖ਼ͳͲ΋ • ηΩϡϦςΟΞοϓσʔτରԠʢϛυϧ΢ΣΞͳͲʣ • ίʔυվળ • ΠϯσοΫε΍SQLͷվળɺΤϥʔϩάͷվળ • etc...  

Slide 18

Slide 18 text

© 2018 Nulab Inc. ͭ·Γʁ • SREͷੑ্࣭ɺͦͷۀ຿ൣғ͸์͓ͬͯ͘ͱࡍݶͳ͘޿͕Δ • ͦΕ͕SREͷָ͍͠ͱ͜ΖͰ΋͋Δ͕……  

Slide 19

Slide 19 text

© 2018 Nulab Inc. Backlogʹ͓͚ΔSREνʔϜͷมભ

Slide 20

Slide 20 text

© 2018 Nulab Inc. Backlogͷྺ࢙͸ΘΓͱ௕͍ • Backlog SREͷྺ࢙͸ΘΓͱ࠷͔ۙΒ   ϕʔλ൛ ϦϦʔε Backlogͷྺ࢙ 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 Backlog SREͷ ྺ࢙

Slide 21

Slide 21 text

© 2018 Nulab Inc. Ϣʔβʔ਺ͷ৳ͼ   ͜ͷ࣌ظ

Slide 22

Slide 22 text

© 2018 Nulab Inc. ಋೖاۀ਺ʢ༗ྉܖ໿਺ʣͷ৳ͼ   ͜ͷ࣌ظ

Slide 23

Slide 23 text

© 2018 Nulab Inc. SREͷϙδγϣϯ͕Ͱ͖ͨ2015ʙ2018೥ • αʔϏεن໛͕2ഒҎ্ʹ৳ͼͨ • τϥϑΟοΫ͕૿͑ͯɺγεςϜ΁ͷෛՙ΋্͕ͬͨ • SRE΋ɺͦͷଞͷΤϯδχΞ΋૿͑ͨ  

Slide 24

Slide 24 text

© 2018 Nulab Inc. 2015೥Ҏલ • ΤϯδχΞશһ͕ӡ༻΋ݟ͍ͯͨ࣌୅ • DevOpsͷπʔϧʢJenkinsͳͲʣΛಋೖ࢝͠Ίͨͷ΋͜ͷࠒ • SREؚΊɺΠϯϑϥઐ೚ͷΤϯδχΞ͸ډͳ͔ͬͨ   SRE 0໊ ༗ྉܖ໿ ʙ3000ࣾ

Slide 25

Slide 25 text

© 2018 Nulab Inc. 2015೥ • Πϯϑϥઐ೚ͷΤϯδχΞ͕ॳೖࣾ • ଞͷΤϯδχΞ͔Βӡ༻ΛҾ͖ܧ͗ͳ͕ΒɺΠϯϑϥվળ • ͜ͷͱ͖͸SREͱ͍͏໊শͰ͸ͳ͔ͬͨ • νʔϜ໊ͱ͔ແ͔ͬͨʢΒ͍͠ʣ • Πϯϑϥઐ೚ऀΛ࠾༻͖͔͚ͨͬ͠͸ʁ • ͳΜͱͳ͘ʮͦΖͦΖཁΔΑͶʯͬͯͳͬͨʢΒ͍͠ʣ   SRE 1໊ ༗ྉܖ໿ 3000ࣾʙ

Slide 26

Slide 26 text

© 2018 Nulab Inc. 2016೥ • SRE͕2໊ʹ૿Ճ • ʮSREνʔϜʯͱ͍͏໊শ͕஀ੜ • ӡ༻͕͜ͳΕͯΞϓϦͷ໰୊͕ݟ͖͑ͯͨ • ӡ༻ͱฒߦͯ͠ɺΞϓϦվળ͕࢝·Δ • ΞϓϦέʔγϣϯαʔόΛεςʔτϨεʹ͢ΔͨΊͷվળΛ͍͔࣮ͭ͘ࢪ • σʔλϕʔεΛEC2্ͷMySQL 5.1͔ΒAWS Aurora΁Ҡߦ • ৄ͘͠͸ɺদӜͷϒϩάهࣄʮີண 24࣌ʂ MySQL 5.1 ͔Β Aurora ΁ͷҠߦ100 ೔ؒ ʙ Backlog ฤ   SRE 2໊ ༗ྉܖ໿ 4000ࣾʙ

Slide 27

Slide 27 text

© 2018 Nulab Inc. 2017೥ • SRE͕3໊ʹ૿Ճ • ͦΕ·ͰSREΛ݉೚͍ͯͨ͠ΤϯδχΞ͕ΞϓϦέʔγϣϯνʔϜʹ໭Γɺ
 3݄ʹࢁ࡚ɺ8݄ʹ౦ژࣄ຿ॴʹ1໊ʢMuziʣೖࣾ • αʔό୆਺͕૿͑ɺτΠϧͷ࡞ۀྔ΋૿Ճ • kernel updateࡇΓͱ͔ • ΞϓϦվળͷൣғ΋޿͕Δ • վળ͕ਐΈɺSRE 1໊Ͱ୲౰͖͠Εͳ͍೉қ౓ͷ՝୊͚͕ͩ࢒ͬͨ • DevͱOpsͷ෼཭͕ڧ͘ͳͬͨ • SREͷਓ਺͕૿͑ͨ݁ՌɺSRE΁ͷظ଴͕ڧ͘ͳΔ • ৽͍͠ΤϯδχΞ͕૿͑ͨ݁Ռɺӡ༻Λ஌Βͳ͍ΤϯδχΞ͕େ൒ʹ   SRE 3໊ ༗ྉܖ໿ 5000ࣾʙ

Slide 28

Slide 28 text

© 2018 Nulab Inc. 2018೥ॳ಄ • SREͷ৬຿ൣғ͕޿͕Γɺӡ༻ϛε͕໨ཱͪ࢝Ίͨ • खॱͷ࿙Ε΍ɺඞཁͳ࡞ۀ͕೺ѲͰ͖͍ͯͳ͔ͬͨͳͲ • DevͱOpsͷ෼཭͕ਐΈɺো֐ൃੜ͔Β෮چ͓ΑͼݪҼڀ໌·Ͱʹ͔͔Δ͕࣌ؒ
 ௕͘ͳͬͨ   SRE 3໊ ༗ྉܖ໿ 6000ࣾʙ

Slide 29

Slide 29 text

© 2018 Nulab Inc. DevͱOpsͷ෼཭͕ى͜͢໰୊ • ฏۉ෮چ࣌ؒʢMean Time To Repair; MTTRʣͷ૿Ճ • ΞϓϦέʔγϣϯͷΞϥʔτ΍ɺސ٬͔Βͷ໰͍߹Θ͕ͤൃੜ͢Δ • SREνʔϜ΍αϙʔτνʔϜ͕ݪҼௐࠪʹಈ͘ • ͠͹Β͘ௐ͔ࠪͯ͠ΒɺϦϦʔεʹΑΔෆ۩߹͕ݪҼͱΘ͔Δ • Կ͕໰୊͔ͩͬͨʁ • ϦϦʔεεέδϡʔϧ΍ϦϦʔε಺༰Λڞ༗Ͱ͖͍ͯͳ͔ͬͨ
 → ϦϦʔε࣌ࠁΛ஌Βͣɺো֐ൃੜ࣌ࠁͱؔ࿈෇͚Ͱ͖ͳ͍
 → ϦϦʔεͷ಺༰Λ஌Βͣɺෆ۩߹ͱͷؔ܎ΛਪଌͰ͖ͳ͍ • ։ൃνʔϜ͕ϦϦʔεޙʹΞϥʔτ΍ϩάΛ؂ࢹ͍ͯ͠ͳ͔ͬͨ
 → SRE͔ΒΤεΧϨʔγϣϯ͢Δ·ͰɺରԠ։࢝Ͱ͖ͳ͍  

Slide 30

Slide 30 text

© 2018 Nulab Inc. ψʔϥϘ BacklogϓϩδΣΫτ Backlogͷ։ൃɾӡ༻ମ੍ʢ࠶ܝʣ • ࣗࣾͰ։ൃͨ͠αʔϏεΛࣗࣾͰӡ༻ • ίϯϙʔωϯτผʹෳ਺ͷ։ൃνʔϜ • ։ൃνʔϜͷ1ͭʹɺվળدΓͷ։ൃνʔϜ΋͋Δ   … ։ൃνʔϜ ։ൃνʔϜ վળدΓͷ ։ൃνʔϜ ɹSREνʔϜ αϙʔτνʔϜ

Slide 31

Slide 31 text

© 2018 Nulab Inc. BacklogϓϩδΣΫτͰߦͬͨରࡦ • SREΛϚτϦοΫεతʹ഑ஔ͢Δ૊৫ʹมߋ   ψʔϥϘ BacklogϓϩδΣΫτ … ։ൃνʔϜ ։ൃνʔϜ (ϦϦʔεස౓ˢ) վળدΓͷ ։ൃνʔϜ SREνʔϜ αϙʔτνʔϜ

Slide 32

Slide 32 text

© 2018 Nulab Inc. ։ൃνʔϜͰͷSREͷ໾໨ʢΞϓϦدΓʣ • ৘ใΛूΊͯɺରԠඞཁͳ໰୊Λൃݟ͠ɺͦͷղܾ·Ͱ௥੻ • ΞϓϦϩάͷվળɺ؂ࢹγεςϜͷվળͳͲ΋   ψʔϥϘ BacklogϓϩδΣΫτ … ։ൃνʔϜ ։ൃνʔϜ (ϦϦʔεස౓ˢ) վળدΓͷ ։ൃνʔϜ SREνʔϜ αϙʔτνʔϜ ো֐ɾෆ۩߹ Λਝ଎ʹΤε ΧϨʔγϣϯ ։ൃɾ
 ϦϦʔεͷঢ়گ
 Λີʹ఻͑Δ ৘ใΛूΊΔ

Slide 33

Slide 33 text

© 2018 Nulab Inc. ։ൃνʔϜͰͷSREͷ໾໨ʢΠϯϑϥدΓʣ • SRE͸ΠϯϑϥدΓͷۀ຿ʹूத • ΞϓϦվળ͕ඞཁͳ࡞ۀ͸ɺ։ൃνʔϜ͕र͏   ψʔϥϘ BacklogϓϩδΣΫτ … ։ൃνʔϜ ։ൃνʔϜ (ϦϦʔεස౓ˢ) վળدΓͷ ։ൃνʔϜ SREνʔϜ αϙʔτνʔϜ ಉ͡νʔϜ಺ Ͱ࡞ۀΛґཔ ͠΍͘͢ SRE๊͕͑ͨ
 ࡞ۀΛ
 र͍΍͘͢

Slide 34

Slide 34 text

© 2018 Nulab Inc. ༗ྉܖ໿ (ະެ։) 2018೥11݄ݱࡏ • νʔϜΛ·͍ͨͩ৘ใڞ༗ͷ໾ׂΛSRE͕୲͏͜ͱͰɺMTTR͕վળͨ͠ • ΞϓϦվળʹؔΘΔλεΫΛ։ൃऀ͕औΔ͜ͱͰɺ࠶ͼӡ༻͕पΓ࢝Ίͯɺ
 ӡ༻ϛε͕ݮͬͨ • ϓϩμΫτͷվળʹSRE͕ඞཁͱͷೝ͕ࣝ޿͕ΓɺSREͷ࠾༻ΛڧԽ • 9݄ʹژ౎ࣄ຿ॴʹ1໊ೖࣾ • ܭଌͱվળͷ࢓૊Έ࡞Γ͕Ճ଎ɻ   SRE 4໊

Slide 35

Slide 35 text

© 2018 Nulab Inc. ࢒͍ͬͯΔ՝୊ • SRE͕ෳ਺νʔϜʹ·͕ͨͬͨঢ়ଶ͸ɺϝϦοτ͕͋Γɺଓ͚Δ΂͖͕ͩ… • SRE͕։ൃνʔϜʹଐ͢Δ͜ͱͰɺεϐʔυ͕མͪΔ໘΋ • ։ൃνʔϜϨϕϧͰ͸DevͱOpsΛ྆ํݟΔΑ͏ʹͳ͕ͬͨɺ
 ݸਓϨϕϧͰ͸DevͱOps͕෼͔Ε͓ͯΓɺOps୲౰ʹෛ୲  

Slide 36

Slide 36 text

© 2018 Nulab Inc. ࠓޙʹ޲͚ͨٞ࿦ • SREνʔϜΛϚτϦοΫεܕͷ··ʹ͢΂͖͔ʁ • ྫ͑͹ɺSREνʔϜΛݩʹ໭্ͨ͠ͰɺϓϩδΣΫτܕ͸Ͳ͏͔ʁ • ։ൃ޻਺͕ඞཁͳਂ͍໰୊ʢΞϓϦվળɺτΠϧͷࣗಈԽʣ͸
 SREͱ։ൃऀͷ྆ํΛؚΉϓϩδΣΫτͱͯ͠੾Γग़͢
 → ݸਓ΁ͷෛ୲Λආ͚ͯɺεϐʔυ΋্͛ΒΕΔʁ • ։ൃऀͱSREͷؒͷϩʔςʔγϣϯ͕ඞཁʁ • ݸਓϨϕϧͰDevͱOps͕෼͔Ε͍ͯΔ͜ͱ΁ͷରࡦͱͯ͠ • ϓϩμΫτͷن໛ʹରͯ͠ɺ·ͩSREͷਓ਺͕଍Γͯͳ͍ʁ • SRE࠾༻ܧଓதɻ౦ژࣄ຿ॴ΋ର৅ • ͍·͸෱Ԭ2໊ɺژ౎1໊ɺ౦ژ1໊  

Slide 37

Slide 37 text

© 2018 Nulab Inc. ·ͱΊ

Slide 38

Slide 38 text

© 2018 Nulab Inc. ·ͱΊ • ΤϯδχΞ૊৫ͷ੒௕ʹ൐ͬͯSREͷඞཁੑ͕૿ͨ͠ • ૊৫͕େ͖͘ͳΔͱɺશһ͕։ൃͱӡ༻Λ݉຿͢Δͷ͸೉͍͠ • SREͷۀ຿ൣғ͸์͓ͬͯ͘ͱ࠶ݱͳ͘޿͕͍ͬͯ͘ • ϓϩμΫτͷ੒௕ͷͨΊʹSRE͸ͳʹΛ͢΂͖͔ʁ • νʔϜ΍෼໺Λ·͙ͨΑ͏ͳ՝୊ͷղܾΛओಋ͢΂͖ • ۀ຿Λ๊͑͗͢ͳ͍ͨΊʹɺଞνʔϜͱͷີͳ࿈ܞ͕ඞཁ • ͦͷͨΊʹSREΛऔΓר͘νʔϜߏ੒ΛͲ͏͢΂͖͔Λ໛ࡧத