© 2018 Nulab Inc. #BDLMPHʹ͓͚Δ
 43&ͷࣄྫ ʙϓϩμΫτͷ੒௕ͷͨΊʹ43&͸ͳʹΛ͢΂͖͔ʙ גࣜձࣾψʔϥϘ
#BDLMPHνʔϜ
4JUF
3FMJBCJMJUZ
&OHJOFFS
ࢁ࡚
ؽ 

© 2018 Nulab Inc. ࣗݾ঺հ • ࢁ࡚ ؽ (@revsystem) • BacklogνʔϜͷSite Reliability Engineer (SRE) • 2017೥3݄ʹψʔϥϘʹೖࣾʢ౦ژࣄ຿ॴˠ2018೥4݄͔Β෱Ԭຊࣾʣ • Ҏલ͸ɺSIۈ຿ɻWebΞϓϦͷઃܭ΍ओʹΦϯϓϨ؀ڥͰͷΠϯϑϥઃܭ/ߏங/ӡ༻
 

© 2018 Nulab Inc. ͜ͷൃදʹ͍ͭͯ • ౦ژࣄ຿ॴͷ Backlog SRE ٢ᖒʢMuziʣ͕
 SRE Lounge #5 Ͱൃදͨ͠಺༰Λݩʹ͍ͯ͠·͢ • ʮSRE Lounge #5 ٢ᖒʯͰݕࡧ͢Δͱ
 ࢿྉͱΠϕϯτϨϙʔτ͕ग़͖ͯ·͢ • SRE Lounge ͸౦ژͰఆظతʹ։࠵͞Ε͍ͯΔ
 SRE ಉ࢜ͷަྲྀͷ৔Ͱ͢ʢ౦ژͷಉ྅ʹͥͻ͝঺հΛʂʣ
 https://sre-lounge.connpass.com/
 ࠷ۙͷ౦ژࣄ຿ॴ ʢϑΟϯΨʔϘʔυങ͍·ͨ͠ʣ 

© 2018 Nulab Inc. ࠓ೔࿩͍ͨ͜͠ͱ • ૊৫͕େ͖͘ͳΔͱɺશһ͕։ൃͱӡ༻Λ݉຿͢Δͷ͸೉͍͠ • SREͷۀ຿ൣғ͸ɺ์͓ͬͯ͘ͱࡍݶͳ͘޿͕͍ͬͯ͘ • ψʔϥϘͰ͸ͲͷΑ͏ͳ͜ͱ͕ى͖ͯɺͲ͏ରॲ͔ͨ͠ʁ
 

© 2018 Nulab Inc. BacklogͱSRE 

© 2018 Nulab Inc.
 

© 2018 Nulab Inc. ਌͠Έ΍͍͢ϢʔβʔΠϯλʔϑΣΠεͰ
 ϝχϡʔΛΫϦοΫ ಺༰Λॻ͍ͯ ௥ՃϘλϯͰ࡞੒    

© 2018 Nulab Inc. ϓϩδΣΫτ؅ཧͷத৺ʹ • ϓϩδΣΫτ؅ཧͷඞਢػೳΛඋ͑ɺଞαʔϏεͷ࿈ܞ΋
 λεΫ؅ཧ • λεΫͷ؅ཧ͕Ͱ͖Δ͔ʁ
• λεΫ͔ΒࣗಈతʹΨϯτνϟʔ τ͕࡞੒Ͱ͖Δ͔ʁ จॻ؅ཧ • จॻΛΦϯϥΠϯ্Ͱ࡞੒ͯ͠ڞ ༗Ͱ͖Δ͔ʁ ϑΝΠϧ؅ཧ • ϑΝΠϧΛڞ༗Ͱ͖Δ͔ʁ
• 8FC%"7Λ࢖ͬͯ1$͔Β௚઀ ϑΝΠϧʹΞΫηεͰ͖Δ͔ʁ ιʔείʔυ؅ཧ • (JUͰιʔείʔυ͕؅ཧͰ͖Δ ͔ʁ
• 4VCWFSTJPOͰιʔείʔυ͕؅ ཧͰ͖Δ͔ʁ ϞόΠϧରԠ • ެࣜͷϞόΠϧΞϓϦ͕ఏڙ͞Ε ͍ͯΔ͔ʁ ηΩϡϦςΟ • άϩʔόϧ*1ΞυϨεʹΑΔαʔ Ϗε΁ͷΞΫηε੍ݶ౳ͷઃఆ͕ Ͱ͖Δ͔ʁ
• ΞΫηεϩάͷఏڙαʔϏε͕͋ Δ͔ʁ ֦ுੑ • "1*͕ఏڙ͞Ε͍ͯΔ͔ʁ
• ৘ใͷߋ৽࣌ʹ֎෦γεςϜʹ௨ ஌͢Δ࢓૊Έ 8FCIPPLͳͲ ͕ ఏڙ͞Ε͍ͯΔ͔ʁ
• ֎෦γεςϜͳͲ͔Βϝʔϧܦ༝ ͰλεΫΛ࡞੒Ͱ͖Δ͔ʁ 

© 2018 Nulab Inc. τϥΠΞϧͷํ๏ • https://backlog.com/ja/ ͷʮແྉτϥΠΞϧʯΛΫϦοΫ
 

© 2018 Nulab Inc. ψʔϥϘ BacklogϓϩδΣΫτ ࠓճͷ࿩ͷഎܠɿBacklogͷ։ൃɾӡ༻ମ੍ • ࣗࣾͰ։ൃͨ͠αʔϏεΛࣗࣾͰӡ༻ • ίϯϙʔωϯτผʹෳ਺ͷ։ൃνʔϜ • ։ൃνʔϜͷ1ͭʹɺվળدΓͷ։ൃνʔϜ΋͋Δ
 … ։ൃνʔϜ ։ൃνʔϜ վળدΓͷ ։ൃνʔϜ ɹSREνʔϜ αϙʔτνʔϜ 

© 2018 Nulab Inc. ࠓճͷ࿩ͷഎܠɿγεςϜߏ੒ͷ֓ཁ • ࠃ֎ɾւ֎ʹෳ਺ͷ؀ڥʢαʔϏεΫϥελʣΛߏங • 2018೥࣌఺Ͱͷαʔό૯਺͸200୆ऑ • ৄ͘͠͸Backlog SREদӜʹΑΔաڈͷϓϨθϯΛࢀর • Backlogͷӡ༻؂ࢹ
 https://speakerdeck.com/nulabinc/operational-monitoring-of-backlog
 

© 2018 Nulab Inc. SRE͸ͳʹΛ͢Δਓ͔ʁ 

© 2018 Nulab Inc. SREͷఆٛ • جຊతʹ͸Opsͷ৬छ • େ͖͘෼͚ͯɺӡ༻ۀ຿ͱվળͷΤϯδχΞϦϯά • αʔϏε͝ͱʹ୲౰͸͋Δ΋ͷͷɺSREࣗମ͸શࣾ ԣஅతͳ૊৫ • SREͷۀ຿ʹؔ͢Δ໌֬ͳఆٛ͸ͳ͍
 https://www.oreilly.co.jp/books/9784873117911/ 

© 2018 Nulab Inc. SREຊͷୈII෦ʮݪଇʯ • ΤϥʔόδΣοτ • αʔϏεϨϕϧ໨ඪʢSLOʣ • τΠϧ • ϞχλϦϯά • ࣗಈԽ • ϦϦʔεΤϯδχΞϦϯά
 https://www.oreilly.co.jp/books/9784873117911/ 

© 2018 Nulab Inc. SRE͸ͳʹΛ͢Δਓ͔ʁ • ιϑτ΢ΣΞΤϯδχΞͱͯ͠ͷεΩϧΛ΋ͬͯɺϓϩμΫτͷ৴པੑΛҰఆҎ্ʹอͭ • ৴པੑΛอͭͨΊʹɺैདྷ͸ΦϖϨʔλ΍ΠϯϑϥΤϯδχΞͳͲ͕୲౰͍ͯͨ͠ • ۀ຿ʴαΛظ଴͞ΕΔ͜ͱ͕ଟ͍ • ΠϯϑϥͱΞϓϦ྆ํͷཧղ͕ඞཁ
 

© 2018 Nulab Inc. Πϯϑϥͷཧղ͕ඞཁͳλεΫ • Πϯϑϥઃܭɾߏங • Πϯϑϥ؂ࢹɺো֐ରԠɺ͓Αͼ؂ࢹγεςϜߏங • ηΩϡϦςΟΞοϓσʔτରԠʢAWSΠϯελϯεɺOSʣ • ͦͷଞͷηΩϡϦςΟରࡦશൠ • AWSͷίετ؅ཧ • Backlog SREࢁ࡚ʹΑΔϒϩάهࣄʮre:dashͰAWSͷίετΛ෼ੳͯ͠Έͨʯ • ISMSରԠɺ؂ࠪରԠͷख఻͍ • WebαΠτߏஙʢWordPressʣ • etc...
 

© 2018 Nulab Inc. ΞϓϦͷཧղ͕ඞཁͳλεΫ • γεςϜઃܭɾߏங • σʔλϕʔεɺϛυϧ΢ΣΞͳͲ • ΞϓϦ؂ࢹɺΤεΧϨʔγϣϯ͓Αͼ؂ࢹγεςϜߏங • ఆৗۀ຿ʢτΠϧʣ͓ΑͼͦͷࣗಈԽ • CI؀ڥͷߏஙɾӡ༻ʢCI؀ڥࣗମͷෆ۩߹ରԠؚΉʣ • CI͕ࣦഊͨ͠ͱ͖ͷݪҼ੾Γ෼͚ͷԆ௕Ͱɺςετͷमਖ਼ͳͲ΋ • ηΩϡϦςΟΞοϓσʔτରԠʢϛυϧ΢ΣΞͳͲʣ • ίʔυվળ • ΠϯσοΫε΍SQLͷվળɺΤϥʔϩάͷվળ • etc...
 

© 2018 Nulab Inc. ͭ·Γʁ • SREͷੑ্࣭ɺͦͷۀ຿ൣғ͸์͓ͬͯ͘ͱࡍݶͳ͘޿͕Δ • ͦΕ͕SREͷָ͍͠ͱ͜ΖͰ΋͋Δ͕……
 

© 2018 Nulab Inc. Backlogʹ͓͚ΔSREνʔϜͷมભ 

© 2018 Nulab Inc. Backlogͷྺ࢙͸ΘΓͱ௕͍ • Backlog SREͷྺ࢙͸ΘΓͱ࠷͔ۙΒ
 ϕʔλ൛ ϦϦʔε Backlogͷྺ࢙ 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 Backlog SREͷ ྺ࢙ 

© 2018 Nulab Inc. Ϣʔβʔ਺ͷ৳ͼ
 ͜ͷ࣌ظ 

© 2018 Nulab Inc. ಋೖاۀ਺ʢ༗ྉܖ໿਺ʣͷ৳ͼ
 ͜ͷ࣌ظ 

© 2018 Nulab Inc. SREͷϙδγϣϯ͕Ͱ͖ͨ2015ʙ2018೥ • αʔϏεن໛͕2ഒҎ্ʹ৳ͼͨ • τϥϑΟοΫ͕૿͑ͯɺγεςϜ΁ͷෛՙ΋্͕ͬͨ • SRE΋ɺͦͷଞͷΤϯδχΞ΋૿͑ͨ
 

© 2018 Nulab Inc. 2015೥Ҏલ • ΤϯδχΞશһ͕ӡ༻΋ݟ͍ͯͨ࣌୅ • DevOpsͷπʔϧʢJenkinsͳͲʣΛಋೖ࢝͠Ίͨͷ΋͜ͷࠒ • SREؚΊɺΠϯϑϥઐ೚ͷΤϯδχΞ͸ډͳ͔ͬͨ
 SRE 0໊ ༗ྉܖ໿ ʙ3000ࣾ 

© 2018 Nulab Inc. 2015೥ • Πϯϑϥઐ೚ͷΤϯδχΞ͕ॳೖࣾ • ଞͷΤϯδχΞ͔Βӡ༻ΛҾ͖ܧ͗ͳ͕ΒɺΠϯϑϥվળ • ͜ͷͱ͖͸SREͱ͍͏໊শͰ͸ͳ͔ͬͨ • νʔϜ໊ͱ͔ແ͔ͬͨʢΒ͍͠ʣ • Πϯϑϥઐ೚ऀΛ࠾༻͖͔͚ͨͬ͠͸ʁ • ͳΜͱͳ͘ʮͦΖͦΖཁΔΑͶʯͬͯͳͬͨʢΒ͍͠ʣ
 SRE 1໊ ༗ྉܖ໿ 3000ࣾʙ 

© 2018 Nulab Inc. 2016೥ • SRE͕2໊ʹ૿Ճ • ʮSREνʔϜʯͱ͍͏໊শ͕஀ੜ • ӡ༻͕͜ͳΕͯΞϓϦͷ໰୊͕ݟ͖͑ͯͨ • ӡ༻ͱฒߦͯ͠ɺΞϓϦվળ͕࢝·Δ • ΞϓϦέʔγϣϯαʔόΛεςʔτϨεʹ͢ΔͨΊͷվળΛ͍͔࣮ͭ͘ࢪ • σʔλϕʔεΛEC2্ͷMySQL 5.1͔ΒAWS Aurora΁Ҡߦ • ৄ͘͠͸ɺদӜͷϒϩάهࣄʮີண 24࣌ʂ MySQL 5.1 ͔Β Aurora ΁ͷҠߦ100 ೔ؒ ʙ Backlog ฤ
 SRE 2໊ ༗ྉܖ໿ 4000ࣾʙ 

© 2018 Nulab Inc. 2017೥ • SRE͕3໊ʹ૿Ճ • ͦΕ·ͰSREΛ݉೚͍ͯͨ͠ΤϯδχΞ͕ΞϓϦέʔγϣϯνʔϜʹ໭Γɺ
 3݄ʹࢁ࡚ɺ8݄ʹ౦ژࣄ຿ॴʹ1໊ʢMuziʣೖࣾ • αʔό୆਺͕૿͑ɺτΠϧͷ࡞ۀྔ΋૿Ճ • kernel updateࡇΓͱ͔ • ΞϓϦվળͷൣғ΋޿͕Δ • վળ͕ਐΈɺSRE 1໊Ͱ୲౰͖͠Εͳ͍೉қ౓ͷ՝୊͚͕ͩ࢒ͬͨ • DevͱOpsͷ෼཭͕ڧ͘ͳͬͨ • SREͷਓ਺͕૿͑ͨ݁ՌɺSRE΁ͷظ଴͕ڧ͘ͳΔ • ৽͍͠ΤϯδχΞ͕૿͑ͨ݁Ռɺӡ༻Λ஌Βͳ͍ΤϯδχΞ͕େ൒ʹ
 SRE 3໊ ༗ྉܖ໿ 5000ࣾʙ 

© 2018 Nulab Inc. 2018೥ॳ಄ • SREͷ৬຿ൣғ͕޿͕Γɺӡ༻ϛε͕໨ཱͪ࢝Ίͨ • खॱͷ࿙Ε΍ɺඞཁͳ࡞ۀ͕೺ѲͰ͖͍ͯͳ͔ͬͨͳͲ • DevͱOpsͷ෼཭͕ਐΈɺো֐ൃੜ͔Β෮چ͓ΑͼݪҼڀ໌·Ͱʹ͔͔Δ͕࣌ؒ
 ௕͘ͳͬͨ
 SRE 3໊ ༗ྉܖ໿ 6000ࣾʙ 

© 2018 Nulab Inc. DevͱOpsͷ෼཭͕ى͜͢໰୊ • ฏۉ෮چ࣌ؒʢMean Time To Repair; MTTRʣͷ૿Ճ • ΞϓϦέʔγϣϯͷΞϥʔτ΍ɺސ٬͔Βͷ໰͍߹Θ͕ͤൃੜ͢Δ • SREνʔϜ΍αϙʔτνʔϜ͕ݪҼௐࠪʹಈ͘ • ͠͹Β͘ௐ͔ࠪͯ͠ΒɺϦϦʔεʹΑΔෆ۩߹͕ݪҼͱΘ͔Δ • Կ͕໰୊͔ͩͬͨʁ • ϦϦʔεεέδϡʔϧ΍ϦϦʔε಺༰Λڞ༗Ͱ͖͍ͯͳ͔ͬͨ
 → ϦϦʔε࣌ࠁΛ஌Βͣɺো֐ൃੜ࣌ࠁͱؔ࿈෇͚Ͱ͖ͳ͍
 → ϦϦʔεͷ಺༰Λ஌Βͣɺෆ۩߹ͱͷؔ܎ΛਪଌͰ͖ͳ͍ • ։ൃνʔϜ͕ϦϦʔεޙʹΞϥʔτ΍ϩάΛ؂ࢹ͍ͯ͠ͳ͔ͬͨ
 → SRE͔ΒΤεΧϨʔγϣϯ͢Δ·ͰɺରԠ։࢝Ͱ͖ͳ͍
 

© 2018 Nulab Inc. ψʔϥϘ BacklogϓϩδΣΫτ Backlogͷ։ൃɾӡ༻ମ੍ʢ࠶ܝʣ • ࣗࣾͰ։ൃͨ͠αʔϏεΛࣗࣾͰӡ༻ • ίϯϙʔωϯτผʹෳ਺ͷ։ൃνʔϜ • ։ൃνʔϜͷ1ͭʹɺվળدΓͷ։ൃνʔϜ΋͋Δ
 … ։ൃνʔϜ ։ൃνʔϜ վળدΓͷ ։ൃνʔϜ ɹSREνʔϜ αϙʔτνʔϜ 

© 2018 Nulab Inc. BacklogϓϩδΣΫτͰߦͬͨରࡦ • SREΛϚτϦοΫεతʹ഑ஔ͢Δ૊৫ʹมߋ
 ψʔϥϘ BacklogϓϩδΣΫτ … ։ൃνʔϜ ։ൃνʔϜ (ϦϦʔεස౓ˢ) վળدΓͷ ։ൃνʔϜ SREνʔϜ αϙʔτνʔϜ 

© 2018 Nulab Inc. ։ൃνʔϜͰͷSREͷ໾໨ʢΞϓϦدΓʣ • ৘ใΛूΊͯɺରԠඞཁͳ໰୊Λൃݟ͠ɺͦͷղܾ·Ͱ௥੻ • ΞϓϦϩάͷվળɺ؂ࢹγεςϜͷվળͳͲ΋
 ψʔϥϘ BacklogϓϩδΣΫτ … ։ൃνʔϜ ։ൃνʔϜ (ϦϦʔεස౓ˢ) վળدΓͷ ։ൃνʔϜ SREνʔϜ αϙʔτνʔϜ ো֐ɾෆ۩߹ Λਝ଎ʹΤε ΧϨʔγϣϯ ։ൃɾ
 ϦϦʔεͷঢ়گ
 Λີʹ఻͑Δ ৘ใΛूΊΔ 

© 2018 Nulab Inc. ։ൃνʔϜͰͷSREͷ໾໨ʢΠϯϑϥدΓʣ • SRE͸ΠϯϑϥدΓͷۀ຿ʹूத • ΞϓϦվળ͕ඞཁͳ࡞ۀ͸ɺ։ൃνʔϜ͕र͏
 ψʔϥϘ BacklogϓϩδΣΫτ … ։ൃνʔϜ ։ൃνʔϜ (ϦϦʔεස౓ˢ) վળدΓͷ ։ൃνʔϜ SREνʔϜ αϙʔτνʔϜ ಉ͡νʔϜ಺ Ͱ࡞ۀΛґཔ ͠΍͘͢ SRE๊͕͑ͨ
 ࡞ۀΛ
 र͍΍͘͢ 

© 2018 Nulab Inc. ༗ྉܖ໿ (ະެ։) 2018೥11݄ݱࡏ • νʔϜΛ·͍ͨͩ৘ใڞ༗ͷ໾ׂΛSRE͕୲͏͜ͱͰɺMTTR͕վળͨ͠ • ΞϓϦվળʹؔΘΔλεΫΛ։ൃऀ͕औΔ͜ͱͰɺ࠶ͼӡ༻͕पΓ࢝Ίͯɺ
 ӡ༻ϛε͕ݮͬͨ • ϓϩμΫτͷվળʹSRE͕ඞཁͱͷೝ͕ࣝ޿͕ΓɺSREͷ࠾༻ΛڧԽ • 9݄ʹژ౎ࣄ຿ॴʹ1໊ೖࣾ • ܭଌͱվળͷ࢓૊Έ࡞Γ͕Ճ଎ɻ
 SRE 4໊ 

© 2018 Nulab Inc. ࢒͍ͬͯΔ՝୊ • SRE͕ෳ਺νʔϜʹ·͕ͨͬͨঢ়ଶ͸ɺϝϦοτ͕͋Γɺଓ͚Δ΂͖͕ͩ… • SRE͕։ൃνʔϜʹଐ͢Δ͜ͱͰɺεϐʔυ͕མͪΔ໘΋ • ։ൃνʔϜϨϕϧͰ͸DevͱOpsΛ྆ํݟΔΑ͏ʹͳ͕ͬͨɺ
 ݸਓϨϕϧͰ͸DevͱOps͕෼͔Ε͓ͯΓɺOps୲౰ʹෛ୲
 

© 2018 Nulab Inc. ࠓޙʹ޲͚ͨٞ࿦ • SREνʔϜΛϚτϦοΫεܕͷ··ʹ͢΂͖͔ʁ • ྫ͑͹ɺSREνʔϜΛݩʹ໭্ͨ͠ͰɺϓϩδΣΫτܕ͸Ͳ͏͔ʁ • ։ൃ޻਺͕ඞཁͳਂ͍໰୊ʢΞϓϦվળɺτΠϧͷࣗಈԽʣ͸
 SREͱ։ൃऀͷ྆ํΛؚΉϓϩδΣΫτͱͯ͠੾Γग़͢
 → ݸਓ΁ͷෛ୲Λආ͚ͯɺεϐʔυ΋্͛ΒΕΔʁ • ։ൃऀͱSREͷؒͷϩʔςʔγϣϯ͕ඞཁʁ • ݸਓϨϕϧͰDevͱOps͕෼͔Ε͍ͯΔ͜ͱ΁ͷରࡦͱͯ͠ • ϓϩμΫτͷن໛ʹରͯ͠ɺ·ͩSREͷਓ਺͕଍Γͯͳ͍ʁ • SRE࠾༻ܧଓதɻ౦ژࣄ຿ॴ΋ର৅ • ͍·͸෱Ԭ2໊ɺژ౎1໊ɺ౦ژ1໊
 

© 2018 Nulab Inc. ·ͱΊ 

© 2018 Nulab Inc. ·ͱΊ • ΤϯδχΞ૊৫ͷ੒௕ʹ൐ͬͯSREͷඞཁੑ͕૿ͨ͠ • ૊৫͕େ͖͘ͳΔͱɺશһ͕։ൃͱӡ༻Λ݉຿͢Δͷ͸೉͍͠ • SREͷۀ຿ൣғ͸์͓ͬͯ͘ͱ࠶ݱͳ͘޿͕͍ͬͯ͘ • ϓϩμΫτͷ੒௕ͷͨΊʹSRE͸ͳʹΛ͢΂͖͔ʁ • νʔϜ΍෼໺Λ·͙ͨΑ͏ͳ՝୊ͷղܾΛओಋ͢΂͖ • ۀ຿Λ๊͑͗͢ͳ͍ͨΊʹɺଞνʔϜͱͷີͳ࿈ܞ͕ඞཁ • ͦͷͨΊʹSREΛऔΓר͘νʔϜߏ੒ΛͲ͏͢΂͖͔Λ໛ࡧத