Upgrade to Pro — share decks privately, control downloads, hide ads and more …

freee のエンジニアは障害から何を学び、どう改善しているのか? / What do freee engineers learn and improve from failures?

freee のエンジニアは障害から何を学び、どう改善しているのか? / What do freee engineers learn and improve from failures?

SRE NEXT 2020 の登壇資料です。 https://sre-next.dev/schedule/#a3

Manabu Sakai

January 25, 2020
Tweet

More Decks by Manabu Sakai

Other Decks in Technology

Transcript

  1. freee のエンジニアは障害から
    何を学び、
    どう改善しているのか?
    43&/&95
    ࡔҪֶ!NBOBCVTBLBJ

    View full-size slide

  2. w GSFFFגࣜձࣾ43&ΤϯδχΞ
    w ೥͔ΒϓϨΠϯάϚωʔδϟʔ
    w ෳۀͰελʔτΞοϓͷٕज़ΞυόΠβʔ
    w 5XJUUFS (JU)VC
    w !NBOBCVTBLBJ
    w ϒϩά
    w NBOBCVTBLBJEFW
    ࡔҪֶ
    .BOBCV4BLBJ

    View full-size slide

  3. εϞʔϧϏδωεΛɺ
    ੈքͷओ໾ʹɻ
    .*44*0/
    ੜ࢈೥ྸਓޱ͕ܶతʹݮগ͠ɺຫੑతͳਓखෆ଍ͱͳΔ೔ຊ
    Ͱ࿑ಇੜ࢈ੑ޲্͸ۓٸͷ՝୊ͱͳ͍ͬͯ·͢ɻ
    GSFFF͸ʮਓ޻஌ೳʯͱʮ౷߹جװۀ຿γεςϜʯΛΫϥ΢υ
    ٕज़Λ׆༻͠ɺۀ຿ޮ཰ԽͷαϙʔτΛଓ͚Δ͜ͱͰɺதݎத
    খاۀͷόοΫΦϑΟεۀ຿ޮ཰ԽΛ໨ࢦ͍ͯ͠·͢ɻ

    View full-size slide

  4. 6
    ૑ۀ͔Β*10·Ͱɺதখاۀ׆ੑԽͷͨΊͷαʔϏεΛҰؾ௨؏Ͱఏڙ
    ձࣾ֓ཁ
    ❂ೲ੫͢Δ
    ⾭ҭͯΔ
    ↻ӡӦ͢Δ
    ✩͸͡ΊΔ
    ձࣾઃཱGSFFF
    ։ۀGSFFF
    Ϋϥ΢υձܭιϑτGSFFF
    ਓࣄ࿑຿GSFFF
    ϚΠφϯόʔ؅ཧGSFFFؚΉ

    Ϋϥ΢υਃࠂGSFFF
    161ԯ603ສԁ (ࢿຊ४උۚ౳ؚΉ)
    ैۀһ਺
    ࣄۀ಺༰
    Ϋϥ΢υܕόοΫΦϑΟεαʔϏεͷ։ൃɾൢച
    ࢿຊۚ
    ઃཱ೥݄೔
    2012೥7݄
    506໊ʢ2019೥6݄຤࣌఺ʣ
    2019೥ʮಇ͖͕͍ͷ͋Δձࣾʯ
    ϥϯΩϯά4Ґ

    View full-size slide

  5. ར༻ࣄۀॴ਺ྦྷܭ

    2014.3 2015.3 2016.3 2017.3 2018.3
    800,000
    600,000
    300,000
    65,000
    1,000,000
    1,000,000

    View full-size slide

  6. 8
    GSFFFͷϓϩμΫτͷಛ௃
    ো֐ʹͲ͏ཱͪ޲͔͖͔ͬͯͨʁ
    ো֐ͷֶͼΛͲ͏޿Ί͖͔ͯͨʁ
    ·ͱΊ
    ΞδΣϯμ

    View full-size slide

  7. 9
    ʮো֐ରԠʹ՝୊Λײ͍ͯ͡Δਓ͕ɺվળͷͨΊͷୈҰาΛ౿Έग़ͦ͏ͱ
    ࢥ͑ΔΑ͏ʹͳΔ͜ͱʯ
    GSFFFͰͷऔΓ૊ΈΛ੺དʑʹ͓࿩͠͠·͕͢·ͩ·ͩ՝୊͸͋Γ·͢ɻ
    ͥͻօ͞ΜͷऔΓ૊Έ΋ڭ͍͑ͯͩ͘͞
    ࠓ೔ͷΰʔϧ

    View full-size slide

  8. GSFFFͷϓϩμΫτͷಛ௃
    01
    Section

    View full-size slide

  9. 11
    w GSFFFͷϓϩμΫτ͸l͓ۚz΍lਓzʹؔΘΔ৘ใ͕ଟ͍
    w ձܭGSFFFݸਓࣄۀ΍๏ਓͷࡒ຿৘ใ
    w ਓࣄ࿑຿GSFFFैۀһͷݸਓ৘ใ΍څ༩৘ใ
    w ϚΠφϯόʔ؅ཧGSFFFैۀһͷϚΠφϯόʔ৘ใ
    GSFFFͷϓϩμΫτ͕ѻ͏৘ใ

    View full-size slide

  10. 12
    w ձܭGSFFF͸ిࢠܾࡁ౳୅ߦۀʹ౰ͨΔ
    w ۜߦ๏౳ʹجͮ͘ొ࿥͕ඞཁͰۚ༥ிʹొ࿥ࡁΈ
    w ࢠձࣾ GSFFFGJOBODFMBC
    ͸ۚ༥αʔϏεࣄۀ΋ߦ͍ͬͯΔ
    w ্৔ʹΑͬͯϓϥΠϕʔτΧϯύχʔ͔ΒύϒϦοΫΧϯύχʔ΁
    ‎Ҏલʹ΋૿ͯ͠ɺো֐ʹରͯ͠ΑΓγϏΞͳରԠ͕ٻΊΒΕΔɻ
    ΑΓࣾձΠϯϑϥʹۙͮ͘

    View full-size slide

  11. ো֐͸θϩʹͰ͖ΔͩΖ͏͔ʁ

    View full-size slide

  12. 14
    w GSFFFͷΑ͏ͳ4BB4Ϗδωεʹͱͬͯ͸ϓϩμΫτͷਐԽ͸໋॓
    w ৽͍͠Ձ஋ΛੜΈग़͢தͰࣦഊʢ㲈ো֐ʣ͸ආ͚ΒΕͳ͍
    w όάͷͳ͍ϓϩάϥϜ͸ଘࡏ͠ͳ͍
    w ਓ͕ӡ༻͢ΔݶΓώϡʔϚϯΤϥʔ͸ى͜Γ͏Δ
    w FUD
    ‎ো֐Λड͚ೖΕͳ͕Β΋҆ఆͨ͠ϓϩμΫτͱ͍͏૬൓͢Δ΋ͷΛ໨ࢦ
    ͢ɻ
    ো֐Λθϩʹ͢Δ͜ͱ͸೉͍͠

    View full-size slide

  13. ো֐ʹͲ͏ཱͪ޲͔͖͔ͬͯͨʁ
    02
    Section

    View full-size slide

  14. 16
    w ΤϯδχΞਓ͘Β͍
    w ໌֬ͳো֐ରԠϑϩʔ͸ͳ͔ͬͨ
    w ϙετϞʔςϜͷΑ͏ͳυΩϡϝϯτ͸΄΅࢒͍ͬͯͳ͍
    w ൿ఻ͷίϚϯυͱ͍ͬͨ҉໧஌
    w ࠜຊతͳରࡦ͸ޙճ͠ʢػೳ։ൃͷํ͕༏ઌ౓͕ߴ͔ͬͨʣ
    w ελʔτΞοϓ͋Δ͋Δͳײ͡
    w αʔϏεͷن໛͕খ͍͞͏ͪ͸͋·Γ໰୊ʹͳΒͳ͍
    ࣗ෼͕ೖࣾͨ͠ࠒʢ೥ʣ

    View full-size slide

  15. 17
    w ϓϩμΫτͷ੒௕ʹ߹Θͤͯ։ൃ૊৫͕֦େ
    w গͣͭ͠ϙετϞʔςϜΛॻ͘श׳͕޿͕Δ
    w ͱ͸͍͑ɺϑΥʔϚοτ΍৘ใͷཻ౓͸όϥόϥ
    w ৘ใ΋ࢄҳ (PPHMF%PDT 2JJUB5FBN $POGMVFODF

    w Өڹ͕খ͍͞ͱ໘౗Ͱॻ͔Εͳ͍͜ͱ΋Α͋ͬͨ͘ʜ
    ‎͜ͷࠒ·Ͱ͸ো֐ରࡦͷֶͼ͕ݸਓʹͱͲ·Γଐਓతʹͳ͍ͬͯͨɻ
    ։ൃ૊৫ͷ֦େʢ೥ʣ

    View full-size slide

  16. 18
    w 40$औಘʹ޲͚ͨ४උͷҰ؀Ͱো֐ରԠϑϩʔ͕ࡦఆ
    w ো֐ͷӨڹ౓߹͍ʹԠͯ͡ɺϨϕϧʙ·Ͱͷো֐ఆٛ
    w FHʮ͘͝Ұ෦ͷࣄۀॴʹ୅ସखஈͰճආෆՄೳͳো֐͕ൃੜʯ
    w ਺ࣈ͕େ͖͍ํ͕Өڹൣғ͕େ͖͍ɻϨϕϧ͸ۓٸটू
    w ো֐൑அͷϒϨ͕ͳ͘ͳͬͨ
    w ϙετϞʔςϜͷϑΥʔϚοτ͕Ͱ͖ͨ͜ͱͰ඼࣭΋վળ
    40$औಘʹ޲͚ͨ४උʢ೥ʣ

    View full-size slide

  17. ͦͯ͠ʹ
    ࣄ݅͸ى͜Δ

    View full-size slide

  18. 21
    w ͱ͋ΔΦϖϛε͕ݪҼͰো֐͕ى͖ɺσʔλෆ੔߹ͷՄೳੑ͕ൃ֮
    w ͓٬༷ͷσʔλอޢΛ࠷༏ઌ͢ΔͨΊʹ΍ΉΛಘͣશαʔϏεఀࢭ
    w SFGIUUQTXXXGSFFFDPKQCMPH@JOGPSNBUJPOIUNM
    w ݄຤ʹ໿࣌ؒ൒ʹ౉Γ͢΂ͯͷػೳ͕࢖͑ͳ͍ঢ়ଶ
    w GSFFFͷϓϩμΫτ͸݄຤݄ॳʹ࠷΋ར༻͞ΕΔ
    ‎͜ͷো֐͔Β͞·͟·ͳֶͼΛಘͯɺશࣾҰؙͱͳͬͯվળ͍ͯ͘͠
    ݄຤ʹશαʔϏεఀࢭʢ೥ʣ

    View full-size slide

  19. ͦͷҰ෦Λ͝঺հ

    View full-size slide

  20. 23
    w 40$औಘʹ޲͚ͯ੔උͨ͠ো֐ରԠϑϩʔΛϒϥογϡΞοϓ
    w ো֐ൃੜ࣌ͷରԠʹ͍ͭͯͷl͢΂ͯzΛ·ͱΊͨυΩϡϝϯτΛ༻ҙ
    w ॳಈରԠ
    w ো֐ରԠʹ͓͚Δ໾ׂ
    w ୂ௕ɺه࿥܎ɺ؅੍ࣨɺҙࢥܾఆऀͷॱং
    w ࢦشΛऔΔਓ΍ҙࢥܾఆऀ͸खΛಈ͔͞ͳ͍
    w ࣾ಺֎ίϛϡχέʔγϣϯ
    w ͞·͟·ͳར֐ؔ܎ऀʹεϜʔζʹ৘ใΛ఻ୡ
    ‎୭͕ԿΛ͢΂͖͔໌֬ʹͨ͜͠ͱͰো֐ରԠ͕εϜʔζʹɻ
    ରԠͷ໌֬Խ

    View full-size slide

  21. 24
    w ো֐ൃੜ࣌ʹ෺ཧతʹू·ΕΔΤϦΞʮϒϦοδʯΛઃஔ
    w େܕσΟεϓϨΠ΍ϏσΦձٞγεςϜͳͲΛৗઃ
    w ো֐࣌͸ϒϦοδ͕ΤϯδχΞͷ࡞ઓج஍ʹͳΔ
    w ৘ใΛҰ͔ॴʹू໿͠ɺ෮چ·Ͱ଩औΓΛ͢Δ
    w ผʑͷ৔ॴͰ࡞ۀ͍ͯ͠ΔͱೝࣝͷζϨ΍͓ݟ߹͍͕ى͖΍͍͢
    ো֐ରԠΤϦΞͷઃஔ

    View full-size slide

  22. 25
    w ো֐࣌͸͍ͭ΋ͷߦಈ͕औΕͳ͍ͷͰɺॳಈΛͱʹ͔͘লྗԽ͢Δ
    w 4MBDLCPU΍8PSLGMPXͷ׆༻
    w ϙετϞʔςϜͷࣗಈ࡞੒
    w ("4Λ࢖ͬͨࣾ಺ࠂ஌ͷࣗಈԽ
    ॳಈͷলྗԽ

    View full-size slide

  23. ো֐ͷֶͼΛͲ͏޿Ί͖͔ͯͨʁ
    03
    Section

    View full-size slide

  24. 27
    w ʮࣦഊͯ͠߈ΊΑ͏ʯ
    w ֶͼͷ͋ΔࣦഊΛͯ͠࠷଎Ͱ੒௕͠Α͏
    w ֶͼͷ͋Δࣦഊͱ͸νʔϜ΍ϓϩμΫτͷ੒௕ʹͭͳ͕Δࣦഊ
    w ֶͼͷ͋ΔࣦഊΛ͍ͯ͠ͳ͍ͷ͸੒௕΁ͷ௅ઓ͕଍Γͳ͍
    ‎૑ۀ౰͔࣌Βࣦഊ͔ΒֶͿͱ͍͏จԽ͕ࠜ෇͍͍ͯͨɻ
    GSFFFͷ։ൃจԽ

    View full-size slide

  25. 28
    w ো֐ͷֶͼΛ։ൃ૊৫શମʹڞ༗͢Δ৔
    w ى͖ͨࣄ৅΍Өڹɺରࡦ಺༰ɺ͔ͦ͜ΒಘֶͨͼΛݐઃతʹڞ༗͢Δ
    w ൜ਓ୳͠΍੹೚௥ٴͷ৔Ͱ͸ͳ͍
    w ϙετϞʔςϜΛ௚઀ؔΘΓͷͳ͍ਓʹ΋஌ͬͯ΋Β͏
    w νʔϜͰͷো֐ৼΓฦΓ͸ผ్ߦ͍ͬͯΔ
    w ো֐͕ͳ͔݄ͬͨ͸ʮण࢘KTʯʹͳΔ
    ࣦഊKT

    View full-size slide

  26. 29
    w খ͞ͳ໰୊͕େ͖͘ͳΔલʹվળ͢Δ༗ࢤͷ׆ಈ
    w ܰඍͳ໰୊΍ώϠϦϋοτΛ์ஔ͢Δͱେ͖ͳ໰୊ʹൃల͢Δ
    w ि࣍Ͱ΋͘΋͘ձΛ։͖όοΫϩάΛফԽ͍ͯ͘͠
    w ͔݄Ͱݸͷ໰୊͕ڍ͕Γɺ͏ͪʹݸ͕ղফࡁΈ
    w ͸ͯͳ͞ΜͷऔΓ૊ΈΛࢀߟʹͨ͠
    w SFGׂΕ૭ཧ࿦Λ8FCΠϯϑϥͷվળʹ׆༻͠ɺνʔϜ಺ͷ஌ࣝڞ༗Λଅਐ͠
    ͍ͯΔ࿩
    ׂΕ૭Λվળ͠ୂ

    View full-size slide

  27. 30
    w ௚ۙिؒͷBMFSUΛৼΓฦΓɺϊ΢ϋ΢Λڞ༗͢Δ׆ಈ
    w ϊΠδʔͳ௨஌͕૿͑ͨΓɺো֐ରԠͷଐਓԽΛ๷͙ͷ͕໨త
    w ࠓ೥͔Β࢝Ίͨͱ͜Ζ
    w ଐਓతͳ΋ͷ͕໌֬ʹͳͬͨΓɺ৽ͨͳׂΕ૭͕ݟ͔ͭͬͨΓ͢Δ
    BMFSUΛৼΓฦΓୂ

    View full-size slide

  28. ·ͱΊ
    04
    Section

    View full-size slide

  29. 32
    w ࣗ෼ͨͪͷ૊৫΍ϓϩμΫτʹ͋ͬͨো֐ରԠϑϩʔΛ࡞Δ
    w ଞࣾͷਅࣅ͚ͩͩͱ͏·͍͔͘ͳ͍͔΋
    w େ͖ͳো֐͸ֶͼͷๅݿ
    w ͳ͍ʹӽͨ͜͠ͱ͸ͳ͍Ͱ͕͢
    w ো֐͔Βͷֶͼ͕ଐਓతʹͳΒͳ͍Α͏ʹ޻෉͢Δ
    w ࣦഊKT
    w ׂΕ૭Λվળ͠ୂ
    w BMFSUৼΓฦΓୂ
    ·ͱΊ

    View full-size slide

  30. @manabusakai

    View full-size slide

  31. ͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View full-size slide