SREグループのマネージャーという立場になって真っ先にやったこと

2022年2月25日（金）
JAWS-UG SRE支部 #2

Transcript

1. 43&άϧʔϓͷϚωʔδϟʔͱ͍͏
   ཱ৔ʹͳͬͯਅͬઌʹ΍ͬͨ͜ͱ
   Isao Shimizu
   JAWS-UG SRE支部 #2
   2022.2.25
   

   View Slide

2. mixi, Inc.
   ࣗݾ঺հ
   ਗ਼ਫ ܄ @isaoshimizu
   ˔ 2011೥ʙ גࣜձࣾϛΫγΟ
   ˓ 2011೥ʙ2014೥ SNSʮmixiʯӡ༻ΤϯδχΞ
   ˓ 2014೥ʙ2018೥ ϞϯελʔετϥΠΫ SRE
   ˓ 2018೥ʙݱࡏ Ո଒ΞϧόϜ ΈͯͶ SRE Ϛωʔδϟʔ
   ˔ CloudNative Days Tokyo 2021 Ωʔϊʔτొஃ
   ˔ SRE NEXT 2020 IN Tokyo ొஃ
   ˔ AWS Summit Tokyo 2014ˍ2019 ొஃ
   ˔ AWS Dev Day Tokyo 2018 LTେձ ϕετεϐʔΧʔ
   ˔ Software Design ಛूهࣄدߘ ͳͲ
   

   View Slide

3. mixi, Inc.
   શੈք ສਓҎ্ͷར༻ऀ
   ※ (ג) ϛΫγΟௐ΂ɻ iOSɾAndroidΞϓϦొ࿥ऀ਺ɺϒϥ΢β൛ొ࿥ऀ਺ͷ߹ܭ
   Ո଒ΞϧόϜ ΈͯͶ
   ଟࠃޠରԠɺੈք͔ࠃҎ্ͰαʔϏεఏڙ
   ೔ຊޠɺӳޠɺதࠃޠʢൟମࣈʣɺؖࠃޠɺϑϥϯεޠ
   υΠπޠɺεϖΠϯޠʹରԠ
   ࠃ಺޲͚
   Ո଒ΞϧόϜ ΈͯͶ
   ւ֎޲͚
   FamilyAlbum
   

   View Slide

4. mixi, Inc.
   ຊ೔࿩͍ͨ͜͠ͱʢ෼ʣ
   ˔ ʮՈ଒ΞϧόϜ ΈͯͶʯͷ43&νʔϜͷੜཱ͍ͪͱ"84ͱͷؔΘΓ
   ˔ 43&νʔϜʹ͓͚Δ͍··ͰͷऔΓ૊Έࣄྫ
   ˔ 43&νʔϜͱͯ͠ͷ՝୊
   ˔ 43&άϧʔϓͷϛογϣϯͱόϦϡʔΛࡦఆ
   ˔ ·ͱΊ
   

   View Slide

5. ʮՈ଒ΞϧόϜ ΈͯͶʯͷ
   43&νʔϜͷੜཱ͍ͪͱ"84ͱͷؔΘΓ
   

   View Slide

6. mixi, Inc.
   ʮՈ଒ΞϧόϜ ΈͯͶʯͷ43&νʔϜͷੜཱ͍ͪ
   ˔ ೥݄ੜ·Ε
   ˔ αʔϏεϦϦʔεޙ ໿೥ܦաͨ͠λΠϛϯά
   ˔ ౰࣌ͷࣄۀ෦ͷ૊৫ن໛͸ʙਓ͘Β͍
   ˔ 43&νʔϜ͕஀ੜͨ͠౰ॳ͸ࢲؚΊ໊ͯମ੍
   ˔ Πϯϑϥͷ՝୊͕ࢁੵ͍ͯͨ͠
   ˓ ʮՈ଒ΞϧόϜ ΈͯͶʯͷΠϯϑϥ͸೥ࠒΑΓ"84Λར༻
   

   View Slide

7. mixi, Inc.
   43&νʔϜͱ"84ͱͷؔΘΓ
   ˔ &OUFSQSJTF4VQQPSU΁ͷՃೖʹΑΓαϙʔτ΋खް͘ͳͬͨ
   ˔ "84͕བྷΉࢪࡦʹ͍ͭͯ͸ɺ"84ͷ5".΍ιϦϡʔγϣϯΞʔΩςΫτͷํʑʹ
   େม͓ੈ࿩ʹͳ͍ͬͯΔ
   ˓ "848FMM"SDIJUFDUFE'SBNFXPSLϨϏϡʔ
   ˓ &,4Ҡߦʹ͋ͨͬͯͷ֤छ૬ஊɾϨϏϡʔ
   ˓ ֤छΞʔΩςΫνϟϨϏϡʔ
   ˓ ίετϨϏϡʔ
   ˓ ͦͷଞ͍Ζ͍Ζ
   

   View Slide

8. mixi, Inc.
   43&νʔϜʹ͓͚Δ͍··ͰͷऔΓ૊Έࣄྫ
   ˔ Πϯϑϥͷෛ࠴ղফ
   ˔ ίϯςφҠߦɺ&,4Ҡߦ
   ˔ $*$%ͷ੔උ
   ˔ ϩά෼ੳج൫ͷ੔උ
   ˔ *B$ͷ੔උɺਪਐ
   ˔ ։ൃ؀ڥͷ੔උ
   ˔ ϞχλϦϯά؀ڥͷ੔උ
   ˔ ίετ࡟ݮ
   ˔ Φϯίʔϧ
   

   View Slide

9. mixi, Inc.
   ΦʔτεέʔϧͱσϓϩΠͷ଎౓Ξοϓ
   EC2ϕʔεͰͷΦʔτεέʔϧɺ
   ChefʹΑΔσϓϩΠ଴ͪʹΑΓ஗͍
   Chef
   Auto
   Scale
   Auto
   Scale
   Container
   Registry
   ίϯϐϡʔτϦιʔεޮ཰Ξοϓɺ
   ίϯςφΠϝʔδͷ഑ஔʹΑΓ଎౓Ξοϓ
   EC2
   EC2
   Container
   ͞·͟·ͳෛՙ܏޲ʹૉૣ͘ରԠ͠΍͘͢ͳͬͨ
   ैདྷ ίϯςφԽ
   AWS OpsWorks
   

   View Slide

10. mixi, Inc.
    *B$Λ҆શʹˍѻ͍΍͘͢
    Provisioningπʔϧ
    Webhook
    Approve
    Plan݁Ռ
    Plan/Apply
    pull
    Terraform
    Configurations
    (.tf files)
    Provisioning
    Account
    Service
    Accounts
    Admin Role
    Terraform
    AWS CodeBuild
    terraform.git
    AWS Lambda
    Invoke
    .tf
    Run
    Provisioningπʔϧ
    GitHub
    

    View Slide

11. mixi, Inc.
    ϝτϦΫεऩूपΓͷ੔උ
    Kubernetes
    Pod
    Prometheus
    (Short Term)
    New Relic
    Pod
    Rails
    (Puma)
    Pod
    KEDA
    Pod New Relic
    Infrastructure
    Kubernetes
    integration
    newrelic_rpm
    Grafana
    Pod
    Exporter
    Internet
    Internet
    yebeda
    Amazon
    Managed
    Service for
    Prometheus
    (Long Term)
    remote
    write
    

    View Slide

12. 43&νʔϜͱͯ͠ͷ՝୊
    

    View Slide

13. mixi, Inc.
    43&νʔϜͱͯ͠ͷ՝୊ײ
    ˔ ೥݄͔Β43&νʔϜͷϦʔμʔʹɻ໨ඪઃఆɺνʔϜϝϯόʔͱͷPOͷ࣮ࢪʢຖिʣͳ
    ͲΛਐΊ͖ͯͨɻ
    ˔ ೥݄ 43&άϧʔϓͷϚωʔδϟʔʹɻϐʔϓϧϚωδϝϯτ͕ΑΓॏཁʹɻ
    ˔ ࠓ·Ͱɺ֤छ੔උ΍ίετ࡟ݮͳͲͰखҰഋͱͳΓ43&ͷϓϥΫςΟεͱͯ͠Ͱ͖ͯͳ͍͜ͱ͕ଟ
    ͔ͬͨɻͦ͏͍͏ϑΣʔζͩͬͨͷ͔΋͠Εͳ͍ɻ
    ˔ ྫ͑͹ɺϙετϞʔςϜ͸ͦΕͳΓʹͰ͖͍ͯΔ͕ɺ4-*4-0ͷࡦఆͱ͔͕͋·ΓͰ͖ͯͳ͍ͳͲɻ
    ˔ 43&ओಋͰ͋Ε͜Ε΍͖ͬͯͯ͠·ͬͨͱ͜Ζ͕͋Δɻ
    ˔ ࠓޙ͸։ൃνʔϜ͕ࣗ཯తʹ͞·͟·ͳ໰୊ΛղܾͰ͖ΔΑ͏ʹ43&νʔϜ͕αϙʔτΛ͍ͯ͠
    ͖͍ͨɻ
    

    View Slide

14. 43&άϧʔϓͷϛογϣϯͱόϦϡʔΛࡦఆ
    

    View Slide

15. mixi, Inc.
    43&άϧʔϓͷϛογϣϯͱόϦϡʔΛࡦఆ
    ˔ 43&ͷνʔϜ͕ԿͷͨΊʹଘࡏ͢Δͷ͔ɺ૊৫΍αʔϏεʹରͯ͠ͲΜͳՁ஋Λ
    ఏڙ͢Δ͍ͯ͠Δͷ͔ɺͲΜͳະདྷΛඳ͍͍ͯΔͷ͔ʹ͍ͭͯɺվΊͯߟ͑ͨɻ
    ˔ 43&ͷϝϯόʔ͓Αͼࣄۀ෦શମ͸΋ͪΖΜɺ૊৫֎ʹ΋ڞ༗Ͱ͖Δ΋ͷʹɻ
    ˔ ࠾༻ʹ͓͍ͯ૊৫ͷߟ͑ΛΘ͔Γ΍͘͢఻͑ɺΦϯϘʔσΟϯάʹ΋໾ཱ͍ͯͨɻ
    ˔ Ϗδϣϯͷઃఆ͸͍ͬͨΜݟૹΓɻͿͬͪΌ͚೉͍͠ɻ
    ˔ ଞࣾͷ43&૊৫ͷϛογϣϯ΍όϦϡʔΛଟ͘ࢀߟʹ͍͖ͤͯͨͩ͞·ͨ͠ɻ
    ˓ ϛογϣϯʹ͍ͭͯͷࣄྫ͸๛෋ɻόϦϡʔͷࣄྫ͸ͱͯ΋গͳ͍ɻ
    ˔ ϛογϣϯɾόϦϡʔ͸ඞཁʹԠͯ͡มԽ͍͖ͯ͠·͢ɻ
    

    View Slide

16. mixi, Inc.
    ʮՈ଒ΞϧόϜ ΈͯͶʯ43&άϧʔϓͷϛογϣϯɾόϦϡʔ
    ˔ ੈքதͷՈ଒͕շద͔ͭ҆৺ʹ࢖͑ΔαʔϏεΛఏڙ͢Δɻ
    ˔ ૊৫͕ࣗ཯తʹ໰୊ղܾͰ͖ΔΑ͏ʹϓϥοτϑΥʔϜΛఏڙ͢Δɻ
    ˔ ࣄۀͷ੒௕Λ્֐͠ͳ͍Α͏ʹΠϯϑϥίετΛ࠷దԽ͢Δɻ
    ˔ ΦʔϓϯͰ͋Δ͜ͱɻ
    ˔ ܭଌ͢Δɻ
    ˔ ࢹ໺Λ޿͛Δɻ
    ˔ มԽ͠ଓ͚Δɻ
    ˔ γϯϓϧʹɻ
    ϛογϣϯ
    όϦϡʔ ˞࣮ࡍʹ͸ͦΕͧΕʹ͍ͭͯͷৄࡉͳઆ໌͕͋Γ·͕͢ৄࡉ͸࣍ͷϖʔδͰɻ
    

    View Slide

17. mixi, Inc.
    ʢࢀߟʣʮՈ଒ΞϧόϜ ΈͯͶʯ43&άϧʔϓͷόϦϡʔͱৄࡉ
    ˔ ΦʔϓϯͰ͋Δ͜ͱɻ
    ˓ ίϛϡχέʔγϣϯɺ՝୊ɺ੒Ռɺࣦഊ͸Ͱ͖ΔݶΓΦʔϓϯʹ͠ɺഓΘΕͨϊ΢ϋ΢͸૊৫಺֎ʹ͓͍ͯΞ΢τϓοτ͠·͠ΐ
    ͏ɻࣦഊ͸ඇ೉ͤͣɺϙετϞʔςϜͱ޲͖߹͍ɺ૊৫શମʹ͓͚Δࠓޙͷྐͱ͠·͠ΐ͏ɻ
    ˔ ܭଌ͢Δɻ
    ˓ ܭଌΛߦ͍ɺ਺ࣈ΍ࣄ࣮ʹج͍ͮͯٞ࿦ɺධՁ͠ɺܧଓతʹվળ͠·͠ΐ͏ɻ
    ˔ ࢹ໺Λ޿͛Δɻ
    ˓ ͨͩͻͱͭͷ͜ͱʹݻࣥͤͣʹશମ૾Λݟͯ෺ࣄΛߟ͑·͠ΐ͏ɻੈքʹ໨Λ޲͚ɺৗʹੈքதͷϢʔβʔʹتΜͰ΋Β͑ΔΑ͏
    ͳࢹ఺Λ࣋ͪ·͠ΐ͏ɻ
    ˔ มԽ͠ଓ͚Δɻ
    ˓ ੈͷத͸ৗʹมԽΛଓ͚͍ͯ·͢ɻಉ࣌ʹ૊৫ɺݸਓɺγεςϜ΋มԽ͠ଓ͚ͳ͍ͱྑ͍αʔϏεΛఏڙ͠ଓ͚ΒΕͳ͍Ͱ
    ͠ΐ͏ɻͦͷ࣌ʑͷ՝୊Λ೺Ѳ͠ɺֶͼɺΞοϓσʔτ͍͖ͯ͠·͠ΐ͏ɻ
    ˔ γϯϓϧʹɻ
    ˓ γϯϓϧͳ΋ͷ͸ࣗ෼ࣗ਎΍ଞͷϝϯόʔʹͱͬͯཧղ͠΍͘͢ɺվળ͠΍͍͢Ͱ͠ΐ͏ɻෳࡶͳ΋ͷ͸ѻ͍ͨ͘ͳ͘ͳΓɺ
    ϝϯςφϯε͕͞Εͳ͘ͳΓɺ΍͕ͯෛ࠴ͱͳ͍͖ͬͯ·͢ɻ
    όϦϡʔ
    

    View Slide

18. mixi, Inc.
    ·ͱΊ
    ˔ SREͷ૊৫͕ͳͥඞཁͳͷ͔ɺͲ͏͍͏Ձ஋؍Λ࣋ͬͯ࢓ࣄΛ͍ͯ͠Δͷ͔ɺϛογ
    ϣϯͱόϦϡʔͱ͍͏ܗͰ૊৫શମʹ͔ͬ͠Γͱڞ༗͓ͯ͘͜͠ͱͰɺνʔϜ͕Կʹ
    ରͯ͠੹೚Λ͍࣋ͬͯΔͷ͔ͷ૬ޓཧղ͕ਐΉɻ
    ˔ SRE૊৫͸։ൃ૊৫ʹൺ΂ͯͦΕ΄Ͳεέʔϧ͢Δ΋ͷͰ͸ͳ͍ͷͰɺ͋Ε΋͜Ε΋
    ͱखΛग़͗͢͠Δͱຊདྷ΍Δ΂͖͜ͱΛݟࣦ͏Մೳੑ͕͋ΔɻΉ͠Ζ։ൃ૊৫͔Βͷ
    ґཔʹԠ͑Δ͚ͩͷ૊৫ʹͳΓಘΔɻ
    ˔ 2021.11.19 SRE Lounge #13 LTͰൃදͨ͠ʮ։ൃऀͱSREͷ໾ׂɺ੹೚ʯ
    https://speakerdeck.com/isaoshimizu/sre-lounge-13-lt Ͱ΋͍ۙ͜ͱΛॻ͍͍ͯ·͢ɻ
    

    View Slide

19. mixi, Inc.
    ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ
    

    View Slide