全世界で成長し続けるサービスのインフラと開発の裏側 / MIXI TECH CONFERENCE 2023

MIXI TECH CONFERENCE 2023
2023年3月3日（金）
#mixitechcon

Transcript

1. ©MIXI
   .*9*5&$)$0/'&3&/$&
   શੈքͰ੒௕͠ଓ͚ΔαʔϏεͷΠϯϑϥͱ։ൃͷཪଆ
   7BOUBHFελδΦ ΈͯͶࣄۀ෦ 43&άϧʔϓ
   ਗ਼ਫ ܄
   

   View Slide

2. ©MIXI
   "CPVU.F
   ਗ਼ਫ ܄ !JTBPTIJNJ[V
   ● ೥ʙ גࣜձࣾϛΫγΟʢݱ.*9*ʣ
   ● ೥ʙ 4/4ʮNJYJʯӡ༻ΤϯδχΞ
   ● ೥ʙ ϞϯελʔετϥΠΫ 43&
   ● ೥ʙ Ո଒ΞϧόϜ ΈͯͶ 43&
   ● ೥ʙ 43&άϧʔϓ Ϛωʔδϟʔ
   ि຤͸ࣾձਓਧ૗ָஂͰͷ׆ಈʢָஂ௕ɺτϩϯϘʔϯ໿೥ɺͨ·ʹࢦشऀʣɻ
   ΩϟϯϓͱΫϥϑτϏʔϧ͕޷͖ɻ
   2
   

   View Slide

3. ©MIXI
   Ո଒ΞϧόϜ ΈͯͶ
   3
   

   View Slide

4. ©MIXI
   Ո଒ΞϧόϜ ΈͯͶ
   4
   ύύɾϚϚ͕ࡱͬͨࢠͲ΋ͷࣸਅ΍ಈըΛɺ૆෕฼΍
   ਌ੰͳͲট଴ͨ͠Ո଒͚ͩʹ؆୯ʹڞ༗Ͱ͖Δࣸਅɾ
   ಈըڞ༗ΞϓϦ
   ϑΥτϒοΫ
   ࣸਅϓϦϯτ
   ঎඼ͷྫ
   

   View Slide

5. ©MIXI
   Ո଒ΞϧόϜ ΈͯͶ
   5
   ΈͯͶΈ·΋Γ(14 ΈͯͶग़ுࡱӨ
   ΈͯͶ೥լঢ় ΈͯͶίʔϧυΫλʔ
   

   View Slide

6. ©MIXI
   Ո଒ΞϧόϜ ΈͯͶ
   ● ೥݄ϦϦʔε
   ● ݱࡏ͸ݴޠɾͷࠃͱ஍ҬͰα
   ʔϏεΛఏڙ
   ● ւ֎Ͱ͸ʮ'BNJMZ"MCVNʯͱ͍͏
   ໊শͰల։த
   ● ೥݄೔ʹར༻ऀ਺͕
   ສਓ˞
   Λಥഁ
   ● ೔ຊࠃ಺Ͱ͸ϚϚ΍ύύͷ൒਺ͱͳ
   Δˋͷํ˞
   ͕͝ར༻
   6
   ˞J04ɾ"OESPJEä ΞϓϦొ࿥ऀ਺ɺϒϥ΢β൛ొ࿥ऀ਺ͷ߹ܭ
   ˞ʮΈͯͶʯొ࿥࣌ʹೖྗ͞Ε͓ͨࢠ͞·ͷ஀ੜ೔ͱްੜ࿑ಇলൃදʮਓޱಈଶ౷ܭʯ͔Βࢉग़ɻ೥݄࣌఺Ͱˋ
   

   View Slide

7. ©MIXI
   γεςϜߏ੒
   7
   

   View Slide

8. ©MIXI
   ΈͯͶΛࢧ͑ΔΫϥ΢υ
   8
   AWS
   EKS / EC2 / S3 / Aurora
   ElastiCache / SQS / SNS など
   Google Cloud
   BigQuery など
   େ൒͕"84Ͱߏ੒͞Ε͍ͯΔ
   

   View Slide

9. ©MIXI
   "84ͷߏ੒ʢུ֓ਤʣ
   9
   Amazon EKS
   Amazon EC2
   Amazon S3 Amazon Aurora
   Amazon
   ElastiCache
   Amazon
   CloudFront
   Application
   Load Balancer
   Amazon
   DynamoDB
   

   View Slide

10. ©MIXI
    γεςϜߏ੒
    ● "84ΞΧ΢ϯτ͸؀ڥ͝ͱʹ׬શʹ෼ׂ
    ● "84ͷϦιʔε͸5FSSBGPSNʹΑͬͯ؅ཧ
    ● "-#ɺ4ͷલ໘ʹ$MPVE'SPOUΛஔ͖ɺωοτϫʔΫܦ࿏ͷ࠷దԽͱΩϟογϡར༻
    ● &,4Ϋϥελʔ͸൒೥ʹҰ౓Ξοϓσʔτ͢Δӡ༻ʢ৽چΫϥελʔೖΕସ͑ʣ
    ● ,VCFSOFUFTͷϦιʔε͸"SHP$%ͱ)FMN$IBSUʹΑͬͯ؅ཧ
    ● &$͸ׂҎ্͕εϙοτΠϯελϯεʢΦϯσϚϯυ͸͘͝Θ͔ͣʣ
    ● &$্Ͱ͸3BJMT"QQMJDBUJPOɺ4JEFLJR8PSLFS͕ಈ࡞
    ● "84-PBE#BMBODFS$POUSPMMFSͱ5BSHFU(SPVQ#JOEJOHʹΑͬͯ"-#͔Β1PE·Ͱ
    ͷϧʔςΟϯάΛίϯτϩʔϧ
    ● "VSPSB͸(MPCBM%BUBCBTFΛར༻ʢৄࡉ͸ޙड़ʣ
    10
    

    View Slide

11. ©MIXI
    ϚϧνϦʔδϣϯߏ੒
    11
    

    View Slide

12. ©MIXI
    ϚϧνϦʔδϣϯߏ੒
    12
    "84
    BQOPSUIFBTU
    "84
    VTFBTU
    Ϣʔβʔ͔Β͍ۙϦʔδϣϯʹΞΫηε͢Δ͜ͱͰ
    ϨεϙϯελΠϜΛ୹͘͢Δ͜ͱ͕Ͱ͖Δ
    

    View Slide

13. ©MIXI
    ϚϧνϦʔδϣϯߏ੒ʢσʔλϕʔεʣ
    13
    ap-northeast-1 us-east-1
    Amazon EKS
    Amazon EC2
    Amazon S3
    Amazon Aurora
    Global Database
    Amazon Aurora
    Global Database
    Amazon EKS
    Amazon EC2
    ௿ϨΠςϯγʢฏۉ ໿NTʣͰಉظ͞ΕΔ
    ಡΈऔΓΞΫηεͷΈରԠ
    ॻ͖ࠐΈɾಡΈऔΓΞΫηεରԠ
    

    View Slide

14. ©MIXI
    ϚϧνϦʔδϣϯߏ੒ʹ͓͚Δ஫ҙ఺
    ● ΈͯͶͰ͸Ϣʔβʔମݧͷ޲্Λ໨తͱͨ͠ϚϧνϦʔδϣϯ
    ○ ͍ΘΏΔ#$1ରࡦͰ͸ͳ͍
    ● ֤ϦʔδϣϯͰಉ༷ͷ؂ࢹΛ͓͜ͳ͏ඞཁ͕͋Δ
    ○ *B$ʹΑͬͯखؒ͸͋·Γ͔͔Βͳ͍
    ● Ϧʔδϣϯ΍";ʹΑͬͯఏڙ͞Εͳ͍αʔϏεɺΠϯελϯελ
    Πϓ͕͋Δ
    ● &$ͷΩϟύγςΟঢ়گ͸Ϧʔδϣϯɺ";ʹΑͬͯେ͖͘มΘΔ
    ● ϦʔδϣϯؒͰωοτϫʔΫো֐͕͋Δͱσʔλϕʔεͷಉظʹ
    େ͖ͳ஗Ԇ͕ى͖Δ
    14
    

    View Slide

15. ©MIXI
    αʔόʔΞϓϦέʔγϣϯͷ
    ։ൃ؀ڥ
    15
    

    View Slide

16. ©MIXI
    αʔόʔΞϓϦέʔγϣϯͷ։ൃ؀ڥ
    16
    Πϯλʔωοτ
    ։ൃऀ͝ͱʹ༻ҙ
    ͞Εͨ؀ڥ
    開発者
    44)
    ͱͯ΋γϯϓϧʹॻ͘ͱ͜Μͳײ͡
    ͜͜Ͱ։ൃ
    

    View Slide

17. ©MIXI
    αʔόʔΞϓϦέʔγϣϯͷ։ൃ؀ڥ
    17
    Πϯλʔωοτ
    ,VCFSOFUFT
    "͞Μͷ
    ։ൃ؀ڥ1PE
    #͞Μͷ
    ։ൃ؀ڥ1PE
    $͞Μͷ
    ։ൃ؀ڥ1PE
    &͞Μͷ
    ։ൃ؀ڥ1PE
    %͞Μͷ
    ։ൃ؀ڥ1PE
    Aさん Bさん Cさん Dさん Eさん
    開発チーム
    

    View Slide

18. ©MIXI
    ։ൃ؀ڥͷ࡞Γํ
    18
    εςοϓ̍ εςοϓ εςοϓ
    :".-ϑΝΠϧΛ
    (JU)VCʹϓογϡ
    ˞:".-ͷத਎͸44)
    ͷެ։伴
    "SHP$%Ͱ4ZODૢ࡞Λ
    ͯ͠։ൃ؀ڥ͕ىಈ
    ˞਺ेඵޙʹ؀ڥ͕࡞੒
    ͞ΕΔ
    44)͢Δ
    ։ൃऀࣗ਎Ͱ࡞੒ɾ࡟আΛࣗ༝ʹߦ͏͜ͱ͕Մೳ
    

    View Slide

19. ©MIXI
    खݩͷ.BD1$
    ։ൃ؀ڥ΁ͷϩάΠϯपΓͷηΩϡϦςΟ
    19
    ,VCFSOFUFTΫϥελ
    ։ൃ؀ڥ 1PE
    ίϯςφ
    sshd
    kubectl exec netcat
    22番ポート
    Πϯλʔωοτ
    ίϯςφ
    ConfigMap
    SSH公開鍵
    ίϯςφ
    ίϯςφ
    ProxyCommand
    .ssh/config
    ssh
    44)ͷϙʔτΛΠϯλʔωοτʹެ։͢Δඞཁ͸ͳ͍
    ։ൃऀ
    ίϯςφ
    "84ೝূ
    

    View Slide

20. ©MIXI
    ։ൃ؀ڥ͕৽͍͠ߏ੒ʹͳͬͯ
    ● Ҏલ͸ݸਓ͝ͱͷ&$ʹ44)ͯ͠%PDLFSΛར༻͍ͯͨ͠
    ○ 44)ͷ伴؅ཧɺ5FSSBGPSNͷద༻ͱߏஙͷεςοϓ͕ଟ͔ͬͨ
    ■ :".-ͷ௥Ճɺ"SHP$%ͷ؅ཧը໘͔Βͷద༻ͱ؆୯ʹ
    ○ खॱ͕Θ͔ΓͮΒ͘43&ϝϯόʔͷαϙʔτ͕ඞཁʹͳͬͨΓ
    ■ ୭Ͱ΋؆୯ʹߏஙɾ࡟আͰ͖ΔΑ͏ʹ
    ○ ൪ϙʔτͷΠϯλʔωοτެ։ʢެ։伴ೝূͱ͸͍͑ةͳ͍ʣ
    ■ ϙʔτެ։ͳ͠Ͱɺ"84ͷೝূʢ440ʣͱެ։伴ೝূͷ૊Έ߹Θͤʹ
    ■ 74$PEF3FNPUF%FWFMPQNFOUʹ΋ରԠ
    ○ ϩά͕ཷ·Γ͗ͨ͢Γɺݕࡧͮ͠Β͔ͬͨΓ
    ■ (SBGBOB-PLJͰݕࡧ͠΍͘͢
    20
    

    View Slide

21. ©MIXI
    ΦϒβʔόϏϦςΟͷ޻෉
    21
    

    View Slide

22. ©MIXI
    ΦϒβʔόϏϦςΟ͸ͳͥඞཁʁ
    ● γεςϜͰԿ͕ى͖͍ͯΔͷ͔Λ೺Ѳ͢ΔͨΊ
    ● ,VCFSOFUFTͳͲίϯςφͷΦʔέετϨʔγϣϯ͕ར༻͞ΕΔ
    ͜ͱ͕૿͑ͨ݁ՌɺγεςϜ಺෦͕ෳࡶʹͳΓɺ֤ίϯϙʔωϯ
    τΛ؍ଌՄೳͳঢ়ଶʹ͢Δඞཁ͕͋Δ
    ● ϝτϦΫεɺϩάɺτϨʔεͳͲͷ৘ใ͸ଟ͋͘Γɺ໰୊ͷݪҼ
    Λௐࠪ͢ΔͨΊʹ͸ͦΕΒΛత֬ʹूܭ͠ɺՄࢹԽ͢Δ࢓૊Έ͕
    ٻΊΒΕΔ
    22
    

    View Slide

23. ©MIXI
    ΦϒβʔόϏϦςΟͷ࣮ݱ
    23
    ϞόΠϧΞϓϦ
    New Relic Mobile
    Firebase Crashlytics
    αʔόʔΞϓϦ
    New Relic APM
    New Relic Browser
    ֤छϩά
    Grafana Loki
    Amazon Athena
    Google BigQuery
    ֤छϝτϦΫε
    Prometheus / Amazon Managed
    Service for Prometheus
    Amazon CloudWatch
    Grafana
    

    View Slide

24. ©MIXI
    ΦϒβʔόϏϦςΟͷશମ૾
    24
    Kubernetes
    Prometheus
    New Relic
    Rails
    newrelic_rpm
    Grafana
    Node Exporter
    Amazon CloudWatch
    Amazon
    Managed
    Service for
    Prometheus
    Grafana Loki
    Promtail
    Ϣʔβʔͷ
    ୺຤
    New
    Relic
    Mobile
    ϞχλϦϯά؀ڥ
    Fluent Bit Amazon Kinesis
    Firehose
    Amazon S3
    Amazon
    Athena
    GCS BigQuery
    ETLॲཧ
    σʔλͷྲྀΕ
    

    View Slide

25. ©MIXI
    43&Λ࣮ફ͢Δ্Ͱ΋
    ΦϒβʔόϏϦςΟ͸ͱͯ΋େࣄ
    25
    

    View Slide

26. ©MIXI
    ·ͱΊ
    26
    

    View Slide

27. ©MIXI
    ·ͱΊ
    ● શੈքͰ ສਓҎ্ͷϢʔβʔʹར༻͞ΕΔαʔϏεͷγε
    ςϜߏ੒͕ࢀߟʹͳΕ͹޾͍Ͱ͢ɻ
    ● "VSPSB(MPCBM%BUBCBTFͷ͓͔͛ͰϚϧνϦʔδϣϯߏ੒Λ࡞
    Δ͜ͱ͕Ͱ͖ɺੈքதͷϢʔβʔମݧΛվળͰ͖·ͨ͠ɻ
    ● ,VCFSOFUFTΫϥελ্ͷ։ൃ؀ڥ͸ηΩϡΞͰѻ͍΍͍͢؀ڥ
    ʹͳΓ·ͨ͠ɻ
    ● ΦϒβʔόϏϦςΟ͸ࠓ·ͰࢼߦࡨޡΛ܁Γฦ͠ɺ43&͚ͩͰͳ
    ͘։ൃऀશମͰར༻Ͱ͖Δ؀ڥΛ࡞Δ͜ͱ͕Ͱ͖·ͨ͠ɻ
    27
    

    View Slide

28. ©MIXI
    8&"3&)*3*/(
    28
    ΈͯͶ ࠾༻ ݕࡧ
    ͞·͟·ͳϙδγϣϯͰืूதͰ͢ʂ
    

    View Slide