月間数千億リクエストをさばく技術 (ArchitectureNight公開用)

https://architecture-night.connpass.com/event/90666/

Transcript

1. ݄ؒ਺ઍԯϦΫΤετΛ͞͹ٕ͘ज़
   גࣜձࣾαΠόʔΤʔδΣϯτΞυςΫຊ෦%ZOBMZTU

   ࠇ࡚༏ଠ!LVSP@N
   

   View Slide

2. ࣗݾ঺հ
   w ࠇ࡚༏ଠ !LVSP@N
   
   w גࣜձࣾαΠόʔΤʔδΣϯτ

   ΞυςΫຊ෦%ZOBMZTUόοΫΤϯυΤϯδχΞ
   w ೥৽ଔೖࣾ
   w ೖࣾҎདྷ%ZOBMZTUͷೖࡳ ഑৴ ܭଌ ूܭ෦෼ͷ։ൃ ΠϯϑϥΛ୲౰
   w ˞ࡱӨ࿥Ի౤ߘېࢭͷϖʔδʹ͸ࠨ্ʹˣͷΞΠίϯΛೖΕͯ͋Γ·͢
   

   View Slide

3. %ZOBMZTUͱ͸
   

   View Slide

4. Πϯλʔωοτ޿ࠂͱ͸
   

   View Slide

5. αΠόʔΤʔδΣϯτʹ͓͚ΔΞυςΫ
   

   View Slide

6. 35#ͷ͘͠Έ
   w 35#3FBM5JNF#JEEJOH
   441
   %41T
   

   View Slide

7. 35#ͷ͘͠Έ
   w 8FCϖʔδ͕ϩʔυ͞Εɺ޿ࠂλά͕ൃՐ
   441
   %41T
   

   View Slide

8. 35#ͷ͘͠Έ
   w ࿮৘ใ΍Ϣʔβ৘ใͱͱ΋ʹCJESFRVFTU͕͘Δ
   441
   %41T
   CJESFRVFTU
   

   View Slide

9. 35#ͷ͘͠Έ
   w ೖࡳֹͱදࣔ͢Δ޿ࠂΛܾఆ
   441
   %41T
   "%
   "%
   "%
   CJESFRVFTU
   ԁ
   ԁ
   ԁ
   

   View Slide

10. 35#ͷ͘͠Έ
    w 0QFO35#ͱ͍͏ϓϩτίϧͰೖࡳ
    441
    %41T
    "%
    "%
    "%
    ԁ
    ԁ
    ԁ
    

    View Slide

11. 35#ͷ͘͠Έ
    w ೖࡳֹ͕Ұ൪ߴ͍ࣄۀऀ͕উར͢Δ
    441
    %41T
    "%
    "%
    "%
    ԁ
    ԁ
    ԁ
    XJO
    

    View Slide

12. 35#ͷ͘͠Έ
    w 441ʹ޿ࠂΫϦΤΠςΟϒ͕ૹΒΕΔ
    441
    %41T
    "%
    "%
    "%
    ԁ
    ԁ
    ԁ
    "%
    XJO
    

    View Slide

13. 35#ͷ͘͠Έ
    w 441ܦ༝Ͱམࡳͨ͠ࣄۀऀͷ޿ࠂ͕ల։͞ΕΔ
    441
    %41T
    "%
    "%
    "%
    ԁ
    ԁ
    ԁ
    "%
    XJO
    "%
    

    View Slide

14. 35#ͷ͘͠Έ
    w ͜ΕΛߴ଎ʹ܁Γฦ͢
    441
    %41T
    "%
    "%
    "%
    ԁ
    ԁ
    ԁ
    "%
    XJO
    "%
    औҾ͸NTҎ಺ʹ׬ྃ͠ͳ͚Ε͹ͳΒͳ͍
    

    View Slide

15. %ZOBMZTUͱ͸
    w %ZOBNJD3FUBSHFUJOHGPS(BNFT
    w εϚϗήʔϜ޲͚޿ࠂ഑৴ϓϥοτϑΥʔϜ
    w τοϓηʔϧε!೔ຊͷεϚϗήʔϜͷதͰ΋ߴ͍γΣΞ
    w ೔ຊɺΞϝϦΧΛؚΉΧࠃʹ഑৴த
    w Ϣʔβ͝ͱʹ࠷దԽͨ͠޿ࠂΛ഑৴
    IUUQXXXEZOBMZTUJP
    

    View Slide

16. ޿ࠂΫϦΤΠςΟϒྫ ࡱӨ࿥Ի౤ߘېࢭ
    
    

    View Slide

17. ݄ؒ਺ઍԯϦΫΤετΛ͞͹ٕ͘ज़
    

    View Slide

18. w "MM"84
    w ओཁར༻ίϯϙʔωϯτ
    %ZOBMZTUͷΞʔΩςΫνϟ
    Amazon
    EC2
    Amazon ECR
    Amazon ECS
    AWS
    Lambda
    Elastic Load
    Balancing
    Amazon
    S3
    Amazon
    RDS
    Amazon
    DynamoDB
    Amazon
    ElastiCache
    Amazon
    Redshift
    Amazon
    CloudFront
    Amazon
    Route 53
    AWS Direct
    Connect
    Amazon
    VPC
    Amazon
    CloudWatch
    AWS
    OpsWorks
    AWS Trusted
    Advisor
    IAM AWS Certificate
    Manager
    Amazon
    Athena
    Amazon
    EMR
    Amazon
    Kinesis
    AWS Data
    Pipeline
    Amazon
    SQS
    Amazon
    SNS
    Amazon
    SES
    Amazon
    WorkMail
    

    View Slide

19. %ZOBMZTUͷΞʔΩςΫνϟ
    w "MM"84
    w BQOPSUIFBTU VTFBTUʹల։
    w ౰ॳΑΓάϩʔόϧͳϓϩμΫτΛ໨ࢦ͍ͯͨ͠ͷͰ"84Λબ୒
    w ໿ݸͷΞϓϦέʔγϣϯίϯϙʔωϯτͰߏ੒
    

    View Slide

20. %ZOBMZTUͷΞʔΩςΫνϟ ུ֓
    
    BQOPSUIFBTU VTFBTU
    ೖࡳ഑৴ܥ
    ೖࡳ഑৴ܥ
    ܭଌܥ
    DBDIFDMVTUFS
    ूܭܥ
    DSPTTSFHJPOSFQMJDBUJPO
    DSPTTSFHJPOSFQMJDBUJPO
    DSPTTSFHJPOSFQMJDBUJPO
    ؅ཧը໘ DBDIFDMVTUFS
    όονܥ όονܥ
    

    View Slide

21. τϥϑΟοΫ֓گ
    ೔ຊͷೖࡳϦΫΤετඵ ΞϝϦΧͷೖࡳϦΫΤετඵ
    ೔ຊΞϝϦΧͷϨεϙϯελΠϜ NTFD
    
    

    View Slide

22. ͪͳΈʹ ࡱӨ࿥Ի౤ߘېࢭ
    
    ͋Δ೔ͷτϥϑΟοΫ
    

    View Slide

23. ͪͳΈʹ
    

    View Slide

24. ΠϯλʔωοττϥϑΟοΫ΋͍͍ͩͨ͜Μͳײ͡
    IUUQXXXNGFFEDPKQTFSWJDFUSB⒏DIUNM
    

    View Slide

25. τϥϑΟοΫ֓گ
    w ͱ͋Δϲ݄ؒͷCJEϦΫΤετ਺ͷ߹ܭ
    

    View Slide

26. w DYMBSHF $(
    Y໿୆ ϐʔΫ 64 +1
    
    w DΠϯελϯε˕
    େྔͷτϥϑΟοΫΛड͚ΔͨΊʹ
    DBDIF
    3%# TMBWF
    
    DBDIF
    3%# TMBWF
    
    BQOPSUIFBTUB
    BQOPSUIFBTUD
    ,74
    3%# NBTUFS
    
    BQOPSUIFBTU
    

    View Slide

27. 4DBMBͱ͍͏બ୒
    w %ZOBMZTUͰ͸΄΅શͯͷΞϓϦέʔγϣϯʹ4DBMBΛ࠾༻
    w ΠϯλϏϡʔهࣄ͋Γ·͢
    http://gihyo.jp/dev/serial/01/cyberagent/0063
    

    View Slide

28. 4DBMBͱ͍͏બ୒
    w 4DBMBΛબ୒ͨ͠ཧ༝
    w νϟϨϯδ ࠓͱͳͬͯ͸ී௨ʁ
    
    w Մಡੑɺϝϯςφϯεੑ͕ߴ͍
    w ϋΠύϑΥʔϚϯε
    w +BWBͷࢿ࢈͕࢖͑Δ
    w ΞυςΫελδΦ͸4DBMBͷ஌ݟͷմ
    X
    

    View Slide

29. 4DBMBʹΑΔฒྻॲཧ
    w ฒྻॲཧ͕ॻ͖΍͍͢
    w 'VUVSFͱ͍͏֓೦
    

    View Slide

30. 4DBMBʹΑΔฒྻॲཧ
    w ฒྻॲཧ͕ॻ͖΍͍͢
    w 'VUVSFͱ͍͏֓೦
    w ྫ3%#͔Βෳ਺ͷςʔϒϧΛҾ͖ͭͭɺ%ZOBNP%#΋Ҿ͘
    ಉ࣌ʹ໰͍߹Θͤͯ
    ฒྻʹϨεϙϯεΛड͚Δ
    NT
    NT
    NT
    ௚ྻॲཧNT

    ฒྻॲཧNT
    

    View Slide

31. ϩά͸શͯ,JOFTJTʹྲྀ͢
    w ೖࡳ΍ܭଌͷϩά͸શͯ,JOFTJT4USFBNTʹྲྀ͢
    w 8"1αʔό͸ϨεϙϯεΛฦ͢͜ͱ͚ͩʹूத
    w ֦ுੑ΍଱ো֐ੑͷ޲্
    %#VQEBUFDPOTVNFS
    MPHBSDIJWFDPOTVNFS
    VQEBUF
    BSDIJWF
    ,JOFTJT4USFBNT
    DPOTVNF
    QSPEVDFMPHT
    BQQTFSWFS
    %ZOBNP%#
    4
    1SPEVDFS 4USFBN $POTVNFS
    

    View Slide

32. 1SPEVDFS
    w ΞϓϦέʔγϣϯαʔό͕ు͍֤ͨछϩάΛ,JOFTJTʹ౤͛Δ
    ,JOFTJT4USFBNT
    qVFOUQMVHJOLJOFTJT
    BQQTFSWFS
    1SPEVDFS 4USFBN
    

    View Slide

33. ,JOFTJT1SPEVDFS-JCSBSZ
    w ,1-ͰτϥϑΟοΫઅ໿
    w ,JOFTJTͷ੍໿γϟʔυ͋ͨΓ SFDPSE165TFDPS.#T
    w ਺,#ͷϩάΛେྔʹసૹ͢Δʹ͸΋͍ͬͨͳ͍
    w ,1-Ͱ,JOFTJTϨίʔυʹෳ਺ϝοηʔδΛ٧ΊࠐΉ QSPUPDPMCV⒎FST
    
    https://speakerdeck.com/kanny/logging-architecture-at-cookpad
    

    View Slide

34. $POTVNFS
    w ,JOFTJT͔ΒϨίʔυΛऔΓग़͢
    w DPOTVNFSΛ૿΍ͤ͹εέʔϧΞ΢τՄೳ
    ,JOFTJT4USFBN
    BQQTFSWFS
    $POTVNFS
    4USFBN
    %ZOBNP%#
    

    View Slide

35. ,JOFTJT$POTVNFS-JCSBSZ
    w ,1-Ͱू໿ͨ͠ϨίʔυΛల։ͯ͘͠ΕΔ
    w XPSLFSຖͷ୲౰γϟʔυͷௐఀػೳ %ZOBNP%#Λར༻
    
    ,JOFTJT4IBSET BQQTFSWFS
    $POTVNFS
    4USFBN
    %ZOBNP%#
    

    View Slide

36. ,JOFTJT$POTVNFS-JCSBSZ
    w ,1-Ͱू໿ͨ͠ϨίʔυΛల։ͯ͘͠ΕΔ
    w XPSLFSຖͷ୲౰γϟʔυͷௐఀػೳ %ZOBNP%#Λར༻
    
    ,JOFTJT4IBSET BQQTFSWFS
    $POTVNFS
    4USFBN
    %ZOBNP%#
    ো֐ൃੜ
    

    View Slide

37. ,JOFTJT$POTVNFS-JCSBSZ
    w ,1-Ͱू໿ͨ͠ϨίʔυΛల։ͯ͘͠ΕΔ
    w XPSLFSຖͷ୲౰γϟʔυͷௐఀػೳ %ZOBNP%#Λར༻
    
    ,JOFTJT4IBSET BQQTFSWFS
    $POTVNFS
    4USFBN
    %ZOBNP%#
    ࣗಈ࠶ׂΓ౰ͯ
    

    View Slide

38. Ϣʔβσʔλͷѻ͍
    w ޿ࠂ༻୺຤*% *%'" "EWFSUJTJOH*%
    ͱϢʔβσʔλͷ૊Έ߹Θͤ
    w ྫ࠷ޙʹϩάΠϯͨ͠೔࣌ˠະϩάΠϯظ͕ؒΘ͔Δ
    w ޿ࠂೖࡳ࣌ʹେྔʹSFBE͕͞ΕΔͷͰਫฏεέʔϧͯ͠ཉ͍͠
    w Ϩίʔυ਺਺ඦԯϨίʔυ
    w αϯϓϧςʔϒϧ ௒؆ུԽ
    
    %&7*$&@*% "%7&35*4&3@*% %&7*$&@5:1& -"45@-0(*/@"5 015065@"5 ʜʜ
    %&7*$&@ "OESPJE ʜʜ
    %&7*$&@ "OESPJE ʜʜ
    %&7*$&@ J04 ʜʜ
    %&7*$&@ "OESPJE ʜʜ
    

    View Slide

39. Ϣʔβσʔλͷѻ͍
    w ,74ͱͯ͠%ZOBNP%#Λ࠾༻
    w )BTI,FZͱ3BOHF,FZΛ૊Έ߹ΘͤΔ
    w )BTI,FZͰҾ͚͹3BOHF,FZͰιʔτ͞Εͨෳ਺Ϩίʔυ͕ฦΔ
    %&7*$&@*% "%7&35*4&3@*% %&7*$&@5:1& -"45@-0(*/@"5 015065@"5 ʜʜ
    %&7*$&@ "OESPJE ʜʜ
    %&7*$&@ "OESPJE ʜʜ
    %&7*$&@ J04 ʜʜ
    %&7*$&@ "OESPJE ʜʜ
    )BTI,FZ 3BOHF,FZ
    

    View Slide

40. ϗοτύʔςΟγϣϯ໰୊
    w ಛఆͷύʔςΟγϣϯʹΞΫηε͕ภΓɺύϑΥʔϚϯε͕ग़ͳ͍
    

    View Slide

41. ϗοτύʔςΟγϣϯ໰୊
    w ,74ͷ໋॓
    w ಛఆͷύʔςΟγϣϯʹ3FBE8SJUF͕ภΔͱੑೳ͕ग़ͳ͍
    w ϗοτσʔλͱίʔϧυσʔλΛࠞͥͳ͍
    w ৽͍͠σʔλΞΫηε͞Ε΍͍͢ ݹ͍σʔλΞΫηε͞Εʹ͍͘
    w ରࡦ
    w σʔλ͕ొ࿥͞Εͨ࣌ظ͝ͱʹςʔϒϧΛ෼͚Δ
    w ҰఆظؒΞΫηε͞Εͳ͍σʔλ͸࡟আ͢Δ
    OPEF OPEF OPEF
    ৽͍͠σʔλ
    ݹ͍σʔλ
    ଟ͍ গͳ͍ ී௨
    ΞΫηεྔ
    

    View Slide

42. %ZOBNP%#ͷΩϟογϡ
    w Ωϟογϡͷಋೖ
    w ϨΠςϯγͷ࡟ݮ
    w %ZOBNP%#3FBE$BQBDJUZͷ࡟ݮ
    w ϗοτύʔςΟγϣϯ໰୊ͷ؇࿨
    w &MBTUJ$BDIF NFNDBDIFE
    Λ࠾༻
    w γϯϓϧɾߴ଎
    w ϚϧνεϨουͰಈ͘
    w SܥΠϯελϯε͸ωοτϫʔΫଳҬ͕(CQT·ͰόʔετՄೳ
    w IUUQTEFWDMBTTNFUIPEKQDMPVEBXTFDSOFUXPSLJOH
    %ZOBNP%#
    NFNDBDIFE
    BQQ
    
    

    View Slide

43. %ZOBNP%#ͷΩϟογϡ
    w Ωϟογϡظ͕ؒ௕͍ͱϢʔβͷঢ়ଶมԽ͕͙͢ʹ൓ө͞Εͳ͍
    w Ϣʔβͷঢ়ଶΛར༻ͨ͠޿ࠂ͕΢ϦͳͷͰଈ࣌൓ө͍ͤͨ͞
    w %ZOBNP%#4USFBNT-BNCEB'VODUJPOͰΩϟογϡͷ࡟আ
    w %ZOBNP%#"DDFMFSBUPS͸ΫΤϦΩϟογϡ͸ࣗಈߋ৽͞Εͳ͍
    w ݱঢ়͸ࣗલͰ࣮૷͢Δ͔͠ͳ͍
    %ZOBNP%#
    61%"5&
    %ZOBNP%#4USFBNT
    %&-&5&
    

    View Slide

44. w ϦΫΤετΛड͚͚ͨͩͰ͸μϝɺूܭ͕ඞཁ
    w ϩάͷྔ5#EBZ
    w "QBDIF4QBSLPO"NB[PO&.3
    w ΫϥελίϯϐϡʔςΟϯάϑϨʔϜϫʔΫ
    w ෼ࢄڞ༗ϝϞϦϞσϧ
    w ϊʔυΛ૿΍͢͜ͱͰεέʔϧ͢Δ
    w 4QBSL.-JCͱ͍͏ػցֶशϥΠϒϥϦ΋
    &.3$MVTUFS
    .BTUFS
    4MBWF
    4MBWF
    4MBWF
    4MBWF
    ϩάूܭ
    

    View Slide

45. ϩάूܭ
    w "QBDIF4QBSLPO&.3
    w ෳࡶͳूܭॲཧ͕ॻ͚Δ
    w ྫ୯७ͳ࣌ؒY޿ࠂओͷ

    ΫϦοΫ਺ूܭ
    42-Ͱॻ͍ͨ৔߹
    4QBSLͰॻ͍ͨ৔߹
    

    View Slide

46. ϋΠύϑΥʔϚϯεͰ৴པੑͷඞཁͳ

    γεςϜΛӡ༻͢ΔͨΊʹ
    

    View Slide

47. ͳͥ৴པੑ͕ٻΊΒΕΔͷ͔
    w ҰൠతͳαʔϏε ϝσΟΞ ήʔϜ
    
    w αʔϏεఏڙ͕Ͱ͖ͳ͍ˠϢʔβͷ཭୤ ऩӹ௿Լ
    w ޿ࠂ഑৴αʔϏε
    w ޿ࠂ͕഑৴Ͱ͖ͳ͍ˠػձଛࣦ ޿ࠂओͷ཭୤ ऩӹ௿Լ
    w ޿ࠂ͕഑৴͞Εͳͯ͘΋Ϣʔβʹରͯ͠͸Өڹ͕ͳ͍
    %ZOBMZTU Ϣʔβ
    ޿ࠂओ
    ൃ஫ ഑৴
    

    View Slide

48. ͳͥ৴པੑ͕ٻΊΒΕΔͷ͔
    w αʔϏε͕੒௕͠ച্ͷن໛͕͕͖͋ͬͯͨˠച্ͷॏཁ౓͸มΘΒͳ͍
    w ച্ສԁ೔Ͱ࣌ؒ഑৴ఀࢭສԁͷଛࣦ
    w ച্ ສԁ೔Ͱ࣌ؒ഑৴ఀࢭ ສԁͷଛࣦ
    w γεςϜن໛΋େ͖͘ͳΔͱ
    w ো֐ͷӨڹ΋େ͖͘ͳ͖ͬͯͨ
    w ҰํίετΧοτͷҖྗ΋େ͖͘ͳ͖ͬͯͨ
    

    View Slide

49. ߴ͍৴པੑ͸ઃܭ͕શͯ
    w ୯Ұো֐఺Λ࡞Βͳ͍
    w Ϋϥ΢υΛ࢖ͬͯΔͳΒΫϥ΢υωΠςΟϒͳϕετϓϥΫςΟεʹدͤΔ
    w ΞϓϦ͸Ͱ͖Δ͚ͩεςʔτϨεʹ
    w εςʔτΛ࣋ͭ෦෼ΛϚωʔδυαʔϏεʹدͤΔ
    w ιϦϡʔγϣϯΞʔΩςΫτʹ૬ஊ͢Δ
    w ͱ͸͍͑ෆଌͷࣄଶ͸ى͜Δʜ
    w ෮چ؇࿨ͷͨΊͷखஈ͸༻ҙ͢Δ΂͖
    

    View Slide

50. ؂ࢹ
    w %ZOBMZTUͰ͸؂ࢹʹ%BUBEPHΛར༻
    w l%ZOBMZTU%BUBEPHzͰݕࡧ
    

    View Slide

51. ؂ࢹ
    

    View Slide

52. ؂ࢹ
    w ޿ࠂओ͝ͱͷ഑৴ϝτϦΫε
    

    View Slide

53. ϦΫΤετ͕٧·ͬͯ͠·ͬͨ࣌
    w Ұ࣌తʹϨεϙϯε͕٧·Δͱʜ
    w αʔόʹϦΫΤετ͕ཷ·Δ
    w ಉ࣌ॲཧϦΫΤετ਺͕രൃ͢Δ
    w εϨουϓʔϧΛա৒ʹ࢖͏
    w Ϩεϙϯε଎౓ͷେ෯ѱԽ
    w ॲཧΛ࢝Ί͔ͯΒҰఆ࣌ؒܦաͨ͠ॲཧΛதஅͯ͠ೖࡳ͠ͳ͍
    w ೖࡳ͠ͳ͍)551 /P$POUFOU
    Λฦ͢
    w 4DBMBͳΒͦ͏͍͏ॲཧ΋ॻ͖΍͍͢ʂ
    

    View Slide

54. ύϑΥʔϚϯεௐࠪ
    w :PVS,JU ঎༻+BWBϓϩϑΝΠϥ
    ΛѪ༻
    

    View Slide

55. ࠓޙ
    

    View Slide

56. ࠓޙ΍͍͖͍ͬͯͨ͜ͱ
    w 4DIFEVMFE4DBMJOH"VUP4DBMJOH
    w "LLB)551
    w "LLB4USFBNΛར༻ͨ͠ॲཧ #BDL1SFTTVSFͱ͔
    
    w Ωϟογϡ %#૚ͷ"VUP4DBMJOH
    w ϏϧυϦϦʔεͷߴ଎Խ
    w &&ςετͷಋೖ
    w ͳͲͳͲʜ
    

    View Slide

57. ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide