AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

Transcript

1. "84ωΠςΟϒͳ&$$3.γεςϜ ӡ༻ʹ͔ܽͤͳ͍ϩάج൫ߏங 
   େ୍ོଠ

2. 
   
   
   DNEFWJP

3. ࣗݾ঺հ
    /BNF
    େ୍ོଠ!UBLJQPOF +PC
    43&!QSJTNBUJY *OUFSFTU
    ωοτϫʔΫσϓϩΠपΓ 'BWPSJUF
   

   "NB[PO
   3PVUF
   ͱ"$.

4. ΞδΣϯμ
     "84ωΠςΟϒͳ&$$3.γεςϜ QSJTNBUJY ͱ͸
    ϩά෼ੳج൫ͷઃܭ
   

   ͭ·͍͍ͮͨͯΔͱ͜Ζ
    ࠓޙͷల๬

5. 
    1. AWSωΠςΟϒͳ
 EC/CRMγεςϜ prismatixͱ͸

6. QSJTNBUJY ҎԼ1[ ͸ɺ&$ͱ$3.γεςϜΛϚΠΫϩαʔϏεͱͯ͠
 ఏڙ͠ݸผʹಋೖՄೳͰ͢ɻ֤γεςϜͷػೳʹ"1*Λ׆༻͢Δ͜ͱͰ
 γεςϜ։ൃͱӡ༻ͷෳࡶੑɺίετɺ࣌ؒΛ࡟ݮ͠·͢ɻ

7. ߏ੒ྫ
    prismatix (API) ECαʔόʔ εϚϗΞϓϦ ECαΠτ

8. ࣄྫ
   
   ύϧί༷
   

9. "84ωΠςΟϒͳΠϯϑϥߏ੒
    ϩʔυόϥϯα (ෛՙ෼ࢄ/TLSऴ୺) ίϯςφΫϥελ (APαʔόʔ) σʔλϕʔε (ϚωʔδυαʔϏε)

10. ϚΠΫϩαʔϏεؒͷ࿈ܞ
     ඇಉظૄ݁߹ͷ͘͠Έ ঎඼αʔϏε Amazon SNS (௨஌) Amazon SQS

    (δϣϒΩϡʔ) ࡏݿαʔϏε (ϫʔΧʔ) ঎඼ొ࿥
    Πϕϯτૹ৴ Πϕϯτ͕
    Ωϡʔʹཷ·Δ ΠϕϯτΛऔಘͯ͠
 ࡏݿσʔλʹ൓ө ΠϕϯτΛసૹ    

11. ϚΠΫϩαʔϏεؒͷ࿈ܞ
     αʔϏεͷϦιʔε૿ݮ
    εέʔϧΠϯΞ΢τ ͠΍͍͢ ঎඼αʔϏεΛ૿ڧ ࡏݿαʔϏεΛ૿ڧ

12. 
     2. ϩά෼ੳج൫ͷઃܭ

13. QSJTNBUJYͷϩά
     ओʹछྨɺ͍ͣΕ΋+40/ܗࣜ
    ✦ ΞϓϦϩά
    4QSJOH +BWB
    ىಈ࣌΍Τϥʔͷϝοηʔδ
    

    ✦ .%$ϩά
    .BQQFE
    %JBHOPTUJD
    $POUFYUT
    ϚΠΫϩαʔϏεͷϦΫΤετͱϨεϙϯεͷҰ෦
    ͋ͱεϩʔΫΤϦϩάͳͲ

14. ϩάͷ༻్
     ✦ τϥϒϧγϡʔςΟϯά
    ‣ ϚΠΫϩαʔϏεͷಈ࡞֬ೝ
    ‣ ঎඼ݕࡧ΍஫จͱͷಥ͖߹Θͤ
    ✦

    ϩά෼ੳ
    ‣ όʔήϯηʔϧ΍4/4ͷεύΠΫͷࣄલରࡦͷجૅࢿྉ औΓ͜΅ͨ͘͠ͳ͍

15. ϩάج൫ͷઃܭࢥ૝
     ✦ ͦΕͳΓʹେྔ
    
    ݄ؒʙ5#
    ✦ 410' 4JOHMF
    1PJOU
    PG
    'BJMVSF Λආ͚͍ͨ
    ✦

    ͳΔ΂͘༗Γ෺Λ࢖͍͍ͨ
    ✦ ༻్ʹΑͬͯҟͳΔಛੑ
    ‣ σʔλϨΠΫ ϦΞϧλΠϜϩάͷ૊Έ߹Θͤ

16. ϩάج൫ͷߏ੒ਤ
     Amazon ECS (ίϯςφΫϥελ) Amazon S3 (σʔλϨΠΫ) CloudWatch

    Logs (ϦΞϧλΠϜϩά) Treasure Data (ϩά෼ੳ)

17. "NB[PO
    &$4
    
    ίϯςφΫϥελ
     ✦ %PDLFSίϯςφΛ؅ཧ͢ΔϚωʔδυαʔϏε
    ‣ ίϯςφΛͲͷΠϯελϯε Ծ૝Ϛγϯ Ͱ࣮ߦ͢Δ ͔ΛΫϥελ͝ͱʹ؅ཧ
    

    ‣ QSJTNBUJYͰ͸'BSHBUF͸ະ࠾༻
    ✦ %PDLFSʹίϯςφϩά ඪ४ग़ྗ Λѻ͏ MPHHJOH
    ESJWFSػೳ͕͋Δ
    ‣ ίϯςφ͸ϩάϑΝΠϧΛѻΘͣɺϩάΛ+40/ܗࣜͰ ඪ४ग़ྗʹग़͢Α͏ΞϓϦΛߏ੒ ECSΫϥελ

18. "NB[PO
    4
    
    σʔλϨΠΫ
     ✦ ΦϯϥΠϯετϨʔδαʔϏε
    ‣ ߴ͍଱ٱੑ
    ‣ ߴ͍εέʔϥϏϦςΟ ༰ྔແ੍ݶ

    
    ‣ ྿Ձɺબ΂ΔετϨʔδΫϥε
    ‣ σʔλ෼ੳαʔϏεͱͷ࿈ܞ

19. "NB[PO
    $MPVE8BUDI
    -PHT
    
    ϦΞϧλΠϜϩά
     ✦ ϚωʔδυͷϩάอଘαʔϏε
    ‣ ߴ͍εέʔϥϏϦςΟ
    ‣ ४ϦΞϧλΠϜࢀর
    ‣

    +40/΁ͷΫΤϦΛαϙʔτ

20. "SN
    5SFBTVSF
    %BUB
    
    ϩά෼ੳ
     ✦ σʔλ෼ੳͷ4BB4αʔϏε ඇ"84
    ‣ ඇߏ଄Խσʔλʹૉૣ͘ΞΫηεͰ͖Δ
 ετϨʔδ

    1MB[NB%#
    ‣ ฒྻΫΤϦΤϯδϯ 1SFTUPͳͲ Ͱ
 ΫΤϦͰ͖Δ

21. ༗Γ෺ʹ͸ݶք͋Γʢʣ
     Amazon ECS Amazon S3 CloudWatch Logs MPHHJOH
    ESJWFSͷ੍໿
    

    ⭕ $MPVE8BUDI
    -PHTΛαϙʔτ
    ❌ 4ʹ͸௚઀޲͚ΒΕͳ͍
    ❌ ෳ਺ग़ྗʹະରԠ ◦ ×

22. ༗Γ෺ʹ͸ݶք͋Γʢʣ
     Amazon S3 Treasure Data %BUB
    $POOFDUPSͱ͍͏5SFBTVSF
    %BUBͷΠϯϙʔτػೳ͕͋Δ
    ⭕

    εέδϡʔϧػೳ͕͋Γɺ೔࣍ͷ
 Πϯϙʔτ͸͜ΕͰ0,
    ❌ Πϯϙʔτ࣌ͷϦιʔε΍ಉ࣮࣌ߦ਺ ʹ੍ݶ͕͋ΓɺॳճΠϯϙʔτʹ͸ن ໛ײ͕߹Θͳ͍
    044ͷ&NCVMLͱ%JHEBH͕தͰ
 ಈ͍͍ͯΔ

23. ෆ଍Λิ͏ͨΊʹ044Λར༻
     Fluentd (ετϦʔϜॲཧ) Embulk (όονॲཧ) ͲͪΒ΋ॊೈͰ๛෋ͳϓϥάΠϯΤίγεςϜ͋Γ
    ͨ·ͨ·5SFBTVSF
    %BUB੡Ͱ5%ͱͷߴ͍਌࿨ੑ

24. %PDLFS
    º
    'MVFOUE
     %PDLFS
    MPHHJOH
    ESJWFS͕'MVFOUE΁ͷ ૹ৴Λαϙʔτ
    ‣ 'MVFOUEࣗମ΋%PDLFSίϯςφͱͯ͠
 &$4ͷ֤ΠϯελϯεͰ࣮ߦ
    ‣ ϚΠΫϩαʔϏεͷίϯςφ͔Β͸


    ಉΠϯελϯεͷ'MVFOUEʹϩάΛૹ৴ Fluentd
 ίϯςφ

25. 'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒
     4ͱ$MPVE8BUDI
    -PHT
 ͷ྆ํʹอଘ Amazon S3 CloudWatch Logs <match

    docker.*.*.*> @type copy <store> @type s3 : </store> <store> @type forward : <server> host fluentd.example.lo port 24224 </server> </store> </match>

26. 'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒
     ✦ ϦΞϧλΠϜϩά͸"HHSFHBUPS ू໿αʔόʔ Λ
 ௥Ճͨ͠ଟஈߏ੒
    ‣ ͋ͱ͔ΒৼΓઌΛม͑΍͍͢Α͏ʹ
    

    ‣ $MPVE8BUDI
    -PHTͷ"1*ίʔϧͷ੍໿ରࡦ CloudWatch Logs Aggregator Forwarder

27. 'MVFOUEͷσʔλՃ޻
     ✦ 'PSXBSEFS
    ‣ +40/ͷύʔε
    ‣ ΞϓϦϩάͱ.%$ϩάͷ
 ۠෼͚
    

    ‣ 4ͷύεϓϨϑΟοΫε
    ✦ "HHSFHBUPS
    ‣ Τϥʔͷநग़
    ‣ $MPVE8BUDI
    -PHTύϥϝʔλ <filter docker.**> @type parser format json key_name log </filter> : <match docker.**> @type rewrite_tag_filter <rule> key marker pattern AUDIT tag ${tag}.audit </rule> <rule> key message pattern .+ tag ${tag}.app </rule> </match>

28. &NCVML
     ✦ 4ˠ5%΁ͷॳظόονΠϯϙʔτ
    ‣ 5%ͷઃఆ͕ͱʹָ͔ͩͬͨ͘
    ‣ %BUB
    $POOFDUPSͷίϯϑΟά͕Ұ෦࢖͍ճͤͨ

29. 
     Embulkͷฒྻ࣮ߦ͸ Ͳ͏͢Δʁ

30. &NCVML
    PO
    "84
    #BUDI
     ✦ "84
    #BUDIͷδϣϒΩϡʔ
 εϙοτΠϯελϯε׆༻
    ‣ δϣϒͷ಺༰͸%PDLFSίϯςφͰ ࣮ߦ͢ΔίϚϯυϥΠϯ
    ✦

    &NCVMLͷ%PDLFSΠϝʔδΛ
 ༻ҙ࣮ͯ͠ߦ
    ‣ IUUQTIVCEPDLFSDPNSDMBTTNFUIPE FNCVMLNFUTUE εϙοτϑϦʔτ δϣϒΩϡʔ AWS Batch

31. &NCVMLͷίϯϑΟά͸؀ڥม਺Λଟ༻
     exec: max_threads: {{ env.MAX_THREADS }} in: type:

    s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH } path_match_patterns: \.gz$ auth_method: instance endpoint: s3-ap-northeast-1.amazonaws.com parser: type: jsonl : decoders: - { type: gzip } out: type: td endpoint: api.treasuredata.com apikey: {{ env.TDAPIKEY }} database: {{ env.TDDATABASE }} table: {{ env.TDTABLE }}

32. 
     3. ͭ·͍ͮͨ/͍ͯΔ
 ͱ͜Ζ

33. 
     Fluentd Aggregatorͷ ৑௕Խ/εέʔϧΞ΢τ CloudWatch Logs Aggregator Forwarder

34. "HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ
     ✦ ෳ਺ͷ"HHSFHBUPSཱ͕ͯΒΕͳ͍
    ‣ $MPVE8BUDI
    -PHTͰ͸ಉҰͷϩάετϦʔϜʹॻ͖ࠐΉ ͱ͖͸TFRVFODF
    UPLFOΛҡ࣋͠ͳ͚Ε͹ͳΒͳ͍ <match **>

    @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name container_id auto_create_stream true </match> ϚΠΫϩαʔϏε
    ͷίϯςφ*% ϩάάϧʔϓ ϩά ετϦʔϜ Aggregator ×

35. "HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ
     ✦ ϩάετϦʔϜ໊ʹͩ͜ΘΒͳ͍
    ‣ "HHSFHBUPS͝ͱʹϩάετϦʔϜΛׂΓ౰ͯΔ
    ‣ ʮΠϕϯτͷݕࡧʯ͔ΒετϦʔϜԣஅͷݕࡧΛར༻ <match

    **> @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name "#{Socket.gethostname}" auto_create_stream true </match> "HHSFHBUPS
    ͷίϯςφ*%

36. %PDLFSͷϗετϙʔτͷ੍໿
     ✦ "HHSFHBUPS΋%PDLFSίϯςφ Ͱ࣮ߦ
    ‣ ϗετϙʔτ͕ݻఆͰϙʔτ Λ઎༗
    ‣

    Πϯελϯεʹίϯςφ͔͠
 ࣮ߦͰ͖ͳ͍
    ✦ ϗετϙʔτΛಈతʹ͢ΔͨΊʹ ϩʔυόϥϯαΛ$-#ˠ/-#ʹ
 Ҡߦ     NLB Aggregator ίϯςφ

37. 
     TD΁ͷΠϯϙʔτͷδϣϒ෼ׂ

38. Πϯϙʔτͷδϣϒ෼ׂ
     ✦ Πϯϙʔτର৅4ϓϨϑΟοΫεΛ޿͘औΔͱ
 ϝϞϦΊͬͪΌ࢖͏
    ‣ %BUB
    $POOFDUPS
    
    ϩά͕ফ͑ͯແݶϦτϥΠ˞೥݄౰࣌
    ‣

    "84
    #BUDI
    
    ϝϞϦׂ౰্ݶ·Ͱ࢖ͬͯ00.
    ,JMMFSൃಈ $ aws s3 ls --profile cm-jp-1 s3://XXXX-infra-logbucket-XXXX/applications
 /condor/app/2018/10/04/05/ 2018-10-04 14:06:24 5964 00_5fbef6f0fdec_0.gz 2018-10-04 14:11:26 5971 05_5fbef6f0fdec_0.gz 2018-10-04 14:16:24 5932 10_5fbef6f0fdec_0.gz 2018-10-04 14:21:25 5954 15_5fbef6f0fdec_0.gz

39. Πϯϙʔτͷδϣϒ෼ׂ
     ✦ ͍Ζ͍Ζࢼͯ͠ɺ೔୯ҐͰδϣϒΛ੾Δͷ͕
 ྑͦ͞͏ͱ͍͏͜ͱʹ
    ‣ ϑΝΠϧαΠζɺϑΝΠϧ਺ʹґଘ͢ΔͷͰ΍ͬͯΈͳ ͍ͱΘ͔Βͳ͍෦෼ ࠓճ͸ϑΝΠϧ਺͕ଟ͔ͬͨ໛༷

    in: type: s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/ {{ env.MONTH }}/{{ env.DAY }}

40. 5SFBTVSF
    %BUBͷ'"2ʹ΋هࡌ͋Γ
     2 %BUB
    $POOFDUPS
    GPS
    4
    KPC
    JT
    SVOOJOH
    GPS
    B
    MPOH
    UJNF
    XIBU
    DBO
    *
    EP
    " $IFDL
    UIF
    DPVOU
    PG
    4
    pMFT
    UIBU
    ZPVS
    DPOOFDUPS
    

    KPC
    JT
    JOHFTUJOH
    *G
    UIFSF
    BSF
    PWFS
     
    pMFT
    UIF
    QFSGPSNBODF
    EFHSBEFT
    
 5P
    NJUJHBUF
    UIJT
    JTTVF
    ZPV
    DBO
    ‣ /BSSPX
    QBUI@QSFpY
    PQUJPO
    BOE
    SFEVDF
    UIF
    DPVOU
    PG
    4
    pMFT
    ‣ 4FU
      
    .#
    UP
    NJO@UBTL@TJ[F
    PQUJPO
    IUUQTTVQQPSUUSFBTVSFEBUBDPNIDFOVTBSUJDMFT%BUB$POOFDUPS GPS"NB[PO4'"2GPSUIF4%BUB$POOFDUPS

41. 
     ϩάαΠζͷ্ݶ

42. ϩάαΠζͷ্ݶ
     ✦ .%$ϩά͕σΧ͍
    ✦ %PDLFS
    MPHHJOH
    ESJWFS͸
 ,#ΑΓେ͖͍ϩάΛ෼ׂ͢Δ
    ‣ 'MVFOUE
    GPSXBSEFSͷ+40/ύʔεʹࣦഊɺࣺͯΒΕΔ
    

    ‣ DPODBUϓϥάΠϯͰ݁߹
    ‣ ϝϞϦ࢖༻ྔ͕௓Ͷͯ00.
    ,JMMFSʹࡴ͞Εͨ
    ΠϚίί
    ✦ $MPVE8BUDI
    -PHTͷ্ݶ͸,#

43. ͓ۚ
     ✦ $MPVE8BUDI
    -PHT͸ϩάσʔλྔͷैྔ՝ۚ
    ‣ ྔ͕ଟ͍ͷͰֹ݄අ༻͕͔͞Ή
    ‣ &$ͳͲଞͷαʔϏεඅ༻ͱτϯτϯͱݴΘΕΔͱͭΒ ͍
    

    ✦ ΞϓϦ͔Βग़͢ϩάͷ෼ྔΛݮΒ͢
    ‣ .%$ϩάͭΒ͍

44. 
     4. ࠓޙͷల๬

45. ࣍ͷҰख
     ✦ ϩά؂ࢹ
    ✦ ϩάू໿ͱͯ͠&MBTUJDTFBSDI,JCBOBͷར༻
    ✦ 5SFBTVSF
    %BUBΛ΋ͬͱ׆༻͍ͨ͠

46. ·ͱΊ
     ✦ εέʔϧ͢Δϩά؅ཧج൫͸Ϋϥ΢υαʔϏεͷ
 ૊Έ߹Θ͕ͤΦεεϝ
    ‣ σʔλϨΠΫͱϦΞϧλΠϜϩάͰͷαʔϏεͷ࢖͍෼͚
    ✦ ૊Έ߹Θ͚ͤͩͰ΋ಈ͘΋ͷ͸Ͱ͖Δ͚Ͳ
    

    ‣ ཁ݅ʹରͯ͠଍Γͳ͍ͱ͜Ζͷิ͕ؒඞཁ
    ‣ σʔλྔɺαΠζͷධՁ΍νϡʔχϯά͕ඞཁ