Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

takipone
October 05, 2018

AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

2018/10/05(金)@UDX秋葉原 Developers.IO 2018のセッション資料

takipone

October 05, 2018
Tweet

More Decks by takipone

Other Decks in Technology

Transcript

  1. "84ωΠςΟϒͳ&$$3.γεςϜ
    ӡ༻ʹ͔ܽͤͳ͍ϩάج൫ߏங
    େ୍ོଠ

    View full-size slide

  2. ࣗݾ঺հ

    /BNF େ୍ོଠ!UBLJQPOF
    +PC 43&!QSJTNBUJY
    *OUFSFTU ωοτϫʔΫσϓϩΠपΓ
    'BWPSJUF "NB[PO3PVUFͱ"$.

    View full-size slide

  3. ΞδΣϯμ

    "84ωΠςΟϒͳ&$$3.γεςϜ QSJTNBUJY
    ͱ͸
    ϩά෼ੳج൫ͷઃܭ
    ͭ·͍͍ͮͨͯΔͱ͜Ζ
    ࠓޙͷల๬

    View full-size slide



  4. 1. AWSωΠςΟϒͳ

    EC/CRMγεςϜ
    prismatixͱ͸

    View full-size slide

  5. QSJTNBUJY ҎԼ1[
    ͸ɺ&$ͱ$3.γεςϜΛϚΠΫϩαʔϏεͱͯ͠

    ఏڙ͠ݸผʹಋೖՄೳͰ͢ɻ֤γεςϜͷػೳʹ"1*Λ׆༻͢Δ͜ͱͰ

    γεςϜ։ൃͱӡ༻ͷෳࡶੑɺίετɺ࣌ؒΛ࡟ݮ͠·͢ɻ

    View full-size slide

  6. ߏ੒ྫ

    prismatix
    (API)
    ECαʔόʔ
    εϚϗΞϓϦ
    ECαΠτ

    View full-size slide

  7. ࣄྫύϧί༷

    View full-size slide

  8. "84ωΠςΟϒͳΠϯϑϥߏ੒

    ϩʔυόϥϯα
    (ෛՙ෼ࢄ/TLSऴ୺)
    ίϯςφΫϥελ
    (APαʔόʔ)
    σʔλϕʔε
    (ϚωʔδυαʔϏε)

    View full-size slide

  9. ϚΠΫϩαʔϏεؒͷ࿈ܞ

    ඇಉظૄ݁߹ͷ͘͠Έ
    ঎඼αʔϏε Amazon SNS
    (௨஌)
    Amazon SQS
    (δϣϒΩϡʔ)
    ࡏݿαʔϏε
    (ϫʔΧʔ)
    ঎඼ొ࿥
    Πϕϯτૹ৴
    Πϕϯτ͕
    Ωϡʔʹཷ·Δ ΠϕϯτΛऔಘͯ͠

    ࡏݿσʔλʹ൓ө
    ΠϕϯτΛసૹ



    View full-size slide

  10. ϚΠΫϩαʔϏεؒͷ࿈ܞ

    αʔϏεͷϦιʔε૿ݮ
    εέʔϧΠϯΞ΢τ
    ͠΍͍͢
    ঎඼αʔϏεΛ૿ڧ
    ࡏݿαʔϏεΛ૿ڧ

    View full-size slide



  11. 2. ϩά෼ੳج൫ͷઃܭ

    View full-size slide

  12. QSJTNBUJYͷϩά

    ओʹछྨɺ͍ͣΕ΋+40/ܗࣜ
    ✦ ΞϓϦϩά 4QSJOH +BWB


    ىಈ࣌΍Τϥʔͷϝοηʔδ
    ✦ .%$ϩά .BQQFE%JBHOPTUJD$POUFYUT

    ϚΠΫϩαʔϏεͷϦΫΤετͱϨεϙϯεͷҰ෦
    ͋ͱεϩʔΫΤϦϩάͳͲ

    View full-size slide

  13. ϩάͷ༻్

    ✦ τϥϒϧγϡʔςΟϯά
    ‣ ϚΠΫϩαʔϏεͷಈ࡞֬ೝ
    ‣ ঎඼ݕࡧ΍஫จͱͷಥ͖߹Θͤ
    ✦ ϩά෼ੳ
    ‣ όʔήϯηʔϧ΍4/4ͷεύΠΫͷࣄલରࡦͷجૅࢿྉ
    औΓ͜΅ͨ͘͠ͳ͍

    View full-size slide

  14. ϩάج൫ͷઃܭࢥ૝

    ✦ ͦΕͳΓʹେྔ݄ؒʙ5#
    ✦ 410' 4JOHMF1PJOUPG'BJMVSF
    Λආ͚͍ͨ
    ✦ ͳΔ΂͘༗Γ෺Λ࢖͍͍ͨ
    ✦ ༻్ʹΑͬͯҟͳΔಛੑ
    ‣ σʔλϨΠΫϦΞϧλΠϜϩάͷ૊Έ߹Θͤ

    View full-size slide

  15. ϩάج൫ͷߏ੒ਤ

    Amazon ECS
    (ίϯςφΫϥελ)
    Amazon S3
    (σʔλϨΠΫ)
    CloudWatch Logs
    (ϦΞϧλΠϜϩά)
    Treasure Data
    (ϩά෼ੳ)

    View full-size slide

  16. "NB[PO&$4ίϯςφΫϥελ

    ✦ %PDLFSίϯςφΛ؅ཧ͢ΔϚωʔδυαʔϏε
    ‣ ίϯςφΛͲͷΠϯελϯε Ծ૝Ϛγϯ
    Ͱ࣮ߦ͢Δ
    ͔ΛΫϥελ͝ͱʹ؅ཧ
    ‣ QSJTNBUJYͰ͸'BSHBUF͸ະ࠾༻
    ✦ %PDLFSʹίϯςφϩά ඪ४ग़ྗ
    Λѻ͏
    MPHHJOHESJWFSػೳ͕͋Δ
    ‣ ίϯςφ͸ϩάϑΝΠϧΛѻΘͣɺϩάΛ+40/ܗࣜͰ
    ඪ४ग़ྗʹग़͢Α͏ΞϓϦΛߏ੒
    ECSΫϥελ

    View full-size slide

  17. "NB[PO4σʔλϨΠΫ

    ✦ ΦϯϥΠϯετϨʔδαʔϏε
    ‣ ߴ͍଱ٱੑ
    ‣ ߴ͍εέʔϥϏϦςΟ ༰ྔແ੍ݶ

    ‣ ྿Ձɺબ΂ΔετϨʔδΫϥε
    ‣ σʔλ෼ੳαʔϏεͱͷ࿈ܞ

    View full-size slide

  18. "NB[PO$MPVE8BUDI-PHTϦΞϧλΠϜϩά

    ✦ ϚωʔδυͷϩάอଘαʔϏε
    ‣ ߴ͍εέʔϥϏϦςΟ
    ‣ ४ϦΞϧλΠϜࢀর
    ‣ +40/΁ͷΫΤϦΛαϙʔτ

    View full-size slide

  19. "SN5SFBTVSF%BUBϩά෼ੳ

    ✦ σʔλ෼ੳͷ4BB4αʔϏε ඇ"84

    ‣ ඇߏ଄Խσʔλʹૉૣ͘ΞΫηεͰ͖Δ

    ετϨʔδ 1MB[NB%#

    ‣ ฒྻΫΤϦΤϯδϯ 1SFTUPͳͲ
    Ͱ

    ΫΤϦͰ͖Δ

    View full-size slide

  20. ༗Γ෺ʹ͸ݶք͋Γʢʣ

    Amazon
    ECS
    Amazon
    S3
    CloudWatch
    Logs
    MPHHJOHESJWFSͷ੍໿
    ⭕ $MPVE8BUDI-PHTΛαϙʔτ
    ❌ 4ʹ͸௚઀޲͚ΒΕͳ͍
    ❌ ෳ਺ग़ྗʹະରԠ

    ×

    View full-size slide

  21. ༗Γ෺ʹ͸ݶք͋Γʢʣ

    Amazon
    S3
    Treasure
    Data
    %BUB$POOFDUPSͱ͍͏5SFBTVSF
    %BUBͷΠϯϙʔτػೳ͕͋Δ
    ⭕ εέδϡʔϧػೳ͕͋Γɺ೔࣍ͷ

    Πϯϙʔτ͸͜ΕͰ0,
    ❌ Πϯϙʔτ࣌ͷϦιʔε΍ಉ࣮࣌ߦ਺
    ʹ੍ݶ͕͋ΓɺॳճΠϯϙʔτʹ͸ن
    ໛ײ͕߹Θͳ͍
    044ͷ&NCVMLͱ%JHEBH͕தͰ

    ಈ͍͍ͯΔ

    View full-size slide

  22. ෆ଍Λิ͏ͨΊʹ044Λར༻

    Fluentd
    (ετϦʔϜॲཧ)
    Embulk
    (όονॲཧ)
    ͲͪΒ΋ॊೈͰ๛෋ͳϓϥάΠϯΤίγεςϜ͋Γ
    ͨ·ͨ·5SFBTVSF%BUB੡Ͱ5%ͱͷߴ͍਌࿨ੑ

    View full-size slide

  23. %PDLFSº'MVFOUE

    %PDLFSMPHHJOHESJWFS͕'MVFOUE΁ͷ
    ૹ৴Λαϙʔτ
    ‣ 'MVFOUEࣗମ΋%PDLFSίϯςφͱͯ͠

    &$4ͷ֤ΠϯελϯεͰ࣮ߦ
    ‣ ϚΠΫϩαʔϏεͷίϯςφ͔Β͸

    ಉΠϯελϯεͷ'MVFOUEʹϩάΛૹ৴
    Fluentd

    ίϯςφ

    View full-size slide

  24. 'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒

    4ͱ$MPVE8BUDI-PHT

    ͷ྆ํʹอଘ
    Amazon
    S3
    CloudWatch
    Logs

    @type copy

    @type s3
    :


    @type forward
    :

    host fluentd.example.lo
    port 24224



    View full-size slide

  25. 'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒

    ✦ ϦΞϧλΠϜϩά͸"HHSFHBUPS ू໿αʔόʔ
    Λ

    ௥Ճͨ͠ଟஈߏ੒
    ‣ ͋ͱ͔ΒৼΓઌΛม͑΍͍͢Α͏ʹ
    ‣ $MPVE8BUDI-PHTͷ"1*ίʔϧͷ੍໿ରࡦ
    CloudWatch
    Logs
    Aggregator
    Forwarder

    View full-size slide

  26. 'MVFOUEͷσʔλՃ޻

    ✦ 'PSXBSEFS
    ‣ +40/ͷύʔε
    ‣ ΞϓϦϩάͱ.%$ϩάͷ

    ۠෼͚
    ‣ 4ͷύεϓϨϑΟοΫε
    ✦ "HHSFHBUPS
    ‣ Τϥʔͷநग़
    ‣ $MPVE8BUDI-PHTύϥϝʔλ

    @type parser
    format json
    key_name log

    :

    @type rewrite_tag_filter

    key marker
    pattern AUDIT
    tag ${tag}.audit


    key message
    pattern .+
    tag ${tag}.app


    View full-size slide

  27. &NCVML

    ✦ 4ˠ5%΁ͷॳظόονΠϯϙʔτ
    ‣ 5%ͷઃఆ͕ͱʹָ͔ͩͬͨ͘
    ‣ %BUB$POOFDUPSͷίϯϑΟά͕Ұ෦࢖͍ճͤͨ

    View full-size slide



  28. Embulkͷฒྻ࣮ߦ͸
    Ͳ͏͢Δʁ

    View full-size slide

  29. &NCVMLPO"84#BUDI

    ✦ "84#BUDIͷδϣϒΩϡʔ

    εϙοτΠϯελϯε׆༻
    ‣ δϣϒͷ಺༰͸%PDLFSίϯςφͰ
    ࣮ߦ͢ΔίϚϯυϥΠϯ
    ✦ &NCVMLͷ%PDLFSΠϝʔδΛ

    ༻ҙ࣮ͯ͠ߦ
    ‣ IUUQTIVCEPDLFSDPNSDMBTTNFUIPE
    FNCVMLNFUTUE
    εϙοτϑϦʔτ
    δϣϒΩϡʔ
    AWS
    Batch

    View full-size slide

  30. &NCVMLͷίϯϑΟά͸؀ڥม਺Λଟ༻

    exec:
    max_threads: {{ env.MAX_THREADS }}
    in:
    type: s3
    bucket: {{ env.S3BUCKET }}
    path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH }
    path_match_patterns: \.gz$
    auth_method: instance
    endpoint: s3-ap-northeast-1.amazonaws.com
    parser:
    type: jsonl
    :
    decoders:
    - { type: gzip }
    out:
    type: td
    endpoint: api.treasuredata.com
    apikey: {{ env.TDAPIKEY }}
    database: {{ env.TDDATABASE }}
    table: {{ env.TDTABLE }}

    View full-size slide



  31. 3. ͭ·͍ͮͨ/͍ͯΔ

    ͱ͜Ζ

    View full-size slide



  32. Fluentd Aggregatorͷ
    ৑௕Խ/εέʔϧΞ΢τ
    CloudWatch
    Logs
    Aggregator
    Forwarder

    View full-size slide

  33. "HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ

    ✦ ෳ਺ͷ"HHSFHBUPSཱ͕ͯΒΕͳ͍
    ‣ $MPVE8BUDI-PHTͰ͸ಉҰͷϩάετϦʔϜʹॻ͖ࠐΉ
    ͱ͖͸TFRVFODFUPLFOΛҡ࣋͠ͳ͚Ε͹ͳΒͳ͍

    @type cloudwatch_logs
    region "#{ENV['AWS_REGION']}"
    log_group_name_key service
    log_stream_name container_id
    auto_create_stream true

    ϚΠΫϩαʔϏε
    ͷίϯςφ*%
    ϩάάϧʔϓ
    ϩά
    ετϦʔϜ
    Aggregator
    ×

    View full-size slide

  34. "HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ

    ✦ ϩάετϦʔϜ໊ʹͩ͜ΘΒͳ͍
    ‣ "HHSFHBUPS͝ͱʹϩάετϦʔϜΛׂΓ౰ͯΔ
    ‣ ʮΠϕϯτͷݕࡧʯ͔ΒετϦʔϜԣஅͷݕࡧΛར༻

    @type cloudwatch_logs
    region "#{ENV['AWS_REGION']}"
    log_group_name_key service
    log_stream_name "#{Socket.gethostname}"
    auto_create_stream true

    "HHSFHBUPS
    ͷίϯςφ*%

    View full-size slide

  35. %PDLFSͷϗετϙʔτͷ੍໿

    ✦ "HHSFHBUPS΋%PDLFSίϯςφ
    Ͱ࣮ߦ
    ‣ ϗετϙʔτ͕ݻఆͰϙʔτ
    Λ઎༗
    ‣ Πϯελϯεʹίϯςφ͔͠

    ࣮ߦͰ͖ͳ͍
    ✦ ϗετϙʔτΛಈతʹ͢ΔͨΊʹ
    ϩʔυόϥϯαΛ$-#ˠ/-#ʹ

    Ҡߦ



    NLB
    Aggregator
    ίϯςφ

    View full-size slide



  36. TD΁ͷΠϯϙʔτͷδϣϒ෼ׂ

    View full-size slide

  37. Πϯϙʔτͷδϣϒ෼ׂ

    ✦ Πϯϙʔτର৅4ϓϨϑΟοΫεΛ޿͘औΔͱ

    ϝϞϦΊͬͪΌ࢖͏
    ‣ %BUB$POOFDUPSϩά͕ফ͑ͯແݶϦτϥΠ˞೥݄౰࣌

    ‣ "84#BUDIϝϞϦׂ౰্ݶ·Ͱ࢖ͬͯ00.,JMMFSൃಈ
    $ aws s3 ls --profile cm-jp-1 s3://XXXX-infra-logbucket-XXXX/applications

    /condor/app/2018/10/04/05/
    2018-10-04 14:06:24 5964 00_5fbef6f0fdec_0.gz
    2018-10-04 14:11:26 5971 05_5fbef6f0fdec_0.gz
    2018-10-04 14:16:24 5932 10_5fbef6f0fdec_0.gz
    2018-10-04 14:21:25 5954 15_5fbef6f0fdec_0.gz

    View full-size slide

  38. Πϯϙʔτͷδϣϒ෼ׂ

    ✦ ͍Ζ͍Ζࢼͯ͠ɺ೔୯ҐͰδϣϒΛ੾Δͷ͕

    ྑͦ͞͏ͱ͍͏͜ͱʹ
    ‣ ϑΝΠϧαΠζɺϑΝΠϧ਺ʹґଘ͢ΔͷͰ΍ͬͯΈͳ
    ͍ͱΘ͔Βͳ͍෦෼ ࠓճ͸ϑΝΠϧ਺͕ଟ͔ͬͨ໛༷

    in:
    type: s3
    bucket: {{ env.S3BUCKET }}
    path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/
    {{ env.MONTH }}/{{ env.DAY }}

    View full-size slide

  39. 5SFBTVSF%BUBͷ'"2ʹ΋هࡌ͋Γ

    2
    %BUB$POOFDUPSGPS4KPCJTSVOOJOHGPSBMPOH
    UJNF XIBUDBO*EP
    "
    $IFDLUIFDPVOUPG4pMFTUIBUZPVSDPOOFDUPS
    KPCJTJOHFTUJOH*GUIFSFBSFPWFS pMFT UIF
    QFSGPSNBODFEFHSBEFT

    5PNJUJHBUFUIJTJTTVF ZPVDBO
    ‣ /BSSPXQBUI@QSFpYPQUJPOBOESFEVDFUIFDPVOUPG4pMFT
    ‣ 4FU .#
    UPNJO@UBTL@TJ[FPQUJPO
    IUUQTTVQQPSUUSFBTVSFEBUBDPNIDFOVTBSUJDMFT%BUB$POOFDUPS
    GPS"NB[PO4'"2GPSUIF4%BUB$POOFDUPS

    View full-size slide



  40. ϩάαΠζͷ্ݶ

    View full-size slide

  41. ϩάαΠζͷ্ݶ

    ✦ .%$ϩά͕σΧ͍
    ✦ %PDLFSMPHHJOHESJWFS͸

    ,#ΑΓେ͖͍ϩάΛ෼ׂ͢Δ
    ‣ 'MVFOUEGPSXBSEFSͷ+40/ύʔεʹࣦഊɺࣺͯΒΕΔ
    ‣ DPODBUϓϥάΠϯͰ݁߹
    ‣ ϝϞϦ࢖༻ྔ͕௓Ͷͯ00.,JMMFSʹࡴ͞ΕͨΠϚίί
    ✦ $MPVE8BUDI-PHTͷ্ݶ͸,#

    View full-size slide

  42. ͓ۚ

    ✦ $MPVE8BUDI-PHT͸ϩάσʔλྔͷैྔ՝ۚ
    ‣ ྔ͕ଟ͍ͷͰֹ݄අ༻͕͔͞Ή
    ‣ &$ͳͲଞͷαʔϏεඅ༻ͱτϯτϯͱݴΘΕΔͱͭΒ
    ͍
    ✦ ΞϓϦ͔Βग़͢ϩάͷ෼ྔΛݮΒ͢
    ‣ .%$ϩάͭΒ͍

    View full-size slide



  43. 4. ࠓޙͷల๬

    View full-size slide

  44. ࣍ͷҰख

    ✦ ϩά؂ࢹ
    ✦ ϩάू໿ͱͯ͠&MBTUJDTFBSDI,JCBOBͷར༻
    ✦ 5SFBTVSF%BUBΛ΋ͬͱ׆༻͍ͨ͠

    View full-size slide

  45. ·ͱΊ

    ✦ εέʔϧ͢Δϩά؅ཧج൫͸Ϋϥ΢υαʔϏεͷ

    ૊Έ߹Θ͕ͤΦεεϝ
    ‣ σʔλϨΠΫͱϦΞϧλΠϜϩάͰͷαʔϏεͷ࢖͍෼͚
    ✦ ૊Έ߹Θ͚ͤͩͰ΋ಈ͘΋ͷ͸Ͱ͖Δ͚Ͳ
    ‣ ཁ݅ʹରͯ͠଍Γͳ͍ͱ͜Ζͷิ͕ؒඞཁ
    ‣ σʔλྔɺαΠζͷධՁ΍νϡʔχϯά͕ඞཁ

    View full-size slide