AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

Ecd0c945f6911dbf75358a8addee15f7?s=47 takipone
October 05, 2018

AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

2018/10/05(金)@UDX秋葉原 Developers.IO 2018のセッション資料

Ecd0c945f6911dbf75358a8addee15f7?s=128

takipone

October 05, 2018
Tweet

Transcript

  1. 10.

    ϚΠΫϩαʔϏεؒͷ࿈ܞ   ඇಉظૄ݁߹ͷ͘͠Έ ঎඼αʔϏε Amazon SNS (௨஌) Amazon SQS

    (δϣϒΩϡʔ) ࡏݿαʔϏε (ϫʔΧʔ) ঎඼ొ࿥ Πϕϯτૹ৴ Πϕϯτ͕ Ωϡʔʹཷ·Δ ΠϕϯτΛऔಘͯ͠
 ࡏݿσʔλʹ൓ө ΠϕϯτΛసૹ    
  2. 13.

    QSJTNBUJYͷϩά   ओʹछྨɺ͍ͣΕ΋+40/ܗࣜ ✦ ΞϓϦϩά 4QSJOH +BWB  ىಈ࣌΍Τϥʔͷϝοηʔδ

    ✦ .%$ϩά .BQQFE%JBHOPTUJD$POUFYUT  ϚΠΫϩαʔϏεͷϦΫΤετͱϨεϙϯεͷҰ෦ ͋ͱεϩʔΫΤϦϩάͳͲ
  3. 14.

    ϩάͷ༻్   ✦ τϥϒϧγϡʔςΟϯά ‣ ϚΠΫϩαʔϏεͷಈ࡞֬ೝ ‣ ঎඼ݕࡧ΍஫จͱͷಥ͖߹Θͤ ✦

    ϩά෼ੳ ‣ όʔήϯηʔϧ΍4/4ͷεύΠΫͷࣄલରࡦͷجૅࢿྉ औΓ͜΅ͨ͘͠ͳ͍
  4. 15.

    ϩάج൫ͷઃܭࢥ૝   ✦ ͦΕͳΓʹେྔ݄ؒʙ5# ✦ 410' 4JOHMF1PJOUPG'BJMVSF Λආ͚͍ͨ ✦

    ͳΔ΂͘༗Γ෺Λ࢖͍͍ͨ ✦ ༻్ʹΑͬͯҟͳΔಛੑ ‣ σʔλϨΠΫ ϦΞϧλΠϜϩάͷ૊Έ߹Θͤ
  5. 17.

    "NB[PO&$4ίϯςφΫϥελ   ✦ %PDLFSίϯςφΛ؅ཧ͢ΔϚωʔδυαʔϏε ‣ ίϯςφΛͲͷΠϯελϯε Ծ૝Ϛγϯ Ͱ࣮ߦ͢Δ ͔ΛΫϥελ͝ͱʹ؅ཧ

    ‣ QSJTNBUJYͰ͸'BSHBUF͸ະ࠾༻ ✦ %PDLFSʹίϯςφϩά ඪ४ग़ྗ Λѻ͏ MPHHJOHESJWFSػೳ͕͋Δ ‣ ίϯςφ͸ϩάϑΝΠϧΛѻΘͣɺϩάΛ+40/ܗࣜͰ ඪ४ग़ྗʹग़͢Α͏ΞϓϦΛߏ੒ ECSΫϥελ
  6. 21.

    ༗Γ෺ʹ͸ݶք͋Γʢʣ   Amazon ECS Amazon S3 CloudWatch Logs MPHHJOHESJWFSͷ੍໿

    ⭕ $MPVE8BUDI-PHTΛαϙʔτ ❌ 4ʹ͸௚઀޲͚ΒΕͳ͍ ❌ ෳ਺ग़ྗʹະରԠ ◦ ×
  7. 22.

    ༗Γ෺ʹ͸ݶք͋Γʢʣ   Amazon S3 Treasure Data %BUB$POOFDUPSͱ͍͏5SFBTVSF %BUBͷΠϯϙʔτػೳ͕͋Δ ⭕

    εέδϡʔϧػೳ͕͋Γɺ೔࣍ͷ
 Πϯϙʔτ͸͜ΕͰ0, ❌ Πϯϙʔτ࣌ͷϦιʔε΍ಉ࣮࣌ߦ਺ ʹ੍ݶ͕͋ΓɺॳճΠϯϙʔτʹ͸ن ໛ײ͕߹Θͳ͍ 044ͷ&NCVMLͱ%JHEBH͕தͰ
 ಈ͍͍ͯΔ
  8. 25.

    'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒   4ͱ$MPVE8BUDI-PHT
 ͷ྆ํʹอଘ Amazon S3 CloudWatch Logs <match

    docker.*.*.*> @type copy <store> @type s3 : </store> <store> @type forward : <server> host fluentd.example.lo port 24224 </server> </store> </match>
  9. 27.

    'MVFOUEͷσʔλՃ޻   ✦ 'PSXBSEFS ‣ +40/ͷύʔε ‣ ΞϓϦϩάͱ.%$ϩάͷ
 ۠෼͚

    ‣ 4ͷύεϓϨϑΟοΫε ✦ "HHSFHBUPS ‣ Τϥʔͷநग़ ‣ $MPVE8BUDI-PHTύϥϝʔλ <filter docker.**> @type parser format json key_name log </filter> : <match docker.**> @type rewrite_tag_filter <rule> key marker pattern AUDIT tag ${tag}.audit </rule> <rule> key message pattern .+ tag ${tag}.app </rule> </match>
  10. 30.

    &NCVMLPO"84#BUDI   ✦ "84#BUDIͷδϣϒΩϡʔ
 εϙοτΠϯελϯε׆༻ ‣ δϣϒͷ಺༰͸%PDLFSίϯςφͰ ࣮ߦ͢ΔίϚϯυϥΠϯ ✦

    &NCVMLͷ%PDLFSΠϝʔδΛ
 ༻ҙ࣮ͯ͠ߦ ‣ IUUQTIVCEPDLFSDPNSDMBTTNFUIPE FNCVMLNFUTUE εϙοτϑϦʔτ δϣϒΩϡʔ AWS Batch
  11. 31.

    &NCVMLͷίϯϑΟά͸؀ڥม਺Λଟ༻   exec: max_threads: {{ env.MAX_THREADS }} in: type:

    s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH } path_match_patterns: \.gz$ auth_method: instance endpoint: s3-ap-northeast-1.amazonaws.com parser: type: jsonl : decoders: - { type: gzip } out: type: td endpoint: api.treasuredata.com apikey: {{ env.TDAPIKEY }} database: {{ env.TDDATABASE }} table: {{ env.TDTABLE }}
  12. 34.

    "HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ   ✦ ෳ਺ͷ"HHSFHBUPSཱ͕ͯΒΕͳ͍ ‣ $MPVE8BUDI-PHTͰ͸ಉҰͷϩάετϦʔϜʹॻ͖ࠐΉ ͱ͖͸TFRVFODFUPLFOΛҡ࣋͠ͳ͚Ε͹ͳΒͳ͍ <match **>

    @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name container_id auto_create_stream true </match> ϚΠΫϩαʔϏε ͷίϯςφ*% ϩάάϧʔϓ ϩά ετϦʔϜ Aggregator ×
  13. 35.

    "HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ   ✦ ϩάετϦʔϜ໊ʹͩ͜ΘΒͳ͍ ‣ "HHSFHBUPS͝ͱʹϩάετϦʔϜΛׂΓ౰ͯΔ ‣ ʮΠϕϯτͷݕࡧʯ͔ΒετϦʔϜԣஅͷݕࡧΛར༻ <match

    **> @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name "#{Socket.gethostname}" auto_create_stream true </match> "HHSFHBUPS ͷίϯςφ*%
  14. 36.

    %PDLFSͷϗετϙʔτͷ੍໿   ✦ "HHSFHBUPS΋%PDLFSίϯςφ Ͱ࣮ߦ ‣ ϗετϙʔτ͕ݻఆͰϙʔτ Λ઎༗ ‣

    Πϯελϯεʹίϯςφ͔͠
 ࣮ߦͰ͖ͳ͍ ✦ ϗετϙʔτΛಈతʹ͢ΔͨΊʹ ϩʔυόϥϯαΛ$-#ˠ/-#ʹ
 Ҡߦ     NLB Aggregator ίϯςφ
  15. 38.

    Πϯϙʔτͷδϣϒ෼ׂ   ✦ Πϯϙʔτର৅4ϓϨϑΟοΫεΛ޿͘औΔͱ
 ϝϞϦΊͬͪΌ࢖͏ ‣ %BUB$POOFDUPSϩά͕ফ͑ͯແݶϦτϥΠ˞೥݄౰࣌  ‣

    "84#BUDIϝϞϦׂ౰্ݶ·Ͱ࢖ͬͯ00.,JMMFSൃಈ $ aws s3 ls --profile cm-jp-1 s3://XXXX-infra-logbucket-XXXX/applications
 /condor/app/2018/10/04/05/ 2018-10-04 14:06:24 5964 00_5fbef6f0fdec_0.gz 2018-10-04 14:11:26 5971 05_5fbef6f0fdec_0.gz 2018-10-04 14:16:24 5932 10_5fbef6f0fdec_0.gz 2018-10-04 14:21:25 5954 15_5fbef6f0fdec_0.gz
  16. 39.
  17. 40.

    5SFBTVSF%BUBͷ'"2ʹ΋هࡌ͋Γ   2 %BUB$POOFDUPSGPS4KPCJTSVOOJOHGPSBMPOH UJNF XIBUDBO*EP  " $IFDLUIFDPVOUPG4pMFTUIBUZPVSDPOOFDUPS

    KPCJTJOHFTUJOH*GUIFSFBSFPWFS pMFT UIF QFSGPSNBODFEFHSBEFT
 5PNJUJHBUFUIJTJTTVF ZPVDBO ‣ /BSSPXQBUI@QSFpYPQUJPOBOESFEVDFUIFDPVOUPG4pMFT ‣ 4FU   .# UPNJO@UBTL@TJ[FPQUJPO IUUQTTVQQPSUUSFBTVSFEBUBDPNIDFOVTBSUJDMFT%BUB$POOFDUPS GPS"NB[PO4'"2GPSUIF4%BUB$POOFDUPS
  18. 42.

    ϩάαΠζͷ্ݶ   ✦ .%$ϩά͕σΧ͍ ✦ %PDLFSMPHHJOHESJWFS͸
 ,#ΑΓେ͖͍ϩάΛ෼ׂ͢Δ ‣ 'MVFOUEGPSXBSEFSͷ+40/ύʔεʹࣦഊɺࣺͯΒΕΔ

    ‣ DPODBUϓϥάΠϯͰ݁߹ ‣ ϝϞϦ࢖༻ྔ͕௓Ͷͯ00.,JMMFSʹࡴ͞ΕͨΠϚίί ✦ $MPVE8BUDI-PHTͷ্ݶ͸,#