AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

Ecd0c945f6911dbf75358a8addee15f7?s=47 takipone
October 05, 2018

AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

2018/10/05(金)@UDX秋葉原 Developers.IO 2018のセッション資料

Ecd0c945f6911dbf75358a8addee15f7?s=128

takipone

October 05, 2018
Tweet

Transcript

  1. "84ωΠςΟϒͳ&$$3.γεςϜ ӡ༻ʹ͔ܽͤͳ͍ϩάج൫ߏங େ୍ོଠ

  2. DNEFWJP

  3. ࣗݾ঺հ   /BNF େ୍ོଠ!UBLJQPOF +PC 43&!QSJTNBUJY *OUFSFTU ωοτϫʔΫσϓϩΠपΓ 'BWPSJUF

    "NB[PO3PVUFͱ"$.
  4. ΞδΣϯμ    "84ωΠςΟϒͳ&$$3.γεςϜ QSJTNBUJY ͱ͸  ϩά෼ੳج൫ͷઃܭ 

    ͭ·͍͍ͮͨͯΔͱ͜Ζ  ࠓޙͷల๬
  5.   1. AWSωΠςΟϒͳ
 EC/CRMγεςϜ prismatixͱ͸

  6. QSJTNBUJY ҎԼ1[ ͸ɺ&$ͱ$3.γεςϜΛϚΠΫϩαʔϏεͱͯ͠
 ఏڙ͠ݸผʹಋೖՄೳͰ͢ɻ֤γεςϜͷػೳʹ"1*Λ׆༻͢Δ͜ͱͰ
 γεςϜ։ൃͱӡ༻ͷෳࡶੑɺίετɺ࣌ؒΛ࡟ݮ͠·͢ɻ

  7. ߏ੒ྫ   prismatix (API) ECαʔόʔ εϚϗΞϓϦ ECαΠτ

  8. ࣄྫύϧί༷  

  9. "84ωΠςΟϒͳΠϯϑϥߏ੒   ϩʔυόϥϯα (ෛՙ෼ࢄ/TLSऴ୺) ίϯςφΫϥελ (APαʔόʔ) σʔλϕʔε (ϚωʔδυαʔϏε)

  10. ϚΠΫϩαʔϏεؒͷ࿈ܞ   ඇಉظૄ݁߹ͷ͘͠Έ ঎඼αʔϏε Amazon SNS (௨஌) Amazon SQS

    (δϣϒΩϡʔ) ࡏݿαʔϏε (ϫʔΧʔ) ঎඼ొ࿥ Πϕϯτૹ৴ Πϕϯτ͕ Ωϡʔʹཷ·Δ ΠϕϯτΛऔಘͯ͠
 ࡏݿσʔλʹ൓ө ΠϕϯτΛసૹ    
  11. ϚΠΫϩαʔϏεؒͷ࿈ܞ   αʔϏεͷϦιʔε૿ݮ εέʔϧΠϯΞ΢τ ͠΍͍͢ ঎඼αʔϏεΛ૿ڧ ࡏݿαʔϏεΛ૿ڧ

  12.   2. ϩά෼ੳج൫ͷઃܭ

  13. QSJTNBUJYͷϩά   ओʹछྨɺ͍ͣΕ΋+40/ܗࣜ ✦ ΞϓϦϩά 4QSJOH +BWB  ىಈ࣌΍Τϥʔͷϝοηʔδ

    ✦ .%$ϩά .BQQFE%JBHOPTUJD$POUFYUT  ϚΠΫϩαʔϏεͷϦΫΤετͱϨεϙϯεͷҰ෦ ͋ͱεϩʔΫΤϦϩάͳͲ
  14. ϩάͷ༻్   ✦ τϥϒϧγϡʔςΟϯά ‣ ϚΠΫϩαʔϏεͷಈ࡞֬ೝ ‣ ঎඼ݕࡧ΍஫จͱͷಥ͖߹Θͤ ✦

    ϩά෼ੳ ‣ όʔήϯηʔϧ΍4/4ͷεύΠΫͷࣄલରࡦͷجૅࢿྉ औΓ͜΅ͨ͘͠ͳ͍
  15. ϩάج൫ͷઃܭࢥ૝   ✦ ͦΕͳΓʹେྔ݄ؒʙ5# ✦ 410' 4JOHMF1PJOUPG'BJMVSF Λආ͚͍ͨ ✦

    ͳΔ΂͘༗Γ෺Λ࢖͍͍ͨ ✦ ༻్ʹΑͬͯҟͳΔಛੑ ‣ σʔλϨΠΫ ϦΞϧλΠϜϩάͷ૊Έ߹Θͤ
  16. ϩάج൫ͷߏ੒ਤ   Amazon ECS (ίϯςφΫϥελ) Amazon S3 (σʔλϨΠΫ) CloudWatch

    Logs (ϦΞϧλΠϜϩά) Treasure Data (ϩά෼ੳ)
  17. "NB[PO&$4ίϯςφΫϥελ   ✦ %PDLFSίϯςφΛ؅ཧ͢ΔϚωʔδυαʔϏε ‣ ίϯςφΛͲͷΠϯελϯε Ծ૝Ϛγϯ Ͱ࣮ߦ͢Δ ͔ΛΫϥελ͝ͱʹ؅ཧ

    ‣ QSJTNBUJYͰ͸'BSHBUF͸ະ࠾༻ ✦ %PDLFSʹίϯςφϩά ඪ४ग़ྗ Λѻ͏ MPHHJOHESJWFSػೳ͕͋Δ ‣ ίϯςφ͸ϩάϑΝΠϧΛѻΘͣɺϩάΛ+40/ܗࣜͰ ඪ४ग़ྗʹग़͢Α͏ΞϓϦΛߏ੒ ECSΫϥελ
  18. "NB[PO4σʔλϨΠΫ   ✦ ΦϯϥΠϯετϨʔδαʔϏε ‣ ߴ͍଱ٱੑ ‣ ߴ͍εέʔϥϏϦςΟ ༰ྔແ੍ݶ

     ‣ ྿Ձɺબ΂ΔετϨʔδΫϥε ‣ σʔλ෼ੳαʔϏεͱͷ࿈ܞ
  19. "NB[PO$MPVE8BUDI-PHTϦΞϧλΠϜϩά   ✦ ϚωʔδυͷϩάอଘαʔϏε ‣ ߴ͍εέʔϥϏϦςΟ ‣ ४ϦΞϧλΠϜࢀর ‣

    +40/΁ͷΫΤϦΛαϙʔτ
  20. "SN5SFBTVSF%BUBϩά෼ੳ   ✦ σʔλ෼ੳͷ4BB4αʔϏε ඇ"84  ‣ ඇߏ଄Խσʔλʹૉૣ͘ΞΫηεͰ͖Δ
 ετϨʔδ

    1MB[NB%#  ‣ ฒྻΫΤϦΤϯδϯ 1SFTUPͳͲ Ͱ
 ΫΤϦͰ͖Δ
  21. ༗Γ෺ʹ͸ݶք͋Γʢʣ   Amazon ECS Amazon S3 CloudWatch Logs MPHHJOHESJWFSͷ੍໿

    ⭕ $MPVE8BUDI-PHTΛαϙʔτ ❌ 4ʹ͸௚઀޲͚ΒΕͳ͍ ❌ ෳ਺ग़ྗʹະରԠ ◦ ×
  22. ༗Γ෺ʹ͸ݶք͋Γʢʣ   Amazon S3 Treasure Data %BUB$POOFDUPSͱ͍͏5SFBTVSF %BUBͷΠϯϙʔτػೳ͕͋Δ ⭕

    εέδϡʔϧػೳ͕͋Γɺ೔࣍ͷ
 Πϯϙʔτ͸͜ΕͰ0, ❌ Πϯϙʔτ࣌ͷϦιʔε΍ಉ࣮࣌ߦ਺ ʹ੍ݶ͕͋ΓɺॳճΠϯϙʔτʹ͸ن ໛ײ͕߹Θͳ͍ 044ͷ&NCVMLͱ%JHEBH͕தͰ
 ಈ͍͍ͯΔ
  23. ෆ଍Λิ͏ͨΊʹ044Λར༻   Fluentd (ετϦʔϜॲཧ) Embulk (όονॲཧ) ͲͪΒ΋ॊೈͰ๛෋ͳϓϥάΠϯΤίγεςϜ͋Γ ͨ·ͨ·5SFBTVSF%BUB੡Ͱ5%ͱͷߴ͍਌࿨ੑ

  24. %PDLFSº'MVFOUE   %PDLFSMPHHJOHESJWFS͕'MVFOUE΁ͷ ૹ৴Λαϙʔτ ‣ 'MVFOUEࣗମ΋%PDLFSίϯςφͱͯ͠
 &$4ͷ֤ΠϯελϯεͰ࣮ߦ ‣ ϚΠΫϩαʔϏεͷίϯςφ͔Β͸


    ಉΠϯελϯεͷ'MVFOUEʹϩάΛૹ৴ Fluentd
 ίϯςφ
  25. 'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒   4ͱ$MPVE8BUDI-PHT
 ͷ྆ํʹอଘ Amazon S3 CloudWatch Logs <match

    docker.*.*.*> @type copy <store> @type s3 : </store> <store> @type forward : <server> host fluentd.example.lo port 24224 </server> </store> </match>
  26. 'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒   ✦ ϦΞϧλΠϜϩά͸"HHSFHBUPS ू໿αʔόʔ Λ
 ௥Ճͨ͠ଟஈߏ੒ ‣ ͋ͱ͔ΒৼΓઌΛม͑΍͍͢Α͏ʹ

    ‣ $MPVE8BUDI-PHTͷ"1*ίʔϧͷ੍໿ରࡦ CloudWatch Logs Aggregator Forwarder
  27. 'MVFOUEͷσʔλՃ޻   ✦ 'PSXBSEFS ‣ +40/ͷύʔε ‣ ΞϓϦϩάͱ.%$ϩάͷ
 ۠෼͚

    ‣ 4ͷύεϓϨϑΟοΫε ✦ "HHSFHBUPS ‣ Τϥʔͷநग़ ‣ $MPVE8BUDI-PHTύϥϝʔλ <filter docker.**> @type parser format json key_name log </filter> : <match docker.**> @type rewrite_tag_filter <rule> key marker pattern AUDIT tag ${tag}.audit </rule> <rule> key message pattern .+ tag ${tag}.app </rule> </match>
  28. &NCVML   ✦ 4ˠ5%΁ͷॳظόονΠϯϙʔτ ‣ 5%ͷઃఆ͕ͱʹָ͔ͩͬͨ͘ ‣ %BUB$POOFDUPSͷίϯϑΟά͕Ұ෦࢖͍ճͤͨ

  29.   Embulkͷฒྻ࣮ߦ͸ Ͳ͏͢Δʁ

  30. &NCVMLPO"84#BUDI   ✦ "84#BUDIͷδϣϒΩϡʔ
 εϙοτΠϯελϯε׆༻ ‣ δϣϒͷ಺༰͸%PDLFSίϯςφͰ ࣮ߦ͢ΔίϚϯυϥΠϯ ✦

    &NCVMLͷ%PDLFSΠϝʔδΛ
 ༻ҙ࣮ͯ͠ߦ ‣ IUUQTIVCEPDLFSDPNSDMBTTNFUIPE FNCVMLNFUTUE εϙοτϑϦʔτ δϣϒΩϡʔ AWS Batch
  31. &NCVMLͷίϯϑΟά͸؀ڥม਺Λଟ༻   exec: max_threads: {{ env.MAX_THREADS }} in: type:

    s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH } path_match_patterns: \.gz$ auth_method: instance endpoint: s3-ap-northeast-1.amazonaws.com parser: type: jsonl : decoders: - { type: gzip } out: type: td endpoint: api.treasuredata.com apikey: {{ env.TDAPIKEY }} database: {{ env.TDDATABASE }} table: {{ env.TDTABLE }}
  32.   3. ͭ·͍ͮͨ/͍ͯΔ
 ͱ͜Ζ

  33.   Fluentd Aggregatorͷ ৑௕Խ/εέʔϧΞ΢τ CloudWatch Logs Aggregator Forwarder

  34. "HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ   ✦ ෳ਺ͷ"HHSFHBUPSཱ͕ͯΒΕͳ͍ ‣ $MPVE8BUDI-PHTͰ͸ಉҰͷϩάετϦʔϜʹॻ͖ࠐΉ ͱ͖͸TFRVFODFUPLFOΛҡ࣋͠ͳ͚Ε͹ͳΒͳ͍ <match **>

    @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name container_id auto_create_stream true </match> ϚΠΫϩαʔϏε ͷίϯςφ*% ϩάάϧʔϓ ϩά ετϦʔϜ Aggregator ×
  35. "HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ   ✦ ϩάετϦʔϜ໊ʹͩ͜ΘΒͳ͍ ‣ "HHSFHBUPS͝ͱʹϩάετϦʔϜΛׂΓ౰ͯΔ ‣ ʮΠϕϯτͷݕࡧʯ͔ΒετϦʔϜԣஅͷݕࡧΛར༻ <match

    **> @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name "#{Socket.gethostname}" auto_create_stream true </match> "HHSFHBUPS ͷίϯςφ*%
  36. %PDLFSͷϗετϙʔτͷ੍໿   ✦ "HHSFHBUPS΋%PDLFSίϯςφ Ͱ࣮ߦ ‣ ϗετϙʔτ͕ݻఆͰϙʔτ Λ઎༗ ‣

    Πϯελϯεʹίϯςφ͔͠
 ࣮ߦͰ͖ͳ͍ ✦ ϗετϙʔτΛಈతʹ͢ΔͨΊʹ ϩʔυόϥϯαΛ$-#ˠ/-#ʹ
 Ҡߦ     NLB Aggregator ίϯςφ
  37.   TD΁ͷΠϯϙʔτͷδϣϒ෼ׂ

  38. Πϯϙʔτͷδϣϒ෼ׂ   ✦ Πϯϙʔτର৅4ϓϨϑΟοΫεΛ޿͘औΔͱ
 ϝϞϦΊͬͪΌ࢖͏ ‣ %BUB$POOFDUPSϩά͕ফ͑ͯແݶϦτϥΠ˞೥݄౰࣌  ‣

    "84#BUDIϝϞϦׂ౰্ݶ·Ͱ࢖ͬͯ00.,JMMFSൃಈ $ aws s3 ls --profile cm-jp-1 s3://XXXX-infra-logbucket-XXXX/applications
 /condor/app/2018/10/04/05/ 2018-10-04 14:06:24 5964 00_5fbef6f0fdec_0.gz 2018-10-04 14:11:26 5971 05_5fbef6f0fdec_0.gz 2018-10-04 14:16:24 5932 10_5fbef6f0fdec_0.gz 2018-10-04 14:21:25 5954 15_5fbef6f0fdec_0.gz
  39. Πϯϙʔτͷδϣϒ෼ׂ   ✦ ͍Ζ͍Ζࢼͯ͠ɺ೔୯ҐͰδϣϒΛ੾Δͷ͕
 ྑͦ͞͏ͱ͍͏͜ͱʹ ‣ ϑΝΠϧαΠζɺϑΝΠϧ਺ʹґଘ͢ΔͷͰ΍ͬͯΈͳ ͍ͱΘ͔Βͳ͍෦෼ ࠓճ͸ϑΝΠϧ਺͕ଟ͔ͬͨ໛༷

    in: type: s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/ {{ env.MONTH }}/{{ env.DAY }}
  40. 5SFBTVSF%BUBͷ'"2ʹ΋هࡌ͋Γ   2 %BUB$POOFDUPSGPS4KPCJTSVOOJOHGPSBMPOH UJNF XIBUDBO*EP  " $IFDLUIFDPVOUPG4pMFTUIBUZPVSDPOOFDUPS

    KPCJTJOHFTUJOH*GUIFSFBSFPWFS pMFT UIF QFSGPSNBODFEFHSBEFT
 5PNJUJHBUFUIJTJTTVF ZPVDBO ‣ /BSSPXQBUI@QSFpYPQUJPOBOESFEVDFUIFDPVOUPG4pMFT ‣ 4FU   .# UPNJO@UBTL@TJ[FPQUJPO IUUQTTVQQPSUUSFBTVSFEBUBDPNIDFOVTBSUJDMFT%BUB$POOFDUPS GPS"NB[PO4'"2GPSUIF4%BUB$POOFDUPS
  41.   ϩάαΠζͷ্ݶ

  42. ϩάαΠζͷ্ݶ   ✦ .%$ϩά͕σΧ͍ ✦ %PDLFSMPHHJOHESJWFS͸
 ,#ΑΓେ͖͍ϩάΛ෼ׂ͢Δ ‣ 'MVFOUEGPSXBSEFSͷ+40/ύʔεʹࣦഊɺࣺͯΒΕΔ

    ‣ DPODBUϓϥάΠϯͰ݁߹ ‣ ϝϞϦ࢖༻ྔ͕௓Ͷͯ00.,JMMFSʹࡴ͞ΕͨΠϚίί ✦ $MPVE8BUDI-PHTͷ্ݶ͸,#
  43. ͓ۚ   ✦ $MPVE8BUDI-PHT͸ϩάσʔλྔͷैྔ՝ۚ ‣ ྔ͕ଟ͍ͷͰֹ݄අ༻͕͔͞Ή ‣ &$ͳͲଞͷαʔϏεඅ༻ͱτϯτϯͱݴΘΕΔͱͭΒ ͍

    ✦ ΞϓϦ͔Βग़͢ϩάͷ෼ྔΛݮΒ͢ ‣ .%$ϩάͭΒ͍
  44.   4. ࠓޙͷల๬

  45. ࣍ͷҰख   ✦ ϩά؂ࢹ ✦ ϩάू໿ͱͯ͠&MBTUJDTFBSDI,JCBOBͷར༻ ✦ 5SFBTVSF%BUBΛ΋ͬͱ׆༻͍ͨ͠

  46. ·ͱΊ   ✦ εέʔϧ͢Δϩά؅ཧج൫͸Ϋϥ΢υαʔϏεͷ
 ૊Έ߹Θ͕ͤΦεεϝ ‣ σʔλϨΠΫͱϦΞϧλΠϜϩάͰͷαʔϏεͷ࢖͍෼͚ ✦ ૊Έ߹Θ͚ͤͩͰ΋ಈ͘΋ͷ͸Ͱ͖Δ͚Ͳ

    ‣ ཁ݅ʹରͯ͠଍Γͳ͍ͱ͜Ζͷิ͕ؒඞཁ ‣ σʔλྔɺαΠζͷධՁ΍νϡʔχϯά͕ඞཁ