AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

"84ωΠςΟϒͳ&$$3.γεςϜ ӡ༻ʹ͔ܽͤͳ͍ϩάج൫ߏங େ୍ོଠ

DNEFWJP

ࣗݾ঺հ /BNF େ୍ོଠ!UBLJQPOF +PC 43&!QSJTNBUJY *OUFSFTU ωοτϫʔΫσϓϩΠपΓ 'BWPSJUF
"NB[PO3PVUFͱ"$.

ΞδΣϯμ "84ωΠςΟϒͳ&$$3.γεςϜ QSJTNBUJY ͱ͸ ϩά෼ੳج൫ͷઃܭ
ͭ·͍͍ͮͨͯΔͱ͜Ζ ࠓޙͷల๬

1. AWSωΠςΟϒͳ  EC/CRMγεςϜ prismatixͱ͸

QSJTNBUJY ҎԼ1[ ͸ɺ&$ͱ$3.γεςϜΛϚΠΫϩαʔϏεͱͯ͠  ఏڙ͠ݸผʹಋೖՄೳͰ͢ɻ֤γεςϜͷػೳʹ"1*Λ׆༻͢Δ͜ͱͰ  γεςϜ։ൃͱӡ༻ͷෳࡶੑɺίετɺ࣌ؒΛ࡟ݮ͠·͢ɻ

ߏ੒ྫ prismatix (API) ECαʔόʔ εϚϗΞϓϦ ECαΠτ

ࣄྫύϧί༷

"84ωΠςΟϒͳΠϯϑϥߏ੒ ϩʔυόϥϯα (ෛՙ෼ࢄ/TLSऴ୺) ίϯςφΫϥελ (APαʔόʔ) σʔλϕʔε (ϚωʔδυαʔϏε)

ϚΠΫϩαʔϏεؒͷ࿈ܞ ඇಉظૄ݁߹ͷ͘͠Έ ঎඼αʔϏε Amazon SNS (௨஌) Amazon SQS
(δϣϒΩϡʔ) ࡏݿαʔϏε (ϫʔΧʔ) ঎඼ొ࿥ Πϕϯτૹ৴ Πϕϯτ͕ Ωϡʔʹཷ·Δ ΠϕϯτΛऔಘͯ͠  ࡏݿσʔλʹ൓ө ΠϕϯτΛసૹ

ϚΠΫϩαʔϏεؒͷ࿈ܞ αʔϏεͷϦιʔε૿ݮ εέʔϧΠϯΞ΢τ ͠΍͍͢ ঎඼αʔϏεΛ૿ڧ ࡏݿαʔϏεΛ૿ڧ

2. ϩά෼ੳج൫ͷઃܭ

QSJTNBUJYͷϩά ओʹछྨɺ͍ͣΕ΋+40/ܗࣜ ✦ ΞϓϦϩά 4QSJOH +BWB ىಈ࣌΍Τϥʔͷϝοηʔδ
✦ .%$ϩά .BQQFE%JBHOPTUJD$POUFYUT ϚΠΫϩαʔϏεͷϦΫΤετͱϨεϙϯεͷҰ෦ ͋ͱεϩʔΫΤϦϩάͳͲ

ϩάͷ༻్ ✦ τϥϒϧγϡʔςΟϯά ‣ ϚΠΫϩαʔϏεͷಈ࡞֬ೝ ‣ ঎඼ݕࡧ΍஫จͱͷಥ͖߹Θͤ ✦
ϩά෼ੳ ‣ όʔήϯηʔϧ΍4/4ͷεύΠΫͷࣄલରࡦͷجૅࢿྉ औΓ͜΅ͨ͘͠ͳ͍

ϩάج൫ͷઃܭࢥ૝ ✦ ͦΕͳΓʹେྔ݄ؒʙ5# ✦ 410' 4JOHMF1PJOUPG'BJMVSF Λආ͚͍ͨ ✦
ͳΔ΂͘༗Γ෺Λ࢖͍͍ͨ ✦ ༻్ʹΑͬͯҟͳΔಛੑ ‣ σʔλϨΠΫ ϦΞϧλΠϜϩάͷ૊Έ߹Θͤ

ϩάج൫ͷߏ੒ਤ Amazon ECS (ίϯςφΫϥελ) Amazon S3 (σʔλϨΠΫ) CloudWatch
Logs (ϦΞϧλΠϜϩά) Treasure Data (ϩά෼ੳ)

"NB[PO&$4ίϯςφΫϥελ ✦ %PDLFSίϯςφΛ؅ཧ͢ΔϚωʔδυαʔϏε ‣ ίϯςφΛͲͷΠϯελϯε Ծ૝Ϛγϯ Ͱ࣮ߦ͢Δ ͔ΛΫϥελ͝ͱʹ؅ཧ
‣ QSJTNBUJYͰ͸'BSHBUF͸ະ࠾༻ ✦ %PDLFSʹίϯςφϩά ඪ४ग़ྗ Λѻ͏ MPHHJOHESJWFSػೳ͕͋Δ ‣ ίϯςφ͸ϩάϑΝΠϧΛѻΘͣɺϩάΛ+40/ܗࣜͰ ඪ४ग़ྗʹग़͢Α͏ΞϓϦΛߏ੒ ECSΫϥελ

"NB[PO4σʔλϨΠΫ ✦ ΦϯϥΠϯετϨʔδαʔϏε ‣ ߴ͍଱ٱੑ ‣ ߴ͍εέʔϥϏϦςΟ ༰ྔແ੍ݶ
‣ ྿Ձɺબ΂ΔετϨʔδΫϥε ‣ σʔλ෼ੳαʔϏεͱͷ࿈ܞ

"NB[PO$MPVE8BUDI-PHTϦΞϧλΠϜϩά ✦ ϚωʔδυͷϩάอଘαʔϏε ‣ ߴ͍εέʔϥϏϦςΟ ‣ ४ϦΞϧλΠϜࢀর ‣
+40/΁ͷΫΤϦΛαϙʔτ

"SN5SFBTVSF%BUBϩά෼ੳ ✦ σʔλ෼ੳͷ4BB4αʔϏε ඇ"84 ‣ ඇߏ଄Խσʔλʹૉૣ͘ΞΫηεͰ͖Δ  ετϨʔδ
1MB[NB%# ‣ ฒྻΫΤϦΤϯδϯ 1SFTUPͳͲ Ͱ  ΫΤϦͰ͖Δ

༗Γ෺ʹ͸ݶք͋Γʢʣ Amazon ECS Amazon S3 CloudWatch Logs MPHHJOHESJWFSͷ੍໿
⭕ $MPVE8BUDI-PHTΛαϙʔτ ❌ 4ʹ͸௚઀޲͚ΒΕͳ͍ ❌ ෳ਺ग़ྗʹະରԠ ◦ ×

༗Γ෺ʹ͸ݶք͋Γʢʣ Amazon S3 Treasure Data %BUB$POOFDUPSͱ͍͏5SFBTVSF %BUBͷΠϯϙʔτػೳ͕͋Δ ⭕
εέδϡʔϧػೳ͕͋Γɺ೔࣍ͷ  Πϯϙʔτ͸͜ΕͰ0, ❌ Πϯϙʔτ࣌ͷϦιʔε΍ಉ࣮࣌ߦ਺ ʹ੍ݶ͕͋ΓɺॳճΠϯϙʔτʹ͸ن ໛ײ͕߹Θͳ͍ 044ͷ&NCVMLͱ%JHEBH͕தͰ  ಈ͍͍ͯΔ

ෆ଍Λิ͏ͨΊʹ044Λར༻ Fluentd (ετϦʔϜॲཧ) Embulk (όονॲཧ) ͲͪΒ΋ॊೈͰ๛෋ͳϓϥάΠϯΤίγεςϜ͋Γ ͨ·ͨ·5SFBTVSF%BUB੡Ͱ5%ͱͷߴ͍਌࿨ੑ

%PDLFSº'MVFOUE %PDLFSMPHHJOHESJWFS͕'MVFOUE΁ͷ ૹ৴Λαϙʔτ ‣ 'MVFOUEࣗମ΋%PDLFSίϯςφͱͯ͠  &$4ͷ֤ΠϯελϯεͰ࣮ߦ ‣ ϚΠΫϩαʔϏεͷίϯςφ͔Β͸ 
ಉΠϯελϯεͷ'MVFOUEʹϩάΛૹ৴ Fluentd  ίϯςφ

'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒ 4ͱ$MPVE8BUDI-PHT  ͷ྆ํʹอଘ Amazon S3 CloudWatch Logs <match
docker.*.*.*> @type copy <store> @type s3 : </store> <store> @type forward : <server> host fluentd.example.lo port 24224 </server> </store> </match>

'MVFOUEͷෳ਺0VUQVUͱଟஈߏ੒ ✦ ϦΞϧλΠϜϩά͸"HHSFHBUPS ू໿αʔόʔ Λ  ௥Ճͨ͠ଟஈߏ੒ ‣ ͋ͱ͔ΒৼΓઌΛม͑΍͍͢Α͏ʹ
‣ $MPVE8BUDI-PHTͷ"1*ίʔϧͷ੍໿ରࡦ CloudWatch Logs Aggregator Forwarder

'MVFOUEͷσʔλՃ޻ ✦ 'PSXBSEFS ‣ +40/ͷύʔε ‣ ΞϓϦϩάͱ.%$ϩάͷ  ۠෼͚
‣ 4ͷύεϓϨϑΟοΫε ✦ "HHSFHBUPS ‣ Τϥʔͷநग़ ‣ $MPVE8BUDI-PHTύϥϝʔλ <filter docker.**> @type parser format json key_name log </filter> : <match docker.**> @type rewrite_tag_filter <rule> key marker pattern AUDIT tag ${tag}.audit </rule> <rule> key message pattern .+ tag ${tag}.app </rule> </match>

&NCVML ✦ 4ˠ5%΁ͷॳظόονΠϯϙʔτ ‣ 5%ͷઃఆ͕ͱʹָ͔ͩͬͨ͘ ‣ %BUB$POOFDUPSͷίϯϑΟά͕Ұ෦࢖͍ճͤͨ

Embulkͷฒྻ࣮ߦ͸ Ͳ͏͢Δʁ

&NCVMLPO"84#BUDI ✦ "84#BUDIͷδϣϒΩϡʔ  εϙοτΠϯελϯε׆༻ ‣ δϣϒͷ಺༰͸%PDLFSίϯςφͰ ࣮ߦ͢ΔίϚϯυϥΠϯ ✦
&NCVMLͷ%PDLFSΠϝʔδΛ  ༻ҙ࣮ͯ͠ߦ ‣ IUUQTIVCEPDLFSDPNSDMBTTNFUIPE FNCVMLNFUTUE εϙοτϑϦʔτ δϣϒΩϡʔ AWS Batch

&NCVMLͷίϯϑΟά͸؀ڥม਺Λଟ༻ exec: max_threads: {{ env.MAX_THREADS }} in: type:
s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH } path_match_patterns: \.gz$ auth_method: instance endpoint: s3-ap-northeast-1.amazonaws.com parser: type: jsonl : decoders: - { type: gzip } out: type: td endpoint: api.treasuredata.com apikey: {{ env.TDAPIKEY }} database: {{ env.TDDATABASE }} table: {{ env.TDTABLE }}

3. ͭ·͍ͮͨ/͍ͯΔ  ͱ͜Ζ

Fluentd Aggregatorͷ ৑௕Խ/εέʔϧΞ΢τ CloudWatch Logs Aggregator Forwarder

"HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ ✦ ෳ਺ͷ"HHSFHBUPSཱ͕ͯΒΕͳ͍ ‣ $MPVE8BUDI-PHTͰ͸ಉҰͷϩάετϦʔϜʹॻ͖ࠐΉ ͱ͖͸TFRVFODFUPLFOΛҡ࣋͠ͳ͚Ε͹ͳΒͳ͍ <match **>
@type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name container_id auto_create_stream true </match> ϚΠΫϩαʔϏε ͷίϯςφ*% ϩάάϧʔϓ ϩά ετϦʔϜ Aggregator ×

"HHSFHBUPSͷ৑௕ԽεέʔϧΞ΢τ ✦ ϩάετϦʔϜ໊ʹͩ͜ΘΒͳ͍ ‣ "HHSFHBUPS͝ͱʹϩάετϦʔϜΛׂΓ౰ͯΔ ‣ ʮΠϕϯτͷݕࡧʯ͔ΒετϦʔϜԣஅͷݕࡧΛར༻ <match
**> @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name "#{Socket.gethostname}" auto_create_stream true </match> "HHSFHBUPS ͷίϯςφ*%

%PDLFSͷϗετϙʔτͷ੍໿ ✦ "HHSFHBUPS΋%PDLFSίϯςφ Ͱ࣮ߦ ‣ ϗετϙʔτ͕ݻఆͰϙʔτ Λ઎༗ ‣
Πϯελϯεʹίϯςφ͔͠  ࣮ߦͰ͖ͳ͍ ✦ ϗετϙʔτΛಈతʹ͢ΔͨΊʹ ϩʔυόϥϯαΛ$-#ˠ/-#ʹ  Ҡߦ NLB Aggregator ίϯςφ

TD΁ͷΠϯϙʔτͷδϣϒ෼ׂ

Πϯϙʔτͷδϣϒ෼ׂ ✦ Πϯϙʔτର৅4ϓϨϑΟοΫεΛ޿͘औΔͱ  ϝϞϦΊͬͪΌ࢖͏ ‣ %BUB$POOFDUPSϩά͕ফ͑ͯແݶϦτϥΠ˞೥݄౰࣌ ‣
"84#BUDIϝϞϦׂ౰্ݶ·Ͱ࢖ͬͯ00.,JMMFSൃಈ $ aws s3 ls --profile cm-jp-1 s3://XXXX-infra-logbucket-XXXX/applications  /condor/app/2018/10/04/05/ 2018-10-04 14:06:24 5964 00_5fbef6f0fdec_0.gz 2018-10-04 14:11:26 5971 05_5fbef6f0fdec_0.gz 2018-10-04 14:16:24 5932 10_5fbef6f0fdec_0.gz 2018-10-04 14:21:25 5954 15_5fbef6f0fdec_0.gz

Πϯϙʔτͷδϣϒ෼ׂ ✦ ͍Ζ͍Ζࢼͯ͠ɺ೔୯ҐͰδϣϒΛ੾Δͷ͕  ྑͦ͞͏ͱ͍͏͜ͱʹ ‣ ϑΝΠϧαΠζɺϑΝΠϧ਺ʹґଘ͢ΔͷͰ΍ͬͯΈͳ ͍ͱΘ͔Βͳ͍෦෼ ࠓճ͸ϑΝΠϧ਺͕ଟ͔ͬͨ໛༷
in: type: s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/ {{ env.MONTH }}/{{ env.DAY }}

5SFBTVSF%BUBͷ'"2ʹ΋هࡌ͋Γ 2 %BUB$POOFDUPSGPS4KPCJTSVOOJOHGPSBMPOH UJNF XIBUDBO*EP " $IFDLUIFDPVOUPG4pMFTUIBUZPVSDPOOFDUPS
KPCJTJOHFTUJOH*GUIFSFBSFPWFS pMFT UIF QFSGPSNBODFEFHSBEFT  5PNJUJHBUFUIJTJTTVF ZPVDBO ‣ /BSSPXQBUI@QSFpYPQUJPOBOESFEVDFUIFDPVOUPG4pMFT ‣ 4FU .# UPNJO@UBTL@TJ[FPQUJPO IUUQTTVQQPSUUSFBTVSFEBUBDPNIDFOVTBSUJDMFT%BUB$POOFDUPS GPS"NB[PO4'"2GPSUIF4%BUB$POOFDUPS

ϩάαΠζͷ্ݶ

ϩάαΠζͷ্ݶ ✦ .%$ϩά͕σΧ͍ ✦ %PDLFSMPHHJOHESJWFS͸  ,#ΑΓେ͖͍ϩάΛ෼ׂ͢Δ ‣ 'MVFOUEGPSXBSEFSͷ+40/ύʔεʹࣦഊɺࣺͯΒΕΔ
‣ DPODBUϓϥάΠϯͰ݁߹ ‣ ϝϞϦ࢖༻ྔ͕௓Ͷͯ00.,JMMFSʹࡴ͞ΕͨΠϚίί ✦ $MPVE8BUDI-PHTͷ্ݶ͸,#

͓ۚ ✦ $MPVE8BUDI-PHT͸ϩάσʔλྔͷैྔ՝ۚ ‣ ྔ͕ଟ͍ͷͰֹ݄අ༻͕͔͞Ή ‣ &$ͳͲଞͷαʔϏεඅ༻ͱτϯτϯͱݴΘΕΔͱͭΒ ͍
✦ ΞϓϦ͔Βग़͢ϩάͷ෼ྔΛݮΒ͢ ‣ .%$ϩάͭΒ͍

4. ࠓޙͷల๬

࣍ͷҰख ✦ ϩά؂ࢹ ✦ ϩάू໿ͱͯ͠&MBTUJDTFBSDI,JCBOBͷར༻ ✦ 5SFBTVSF%BUBΛ΋ͬͱ׆༻͍ͨ͠

·ͱΊ ✦ εέʔϧ͢Δϩά؅ཧج൫͸Ϋϥ΢υαʔϏεͷ  ૊Έ߹Θ͕ͤΦεεϝ ‣ σʔλϨΠΫͱϦΞϧλΠϜϩάͰͷαʔϏεͷ࢖͍෼͚ ✦ ૊Έ߹Θ͚ͤͩͰ΋ಈ͘΋ͷ͸Ͱ͖Δ͚Ͳ
‣ ཁ݅ʹରͯ͠଍Γͳ͍ͱ͜Ζͷิ͕ؒඞཁ ‣ σʔλྔɺαΠζͷධՁ΍νϡʔχϯά͕ඞཁ

AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra

More Decks by takipone

Other Decks in Technology

Featured

Transcript