Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra
takipone
October 05, 2018
Technology
7
2.3k
AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra
2018/10/05(金)@UDX秋葉原 Developers.IO 2018のセッション資料
takipone
October 05, 2018
Tweet
Share
More Decks by takipone
See All by takipone
AWSネットワークリソースをサービスでマルチテナントに 提供するときのスケーラビリティ / nwjaws202112-takipone
takipone
1
350
ネットワークがクラウドに載って変わることと変わらないこと /nwjaws201909-takipone
takipone
0
1.5k
日本🇯🇵でAWS Direct Connect を利用する話/akiba-aws-dx
takipone
0
1.9k
CMブートキャンプ(社内勉強会) DNS 第4回 AWSのDNSサービス /cm-dnsstudy-4
takipone
2
17k
AWSクラウドによる ビッグデータ分析、業務システム連携の事例とIoT活用 / aws-data-analytics-iot
takipone
0
550
AWSネットワークのL4以下の話 / aws-network-small-talks
takipone
3
3.1k
AWS IoT プラットフォームの機能・利用事例紹介/aws-iot-platform
takipone
1
1.9k
AWSのラッパー系 サービス事例から見る アプリケーション ライフサイクルの考察/innomeetup201701
takipone
1
1.2k
IoTの実証実験(PoC)を 迅速かつリーズナブルに 進めるためのコツ / cm-iot-seminar-tokyo
takipone
0
1.1k
Other Decks in Technology
See All in Technology
LINEのData Platform室が実践する大規模分散環境のCapacity Planning
line_developers
PRO
0
310
數據的多重宇宙 @ LINE Taiwan
line_developers_tw
PRO
0
620
エンジニアと気軽に繋がれるプラットフォーム「ハッカー飯」で行った セキュリティ・モニタリングに関する取り組みについて
nobuakikikuchi
0
350
How We Foster Reliability in Diversity
nari_ex
PRO
9
2.4k
アルプの 認証/認可分離戦略と手法
ma2k8
PRO
1
280
msal.jsのあれこれ
takas0522
0
1.5k
OSINT/GEOINT ワークショップ 20220514 古橋資料
furuhashilab
2
300
JAWS-UG 朝会 #33 登壇資料
takakuni
0
380
A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion
takmin
0
200
jaws-ug-asa-datasync-20220510
hiashisan
0
470
srenext2022-skaru
mixi_engineers
0
450
ITエンジニアを取り巻く環境とキャリアパス / A career path for Japanese IT engineers
takatama
0
580
Featured
See All Featured
How STYLIGHT went responsive
nonsquared
85
3.9k
Web development in the modern age
philhawksworth
197
9.3k
How GitHub (no longer) Works
holman
296
140k
The Most Common Mistakes in Cover Letters
jrick
PRO
4
24k
Automating Front-end Workflow
addyosmani
1351
200k
The Illustrated Children's Guide to Kubernetes
chrisshort
14
34k
How to train your dragon (web standard)
notwaldorf
57
3.8k
Code Reviewing Like a Champion
maltzj
506
37k
In The Pink: A Labor of Love
frogandcode
130
21k
Writing Fast Ruby
sferik
612
57k
A Tale of Four Properties
chriscoyier
149
20k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
655
120k
Transcript
"84ωΠςΟϒͳ&$$3.γεςϜ ӡ༻ʹ͔ܽͤͳ͍ϩάج൫ߏங େ୍ོଠ
DNEFWJP
ࣗݾհ /BNF େ୍ོଠ!UBLJQPOF +PC 43&!QSJTNBUJY *OUFSFTU ωοτϫʔΫσϓϩΠपΓ 'BWPSJUF
"NB[PO3PVUFͱ"$.
ΞδΣϯμ "84ωΠςΟϒͳ&$$3.γεςϜ QSJTNBUJY ͱ ϩάੳج൫ͷઃܭ
ͭ·͍͍ͮͨͯΔͱ͜Ζ ࠓޙͷల
1. AWSωΠςΟϒͳ EC/CRMγεςϜ prismatixͱ
QSJTNBUJY ҎԼ1[ ɺ&$ͱ$3.γεςϜΛϚΠΫϩαʔϏεͱͯ͠ ఏڙ͠ݸผʹಋೖՄೳͰ͢ɻ֤γεςϜͷػೳʹ"1*Λ׆༻͢Δ͜ͱͰ γεςϜ։ൃͱӡ༻ͷෳࡶੑɺίετɺ࣌ؒΛݮ͠·͢ɻ
ߏྫ prismatix (API) ECαʔόʔ εϚϗΞϓϦ ECαΠτ
ࣄྫύϧί༷
"84ωΠςΟϒͳΠϯϑϥߏ ϩʔυόϥϯα (ෛՙࢄ/TLSऴ) ίϯςφΫϥελ (APαʔόʔ) σʔλϕʔε (ϚωʔδυαʔϏε)
ϚΠΫϩαʔϏεؒͷ࿈ܞ ඇಉظૄ݁߹ͷ͘͠Έ αʔϏε Amazon SNS (௨) Amazon SQS
(δϣϒΩϡʔ) ࡏݿαʔϏε (ϫʔΧʔ) ొ Πϕϯτૹ৴ Πϕϯτ͕ Ωϡʔʹཷ·Δ ΠϕϯτΛऔಘͯ͠ ࡏݿσʔλʹө ΠϕϯτΛసૹ
ϚΠΫϩαʔϏεؒͷ࿈ܞ αʔϏεͷϦιʔε૿ݮ εέʔϧΠϯΞτ ͍͢͠ αʔϏεΛ૿ڧ ࡏݿαʔϏεΛ૿ڧ
2. ϩάੳج൫ͷઃܭ
QSJTNBUJYͷϩά ओʹछྨɺ͍ͣΕ+40/ܗࣜ ✦ ΞϓϦϩά 4QSJOH +BWB ىಈ࣌Τϥʔͷϝοηʔδ
✦ .%$ϩά .BQQFE%JBHOPTUJD$POUFYUT ϚΠΫϩαʔϏεͷϦΫΤετͱϨεϙϯεͷҰ෦ ͋ͱεϩʔΫΤϦϩάͳͲ
ϩάͷ༻్ ✦ τϥϒϧγϡʔςΟϯά ‣ ϚΠΫϩαʔϏεͷಈ࡞֬ೝ ‣ ݕࡧจͱͷಥ͖߹Θͤ ✦
ϩάੳ ‣ όʔήϯηʔϧ4/4ͷεύΠΫͷࣄલରࡦͷجૅࢿྉ औΓ͜΅ͨ͘͠ͳ͍
ϩάج൫ͷઃܭࢥ ✦ ͦΕͳΓʹେྔ݄ؒʙ5# ✦ 410' 4JOHMF1PJOUPG'BJMVSF Λආ͚͍ͨ ✦
ͳΔ͘༗ΓΛ͍͍ͨ ✦ ༻్ʹΑͬͯҟͳΔಛੑ ‣ σʔλϨΠΫ ϦΞϧλΠϜϩάͷΈ߹Θͤ
ϩάج൫ͷߏਤ Amazon ECS (ίϯςφΫϥελ) Amazon S3 (σʔλϨΠΫ) CloudWatch
Logs (ϦΞϧλΠϜϩά) Treasure Data (ϩάੳ)
"NB[PO&$4ίϯςφΫϥελ ✦ %PDLFSίϯςφΛཧ͢ΔϚωʔδυαʔϏε ‣ ίϯςφΛͲͷΠϯελϯε ԾϚγϯ Ͱ࣮ߦ͢Δ ͔ΛΫϥελ͝ͱʹཧ
‣ QSJTNBUJYͰ'BSHBUFະ࠾༻ ✦ %PDLFSʹίϯςφϩά ඪ४ग़ྗ Λѻ͏ MPHHJOHESJWFSػೳ͕͋Δ ‣ ίϯςφϩάϑΝΠϧΛѻΘͣɺϩάΛ+40/ܗࣜͰ ඪ४ग़ྗʹग़͢Α͏ΞϓϦΛߏ ECSΫϥελ
"NB[PO4σʔλϨΠΫ ✦ ΦϯϥΠϯετϨʔδαʔϏε ‣ ߴ͍ٱੑ ‣ ߴ͍εέʔϥϏϦςΟ ༰ྔແ੍ݶ
‣ ྿ՁɺબΔετϨʔδΫϥε ‣ σʔλੳαʔϏεͱͷ࿈ܞ
"NB[PO$MPVE8BUDI-PHTϦΞϧλΠϜϩά ✦ ϚωʔδυͷϩάอଘαʔϏε ‣ ߴ͍εέʔϥϏϦςΟ ‣ ४ϦΞϧλΠϜࢀর ‣
+40/ͷΫΤϦΛαϙʔτ
"SN5SFBTVSF%BUBϩάੳ ✦ σʔλੳͷ4BB4αʔϏε ඇ"84 ‣ ඇߏԽσʔλʹૉૣ͘ΞΫηεͰ͖Δ ετϨʔδ
1MB[NB%# ‣ ฒྻΫΤϦΤϯδϯ 1SFTUPͳͲ Ͱ ΫΤϦͰ͖Δ
༗Γʹݶք͋Γʢʣ Amazon ECS Amazon S3 CloudWatch Logs MPHHJOHESJWFSͷ੍
⭕ $MPVE8BUDI-PHTΛαϙʔτ ❌ 4ʹ͚ΒΕͳ͍ ❌ ෳग़ྗʹະରԠ ◦ ×
༗Γʹݶք͋Γʢʣ Amazon S3 Treasure Data %BUB$POOFDUPSͱ͍͏5SFBTVSF %BUBͷΠϯϙʔτػೳ͕͋Δ ⭕
εέδϡʔϧػೳ͕͋Γɺ࣍ͷ Πϯϙʔτ͜ΕͰ0, ❌ Πϯϙʔτ࣌ͷϦιʔεಉ࣮࣌ߦ ʹ੍ݶ͕͋ΓɺॳճΠϯϙʔτʹن ײ͕߹Θͳ͍ 044ͷ&NCVMLͱ%JHEBH͕தͰ ಈ͍͍ͯΔ
ෆΛิ͏ͨΊʹ044Λར༻ Fluentd (ετϦʔϜॲཧ) Embulk (όονॲཧ) ͲͪΒॊೈͰ๛ͳϓϥάΠϯΤίγεςϜ͋Γ ͨ·ͨ·5SFBTVSF%BUBͰ5%ͱͷߴ͍ੑ
%PDLFSº'MVFOUE %PDLFSMPHHJOHESJWFS͕'MVFOUEͷ ૹ৴Λαϙʔτ ‣ 'MVFOUEࣗମ%PDLFSίϯςφͱͯ͠ &$4ͷ֤ΠϯελϯεͰ࣮ߦ ‣ ϚΠΫϩαʔϏεͷίϯςφ͔Β
ಉΠϯελϯεͷ'MVFOUEʹϩάΛૹ৴ Fluentd ίϯςφ
'MVFOUEͷෳ0VUQVUͱଟஈߏ 4ͱ$MPVE8BUDI-PHT ͷ྆ํʹอଘ Amazon S3 CloudWatch Logs <match
docker.*.*.*> @type copy <store> @type s3 : </store> <store> @type forward : <server> host fluentd.example.lo port 24224 </server> </store> </match>
'MVFOUEͷෳ0VUQVUͱଟஈߏ ✦ ϦΞϧλΠϜϩά"HHSFHBUPS ूαʔόʔ Λ Ճͨ͠ଟஈߏ ‣ ͋ͱ͔ΒৼΓઌΛม͍͑͢Α͏ʹ
‣ $MPVE8BUDI-PHTͷ"1*ίʔϧͷ੍ରࡦ CloudWatch Logs Aggregator Forwarder
'MVFOUEͷσʔλՃ ✦ 'PSXBSEFS ‣ +40/ͷύʔε ‣ ΞϓϦϩάͱ.%$ϩάͷ ͚۠
‣ 4ͷύεϓϨϑΟοΫε ✦ "HHSFHBUPS ‣ Τϥʔͷநग़ ‣ $MPVE8BUDI-PHTύϥϝʔλ <filter docker.**> @type parser format json key_name log </filter> : <match docker.**> @type rewrite_tag_filter <rule> key marker pattern AUDIT tag ${tag}.audit </rule> <rule> key message pattern .+ tag ${tag}.app </rule> </match>
&NCVML ✦ 4ˠ5%ͷॳظόονΠϯϙʔτ ‣ 5%ͷઃఆ͕ͱʹָ͔ͩͬͨ͘ ‣ %BUB$POOFDUPSͷίϯϑΟά͕Ұ෦͍ճͤͨ
Embulkͷฒྻ࣮ߦ Ͳ͏͢Δʁ
&NCVMLPO"84#BUDI ✦ "84#BUDIͷδϣϒΩϡʔ εϙοτΠϯελϯε׆༻ ‣ δϣϒͷ༰%PDLFSίϯςφͰ ࣮ߦ͢ΔίϚϯυϥΠϯ ✦
&NCVMLͷ%PDLFSΠϝʔδΛ ༻ҙ࣮ͯ͠ߦ ‣ IUUQTIVCEPDLFSDPNSDMBTTNFUIPE FNCVMLNFUTUE εϙοτϑϦʔτ δϣϒΩϡʔ AWS Batch
&NCVMLͷίϯϑΟάڥมΛଟ༻ exec: max_threads: {{ env.MAX_THREADS }} in: type:
s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH } path_match_patterns: \.gz$ auth_method: instance endpoint: s3-ap-northeast-1.amazonaws.com parser: type: jsonl : decoders: - { type: gzip } out: type: td endpoint: api.treasuredata.com apikey: {{ env.TDAPIKEY }} database: {{ env.TDDATABASE }} table: {{ env.TDTABLE }}
3. ͭ·͍ͮͨ/͍ͯΔ ͱ͜Ζ
Fluentd Aggregatorͷ Խ/εέʔϧΞτ CloudWatch Logs Aggregator Forwarder
"HHSFHBUPSͷԽεέʔϧΞτ ✦ ෳͷ"HHSFHBUPSཱ͕ͯΒΕͳ͍ ‣ $MPVE8BUDI-PHTͰಉҰͷϩάετϦʔϜʹॻ͖ࠐΉ ͱ͖TFRVFODFUPLFOΛҡ࣋͠ͳ͚ΕͳΒͳ͍ <match **>
@type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name container_id auto_create_stream true </match> ϚΠΫϩαʔϏε ͷίϯςφ*% ϩάάϧʔϓ ϩά ετϦʔϜ Aggregator ×
"HHSFHBUPSͷԽεέʔϧΞτ ✦ ϩάετϦʔϜ໊ʹͩ͜ΘΒͳ͍ ‣ "HHSFHBUPS͝ͱʹϩάετϦʔϜΛׂΓͯΔ ‣ ʮΠϕϯτͷݕࡧʯ͔ΒετϦʔϜԣஅͷݕࡧΛར༻ <match
**> @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name "#{Socket.gethostname}" auto_create_stream true </match> "HHSFHBUPS ͷίϯςφ*%
%PDLFSͷϗετϙʔτͷ੍ ✦ "HHSFHBUPS%PDLFSίϯςφ Ͱ࣮ߦ ‣ ϗετϙʔτ͕ݻఆͰϙʔτ Λ༗ ‣
Πϯελϯεʹίϯςφ͔͠ ࣮ߦͰ͖ͳ͍ ✦ ϗετϙʔτΛಈతʹ͢ΔͨΊʹ ϩʔυόϥϯαΛ$-#ˠ/-#ʹ Ҡߦ NLB Aggregator ίϯςφ
TDͷΠϯϙʔτͷδϣϒׂ
Πϯϙʔτͷδϣϒׂ ✦ Πϯϙʔτର4ϓϨϑΟοΫεΛ͘औΔͱ ϝϞϦΊͬͪΌ͏ ‣ %BUB$POOFDUPSϩά͕ফ͑ͯແݶϦτϥΠ˞݄࣌ ‣
"84#BUDIϝϞϦ্ׂݶ·Ͱͬͯ00.,JMMFSൃಈ $ aws s3 ls --profile cm-jp-1 s3://XXXX-infra-logbucket-XXXX/applications /condor/app/2018/10/04/05/ 2018-10-04 14:06:24 5964 00_5fbef6f0fdec_0.gz 2018-10-04 14:11:26 5971 05_5fbef6f0fdec_0.gz 2018-10-04 14:16:24 5932 10_5fbef6f0fdec_0.gz 2018-10-04 14:21:25 5954 15_5fbef6f0fdec_0.gz
Πϯϙʔτͷδϣϒׂ ✦ ͍Ζ͍Ζࢼͯ͠ɺ୯ҐͰδϣϒΛΔͷ͕ ྑͦ͞͏ͱ͍͏͜ͱʹ ‣ ϑΝΠϧαΠζɺϑΝΠϧʹґଘ͢ΔͷͰͬͯΈͳ ͍ͱΘ͔Βͳ͍෦ ࠓճϑΝΠϧ͕ଟ͔༷ͬͨ
in: type: s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/ {{ env.MONTH }}/{{ env.DAY }}
5SFBTVSF%BUBͷ'"2ʹهࡌ͋Γ 2 %BUB$POOFDUPSGPS4KPCJTSVOOJOHGPSBMPOH UJNF XIBUDBO*EP " $IFDLUIFDPVOUPG4pMFTUIBUZPVSDPOOFDUPS
KPCJTJOHFTUJOH*GUIFSFBSFPWFS pMFT UIF QFSGPSNBODFEFHSBEFT 5PNJUJHBUFUIJTJTTVF ZPVDBO ‣ /BSSPXQBUI@QSFpYPQUJPOBOESFEVDFUIFDPVOUPG4pMFT ‣ 4FU .# UPNJO@UBTL@TJ[FPQUJPO IUUQTTVQQPSUUSFBTVSFEBUBDPNIDFOVTBSUJDMFT%BUB$POOFDUPS GPS"NB[PO4'"2GPSUIF4%BUB$POOFDUPS
ϩάαΠζͷ্ݶ
ϩάαΠζͷ্ݶ ✦ .%$ϩά͕σΧ͍ ✦ %PDLFSMPHHJOHESJWFS ,#ΑΓେ͖͍ϩάΛׂ͢Δ ‣ 'MVFOUEGPSXBSEFSͷ+40/ύʔεʹࣦഊɺࣺͯΒΕΔ
‣ DPODBUϓϥάΠϯͰ݁߹ ‣ ϝϞϦ༻ྔ͕Ͷͯ00.,JMMFSʹࡴ͞ΕͨΠϚίί ✦ $MPVE8BUDI-PHTͷ্ݶ,#
͓ۚ ✦ $MPVE8BUDI-PHTϩάσʔλྔͷैྔ՝ۚ ‣ ྔ͕ଟ͍ͷͰֹ݄අ༻͕͔͞Ή ‣ &$ͳͲଞͷαʔϏεඅ༻ͱτϯτϯͱݴΘΕΔͱͭΒ ͍
✦ ΞϓϦ͔Βग़͢ϩάͷྔΛݮΒ͢ ‣ .%$ϩάͭΒ͍
4. ࠓޙͷల
࣍ͷҰख ✦ ϩάࢹ ✦ ϩάूͱͯ͠&MBTUJDTFBSDI,JCBOBͷར༻ ✦ 5SFBTVSF%BUBΛͬͱ׆༻͍ͨ͠
·ͱΊ ✦ εέʔϧ͢Δϩάཧج൫ΫϥυαʔϏεͷ Έ߹Θ͕ͤΦεεϝ ‣ σʔλϨΠΫͱϦΞϧλΠϜϩάͰͷαʔϏεͷ͍͚ ✦ Έ߹Θ͚ͤͩͰಈ͘ͷͰ͖Δ͚Ͳ
‣ ཁ݅ʹରͯ͠Γͳ͍ͱ͜Ζͷิ͕ؒඞཁ ‣ σʔλྔɺαΠζͷධՁνϡʔχϯά͕ඞཁ