Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018...
Search
takipone
October 05, 2018
Technology
7
3.1k
AWSネイティブなEC/CRMシステム運用に欠かせないログ基盤構築 / cmdevio2018-aws-log-infra
2018/10/05(金)@UDX秋葉原 Developers.IO 2018のセッション資料
takipone
October 05, 2018
Tweet
Share
More Decks by takipone
See All by takipone
OJTに夢を見すぎていませんか? ロールプレイ研修の試行錯誤/tryanderror-in-roleplaying-training
takipone
1
160
ロールプレイトレーニングでクラウドエンジニアを短期育成した知見を一挙公開 / roll playing training benefits and cases
takipone
0
630
スキル育成に生かすLlamaIndexの活用 /using-llamaIndex-to-enhance-ones-skills
takipone
1
3.4k
DNSにちょっとだけ詳しくなりたい人に贈る少し突っ込んだDNSの話 / devioday1_dns
takipone
2
11k
Amazon VPC Latticeに期待する / look-for-vpc-lattice
takipone
0
1.1k
DevelopersIOのアウトプット文化をハックして人材育成に繋げる話 /deviosapporo2022fall_takipone
takipone
0
960
AWSネットワークリソースをサービスでマルチテナントに 提供するときのスケーラビリティ / nwjaws202112-takipone
takipone
1
740
ネットワークがクラウドに載って変わることと変わらないこと /nwjaws201909-takipone
takipone
0
2.1k
日本🇯🇵でAWS Direct Connect を利用する話/akiba-aws-dx
takipone
1
2.4k
Other Decks in Technology
See All in Technology
LangSmith×Webhook連携で実現するプロンプトドリブンCI/CD
sergicalsix
1
200
赤煉瓦倉庫勉強会「Databricksを選んだ理由と、絶賛真っ只中のデータ基盤移行体験記」
ivry_presentationmaterials
2
290
Core Audio tapを使ったリアルタイム音声処理のお話
yuta0306
0
180
Geminiとv0による高速プロトタイピング
shinya337
0
230
KubeCon + CloudNativeCon Japan 2025 Recap
ren510dev
1
360
AIの全社活用を推進するための安全なレールを敷いた話
shoheimitani
2
310
AI専用のリンターを作る #yumemi_patch
bengo4com
5
3.9k
AIとともに進化するエンジニアリング / Engineering-Evolving-with-AI_final.pdf
lycorptech_jp
PRO
0
150
生成AI時代の開発組織・技術・プロセス 〜 ログラスの挑戦と考察 〜
itohiro73
1
410
Witchcraft for Memory
pocke
1
740
マネジメントって難しい、けどおもしろい / Management is tough, but fun! #em_findy
ar_tama
5
680
Lambda Web Adapterについて自分なりに理解してみた
smt7174
6
160
Featured
See All Featured
Optimizing for Happiness
mojombo
379
70k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
Testing 201, or: Great Expectations
jmmastey
42
7.6k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Writing Fast Ruby
sferik
628
62k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.4k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
A Modern Web Designer's Workflow
chriscoyier
694
190k
Embracing the Ebb and Flow
colly
86
4.7k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
20
1.3k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
The Language of Interfaces
destraynor
158
25k
Transcript
"84ωΠςΟϒͳ&$$3.γεςϜ ӡ༻ʹ͔ܽͤͳ͍ϩάج൫ߏங େ୍ོଠ
DNEFWJP
ࣗݾհ /BNF େ୍ོଠ!UBLJQPOF +PC 43&!QSJTNBUJY *OUFSFTU ωοτϫʔΫσϓϩΠपΓ 'BWPSJUF
"NB[PO3PVUFͱ"$.
ΞδΣϯμ "84ωΠςΟϒͳ&$$3.γεςϜ QSJTNBUJY ͱ ϩάੳج൫ͷઃܭ
ͭ·͍͍ͮͨͯΔͱ͜Ζ ࠓޙͷల
1. AWSωΠςΟϒͳ EC/CRMγεςϜ prismatixͱ
QSJTNBUJY ҎԼ1[ ɺ&$ͱ$3.γεςϜΛϚΠΫϩαʔϏεͱͯ͠ ఏڙ͠ݸผʹಋೖՄೳͰ͢ɻ֤γεςϜͷػೳʹ"1*Λ׆༻͢Δ͜ͱͰ γεςϜ։ൃͱӡ༻ͷෳࡶੑɺίετɺ࣌ؒΛݮ͠·͢ɻ
ߏྫ prismatix (API) ECαʔόʔ εϚϗΞϓϦ ECαΠτ
ࣄྫύϧί༷
"84ωΠςΟϒͳΠϯϑϥߏ ϩʔυόϥϯα (ෛՙࢄ/TLSऴ) ίϯςφΫϥελ (APαʔόʔ) σʔλϕʔε (ϚωʔδυαʔϏε)
ϚΠΫϩαʔϏεؒͷ࿈ܞ ඇಉظૄ݁߹ͷ͘͠Έ αʔϏε Amazon SNS (௨) Amazon SQS
(δϣϒΩϡʔ) ࡏݿαʔϏε (ϫʔΧʔ) ొ Πϕϯτૹ৴ Πϕϯτ͕ Ωϡʔʹཷ·Δ ΠϕϯτΛऔಘͯ͠ ࡏݿσʔλʹө ΠϕϯτΛసૹ
ϚΠΫϩαʔϏεؒͷ࿈ܞ αʔϏεͷϦιʔε૿ݮ εέʔϧΠϯΞτ ͍͢͠ αʔϏεΛ૿ڧ ࡏݿαʔϏεΛ૿ڧ
2. ϩάੳج൫ͷઃܭ
QSJTNBUJYͷϩά ओʹछྨɺ͍ͣΕ+40/ܗࣜ ✦ ΞϓϦϩά 4QSJOH +BWB ىಈ࣌Τϥʔͷϝοηʔδ
✦ .%$ϩά .BQQFE%JBHOPTUJD$POUFYUT ϚΠΫϩαʔϏεͷϦΫΤετͱϨεϙϯεͷҰ෦ ͋ͱεϩʔΫΤϦϩάͳͲ
ϩάͷ༻్ ✦ τϥϒϧγϡʔςΟϯά ‣ ϚΠΫϩαʔϏεͷಈ࡞֬ೝ ‣ ݕࡧจͱͷಥ͖߹Θͤ ✦
ϩάੳ ‣ όʔήϯηʔϧ4/4ͷεύΠΫͷࣄલରࡦͷجૅࢿྉ औΓ͜΅ͨ͘͠ͳ͍
ϩάج൫ͷઃܭࢥ ✦ ͦΕͳΓʹେྔ݄ؒʙ5# ✦ 410' 4JOHMF1PJOUPG'BJMVSF Λආ͚͍ͨ ✦
ͳΔ͘༗ΓΛ͍͍ͨ ✦ ༻్ʹΑͬͯҟͳΔಛੑ ‣ σʔλϨΠΫ ϦΞϧλΠϜϩάͷΈ߹Θͤ
ϩάج൫ͷߏਤ Amazon ECS (ίϯςφΫϥελ) Amazon S3 (σʔλϨΠΫ) CloudWatch
Logs (ϦΞϧλΠϜϩά) Treasure Data (ϩάੳ)
"NB[PO&$4ίϯςφΫϥελ ✦ %PDLFSίϯςφΛཧ͢ΔϚωʔδυαʔϏε ‣ ίϯςφΛͲͷΠϯελϯε ԾϚγϯ Ͱ࣮ߦ͢Δ ͔ΛΫϥελ͝ͱʹཧ
‣ QSJTNBUJYͰ'BSHBUFະ࠾༻ ✦ %PDLFSʹίϯςφϩά ඪ४ग़ྗ Λѻ͏ MPHHJOHESJWFSػೳ͕͋Δ ‣ ίϯςφϩάϑΝΠϧΛѻΘͣɺϩάΛ+40/ܗࣜͰ ඪ४ग़ྗʹग़͢Α͏ΞϓϦΛߏ ECSΫϥελ
"NB[PO4σʔλϨΠΫ ✦ ΦϯϥΠϯετϨʔδαʔϏε ‣ ߴ͍ٱੑ ‣ ߴ͍εέʔϥϏϦςΟ ༰ྔແ੍ݶ
‣ ྿ՁɺબΔετϨʔδΫϥε ‣ σʔλੳαʔϏεͱͷ࿈ܞ
"NB[PO$MPVE8BUDI-PHTϦΞϧλΠϜϩά ✦ ϚωʔδυͷϩάอଘαʔϏε ‣ ߴ͍εέʔϥϏϦςΟ ‣ ४ϦΞϧλΠϜࢀর ‣
+40/ͷΫΤϦΛαϙʔτ
"SN5SFBTVSF%BUBϩάੳ ✦ σʔλੳͷ4BB4αʔϏε ඇ"84 ‣ ඇߏԽσʔλʹૉૣ͘ΞΫηεͰ͖Δ ετϨʔδ
1MB[NB%# ‣ ฒྻΫΤϦΤϯδϯ 1SFTUPͳͲ Ͱ ΫΤϦͰ͖Δ
༗Γʹݶք͋Γʢʣ Amazon ECS Amazon S3 CloudWatch Logs MPHHJOHESJWFSͷ੍
⭕ $MPVE8BUDI-PHTΛαϙʔτ ❌ 4ʹ͚ΒΕͳ͍ ❌ ෳग़ྗʹະରԠ ◦ ×
༗Γʹݶք͋Γʢʣ Amazon S3 Treasure Data %BUB$POOFDUPSͱ͍͏5SFBTVSF %BUBͷΠϯϙʔτػೳ͕͋Δ ⭕
εέδϡʔϧػೳ͕͋Γɺ࣍ͷ Πϯϙʔτ͜ΕͰ0, ❌ Πϯϙʔτ࣌ͷϦιʔεಉ࣮࣌ߦ ʹ੍ݶ͕͋ΓɺॳճΠϯϙʔτʹن ײ͕߹Θͳ͍ 044ͷ&NCVMLͱ%JHEBH͕தͰ ಈ͍͍ͯΔ
ෆΛิ͏ͨΊʹ044Λར༻ Fluentd (ετϦʔϜॲཧ) Embulk (όονॲཧ) ͲͪΒॊೈͰ๛ͳϓϥάΠϯΤίγεςϜ͋Γ ͨ·ͨ·5SFBTVSF%BUBͰ5%ͱͷߴ͍ੑ
%PDLFSº'MVFOUE %PDLFSMPHHJOHESJWFS͕'MVFOUEͷ ૹ৴Λαϙʔτ ‣ 'MVFOUEࣗମ%PDLFSίϯςφͱͯ͠ &$4ͷ֤ΠϯελϯεͰ࣮ߦ ‣ ϚΠΫϩαʔϏεͷίϯςφ͔Β
ಉΠϯελϯεͷ'MVFOUEʹϩάΛૹ৴ Fluentd ίϯςφ
'MVFOUEͷෳ0VUQVUͱଟஈߏ 4ͱ$MPVE8BUDI-PHT ͷ྆ํʹอଘ Amazon S3 CloudWatch Logs <match
docker.*.*.*> @type copy <store> @type s3 : </store> <store> @type forward : <server> host fluentd.example.lo port 24224 </server> </store> </match>
'MVFOUEͷෳ0VUQVUͱଟஈߏ ✦ ϦΞϧλΠϜϩά"HHSFHBUPS ूαʔόʔ Λ Ճͨ͠ଟஈߏ ‣ ͋ͱ͔ΒৼΓઌΛม͍͑͢Α͏ʹ
‣ $MPVE8BUDI-PHTͷ"1*ίʔϧͷ੍ରࡦ CloudWatch Logs Aggregator Forwarder
'MVFOUEͷσʔλՃ ✦ 'PSXBSEFS ‣ +40/ͷύʔε ‣ ΞϓϦϩάͱ.%$ϩάͷ ͚۠
‣ 4ͷύεϓϨϑΟοΫε ✦ "HHSFHBUPS ‣ Τϥʔͷநग़ ‣ $MPVE8BUDI-PHTύϥϝʔλ <filter docker.**> @type parser format json key_name log </filter> : <match docker.**> @type rewrite_tag_filter <rule> key marker pattern AUDIT tag ${tag}.audit </rule> <rule> key message pattern .+ tag ${tag}.app </rule> </match>
&NCVML ✦ 4ˠ5%ͷॳظόονΠϯϙʔτ ‣ 5%ͷઃఆ͕ͱʹָ͔ͩͬͨ͘ ‣ %BUB$POOFDUPSͷίϯϑΟά͕Ұ෦͍ճͤͨ
Embulkͷฒྻ࣮ߦ Ͳ͏͢Δʁ
&NCVMLPO"84#BUDI ✦ "84#BUDIͷδϣϒΩϡʔ εϙοτΠϯελϯε׆༻ ‣ δϣϒͷ༰%PDLFSίϯςφͰ ࣮ߦ͢ΔίϚϯυϥΠϯ ✦
&NCVMLͷ%PDLFSΠϝʔδΛ ༻ҙ࣮ͯ͠ߦ ‣ IUUQTIVCEPDLFSDPNSDMBTTNFUIPE FNCVMLNFUTUE εϙοτϑϦʔτ δϣϒΩϡʔ AWS Batch
&NCVMLͷίϯϑΟάڥมΛଟ༻ exec: max_threads: {{ env.MAX_THREADS }} in: type:
s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/{{ env.MONTH } path_match_patterns: \.gz$ auth_method: instance endpoint: s3-ap-northeast-1.amazonaws.com parser: type: jsonl : decoders: - { type: gzip } out: type: td endpoint: api.treasuredata.com apikey: {{ env.TDAPIKEY }} database: {{ env.TDDATABASE }} table: {{ env.TDTABLE }}
3. ͭ·͍ͮͨ/͍ͯΔ ͱ͜Ζ
Fluentd Aggregatorͷ Խ/εέʔϧΞτ CloudWatch Logs Aggregator Forwarder
"HHSFHBUPSͷԽεέʔϧΞτ ✦ ෳͷ"HHSFHBUPSཱ͕ͯΒΕͳ͍ ‣ $MPVE8BUDI-PHTͰಉҰͷϩάετϦʔϜʹॻ͖ࠐΉ ͱ͖TFRVFODFUPLFOΛҡ࣋͠ͳ͚ΕͳΒͳ͍ <match **>
@type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name container_id auto_create_stream true </match> ϚΠΫϩαʔϏε ͷίϯςφ*% ϩάάϧʔϓ ϩά ετϦʔϜ Aggregator ×
"HHSFHBUPSͷԽεέʔϧΞτ ✦ ϩάετϦʔϜ໊ʹͩ͜ΘΒͳ͍ ‣ "HHSFHBUPS͝ͱʹϩάετϦʔϜΛׂΓͯΔ ‣ ʮΠϕϯτͷݕࡧʯ͔ΒετϦʔϜԣஅͷݕࡧΛར༻ <match
**> @type cloudwatch_logs region "#{ENV['AWS_REGION']}" log_group_name_key service log_stream_name "#{Socket.gethostname}" auto_create_stream true </match> "HHSFHBUPS ͷίϯςφ*%
%PDLFSͷϗετϙʔτͷ੍ ✦ "HHSFHBUPS%PDLFSίϯςφ Ͱ࣮ߦ ‣ ϗετϙʔτ͕ݻఆͰϙʔτ Λ༗ ‣
Πϯελϯεʹίϯςφ͔͠ ࣮ߦͰ͖ͳ͍ ✦ ϗετϙʔτΛಈతʹ͢ΔͨΊʹ ϩʔυόϥϯαΛ$-#ˠ/-#ʹ Ҡߦ NLB Aggregator ίϯςφ
TDͷΠϯϙʔτͷδϣϒׂ
Πϯϙʔτͷδϣϒׂ ✦ Πϯϙʔτର4ϓϨϑΟοΫεΛ͘औΔͱ ϝϞϦΊͬͪΌ͏ ‣ %BUB$POOFDUPSϩά͕ফ͑ͯແݶϦτϥΠ˞݄࣌ ‣
"84#BUDIϝϞϦ্ׂݶ·Ͱͬͯ00.,JMMFSൃಈ $ aws s3 ls --profile cm-jp-1 s3://XXXX-infra-logbucket-XXXX/applications /condor/app/2018/10/04/05/ 2018-10-04 14:06:24 5964 00_5fbef6f0fdec_0.gz 2018-10-04 14:11:26 5971 05_5fbef6f0fdec_0.gz 2018-10-04 14:16:24 5932 10_5fbef6f0fdec_0.gz 2018-10-04 14:21:25 5954 15_5fbef6f0fdec_0.gz
Πϯϙʔτͷδϣϒׂ ✦ ͍Ζ͍Ζࢼͯ͠ɺ୯ҐͰδϣϒΛΔͷ͕ ྑͦ͞͏ͱ͍͏͜ͱʹ ‣ ϑΝΠϧαΠζɺϑΝΠϧʹґଘ͢ΔͷͰͬͯΈͳ ͍ͱΘ͔Βͳ͍෦ ࠓճϑΝΠϧ͕ଟ͔༷ͬͨ
in: type: s3 bucket: {{ env.S3BUCKET }} path_prefix: applications/{{ env.MET_SERVICE }}/audit/{{ env.YEAR }}/ {{ env.MONTH }}/{{ env.DAY }}
5SFBTVSF%BUBͷ'"2ʹهࡌ͋Γ 2 %BUB$POOFDUPSGPS4KPCJTSVOOJOHGPSBMPOH UJNF XIBUDBO*EP " $IFDLUIFDPVOUPG4pMFTUIBUZPVSDPOOFDUPS
KPCJTJOHFTUJOH*GUIFSFBSFPWFS pMFT UIF QFSGPSNBODFEFHSBEFT 5PNJUJHBUFUIJTJTTVF ZPVDBO ‣ /BSSPXQBUI@QSFpYPQUJPOBOESFEVDFUIFDPVOUPG4pMFT ‣ 4FU .# UPNJO@UBTL@TJ[FPQUJPO IUUQTTVQQPSUUSFBTVSFEBUBDPNIDFOVTBSUJDMFT%BUB$POOFDUPS GPS"NB[PO4'"2GPSUIF4%BUB$POOFDUPS
ϩάαΠζͷ্ݶ
ϩάαΠζͷ্ݶ ✦ .%$ϩά͕σΧ͍ ✦ %PDLFSMPHHJOHESJWFS ,#ΑΓେ͖͍ϩάΛׂ͢Δ ‣ 'MVFOUEGPSXBSEFSͷ+40/ύʔεʹࣦഊɺࣺͯΒΕΔ
‣ DPODBUϓϥάΠϯͰ݁߹ ‣ ϝϞϦ༻ྔ͕Ͷͯ00.,JMMFSʹࡴ͞ΕͨΠϚίί ✦ $MPVE8BUDI-PHTͷ্ݶ,#
͓ۚ ✦ $MPVE8BUDI-PHTϩάσʔλྔͷैྔ՝ۚ ‣ ྔ͕ଟ͍ͷͰֹ݄අ༻͕͔͞Ή ‣ &$ͳͲଞͷαʔϏεඅ༻ͱτϯτϯͱݴΘΕΔͱͭΒ ͍
✦ ΞϓϦ͔Βग़͢ϩάͷྔΛݮΒ͢ ‣ .%$ϩάͭΒ͍
4. ࠓޙͷల
࣍ͷҰख ✦ ϩάࢹ ✦ ϩάूͱͯ͠&MBTUJDTFBSDI,JCBOBͷར༻ ✦ 5SFBTVSF%BUBΛͬͱ׆༻͍ͨ͠
·ͱΊ ✦ εέʔϧ͢Δϩάཧج൫ΫϥυαʔϏεͷ Έ߹Θ͕ͤΦεεϝ ‣ σʔλϨΠΫͱϦΞϧλΠϜϩάͰͷαʔϏεͷ͍͚ ✦ Έ߹Θ͚ͤͩͰಈ͘ͷͰ͖Δ͚Ͳ
‣ ཁ݅ʹରͯ͠Γͳ͍ͱ͜Ζͷิ͕ؒඞཁ ‣ σʔλྔɺαΠζͷධՁνϡʔχϯά͕ඞཁ