Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ELKstackとAthenaの素敵な関係
Search
遊
February 10, 2017
Technology
0
950
ELKstackとAthenaの素敵な関係
2017/02/10 JAWS-UG#9で発表した資料になります
遊
February 10, 2017
Tweet
Share
More Decks by 遊
See All by 遊
applibotのDevOpsを支える terraform/packer
gacharu
1
3.2k
ゲームのインフラ6年やっててよく聞かれること
gacharu
1
1.2k
Other Decks in Technology
See All in Technology
Git 研修 Basic【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
310
Azure OpenAI Service Dev Day / LLMでできる!使える!生成AIエージェント
masahiro_nishimi
3
740
公共領域から学ぶ クラウド移行についてエンジニアが意識していること
kawakawa2222
0
140
運用改善、不都合な真実 / 20240722-ssmjp-kaizen
opelab
17
8k
ギークの理想が7つ集まるエムスリーで夢を叶えよう - エムスリー株式会社
m3_engineering
1
260
コミュニティサービスに「あなたへ」フィードを リリースするまでの試行錯誤
takapy
1
140
What is DRE? - Road to SRE NEXT@広島
chanyou0311
3
620
LLMアプリケーションの評価の実践と課題 ~PharmaXにおける今後の展望~
pharma_x_tech
2
160
dxd2024-生成AIに振り回された3か月間の成功と失敗/dxd2024-link-and-motivation
lmi
2
260
AWSで”最小権限の原則”を実現するための考え方 /20240722-ssmjp-aws-least-privilege
opelab
10
4.3k
ここがすごいよ! AWS Systems Manager!
saichan11
0
1.8k
大規模ドラレコデータ収集・機械学習基盤を支える AWS CDK 〜導入・運用事例紹介〜
pemugi
0
110
Featured
See All Featured
Building a Scalable Design System with Sketch
lauravandoore
458
32k
Designing with Data
zakiwarfel
96
5k
Facilitating Awesome Meetings
lara
46
5.8k
Designing the Hi-DPI Web
ddemaree
276
34k
Imperfection Machines: The Place of Print at Facebook
scottboms
262
13k
Web development in the modern age
philhawksworth
203
10k
Building Better People: How to give real-time feedback that sticks.
wjessup
357
18k
Code Reviewing Like a Champion
maltzj
517
39k
Practical Orchestrator
shlominoach
185
10k
What's in a price? How to price your products and services
michaelherold
239
11k
The Illustrated Children's Guide to Kubernetes
chrisshort
39
47k
What's new in Ruby 2.0
geeforr
338
31k
Transcript
& - , T U B D L ͱ "
U I F O B ͷ ૉ ఢ ͳ ؔ JAWS-UGԣ #9 - Operational Excellence 2017/02/10 ଜ ༡
˞ ຊ εϥ Π υ & - ,
T U B D L ͱ " N B [ P O " U I F O B Λ ٕ ज़ త ʹ ࿈ ܞ ͞ ͤ ͨ Ͱ ͋ Γ · ͤ Μ ͢ ͍ · ͤ Μ ʂ ͱ ͍ ͏ ͔ ٕ ज़ త ͳ ͋ Γ · ͤ Μ
& - , T U B D L Ͱ ͷ
ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ " N B [ P O " U I F O B ͕ औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏
͔ ͍ ͠ Ό ͍ Μ ͤ ͍
ͭ ɿ ͓ ͱ ͜ Ϩ ϕ ϧ ɿ ) 1 . 1 ࣗ ݾ հ ࣾ ձ ਓ ྺ d ɹ % $ Ͱ ࢹ Φϖ Ϩ ʔ λ ʔ d ɹ ι ʔ γ ϟϧ ήʔϜ ܥ ձ ࣾ d / 0 8 ɹ ɹ ג ࣜ ձ ࣾ " 1 1 - * # 05 "8 4 ྺ ͳ · ͑ ɹ ɹ ɿ ଜ ɹ ༡ ͭ ͍ ͬ ͨ ʔ ɿ !HBDIBSJPO
None
None
&-,TUBDL "NB[PO"UIFOB FMBTUJDTFBSDI LJCBOBͰͷϩάࢀর ٹੈओ"UIFOB ӡ༻5JQTతͳ ҰԠ৮Ε͓͖ͯ͘͜ͱ ·ͱΊ
࣍
& - , T U B D L • elastic
search + log stash + kibana = elk stack • L ͷ෦ʢσʔλೖʣfluentdͩͬͨΓ͢Δͱ EFKͱͳΔΒ͍͠ɻ • elasticsearchʹೖΕͯkibanaͰݟͯΔͷͰ εϥΠυͰ૯শͱͯͬͯ͠·͢ɻ
" N B [ P O " U I
F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon) Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠ ΫΤϦΛ͛ΔͱɺඵͰ݁Ռ͕ฦͬͯ͘Δɻ
" N B [ P O " U I
F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon) Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠ ΫΤϦΛ͛ΔͱɺඵͰ݁Ռ͕ฦͬͯ͘Δɻ ˢϚδͰ ਆ͡Όͳ͍͔͢
FMBTUJDTFBSDI LJCBOB Ͱͷϩάࢀর
Ͷ Β ͍ • ͱΓ͋͑ͣௐࠪʹ͏ϩάelasticsearch • ֤αʔόͷϩάΛfluentdͰूɺೖ • ݸʑͷαʔόͷΈʹ͔͠Βͳ͍ϩάΛͳ͘͢ త͋Γ
• ͨ·ʹlogstash • ΫΤϦʹΑΔΞϥʔτઃఆʢࣗલεΫϦϓτʣ
$ V T U P N - 0 (
& - #
3 % 4
F M B T U J D T F B
S D I ӡ ༻ ϧ ʔ ϧ త ͳ • ೖΕΔσʔλϑΝΠϧͱͯ͠ॻ͖ग़͓ͯ͘͠ʢJSONʣ • dailyͰgzipͯ͠S3backup • ྔͷଟ͍ϩάʢ110GBҎ্ʣஷ·Δϩά3Ͱindexຖআ • ҎલɺظͰݟΕΔΑ͏ʹ͍͕ͯͨ͠ɺkibanaද͕͔ࣔͳΓ͘ͳͬͯ͠·ͬͨ • ͦͦͦΜͳʹաڈͷΛݟͳ͔ͬͨ • ετϨʔδ༰ྔΛऔΒͳ͍͚ͯ͘ͳ͍ͨΊظؒอ࣋ͨ͘͠ͳ͔ͬͨ • εφοϓγϣοτͱͬͨΓͱͬͯͳ͔ͬͨΓ(ũųųƅƁ • ͱ͍ͬͯͨͭҰिؒ΄ͲͰফ͍ͯͨ͠ɻ • ࠓrundeck͞ΜͰຖࣗಈ࣮ߦཧɻS3ʹอଘɻ
• indexআޙͷௐࠪ • ݟ͍ͨϩά΄ΜͷҰ෦ • snapshotͷϦετΞͰ͖ͳ͍߹ •
ेGBͷϩάΛDLͯ͠ղౚͯ͠grepͯ͠ɾɾɾ
• elasticsearchʹೖΕΔͷͦΕͳΓʹ༻ҙ͕ඞཁ • ύʔαʔ୳ͨ͠Γɺॻ͍ͨΓ • template࡞ͬͨΓ(dynamic templateͰେମରԠ)
• ٸͳґཔʹରԠ͖͠Εͳ͍
ٹ ੈ ओ
ࣄ ྫ ͍ ߹ Θ ͤ ௐ ࠪ
ͷ ҝ ɺ ա ڈ ͷ " 1 * ϩ ά ͕ Έ ͨ ͍
ͦ Ε ͳ Β L J C B O B
Ͱ ݟ Ε · ͬ ͤ ʂ
ͦ Ε ͳ Β L J C B O B
Ͱ ݟ Ε · ͬ ͤ ʂ Ұिؒ΄ͲલͳΜͰ͕͢ʜ
T O B Q T I P U ͏
ͳ ͍ Θ ʜ Ұिؒ΄ͲલͳΜͰ͕͢ʜ
4͔Βϩά (# Λ%-ͯ͠ղౚͯ͠HSFQ ͭΒ͍
ٹ ੈ ओ
\ EBUFz UISFBEll MFWFM*/'0 IBOEMFSll 64&3@*%ll
WFSTJPOll 4&44*0/@*%ll OB[P NFTTBHFl63-IUUQ 1BSBNFUFS\dུd^ IPTUOBNFlBQJTEGTGPN ^ ݩϩά +40/ܗࣜ
$3&"5&&95&3/"-5"#-&*'/05&9*454BQJTFSWFSBQJMPH@@ AEBUFAUJNFTUBNQ AUISFBEATUSJOH AMFWFMATUSJOH AIBOEMFSATUSJOH
A64&3@*%ATUSJOH AWFSTJPOATUSJOH A4&44*0/@*%ATUSJOH AOB[PATUSJOH ANFTTBHFATUSJOH AIPTUOBNFATUSJOH 308'03."54&3%&PSHPQFOYEBUBKTPOTFSEF+TPO4FS%F 8*5)4&3%&1301&35*&4 TFSJBMJ[BUJPOGPSNBU -0$"5*0/TMPHCVDLFUBQJTFSWFS %#5BCMF࡞
4&-&$5 '30.BQJMPH@@ 8)&3&VTFS@JE ΫΤϦ࣮ߦ
݁Ռ DTWͰͷ%-Մೳ
Β͘Β͘ڞ༗
ࣄ ྫ 4 ͷ ΞΫ η ε
ϩ ά ͕ ݟ ͨ ͍
%FW 0QT -BNCEBͰը૾ॲཧͯ͠Δ͠ "1*͔Βૢ࡞͍ͯ͠ΔͷͰɺ 4ΞΫηεϩάΈ͍ͨͰ͢ʂ XFCϖʔδೖͬͯΔΘ͚Ͱͳ͍͔ Β0/ʹͯ͠ͳ͔ͬͨΘʜ ઃఆ͠·͢
MPHHJOH0/ 0QT ϩάग़Δ·Ͱʹ͔͔࣌ؒΔͱॻ͍ͯ͋Δ ˞IUUQEPDTBXTBNB[PODPN ϩάLJCBOBͰݟΕ·͔͢Ͷʁ %&-&5&Λߦͳ͍ͬͯΔϦΫΤετ͕ݟ͍ͨ ɹ;Ή ɹFMBTUJDTFBSDIʹೖΕͯ ɹݕࡧͰ͖ΔΑ͏ʹ͠Α͏ %FW
ͯ͞FMBTUJDTFBSDIʹೖΕΔ४උ͠ͱ͔͘ɻ ͲΜͳײ͡ͷ͚ͭͩͬʁ 0QT φχίϨ IUUQEPDTBXTBNB[PODPN KB@KQ"NB[PO4MBUFTUEFW 4ΞΫηεϩάϩάܗࣜαϯϓϧ BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCFNZCVDLFU<'FC >
BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCF&'&9".1-& 3&45(&57&34*0/*/((&5NZCVDLFU WFSTJPOJOH)5514$POTPMF
MPHTUBTIͰύʔεͰ͖Δ༷ʹͯ͠ʜ FMBTUJDTFBSDIʹೖΕͯʜ LJCBOBͰEBTICPBSE࡞ͬͯʜ
%&-&5&ͷϩάΛݟ͍͚ͨͩͳͷʹʜ ࣌ؒʜ͔͔Δʜͳ͊
ٹ ੈ ओ
$3&"5&&95&3/"-5"#-&*'/05&9*454T@BDDFTTMPHSFTPVSTF@CVDLFU #VDLFU0XOFSTUSJOH #VDLFUTUSJOH 3FRVFTU%BUF5JNFTUSJOH 3FNPUF*1TUSJOH
3FRVFTUFSTUSJOH 3FRVFTU*%TUSJOH 0QFSBUJPOTUSJOH ,FZTUSJOH 3FRVFTU63*@PQFSBUJPOTUSJOH 3FRVFTU63*@LFZTUSJOH 3FRVFTU63*@IUUQ1SPUPWFSTJPOTUSJOH )551TUBUVTTUSJOH &SSPS$PEFTUSJOH #ZUFT4FOUTUSJOH 0CKFDU4J[FTUSJOH 5PUBM5JNFTUSJOH 5VSO"SPVOE5JNFTUSJOH 3FGFSSFSTUSJOH 6TFS"HFOUTUSJOH 7FSTJPO*ETUSJOH 308'03."54&3%&PSHBQBDIFIBEPPQIJWFTFSEF3FHFY4FS%F 8*5)4&3%&1301&35*&4 TFSJBMJ[BUJPOGPSNBU JOQVUSFHFY <?> <?> aa< aa> <?> <?> <?> <?> <?> aaa <?> <?> c<?> aaa c<> <?> <?> <?> <?> <?> <?> a<?a> a <?> -0$"5*0/bTBXTMPH4SFTPVSTF@CVDLFU %#5BCMF ࡞
4&-&$5SFNPUFJQ SFRVFTUFS DPVOU DPVOU '30.SFTPVSTF@CVDLFU 8)&3&SFRVFTUVSJ@PQFSBUJPO%&-&5& HSPVQCZSFNPUFJQ
SFRVFTUFS ΫΤϦ࣮ߦ
SFNPUFJQSFRVFTUFS DPVOU BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS
BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS dུd BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ݁Ռ
·ͱΊ
· ͱ Ί • Amazon AthenaͰS3্ͷϩά͕ଈղੳՄೳঢ়ଶʹ AthenaͰݟΔͨΊʹ~ΛS3ʹ͘Α͏ʹͯ͠…ͱ͔Ͱͳ͘ɺ ͢ͰʹόοΫΞοϓͱ্͍ͯ͛ͯͨ͠ϩάશ͕ͯࣗಈతʹରͱݴ͑Δɻ AWSͷαʔϏε(S3 /
CloudFront / ELB / CloudTrail)͔Βॻ͖ग़͢ϩάɺ ௐ͚ࠪͩͳΒ͜ΕͰेɻ • elasticsearch͔Βআͯ͠S3ʹϩάϑΝΠϧ͋ΕௐࠪOK ظؒݟΕΔ༷ʹ͢ΔͨΊʹ αʔόεϖοΫΛ্͓͛ͯ͘ඞཁແ͘ͳͬͨɻ ˠ ຊʹඞཁͳظؒͷΈͰΑ͘ͳͬͨɻ ɹ→ snapshotͪΌΜͱऔΖ͏+͋Δఔظؒ࣋ͬͯͯΑ͔ͬͨ(ল)
· ͱ Ί • ϦΞϧλΠϜͷϩάղੳ → elasticsearch + kibana •
S3ʹ͓͍ͨϩάௐࠪ → Amazon Athena
& - , T U B D L Ͱ ͷ
ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ " N B [ P O " U I F O B ͕ औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏ Ͱ ͠ ͨ
ޚ ਗ਼ ௌ ͋ Γ ͕ ͱ ͏ ͝
͟ ͍ · ͠ ͨ