ELKstackとAthenaの素敵な関係

& - , T U B D L ͱ "
U I F O B ͷ ૉ ఢ ͳ ؔ ܎ JAWS-UGԣ඿ #9 - Operational Excellence 2017/02/10 ੢ଜ ༡

˞ ຊ εϥ Π υ ͸ & - ,
T U B D L ͱ " N B [ P O " U I F O B Λ ٕ ज़ త ʹ ࿈ ܞ ͞ ͤ ͨ ࿩ Ͱ ͸ ͋ Γ · ͤ Μ ͢ ͍ · ͤ Μ ʂ ͱ ͍ ͏ ͔ ٕ ज़ త ͳ ࿩ ͋ Γ · ͤ Μ

& - , T U B D L Ͱ ͷ
ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ " N B [ P O " U I F O B ͕ औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏ ࿩

͔ ͍ ͠ Ό ͍ Μ ͤ ͍ ΂
ͭ ɿ ͓ ͱ ͜ Ϩ ϕ ϧ ɿ ) 1 . 1 ࣗ ݾ ঺ հ ࣾ ձ ਓ ྺ d ɹ % $ Ͱ ؂ ࢹ Φϖ Ϩ ʔ λ ʔ d ɹ ๭ ι ʔ γ ϟϧ ήʔϜ ܥ ձ ࣾ d / 0 8 ɹ ɹ ג ࣜ ձ ࣾ " 1 1 - * # 05 "8 4 ྺ ໿ ೥ ͳ · ͑ ɹ ɹ ɿ ੢ ଜ ɹ ༡ ͭ ͍ ͬ ͨ ʔ ɿ !HBDIBSJPO

&-,TUBDL "NB[PO"UIFOB FMBTUJDTFBSDI LJCBOBͰͷϩάࢀর ٹੈओ"UIFOB ӡ༻5JQTతͳ ҰԠ৮Ε͓ͯ͘΂͖͜ͱ ·ͱΊ ໨
࣍

& - , T U B D L • elastic
search + log stash + kibana = elk stack • L ͷ෦෼ʢσʔλ౤ೖʣ͸fluentdͩͬͨΓ͢Δͱ EFKͱͳΔΒ͍͠ɻ • elasticsearchʹೖΕͯkibanaͰݟͯΔͷͰ  εϥΠυ಺Ͱ͸૯শͱͯ͠࢖ͬͯ·͢ɻ

" N B [ P O " U I
F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon)  Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠  ΫΤϦΛ౤͛Δͱɺ਺ඵͰ݁Ռ͕ฦͬͯ͘Δɻ

" N B [ P O " U I
F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon)  Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠  ΫΤϦΛ౤͛Δͱɺ਺ඵͰ݁Ռ͕ฦͬͯ͘Δɻ ˢϚδͰ ਆ͡Όͳ͍͔͢

FMBTUJDTFBSDI LJCBOB Ͱͷϩάࢀর

Ͷ Β ͍ • ͱΓ͋͑ͣௐࠪʹ࢖͏ϩά͸elasticsearch • ֤αʔόͷϩάΛfluentdͰू໿ɺ౤ೖ • ݸʑͷαʔόͷΈʹ͔͠࢒Βͳ͍ϩάΛͳ͘͢  ໨త΋͋Γ
• ͨ·ʹlogstash • ΫΤϦʹΑΔΞϥʔτઃఆʢࣗલεΫϦϓτʣ

$ V T U P N - 0 (

F M B T U J D T F B
S D I ӡ ༻ ϧ ʔ ϧ త ͳ • ೖΕΔσʔλ͸ϑΝΠϧͱͯ͠΋ॻ͖ग़͓ͯ͘͠ʢJSONʣ • dailyͰgzipͯ͠S3΁backup • ྔͷଟ͍ϩάʢ1೔10GBҎ্ʣஷ·Δϩά͸3೔Ͱindexຖ࡟আ • Ҏલɺ௕ظͰݟΕΔΑ͏ʹ͍͕ͯͨ͠ɺkibanaද͕͔ࣔͳΓ஗͘ͳͬͯ͠·ͬͨ • ͦ΋ͦ΋ͦΜͳʹաڈͷΛݟͳ͔ͬͨ • ετϨʔδ΋༰ྔΛऔΒͳͯ͘͸͍͚ͳ͍ͨΊ௕ظؒอ࣋ͨ͘͠ͳ͔ͬͨ • εφοϓγϣοτ͸ͱͬͨΓͱͬͯͳ͔ͬͨΓ(ũųųƅƁ • ͱ͍ͬͯͨ΍ͭ΋Ұिؒ΄ͲͰফ͍ͯͨ͠ɻ • ࠓ͸rundeck͞ΜͰຖ೔ࣗಈ࣮ߦ؅ཧɻS3ʹอଘɻ

໰୊఺

໰ ୊ ఺ • index࡟আޙͷௐࠪ • ݟ͍ͨϩά͸΄ΜͷҰ෦෼ • snapshotͷϦετΞͰ͖ͳ͍৔߹ •
਺ेGBͷϩάΛDLͯ͠ղౚͯ͠grepͯ͠ɾɾɾ

໰ ୊ ఺ • elasticsearchʹೖΕΔͷ΋ͦΕͳΓʹ༻ҙ͕ඞཁ • ύʔαʔ୳ͨ͠Γɺॻ͍ͨΓ • template࡞ͬͨΓ(dynamic templateͰେମରԠ)
• ٸͳґཔʹରԠ͖͠Εͳ͍

ٹ ੈ ओ

ࣄ ྫ ໰ ͍ ߹ Θ ͤ ௐ ࠪ
ͷ ҝ ɺ ա ڈ ͷ " 1 * ϩ ά ͕ Έ ͨ ͍

ͦ Ε ͳ Β L J C B O B
Ͱ ݟ Ε · ͬ ͤ ʂ

ͦ Ε ͳ Β L J C B O B
Ͱ ݟ Ε · ͬ ͤ ʂ Ұिؒ΄ͲલͳΜͰ͕͢ʜ

T O B Q T I P U ΋ ͏
ͳ ͍ Θ ʜ Ұिؒ΄ͲલͳΜͰ͕͢ʜ

4͔Βϩά (#௒ Λ%-ͯ͠ղౚͯ͠HSFQ ͭΒ͍

ٹ ੈ ओ

\ EBUFz UISFBEll MFWFM*/'0 IBOEMFSll 64&3@*%ll
WFSTJPOll 4&44*0/@*%ll OB[P NFTTBHFl63-IUUQ 1BSBNFUFS\dུd^ IPTUOBNFlBQJTEGTGPN ^ ݩϩά +40/ܗࣜ

$3&"5&&95&3/"-5"#-&*'/05&9*454BQJTFSWFSBQJMPH@@ AEBUFAUJNFTUBNQ AUISFBEATUSJOH AMFWFMATUSJOH AIBOEMFSATUSJOH
A64&3@*%ATUSJOH AWFSTJPOATUSJOH A4&44*0/@*%ATUSJOH AOB[PATUSJOH ANFTTBHFATUSJOH AIPTUOBNFATUSJOH 308'03."54&3%&PSHPQFOYEBUBKTPOTFSEF+TPO4FS%F 8*5)4&3%&1301&35*&4 TFSJBMJ[BUJPOGPSNBU -0$"5*0/TMPHCVDLFUBQJTFSWFS %#5BCMF࡞੒

4&-&$5 '30.BQJMPH@@ 8)&3&VTFS@JE ΫΤϦ࣮ߦ

݁Ռ DTWͰͷ%-΋Մೳ

Β͘Β͘ڞ༗

ࣄ ྫ 4 ͷ ΞΫ η ε
ϩ ά ͕ ݟ ͨ ͍

%FW 0QT -BNCEBͰը૾ॲཧͯ͠Δ͠ "1*͔Β΋ૢ࡞͍ͯ͠ΔͷͰɺ 4ΞΫηεϩάΈ͍ͨͰ͢ʂ XFCϖʔδೖͬͯΔΘ͚Ͱ΋ͳ͍͔ Β0/ʹͯ͠ͳ͔ͬͨΘʜ ઃఆ͠·͢

MPHHJOH0/ 0QT ϩάग़Δ·Ͱʹ਺͔͔࣌ؒΔͱॻ͍ͯ͋Δ ˞IUUQEPDTBXTBNB[PODPN ϩάLJCBOBͰݟΕ·͔͢Ͷʁ %&-&5&Λߦͳ͍ͬͯΔϦΫΤετ͕ݟ͍ͨ ɹ;Ή ɹFMBTUJDTFBSDIʹೖΕͯ ɹݕࡧͰ͖ΔΑ͏ʹ͠Α͏ %FW

ͯ͞FMBTUJDTFBSDIʹೖΕΔ४උ͠ͱ͔͘ɻ ͲΜͳײ͡ͷ΍͚ͭͩͬʁ 0QT φχίϨ IUUQEPDTBXTBNB[PODPN KB@KQ"NB[PO4MBUFTUEFW 4ΞΫηεϩάϩάܗࣜαϯϓϧ BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCFNZCVDLFU<'FC >
BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCF&'&9".1-& 3&45(&57&34*0/*/((&5NZCVDLFU WFSTJPOJOH)5514$POTPMF

MPHTUBTIͰύʔεͰ͖Δ༷ʹͯ͠ʜ FMBTUJDTFBSDIʹೖΕͯʜ LJCBOBͰEBTICPBSE࡞ͬͯʜ

%&-&5&ͷϩάΛݟ͍͚ͨͩͳͷʹʜ ࣌ؒʜ͔͔Δʜͳ͊

ٹ ੈ ओ

$3&"5&&95&3/"-5"#-&*'/05&9*454T@BDDFTTMPHSFTPVSTF@CVDLFU #VDLFU0XOFSTUSJOH #VDLFUTUSJOH 3FRVFTU%BUF5JNFTUSJOH 3FNPUF*1TUSJOH
3FRVFTUFSTUSJOH 3FRVFTU*%TUSJOH 0QFSBUJPOTUSJOH ,FZTUSJOH 3FRVFTU63*@PQFSBUJPOTUSJOH 3FRVFTU63*@LFZTUSJOH 3FRVFTU63*@IUUQ1SPUPWFSTJPOTUSJOH )551TUBUVTTUSJOH &SSPS$PEFTUSJOH #ZUFT4FOUTUSJOH 0CKFDU4J[FTUSJOH 5PUBM5JNFTUSJOH 5VSO"SPVOE5JNFTUSJOH 3FGFSSFSTUSJOH 6TFS"HFOUTUSJOH 7FSTJPO*ETUSJOH 308'03."54&3%&PSHBQBDIFIBEPPQIJWFTFSEF3FHFY4FS%F 8*5)4&3%&1301&35*&4 TFSJBMJ[BUJPOGPSNBU JOQVUSFHFY <?> <?> aa< aa> <?> <?> <?> <?> <?> aaa <?> <?> c<?> aaa c<> <?> <?> <?> <?> <?> <?> a<?a> a <?> -0$"5*0/bTBXTMPH4SFTPVSTF@CVDLFU %#5BCMF ࡞੒

4&-&$5SFNPUFJQ SFRVFTUFS DPVOU DPVOU '30.SFTPVSTF@CVDLFU 8)&3&SFRVFTUVSJ@PQFSBUJPO%&-&5& HSPVQCZSFNPUFJQ
SFRVFTUFS ΫΤϦ࣮ߦ

SFNPUFJQSFRVFTUFS DPVOU BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS
BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS dུd BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ݁Ռ

·ͱΊ

· ͱ Ί • Amazon AthenaͰS3্ͷϩά͕ଈղੳՄೳঢ়ଶʹ  AthenaͰݟΔͨΊʹ~ΛS3ʹ͸͘Α͏ʹͯ͠…ͱ͔Ͱ͸ͳ͘ɺ  ͢ͰʹόοΫΞοϓͱ্͍ͯ͛ͯͨ͠ϩάશ͕ͯࣗಈతʹର৅ͱݴ͑Δɻ  AWSͷαʔϏε(S3 /
CloudFront / ELB / CloudTrail)͔Βॻ͖ग़͢ϩά΋ɺ ௐ͚ࠪͩͳΒ͜ΕͰे෼ɻ • elasticsearch͔Β࡟আͯ͠΋S3ʹϩάϑΝΠϧ͋Ε͹ௐࠪ͸OK  ௕ظؒݟΕΔ༷ʹ͢ΔͨΊʹ  αʔόεϖοΫΛ্͓͛ͯ͘ඞཁ͸ແ͘ͳͬͨɻ  ˠ ຊ౰ʹඞཁͳظؒͷΈͰΑ͘ͳͬͨɻ  ɹ→ snapshot͸ͪΌΜͱऔΖ͏+͋Δఔ౓ظؒ࣋ͬͯͯ΋Α͔ͬͨ(൓ল)

· ͱ Ί • ϦΞϧλΠϜͷϩάղੳ → elasticsearch + kibana •
S3ʹ͓͍ͨϩάௐࠪ → Amazon Athena

& - , T U B D L Ͱ ͷ
ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ " N B [ P O " U I F O B ͕ औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏ ࿩ Ͱ ͠ ͨ

ޚ ਗ਼ ௌ ͋ Γ ͕ ͱ ͏ ͝
͟ ͍ · ͠ ͨ

ELKstackとAthenaの素敵な関係

ELKstackとAthenaの素敵な関係

More Decks by 遊

Other Decks in Technology

Featured

Transcript