ELKstackとAthenaの素敵な関係

by 遊

Slide 1

Slide 1 text

& - , T U B D L ͱ " U I F O B ͷ ૉ ఢ ͳ ؔ ܎ JAWS-UGԣ඿ #9 - Operational Excellence 2017/02/10 ੢ଜ ༡

Slide 2

Slide 2 text

˞ ຊ εϥ Π υ ͸ & - , T U B D L ͱ " N B [ P O " U I F O B Λ ٕ ज़ త ʹ ࿈ ܞ ͞ ͤ ͨ ࿩ Ͱ ͸ ͋ Γ · ͤ Μ ͢ ͍ · ͤ Μ ʂ ͱ ͍ ͏ ͔ ٕ ज़ త ͳ ࿩ ͋ Γ · ͤ Μ

Slide 3

Slide 3 text

& - , T U B D L Ͱ ͷ ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ " N B [ P O " U I F O B ͕ औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏ ࿩

Slide 4

Slide 4 text

͔ ͍ ͠ Ό ͍ Μ ͤ ͍ ΂ ͭ ɿ ͓ ͱ ͜ Ϩ ϕ ϧ ɿ ) 1 . 1 ࣗ ݾ ঺ հ ࣾ ձ ਓ ྺ d ɹ % $ Ͱ ؂ ࢹ Φϖ Ϩ ʔ λ ʔ d ɹ ๭ ι ʔ γ ϟϧ ήʔϜ ܥ ձ ࣾ d / 0 8 ɹ ɹ ג ࣜ ձ ࣾ " 1 1 - * # 05 "8 4 ྺ ໿ ೥ ͳ · ͑ ɹ ɹ ɿ ੢ ଜ ɹ ༡ ͭ ͍ ͬ ͨ ʔ ɿ !HBDIBSJPO

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

&-,TUBDL "NB[PO"UIFOB FMBTUJDTFBSDILJCBOBͰͷϩάࢀর ٹੈओ"UIFOB ӡ༻5JQTతͳ ҰԠ৮Ε͓ͯ͘΂͖͜ͱ ·ͱΊ ໨ ࣍

Slide 8

Slide 8 text

& - , T U B D L • elastic search + log stash + kibana = elk stack • L ͷ෦෼ʢσʔλ౤ೖʣ͸fluentdͩͬͨΓ͢Δͱ EFKͱͳΔΒ͍͠ɻ • elasticsearchʹೖΕͯkibanaͰݟͯΔͷͰ  εϥΠυ಺Ͱ͸૯শͱͯ͠࢖ͬͯ·͢ɻ

Slide 9

Slide 9 text

" N B [ P O " U I F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon)  Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠  ΫΤϦΛ౤͛Δͱɺ਺ඵͰ݁Ռ͕ฦͬͯ͘Δɻ

Slide 10

Slide 10 text

" N B [ P O " U I F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon)  Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠  ΫΤϦΛ౤͛Δͱɺ਺ඵͰ݁Ռ͕ฦͬͯ͘Δɻ ˢϚδͰ ਆ͡Όͳ͍͔͢

Slide 11

Slide 11 text

FMBTUJDTFBSDILJCBOB Ͱͷϩάࢀর

Slide 12

Slide 12 text

Ͷ Β ͍ • ͱΓ͋͑ͣௐࠪʹ࢖͏ϩά͸elasticsearch • ֤αʔόͷϩάΛfluentdͰू໿ɺ౤ೖ • ݸʑͷαʔόͷΈʹ͔͠࢒Βͳ͍ϩάΛͳ͘͢  ໨త΋͋Γ • ͨ·ʹlogstash • ΫΤϦʹΑΔΞϥʔτઃఆʢࣗલεΫϦϓτʣ

Slide 13

Slide 13 text

$ V T U P N - 0 (

Slide 14

Slide 14 text

& - #

Slide 15

Slide 15 text

3 % 4

Slide 16

Slide 16 text

F M B T U J D T F B S D I ӡ ༻ ϧ ʔ ϧ త ͳ • ೖΕΔσʔλ͸ϑΝΠϧͱͯ͠΋ॻ͖ग़͓ͯ͘͠ʢJSONʣ • dailyͰgzipͯ͠S3΁backup • ྔͷଟ͍ϩάʢ1೔10GBҎ্ʣஷ·Δϩά͸3೔Ͱindexຖ࡟আ • Ҏલɺ௕ظͰݟΕΔΑ͏ʹ͍͕ͯͨ͠ɺkibanaද͕͔ࣔͳΓ஗͘ͳͬͯ͠·ͬͨ • ͦ΋ͦ΋ͦΜͳʹաڈͷΛݟͳ͔ͬͨ • ετϨʔδ΋༰ྔΛऔΒͳͯ͘͸͍͚ͳ͍ͨΊ௕ظؒอ࣋ͨ͘͠ͳ͔ͬͨ • εφοϓγϣοτ͸ͱͬͨΓͱͬͯͳ͔ͬͨΓ(ũųųƅƁ • ͱ͍ͬͯͨ΍ͭ΋Ұिؒ΄ͲͰফ͍ͯͨ͠ɻ • ࠓ͸rundeck͞ΜͰຖ೔ࣗಈ࣮ߦ؅ཧɻS3ʹอଘɻ

Slide 17

Slide 17 text

໰୊఺

Slide 18

Slide 18 text

໰ ୊ ఺ • index࡟আޙͷௐࠪ • ݟ͍ͨϩά͸΄ΜͷҰ෦෼ • snapshotͷϦετΞͰ͖ͳ͍৔߹ • ਺ेGBͷϩάΛDLͯ͠ղౚͯ͠grepͯ͠ɾɾɾ

Slide 19

Slide 19 text

໰ ୊ ఺ • elasticsearchʹೖΕΔͷ΋ͦΕͳΓʹ༻ҙ͕ඞཁ • ύʔαʔ୳ͨ͠Γɺॻ͍ͨΓ • template࡞ͬͨΓ(dynamic templateͰେମରԠ) • ٸͳґཔʹରԠ͖͠Εͳ͍

Slide 20

Slide 20 text

ٹ ੈ ओ

Slide 21

Slide 21 text

ࣄ ྫ ໰ ͍ ߹ Θ ͤ ௐ ࠪ ͷ ҝ ɺ ա ڈ ͷ " 1 * ϩ ά ͕ Έ ͨ ͍

Slide 22

Slide 22 text

ͦ Ε ͳ Β L J C B O B Ͱ ݟ Ε · ͬ ͤ ʂ

Slide 23

Slide 23 text

ͦ Ε ͳ Β L J C B O B Ͱ ݟ Ε · ͬ ͤ ʂ Ұिؒ΄ͲલͳΜͰ͕͢ʜ

Slide 24

Slide 24 text

T O B Q T I P U ΋ ͏ ͳ ͍ Θ ʜ Ұिؒ΄ͲલͳΜͰ͕͢ʜ

Slide 25

Slide 25 text

4͔Βϩά (#௒ Λ%-ͯ͠ղౚͯ͠HSFQ ͭΒ͍

Slide 26

Slide 26 text

ٹ ੈ ओ

Slide 27

Slide 27 text

\ EBUFz UISFBEll MFWFM*/'0 IBOEMFSll 64&3@*%ll WFSTJPOll 4&44*0/@*%ll OB[P NFTTBHFl63-IUUQ1BSBNFUFS\dུd^ IPTUOBNFlBQJTEGTGPN ^ ݩϩά +40/ܗࣜ

Slide 28

Slide 28 text

$3&"5&&95&3/"-5"#-&*'/05&9*454BQJTFSWFSBQJMPH@@ AEBUFAUJNFTUBNQ AUISFBEATUSJOH AMFWFMATUSJOH AIBOEMFSATUSJOH A64&3@*%ATUSJOH AWFSTJPOATUSJOH A4&44*0/@*%ATUSJOH AOB[PATUSJOH ANFTTBHFATUSJOH AIPTUOBNFATUSJOH 308'03."54&3%&PSHPQFOYEBUBKTPOTFSEF+TPO4FS%F 8*5)4&3%&1301&35*&4 TFSJBMJ[BUJPOGPSNBU -0$"5*0/TMPHCVDLFUBQJTFSWFS %#5BCMF࡞੒

Slide 29

Slide 29 text

4&-&$5 '30.BQJMPH@@ 8)&3&VTFS@JE ΫΤϦ࣮ߦ

Slide 30

Slide 30 text

݁Ռ DTWͰͷ%-΋Մೳ

Slide 31

Slide 31 text

Β͘Β͘ڞ༗

Slide 32

Slide 32 text

ࣄ ྫ 4 ͷ ΞΫ η ε ϩ ά ͕ ݟ ͨ ͍

Slide 33

Slide 33 text

%FW 0QT -BNCEBͰը૾ॲཧͯ͠Δ͠ "1*͔Β΋ૢ࡞͍ͯ͠ΔͷͰɺ 4ΞΫηεϩάΈ͍ͨͰ͢ʂ XFCϖʔδೖͬͯΔΘ͚Ͱ΋ͳ͍͔ Β0/ʹͯ͠ͳ͔ͬͨΘʜ ઃఆ͠·͢

Slide 34

Slide 34 text

MPHHJOH0/ 0QT ϩάग़Δ·Ͱʹ਺͔͔࣌ؒΔͱॻ͍ͯ͋Δ ˞IUUQEPDTBXTBNB[PODPN ϩάLJCBOBͰݟΕ·͔͢Ͷʁ %&-&5&Λߦͳ͍ͬͯΔϦΫΤετ͕ݟ͍ͨ ɹ;Ή ɹFMBTUJDTFBSDIʹೖΕͯ ɹݕࡧͰ͖ΔΑ͏ʹ͠Α͏ %FW

Slide 35

Slide 35 text

ͯ͞FMBTUJDTFBSDIʹೖΕΔ४උ͠ͱ͔͘ɻ ͲΜͳײ͡ͷ΍͚ͭͩͬʁ 0QT φχίϨ IUUQEPDTBXTBNB[PODPN KB@KQ"NB[PO4MBUFTUEFW 4ΞΫηεϩάϩάܗࣜαϯϓϧ BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCFNZCVDLFU<'FC > BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCF&'&9".1-& 3&45(&57&34*0/*/((&5NZCVDLFU WFSTJPOJOH)5514$POTPMF

Slide 36

Slide 36 text

MPHTUBTIͰύʔεͰ͖Δ༷ʹͯ͠ʜ FMBTUJDTFBSDIʹೖΕͯʜ LJCBOBͰEBTICPBSE࡞ͬͯʜ

Slide 37

Slide 37 text

%&-&5&ͷϩάΛݟ͍͚ͨͩͳͷʹʜ ࣌ؒʜ͔͔Δʜͳ͊

Slide 38

Slide 38 text

ٹ ੈ ओ

Slide 39

Slide 39 text

$3&"5&&95&3/"-5"#-&*'/05&9*454T@BDDFTTMPHSFTPVSTF@CVDLFU #VDLFU0XOFSTUSJOH #VDLFUTUSJOH 3FRVFTU%BUF5JNFTUSJOH 3FNPUF*1TUSJOH 3FRVFTUFSTUSJOH 3FRVFTU*%TUSJOH 0QFSBUJPOTUSJOH ,FZTUSJOH 3FRVFTU63*@PQFSBUJPOTUSJOH 3FRVFTU63*@LFZTUSJOH 3FRVFTU63*@IUUQ1SPUPWFSTJPOTUSJOH )551TUBUVTTUSJOH &SSPS$PEFTUSJOH #ZUFT4FOUTUSJOH 0CKFDU4J[FTUSJOH 5PUBM5JNFTUSJOH 5VSO"SPVOE5JNFTUSJOH 3FGFSSFSTUSJOH 6TFS"HFOUTUSJOH 7FSTJPO*ETUSJOH 308'03."54&3%&PSHBQBDIFIBEPPQIJWFTFSEF3FHFY4FS%F 8*5)4&3%&1301&35*&4 TFSJBMJ[BUJPOGPSNBU JOQVUSFHFY aa< aa> aaa c aaa c<> aa -0$"5*0/bTBXTMPH4SFTPVSTF@CVDLFU %#5BCMF ࡞੒

Slide 40

Slide 40 text

4&-&$5SFNPUFJQ SFRVFTUFS DPVOU DPVOU '30.SFTPVSTF@CVDLFU 8)&3&SFRVFTUVSJ@PQFSBUJPO%&-&5& HSPVQCZSFNPUFJQ SFRVFTUFS ΫΤϦ࣮ߦ

Slide 41

Slide 41 text

SFNPUFJQSFRVFTUFS DPVOU BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS dུd BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ݁Ռ

Slide 42

Slide 42 text

·ͱΊ

Slide 43

Slide 43 text

· ͱ Ί • Amazon AthenaͰS3্ͷϩά͕ଈղੳՄೳঢ়ଶʹ  AthenaͰݟΔͨΊʹ~ΛS3ʹ͸͘Α͏ʹͯ͠…ͱ͔Ͱ͸ͳ͘ɺ  ͢ͰʹόοΫΞοϓͱ্͍ͯ͛ͯͨ͠ϩάશ͕ͯࣗಈతʹର৅ͱݴ͑Δɻ  AWSͷαʔϏε(S3 / CloudFront / ELB / CloudTrail)͔Βॻ͖ग़͢ϩά΋ɺ ௐ͚ࠪͩͳΒ͜ΕͰे෼ɻ • elasticsearch͔Β࡟আͯ͠΋S3ʹϩάϑΝΠϧ͋Ε͹ௐࠪ͸OK  ௕ظؒݟΕΔ༷ʹ͢ΔͨΊʹ  αʔόεϖοΫΛ্͓͛ͯ͘ඞཁ͸ແ͘ͳͬͨɻ  ˠ ຊ౰ʹඞཁͳظؒͷΈͰΑ͘ͳͬͨɻ  ɹ→ snapshot͸ͪΌΜͱऔΖ͏+͋Δఔ౓ظؒ࣋ͬͯͯ΋Α͔ͬͨ(൓ল)

Slide 44

Slide 44 text

· ͱ Ί • ϦΞϧλΠϜͷϩάղੳ → elasticsearch + kibana • S3ʹ͓͍ͨϩάௐࠪ → Amazon Athena

Slide 45

Slide 45 text

& - , T U B D L Ͱ ͷ ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ " N B [ P O " U I F O B ͕ औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏ ࿩ Ͱ ͠ ͨ

Slide 46

Slide 46 text

ޚ ਗ਼ ௌ ͋ Γ ͕ ͱ ͏ ͝ ͟ ͍ · ͠ ͨ