Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ELKstackとAthenaの素敵な関係

Ef94c9a26592ba254c9c774cc0a4515c?s=47
February 10, 2017

 ELKstackとAthenaの素敵な関係

2017/02/10 JAWS-UG#9で発表した資料になります

Ef94c9a26592ba254c9c774cc0a4515c?s=128

February 10, 2017
Tweet

More Decks by 遊

Other Decks in Technology

Transcript

  1. & - , T U B D L ͱ "

    U I F O B ͷ  ૉ ఢ ͳ ؔ ܎ JAWS-UGԣ඿ #9 - Operational Excellence 2017/02/10 ੢ଜ ༡
  2. ˞ ຊ εϥ Π υ ͸  & - ,

    T U B D L ͱ " N B [ P O " U I F O B Λ  ٕ ज़ త ʹ ࿈ ܞ ͞ ͤ ͨ ࿩ Ͱ ͸ ͋ Γ · ͤ Μ  ͢ ͍ · ͤ Μ ʂ  ͱ ͍ ͏ ͔ ٕ ज़ త ͳ ࿩ ͋ Γ · ͤ Μ
  3. & - , T U B D L Ͱ ͷ

    ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ  " N B [ P O  " U I F O B ͕  औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ  ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏ ࿩
  4. ͔ ͍ ͠ Ό ͍ Μ  ͤ ͍ ΂

    ͭ ɿ ͓ ͱ ͜  Ϩ ϕ ϧ ɿ    ) 1         . 1       ࣗ ݾ ঺ հ ࣾ ձ ਓ ྺ         d        ɹ % $ Ͱ ؂ ࢹ Φϖ Ϩ ʔ λ ʔ         d        ɹ ๭ ι ʔ γ ϟϧ ήʔϜ ܥ ձ ࣾ         d  / 0 8 ɹ ɹ  ג ࣜ ձ ࣾ " 1 1 - * # 05  "8 4 ྺ    ໿  ೥ ͳ · ͑ ɹ ɹ ɿ ੢ ଜ ɹ ༡  ͭ ͍ ͬ ͨ ʔ ɿ !HBDIBSJPO
  5. None
  6. None
  7. &-,TUBDL "NB[PO"UIFOB  FMBTUJDTFBSDI LJCBOBͰͷϩάࢀর ٹੈओ"UIFOB ӡ༻5JQTతͳ ҰԠ৮Ε͓ͯ͘΂͖͜ͱ ·ͱΊ ໨

  8. & - , T U B D L • elastic

    search + log stash + kibana = elk stack • L ͷ෦෼ʢσʔλ౤ೖʣ͸fluentdͩͬͨΓ͢Δͱ EFKͱͳΔΒ͍͠ɻ • elasticsearchʹೖΕͯkibanaͰݟͯΔͷͰ
 εϥΠυ಺Ͱ͸૯শͱͯ͠࢖ͬͯ·͢ɻ
  9. " N B [ P O  " U I

    F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon)
 Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠
 ΫΤϦΛ౤͛Δͱɺ਺ඵͰ݁Ռ͕ฦͬͯ͘Δɻ
  10. " N B [ P O  " U I

    F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon)
 Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠
 ΫΤϦΛ౤͛Δͱɺ਺ඵͰ݁Ռ͕ฦͬͯ͘Δɻ ˢϚδͰ ਆ͡Όͳ͍͔͢
  11. FMBTUJDTFBSDI LJCBOB Ͱͷϩάࢀর

  12. Ͷ Β ͍ • ͱΓ͋͑ͣௐࠪʹ࢖͏ϩά͸elasticsearch • ֤αʔόͷϩάΛfluentdͰू໿ɺ౤ೖ • ݸʑͷαʔόͷΈʹ͔͠࢒Βͳ͍ϩάΛͳ͘͢
 ໨త΋͋Γ

    • ͨ·ʹlogstash • ΫΤϦʹΑΔΞϥʔτઃఆʢࣗલεΫϦϓτʣ
  13. $ V T U P N  - 0 (

  14. & - #

  15. 3 % 4  

  16. F M B T U J D T F B

    S D I ӡ ༻ ϧ ʔ ϧ త ͳ • ೖΕΔσʔλ͸ϑΝΠϧͱͯ͠΋ॻ͖ग़͓ͯ͘͠ʢJSONʣ • dailyͰgzipͯ͠S3΁backup • ྔͷଟ͍ϩάʢ1೔10GBҎ্ʣஷ·Δϩά͸3೔Ͱindexຖ࡟আ • Ҏલɺ௕ظͰݟΕΔΑ͏ʹ͍͕ͯͨ͠ɺkibanaද͕͔ࣔͳΓ஗͘ͳͬͯ͠·ͬͨ • ͦ΋ͦ΋ͦΜͳʹաڈͷΛݟͳ͔ͬͨ • ετϨʔδ΋༰ྔΛऔΒͳͯ͘͸͍͚ͳ͍ͨΊ௕ظؒอ࣋ͨ͘͠ͳ͔ͬͨ • εφοϓγϣοτ͸ͱͬͨΓͱͬͯͳ͔ͬͨΓ(ũųųƅƁ • ͱ͍ͬͯͨ΍ͭ΋Ұिؒ΄ͲͰফ͍ͯͨ͠ɻ • ࠓ͸rundeck͞ΜͰຖ೔ࣗಈ࣮ߦ؅ཧɻS3ʹอଘɻ
  17. ໰୊఺

  18. ໰ ୊ ఺ • index࡟আޙͷௐࠪ • ݟ͍ͨϩά͸΄ΜͷҰ෦෼ • snapshotͷϦετΞͰ͖ͳ͍৔߹ •

    ਺ेGBͷϩάΛDLͯ͠ղౚͯ͠grepͯ͠ɾɾɾ
  19. ໰ ୊ ఺ • elasticsearchʹೖΕΔͷ΋ͦΕͳΓʹ༻ҙ͕ඞཁ • ύʔαʔ୳ͨ͠Γɺॻ͍ͨΓ • template࡞ͬͨΓ(dynamic templateͰେମରԠ)

    • ٸͳґཔʹରԠ͖͠Εͳ͍
  20. ٹ ੈ ओ

  21. ࣄ ྫ  ໰ ͍ ߹ Θ ͤ ௐ ࠪ

    ͷ ҝ ɺ  ա ڈ ͷ " 1 * ϩ ά ͕ Έ ͨ ͍
  22. ͦ Ε ͳ Β L J C B O B

    Ͱ ݟ Ε · ͬ ͤ ʂ
  23. ͦ Ε ͳ Β L J C B O B

    Ͱ ݟ Ε · ͬ ͤ ʂ Ұिؒ΄ͲલͳΜͰ͕͢ʜ
  24. T O B Q T I P U ΋ ͏

    ͳ ͍ Θ ʜ Ұिؒ΄ͲલͳΜͰ͕͢ʜ
  25. 4͔Βϩά (#௒ Λ%-ͯ͠ղౚͯ͠HSFQ ͭΒ͍

  26. ٹ ੈ ओ

  27. \ EBUFz  UISFBElŠl  MFWFM*/'0  IBOEMFSlŠl  64&3@*%lŠl

     WFSTJPOlŠl  4&44*0/@*%lŠl  OB[P  NFTTBHFl63-IUUQ  1BSBNFUFS\dུd^  IPTUOBNFlBQJTEGTGPN ^ ݩϩά +40/ܗࣜ
  28. $3&"5&&95&3/"-5"#-&*'/05&9*454BQJTFSWFSBQJMPH@@  AEBUFAUJNFTUBNQ  AUISFBEATUSJOH  AMFWFMATUSJOH  AIBOEMFSATUSJOH 

    A64&3@*%ATUSJOH  AWFSTJPOATUSJOH  A4&44*0/@*%ATUSJOH  AOB[PATUSJOH  ANFTTBHFATUSJOH  AIPTUOBNFATUSJOH  308'03."54&3%&PSHPQFOYEBUBKTPOTFSEF+TPO4FS%F 8*5)4&3%&1301&35*&4  TFSJBMJ[BUJPOGPSNBU -0$"5*0/TMPHCVDLFUBQJTFSWFS %#5BCMF࡞੒
  29. 4&-&$5  '30.BQJMPH@@ 8)&3&VTFS@JE ΫΤϦ࣮ߦ

  30. ݁Ռ DTWͰͷ%-΋Մೳ

  31. Β͘Β͘ڞ༗

  32. ࣄ ྫ   4  ͷ ΞΫ η ε

    ϩ ά ͕ ݟ ͨ ͍
  33. %FW 0QT -BNCEBͰը૾ॲཧͯ͠Δ͠ "1*͔Β΋ૢ࡞͍ͯ͠ΔͷͰɺ 4ΞΫηεϩάΈ͍ͨͰ͢ʂ XFCϖʔδೖͬͯΔΘ͚Ͱ΋ͳ͍͔ Β0/ʹͯ͠ͳ͔ͬͨΘʜ  ઃఆ͠·͢

  34. MPHHJOH0/ 0QT ϩάग़Δ·Ͱʹ਺͔͔࣌ؒΔͱॻ͍ͯ͋Δ ˞IUUQEPDTBXTBNB[PODPN ϩάLJCBOBͰݟΕ·͔͢Ͷʁ %&-&5&Λߦͳ͍ͬͯΔϦΫΤετ͕ݟ͍ͨ ɹ;Ή ɹFMBTUJDTFBSDIʹೖΕͯ ɹݕࡧͰ͖ΔΑ͏ʹ͠Α͏ %FW

  35. ͯ͞FMBTUJDTFBSDIʹೖΕΔ४උ͠ͱ͔͘ɻ ͲΜͳײ͡ͷ΍͚ͭͩͬʁ 0QT φχίϨ IUUQEPDTBXTBNB[PODPN KB@KQ"NB[PO4MBUFTUEFW 4ΞΫηεϩάϩάܗࣜαϯϓϧ BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCFNZCVDLFU<'FC  >

    BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCF&'&9".1-& 3&45(&57&34*0/*/((&5NZCVDLFU WFSTJPOJOH)5514$POTPMF
  36. MPHTUBTIͰύʔεͰ͖Δ༷ʹͯ͠ʜ FMBTUJDTFBSDIʹೖΕͯʜ LJCBOBͰEBTICPBSE࡞ͬͯʜ

  37. %&-&5&ͷϩάΛݟ͍͚ͨͩͳͷʹʜ ࣌ؒʜ͔͔Δʜͳ͊

  38. ٹ ੈ ओ

  39. $3&"5&&95&3/"-5"#-&*'/05&9*454T@BDDFTTMPHSFTPVSTF@CVDLFU  #VDLFU0XOFSTUSJOH  #VDLFUTUSJOH  3FRVFTU%BUF5JNFTUSJOH  3FNPUF*1TUSJOH 

    3FRVFTUFSTUSJOH  3FRVFTU*%TUSJOH  0QFSBUJPOTUSJOH  ,FZTUSJOH  3FRVFTU63*@PQFSBUJPOTUSJOH  3FRVFTU63*@LFZTUSJOH  3FRVFTU63*@IUUQ1SPUPWFSTJPOTUSJOH  )551TUBUVTTUSJOH  &SSPS$PEFTUSJOH  #ZUFT4FOUTUSJOH  0CKFDU4J[FTUSJOH  5PUBM5JNFTUSJOH  5VSO"SPVOE5JNFTUSJOH  3FGFSSFSTUSJOH  6TFS"HFOUTUSJOH  7FSTJPO*ETUSJOH  308'03."54&3%&PSHBQBDIFIBEPPQIJWFTFSEF3FHFY4FS%F 8*5)4&3%&1301&35*&4  TFSJBMJ[BUJPOGPSNBU JOQVUSFHFY <?>  <?> aa<  aa> <?>  <?>  <?>  <?>  <?> aaa <?>  <?>  c<?> aaa c<>  <?>  <?>  <?>  <?>  <?>  <?>  a<?a> a  <?>  -0$"5*0/bTBXTMPH4SFTPVSTF@CVDLFU %#5BCMF ࡞੒
  40. 4&-&$5SFNPUFJQ  SFRVFTUFS  DPVOU  DPVOU '30.SFTPVSTF@CVDLFU 8)&3&SFRVFTUVSJ@PQFSBUJPO%&-&5& HSPVQCZSFNPUFJQ

    SFRVFTUFS ΫΤϦ࣮ߦ
  41.  SFNPUFJQSFRVFTUFS DPVOU   BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS    

    BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS    BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS  dུd  BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS   ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS   ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS   ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS   ݁Ռ
  42. ·ͱΊ

  43. · ͱ Ί • Amazon AthenaͰS3্ͷϩά͕ଈղੳՄೳঢ়ଶʹ
 AthenaͰݟΔͨΊʹ~ΛS3ʹ͸͘Α͏ʹͯ͠…ͱ͔Ͱ͸ͳ͘ɺ
 ͢ͰʹόοΫΞοϓͱ্͍ͯ͛ͯͨ͠ϩάશ͕ͯࣗಈతʹର৅ͱݴ͑Δɻ
 AWSͷαʔϏε(S3 /

    CloudFront / ELB / CloudTrail)͔Βॻ͖ग़͢ϩά΋ɺ ௐ͚ࠪͩͳΒ͜ΕͰे෼ɻ • elasticsearch͔Β࡟আͯ͠΋S3ʹϩάϑΝΠϧ͋Ε͹ௐࠪ͸OK
 ௕ظؒݟΕΔ༷ʹ͢ΔͨΊʹ
 αʔόεϖοΫΛ্͓͛ͯ͘ඞཁ͸ແ͘ͳͬͨɻ
 ˠ ຊ౰ʹඞཁͳظؒͷΈͰΑ͘ͳͬͨɻ
 ɹ→ snapshot͸ͪΌΜͱऔΖ͏+͋Δఔ౓ظؒ࣋ͬͯͯ΋Α͔ͬͨ(൓ল)
  44. · ͱ Ί • ϦΞϧλΠϜͷϩάղੳ → elasticsearch + kibana •

    S3ʹ͓͍ͨϩάௐࠪ → Amazon Athena
  45. & - , T U B D L Ͱ ͷ

    ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ  " N B [ P O  " U I F O B ͕  औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ  ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏ ࿩ Ͱ ͠ ͨ
  46. ޚ ਗ਼ ௌ  ͋ Γ ͕ ͱ ͏ ͝

    ͟ ͍ · ͠ ͨ