AWSのマネージドサービスを活用したログ可視化

by Shinji Fujimoto

Slide 1

Slide 1 text

AWS ͷ αʔϏεΛ׆༻ͨ͠ ϩάՄࢹԽ ߹ಉษڧձ in ෱Ԭ 2017/02/11 ౻ຊ ਅ࢘

Slide 2

Slide 2 text

ࣗݾ঺հ • ౻ຊ ਅ࢘ • Work  AWS Solutions Architect  @Classmethod, Inc. • Interest  Elastic Stack

Slide 3

Slide 3 text

ձࣾ঺հ

Slide 4

Slide 4 text

ձࣾ঺հ "84΍ϞόΠϧɺϏοάσʔλͳͲ ࣾһ૯ग़Ͱ޷͖ͳΑ͏ʹϒϩάॻ͍͍ͯ·͢

Slide 5

Slide 5 text

AWSࣄۀ෦঺հ

Slide 6

Slide 6 text

AWSࣄۀ෦঺հ ओʹ"84ʹؔ͢Δϒϩάॻ͍͍ͯ·͢

Slide 7

Slide 7 text

ຊ೔఻͍͑ͨ͜ͱ

Slide 8

Slide 8 text

ϩάͷՄࢹԽ͕ ؆୯ʹͳΓ·ͨ͠

Slide 9

Slide 9 text

ΞδΣϯμ • ϩάՄࢹԽͬͯʁ • AWSͷαʔϏεΛ׆༻ͨ͠ϩάՄࢹԽ • Demo

Slide 10

Slide 10 text

ϩάՄࢹԽͬͯʁ

Slide 11

Slide 11 text

Έͳ͞Μ ϩάΛ׆༻͍ͯ͠·͔͢ʁ

Slide 12

Slide 12 text

ϩάͬͯԿ͚ͩͬʁ • ޠݯ • ߤւͷધͷ଎౓ΛଌΔͨΊʹ  ؙଠ(log)ʹҰఆִؒͷ͋ΔೄΛׅΓ෇͚ ͨ΋ͷ  ʢग़యɿWikipediaʣ

Slide 13

Slide 13 text

ԿͰϩά͕ඞཁʁ • ؂ࢹɺΞϥʔςΟϯά • ো֐ௐࠪ • ύϑΥʔϚϯεղੳ • σʔλ෼ੳ

Slide 14

Slide 14 text

ԿͰϩάͷՄࢹԽ͕ඞཁʁ • มԽͷؾ͖ͮΛಘΔ • ো֐ͷ༧ஹɺधཁͷ༧ଌ • આ໌ͷઆಘྗ • จࣈ਺ࣈΑΓ΋Ұ໨ྎવ

Slide 15

Slide 15 text

ϩάՄࢹԽͷ՝୊

Slide 16

Slide 16 text

ϩάՄࢹԽͷ՝୊ • δϣϒεέδϡʔϦϯά • ETL • ՄࢹԽπʔϧͷબఆ

Slide 17

Slide 17 text

ϩάՄࢹԽͷ՝୊ • δϣϒεέδϡʔϥ • ϦΞϧλΠϜͳͷ͔ɺόονͳͷ͔ • ੒ޭɺࣦഊͷ൑ఆ • ϦτϥΠॲཧ

Slide 18

Slide 18 text

ϩάՄࢹԽͷ՝୊ • Extract • Ͳ͔͜ΒϩάΛऔಘ͢Δ͔ • Transform • ՄࢹԽ͍ͨ͠৘ใɺܗࣜ΁ͷม׵ • Load • ՄࢹԽπʔϧ΁ͷσʔλ౤ೖ

Slide 19

Slide 19 text

ϩάՄࢹԽͷ՝୊ • ՄࢹԽπʔϧͷબఆ • Kibana • Grafana • re:Dash • Superset • Google Data Studio • etc.. etc..

Slide 20

Slide 20 text

AWS ʹ͓͚ΔϩάՄࢹԽ

Slide 21

Slide 21 text

Amazon Elasticsearch Service

Slide 22

Slide 22 text

Amazon Elasticsearch Service • Elasticsearch • ݕࡧʗ෼ੳΤϯδϯ • Kibana • ՄࢹԽɺμογϡϘʔυԽ • ४ϦΞϧλΠϜʹՄࢹԽ

Slide 23

Slide 23 text

Amazon Elasticsearch Service • Elasticsearch + Kibana ͷϚωʔδυαʔϏε • ਺ΫϦοΫͰ؀ڥߏங • ӡ༻͸ AWS ʹ͓೚ͤ • ೔࣍εφοϓγϣοτͷऔಘ • ো֐࣌ͷ෮چରԠ

Slide 24

Slide 24 text

Amazon Elasticsearch Service Lambda S3 EC2 Elasticsearch Service

Slide 25

Slide 25 text

Amazon Elasticsearch Service • 2017/01/31 Ξοϓσʔτ • Elasticsearch / Kibana 5 ܥʹରԠ • εΫϦϓτͷར༻Մ • Ingest Node • Kibana ͷ DevTools ΍ Timelion Λར༻Մ

Slide 26

Slide 26 text

Amazon Athena & Amazon QuickSight

Slide 27

Slide 27 text

͜ͷαʔϏε஌͍ͬͯ·͔͢ʁ Amazon Athena Amazon QuickSight

Slide 28

Slide 28 text

Amazon Athena • AWS re:Invent 2016 ൃදˍϦϦʔε • S3 ͷඇߏ଄Խσʔλʹ SQL Ͱߏ଄Խσʔλ ΛऔΓग़͢

Slide 29

Slide 29 text

Amazon Athena • αʔόʔϨε • ӡ༻ෆཁɻ࢖͍͍ͨ࣌ʹ࢖͑Δ • ߴ଎ͳύϑΥʔϚϯε • 1000ສ݅ͷςΩετͷूܭ͕਺ඵ • ௿ίετ • εΩϟϯର৅αΠζ՝ۚɹ5$ / TB

Slide 30

Slide 30 text

Amazon QuickSight • AWS re:Invent 2015 ൃදɻ2016/11ϦϦʔε • AWS ۘ੡ BI πʔϧ

Slide 31

Slide 31 text

Amazon QuickSight • αʔόʔϨε • SPICE ʹΑΔߴ଎ͳՄࢹԽ • ଟ͘ͷ AWS αʔϏεʹରԠ • ௿ίετͳ BI πʔϧ

Slide 32

Slide 32 text

Amazon Athena & QuickSight

Slide 33

Slide 33 text

࢖͍ॲ ʢ˞ϩάՄࢹԽͷ؍఺ʣ

Slide 34

Slide 34 text

Ϣʔεέʔε • Elasticsearch Service • χΞϦΞϧλΠϜͳଟ࣍ݩ෼ੳ • Athena & QuickSight • ΞυϗοΫͳ෼ੳ

Slide 35

Slide 35 text

ར༻ίετ • Elasticsearch Service • ElasticsearchΫϥελͷىಈ࣌ؒ • ྫ͑͹ɺr3.large 50GBσΟεΫ x 3୆ ౦ژϦʔδϣϯͷ৔߹ɺ$659 / ݄

Slide 36

Slide 36 text

ར༻ίετ • Athena & QuickSight • Athena ͸ΫΤϦͨ͠ϑΝΠϧαΠζ • QuickSight ͸Ϣʔβʔˍ௥ՃσΟεΫ • ྫ͑͹ɺ5Ϣʔβʔ 10TB εΩϟϯ  $95 / ݄

Slide 37

Slide 37 text

ͦͷଞίετ • Elasticsearch Service • Elasticsearch ΁ϩάΛೖΕΔͨΊͷ࢓૊Έ ͕ඞཁ • Kibana ͸ॳݟͰ৮Δʹ͸ਏ͍ • Athena & QuickSight • Athena ͸ SQL ͕৮ΕΕ͹࢖͑ͨ • QuickSight ͸ॳݟͰ৮Εͨ

Slide 38

Slide 38 text

Demo

Slide 39

Slide 39 text

·ͱΊ • ϩάͬͯ஍ຯ͚ͩͲେࣄ • ϩάՄࢹԽͷબ୒ࢶ͕௥Ճ͞Εͨ • ༻్ʹ͋ͬͨՄࢹԽΛબ୒ • ՄࢹԽָ͍ͬͯ͠