Tweet
Tweet

Slide 1

Slide 1 text

Intro  to  the  ELK  Stack   Log  Analy4cs  …  and  beyond   Dave  Erickson   dave@elas-c.co   May  20th,  2015  

Slide 2

Slide 2 text

According  to  the  Meet-­‐up  Agenda…   •  What  the  Meetup  Said   –  “Through  The  Lens  of  Github,  Mozilla,  Wikipedia”   –  “Key  Metrics  that  will  make  your  projects  more   successful”   –  Elas4csearch  &  .NET   •  I’m  Going  to  Go  Off-­‐Script   –  ELK  Stack   –  Live  Demo   –  Fun  Use  Cases!   –  Awesome  stuff  in  the  Roadmaps  (we  have  many)   –  A  Tale  of  Two  .NET  Clients  

Slide 3

Slide 3 text

It’s  not  E.L.K.,                  it’s  elk  

Slide 4

Slide 4 text

Why  is  geWng  value  out  of  logs  hard?   93.114.45.13  -­‐  -­‐  [16/Feb/2014:09:47:04  -­‐0500]  "GET  /favicon.ico  HTTP/1.1"  200  3638  "-­‐"  "Mozilla/5.0  (X11;  Linux  x86_64;  rv:25.0)  Gecko/ 20100101  Firefox/25.0"   93.114.45.13  -­‐  -­‐  [16/Feb/2014:09:47:04  -­‐0500]  "GET  /images/jordan-­‐80.png  HTTP/1.1"  200  6146  "hmp://www.semicomplete.com/ar4cles/ dynamic-­‐dns-­‐with-­‐dhcp/"  "Mozilla/5.0  (X11;  Linux  x86_64;  rv:25.0)  Gecko/20100101  Firefox/25.0"   93.114.45.13  -­‐  -­‐  [16/Feb/2014:09:47:04  -­‐0500]  "GET  /images/web/2009/banner.png  HTTP/1.1"  200  52315  "hmp://www.semicomplete.com/ style2.css"  "Mozilla/5.0  (X11;  Linux  x86_64;  rv:25.0)  Gecko/20100101  Firefox/25.0"   66.249.73.135  -­‐  -­‐  [16/Feb/2014:09:47:34  -­‐0500]  "GET  /blog/tags/ipv6  HTTP/1.1"  200  12251  "-­‐"  "Mozilla/5.0  (iPhone;  CPU  iPhone  OS  6_0  like  Mac   OS  X)  AppleWebKit/536.26  (KHTML,  like  Gecko)  Version/6.0  Mobile/10A5376e  Safari/8536.25  (compa4ble;  Googlebot/2.1;  +hmp:// www.google.com/bot.html)"   50.16.19.13  -­‐  -­‐  [16/Feb/2014:09:47:46  -­‐0500]  "GET  /blog/tags/puppet?flav=rss20  HTTP/1.1"  200  14872  "hmp://www.semicomplete.com/blog/ tags/puppet?flav=rss20"  "Tiny  Tiny  RSS/1.11  (hmp://m-­‐rss.org/)"   66.249.73.185  -­‐  -­‐  [16/Feb/2014:09:47:54  -­‐0500]  "GET  /  HTTP/1.1"  200  37932  "-­‐"  "Mozilla/5.0  (compa4ble;  Googlebot/2.1;  +hmp:// www.google.com/bot.html)"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:42  -­‐0500]  "GET  /projects/xdotool/  HTTP/1.1"  200  12292  "hmp://www.google.com/url? sa=t&rct=j&q=&esrc=s&source=web&cd=5&cad=rja&sqi=2&ved=0CFYQFjAE&url=hmp%3A%2F%2Fwww.semicomplete.com%2Fprojects %2Fxdotool%2F&ei=6cwAU_bRHo6urAeI0YD4Ag&usg=AFQjCNE3V_aCf3-­‐gfNcbS924S6jZ6FqffA&bvm=bv.61535280,d.bmk"  "Mozilla/5.0   (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"   46.105.14.53  -­‐  -­‐  [16/Feb/2014:09:48:48  -­‐0500]  "GET  /blog/tags/puppet?flav=rss20  HTTP/1.1"  200  14872  "-­‐"  "UniversalFeedParser/4.2-­‐pre-­‐314-­‐ svn  +hmp://feedparser.org/"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /reset.css  HTTP/1.1"  200  1015  "hmp://www.semicomplete.com/projects/xdotool/"   "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /style2.css  HTTP/1.1"  200  4877  "hmp://www.semicomplete.com/projects/xdotool/"   "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /favicon.ico  HTTP/1.1"  200  3638  "-­‐"  "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)   Gecko/20100101  Firefox/28.0"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /images/jordan-­‐80.png  HTTP/1.1"  200  6146  "hmp://www.semicomplete.com/projects/ xdotool/"  "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"   123.125.71.35  -­‐  -­‐  [16/Feb/2014:09:49:02  -­‐0500]  "GET  /blog/tags/release  HTTP/1.1"  200  40693  "-­‐"  "Mozilla/5.0  (compa4ble;  Baiduspider/2.0;   +hmp://www.baidu.com/search/spider.html)"   110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /images/web/2009/banner.png  HTTP/1.1"  200  52315  "hmp://www.semicomplete.com/ style2.css"  "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"   Logs  are  have  no  standard  format   There  is  no  consistency   You  have  to  be  an  expert  to  read  them   Almost  always  stored  somewhere  that’s  hard  for   the  organiza4on  to  get  to.   grep  and  regular  expressions  don’t  scale   I  can  help!  

Slide 5

Slide 5 text

ELK  Stack   Elas4csearch,  Logstash,  and  Kibana   Logstash   Elas4csearch   Kibana   Key:  

Slide 6

Slide 6 text

Logstash   •  Input  from  many  Sources   –  It’s  really  good  at  parsing  logs  (shocking!)   –  Other  sources  too   •  Files,  Queues,  Messages,  Databases,  etc   –  Hundreds  of  plugins   •  Transform  and  Enrich   –  GROK   –  IP  -­‐-­‐>  Geospa4al   –  Conver4ng  to  JSON  is  very  popular   •  Output  to  Many  Des4na4ons   –  Databases,  Dashboards,  Elas4csearch  …   –  many  others  

Slide 7

Slide 7 text

Elas4csearch   •  Based  on  Lucene   –  Wicked  fast   –  Good  at  search   –  Good  at  analy4cs   •  Things  it  adds   –  Horizontal  Scaling   –  High  Availability   –  Ease  of  Use   –   “Near  Real-­‐Time”  

Slide 8

Slide 8 text

Kibana   •  UI  for  Elas4csearch   –  Lightweight  Search  Interface   –  Discover   –  Visualize   –  Dashboards   •  Explore  and  Understand  your   data   •  Scale  and  Speed   –  Heavy  li|ing  s4ll  done  in   Elas4csearch  

Slide 9

Slide 9 text

Let’s  take  an  example  

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

Regular  Expression  

Slide 12

Slide 12 text

Grok  PaJern  

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

{          4me:          “120819”,          query_4me:      27.115751,          lock_4me:        0.00007          rows_sent:        55996,          rows_examined:    56000          query_4mestamp:    “1345373510”,          query:          “SELECT  ID  FROM  wp_posts  WHERE  …”   }  

Slide 19

Slide 19 text

Logstash  Pipeline  

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

ELK  Stack   Elas4csearch,  Logstash,  and  Kibana   Logstash   Elas4csearch   Kibana   Key:  

Slide 23

Slide 23 text

What  usually  ends  up  happening  once   someone  makes  something  useful   Logstash   Elas4csearch   Kibana   Key:   ES-­‐Hadoop   Rela-onal   Hadoop   Non-­‐Rela-onal   Apps  and     Mobile   Language   Clients  

Slide 24

Slide 24 text

Scaled  Architectures   Logstash   Elas4csearch   Kibana   Key:   master   master   master   …   …   …   queue   queue   queue   …   Agents   (shippers)   route  /  buffer   collect   process   Index  /  alert   view  

Slide 25

Slide 25 text

Going  Beyond  Logs   •  Search   – Relevance   – Faceted  Naviga4on   – Human  Language   – Unstructured  Data   – Rich  Syntax   •  Analy4cs   –  Real  Time   –  Significant  Terms   –  En4ty  Oriented  Indexing   –  Make  data  available  to   the  whole  organiza4on   –  Use  ELK  to  figure  out   what  ques4ons  to  start   asking  

Slide 26

Slide 26 text

Use  cases   hmps://www.elas4c.co/elas4con/2015/sf/videos/  

Slide 27

Slide 27 text

Use  Case:  MozDef   •  Large  Scale  Security  Challenge   –  300  Million  events  per  day   •  Amackers   –  Innova4ve   –  Real  Time   –  Adap4ve   •  Exis4ng  SIEM  op4ons   –  Closed  Systems   –  Proprietary   –  Lack  of  API’s  &  Endpoints   •  Solu4on:  built  their  own   –  Mozilla  Defense  Pla€orm   –  Open  Source  SIEM  overlay  for   Elas4csearch  

Slide 28

Slide 28 text

Use  Case:  Wikimedia   •  265+  Languages   •  3.1  B  prefix  searches  /   month   •  870  M  text  searches  /  month   •  Real  Time   –  Rewards  Contributors   –  Fixes  Vandalism   •  Plugins   –  Contributed  many   •  Expressive  Syntax   –  Fix  search  without   redeployment  of  search   infrastructure   •  No  Down4me  Management   –  Aliases  

Slide 29

Slide 29 text

Use  Case:  Github  Search   A  Story  About  Learning  to  Scale  

Slide 30

Slide 30 text

Use  Case:  NASA  JPL  Telemetry  

Slide 31

Slide 31 text

Awesome  Stuff  in  the  road  map   •  Marvel  2.0  &  Shield  2.0   •  Watcher    (Aler4ng  for  Elas4csearch)   •  Logstash   –  Cleaner  Plugins   –  Bemer  Windows  Support   –  More  inputs  (Ka•a,  Avro  ...  etc.)   –  Buffering,  Clustering,  Resiliency   •  Elas4csearch   –  Aggrega4ons  2.0  (name  pending)   –  Changes  API,  Re-­‐indexing  API   •  Elas4csearch  Hadoop  Connector   –  Na4ve  support  for  Spark,  Storm,  Yarn   –  Exposing  ES  internals  to  Machine  Learning   •  Kibana   –  Rapidly  evolving  

Slide 32

Slide 32 text

Elas4csearch  &  .Net  

Slide 33

Slide 33 text

Two  Clients  

Slide 34

Slide 34 text

•  Low  Level:  Elas4csearch.NET   –  No  dependencies,  load  balancing,  failover   •  High  Level:  NEST  (v1.5  just  released)   –  Strongly  Typed  /  Fluent  API   –  Dependencies:  Json.NET,  Elas4csearch.NET   –  Plain  Old  C#  Objects  (POCOs)   –  Annota4on  driven  Mappings  (Don’t  repeat  yourself!)   –  Wraps  and  Exposes!   •  Directly  Input  Boolean  filters  as  DSL   •  Boolean  bitwise  operators  mapped  to  Elas4csearch  DSL  

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

Magic,  when  you  need  it  

Slide 37

Slide 37 text

Key  things  to  make  your  projects  more   successful?   •  Understand  Distributed  Systems  lingo  /  invarients   –  “Master”  =/=  “Primary”   –  Don’t  model  your  data  rela4onally   •  Read  the  book   –  HTML  version  of  O’Reilly  book  is  on  Elas4c.co  >  Learn   (Elas4csearch  –  The  Defini4ve  Guide)    …  it’s  really  good         •  It’s  not  a  silver  bullet   –  No  Such  Thing  As  a  Free  Lunch  

Slide 38

Slide 38 text

THANKS!    QUESTIONS?   dave@elas4c.co