Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intro to ELK Stack for Chicago .Net Meetup

Intro to ELK Stack for Chicago .Net Meetup

Dave Erickson 2015-05-20

Elasticsearch Inc

May 20, 2015
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. Intro  to  the  ELK  Stack  
    Log  Analy4cs  …  and  beyond  
    Dave  Erickson  
    [email protected]  
    May  20th,  2015  

    View full-size slide

  2. According  to  the  Meet-­‐up  Agenda…  
    •  What  the  Meetup  Said  
    –  “Through  The  Lens  of  Github,  Mozilla,  Wikipedia”  
    –  “Key  Metrics  that  will  make  your  projects  more  
    successful”  
    –  Elas4csearch  &  .NET  
    •  I’m  Going  to  Go  Off-­‐Script  
    –  ELK  Stack  
    –  Live  Demo  
    –  Fun  Use  Cases!  
    –  Awesome  stuff  in  the  Roadmaps  (we  have  many)  
    –  A  Tale  of  Two  .NET  Clients  

    View full-size slide

  3. It’s  not  E.L.K.,    
                 it’s  elk  

    View full-size slide

  4. Why  is  geWng  value  out  of  logs  hard?  
    93.114.45.13  -­‐  -­‐  [16/Feb/2014:09:47:04  -­‐0500]  "GET  /favicon.ico  HTTP/1.1"  200  3638  "-­‐"  "Mozilla/5.0  (X11;  Linux  x86_64;  rv:25.0)  Gecko/
    20100101  Firefox/25.0"  
    93.114.45.13  -­‐  -­‐  [16/Feb/2014:09:47:04  -­‐0500]  "GET  /images/jordan-­‐80.png  HTTP/1.1"  200  6146  "hmp://www.semicomplete.com/ar4cles/
    dynamic-­‐dns-­‐with-­‐dhcp/"  "Mozilla/5.0  (X11;  Linux  x86_64;  rv:25.0)  Gecko/20100101  Firefox/25.0"  
    93.114.45.13  -­‐  -­‐  [16/Feb/2014:09:47:04  -­‐0500]  "GET  /images/web/2009/banner.png  HTTP/1.1"  200  52315  "hmp://www.semicomplete.com/
    style2.css"  "Mozilla/5.0  (X11;  Linux  x86_64;  rv:25.0)  Gecko/20100101  Firefox/25.0"  
    66.249.73.135  -­‐  -­‐  [16/Feb/2014:09:47:34  -­‐0500]  "GET  /blog/tags/ipv6  HTTP/1.1"  200  12251  "-­‐"  "Mozilla/5.0  (iPhone;  CPU  iPhone  OS  6_0  like  Mac  
    OS  X)  AppleWebKit/536.26  (KHTML,  like  Gecko)  Version/6.0  Mobile/10A5376e  Safari/8536.25  (compa4ble;  Googlebot/2.1;  +hmp://
    www.google.com/bot.html)"  
    50.16.19.13  -­‐  -­‐  [16/Feb/2014:09:47:46  -­‐0500]  "GET  /blog/tags/puppet?flav=rss20  HTTP/1.1"  200  14872  "hmp://www.semicomplete.com/blog/
    tags/puppet?flav=rss20"  "Tiny  Tiny  RSS/1.11  (hmp://m-­‐rss.org/)"  
    66.249.73.185  -­‐  -­‐  [16/Feb/2014:09:47:54  -­‐0500]  "GET  /  HTTP/1.1"  200  37932  "-­‐"  "Mozilla/5.0  (compa4ble;  Googlebot/2.1;  +hmp://
    www.google.com/bot.html)"  
    110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:42  -­‐0500]  "GET  /projects/xdotool/  HTTP/1.1"  200  12292  "hmp://www.google.com/url?
    sa=t&rct=j&q=&esrc=s&source=web&cd=5&cad=rja&sqi=2&ved=0CFYQFjAE&url=hmp%3A%2F%2Fwww.semicomplete.com%2Fprojects
    %2Fxdotool%2F&ei=6cwAU_bRHo6urAeI0YD4Ag&usg=AFQjCNE3V_aCf3-­‐gfNcbS924S6jZ6FqffA&bvm=bv.61535280,d.bmk"  "Mozilla/5.0  
    (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"  
    46.105.14.53  -­‐  -­‐  [16/Feb/2014:09:48:48  -­‐0500]  "GET  /blog/tags/puppet?flav=rss20  HTTP/1.1"  200  14872  "-­‐"  "UniversalFeedParser/4.2-­‐pre-­‐314-­‐
    svn  +hmp://feedparser.org/"  
    110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /reset.css  HTTP/1.1"  200  1015  "hmp://www.semicomplete.com/projects/xdotool/"  
    "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"  
    110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /style2.css  HTTP/1.1"  200  4877  "hmp://www.semicomplete.com/projects/xdotool/"  
    "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"  
    110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /favicon.ico  HTTP/1.1"  200  3638  "-­‐"  "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  
    Gecko/20100101  Firefox/28.0"  
    110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /images/jordan-­‐80.png  HTTP/1.1"  200  6146  "hmp://www.semicomplete.com/projects/
    xdotool/"  "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"  
    123.125.71.35  -­‐  -­‐  [16/Feb/2014:09:49:02  -­‐0500]  "GET  /blog/tags/release  HTTP/1.1"  200  40693  "-­‐"  "Mozilla/5.0  (compa4ble;  Baiduspider/2.0;  
    +hmp://www.baidu.com/search/spider.html)"  
    110.136.166.128  -­‐  -­‐  [16/Feb/2014:09:48:53  -­‐0500]  "GET  /images/web/2009/banner.png  HTTP/1.1"  200  52315  "hmp://www.semicomplete.com/
    style2.css"  "Mozilla/5.0  (Windows  NT  6.2;  WOW64;  rv:28.0)  Gecko/20100101  Firefox/28.0"  
    Logs  are  have  no  standard  format  
    There  is  no  consistency  
    You  have  to  be  an  expert  to  read  them  
    Almost  always  stored  somewhere  that’s  hard  for  
    the  organiza4on  to  get  to.  
    grep  and  regular  expressions  don’t  scale  
    I  can  help!  

    View full-size slide

  5. ELK  Stack  
    Elas4csearch,  Logstash,  and  Kibana  
    Logstash   Elas4csearch   Kibana  
    Key:  

    View full-size slide

  6. Logstash  
    •  Input  from  many  Sources  
    –  It’s  really  good  at  parsing  logs  (shocking!)  
    –  Other  sources  too  
    •  Files,  Queues,  Messages,  Databases,  etc  
    –  Hundreds  of  plugins  
    •  Transform  and  Enrich  
    –  GROK  
    –  IP  -­‐-­‐>  Geospa4al  
    –  Conver4ng  to  JSON  is  very  popular  
    •  Output  to  Many  Des4na4ons  
    –  Databases,  Dashboards,  Elas4csearch  …  
    –  many  others  

    View full-size slide

  7. Elas4csearch  
    •  Based  on  Lucene  
    –  Wicked  fast  
    –  Good  at  search  
    –  Good  at  analy4cs  
    •  Things  it  adds  
    –  Horizontal  Scaling  
    –  High  Availability  
    –  Ease  of  Use  
    –   “Near  Real-­‐Time”  

    View full-size slide

  8. Kibana  
    •  UI  for  Elas4csearch  
    –  Lightweight  Search  Interface  
    –  Discover  
    –  Visualize  
    –  Dashboards  
    •  Explore  and  Understand  your  
    data  
    •  Scale  and  Speed  
    –  Heavy  li|ing  s4ll  done  in  
    Elas4csearch  

    View full-size slide

  9. Let’s  take  an  example  

    View full-size slide

  10. Regular  Expression  

    View full-size slide

  11. Grok  PaJern  

    View full-size slide

  12. {  
           4me:          “120819”,  
           query_4me:      27.115751,  
           lock_4me:        0.00007  
           rows_sent:        55996,  
           rows_examined:    56000  
           query_4mestamp:    “1345373510”,  
           query:          “SELECT  ID  FROM  wp_posts  WHERE  …”  
    }  

    View full-size slide

  13. Logstash  Pipeline  

    View full-size slide

  14. ELK  Stack  
    Elas4csearch,  Logstash,  and  Kibana  
    Logstash   Elas4csearch   Kibana  
    Key:  

    View full-size slide

  15. What  usually  ends  up  happening  once  
    someone  makes  something  useful  
    Logstash   Elas4csearch   Kibana  
    Key:  
    ES-­‐Hadoop  
    Rela-onal  
    Hadoop  
    Non-­‐Rela-onal  
    Apps  and    
    Mobile   Language  
    Clients  

    View full-size slide

  16. Scaled  Architectures  
    Logstash   Elas4csearch   Kibana  
    Key:  
    master  
    master  
    master  
    …  
    …  
    …  
    queue  
    queue  
    queue  
    …  
    Agents  
    (shippers)  
    route  /  buffer  
    collect   process   Index  /  alert   view  

    View full-size slide

  17. Going  Beyond  Logs  
    •  Search  
    – Relevance  
    – Faceted  Naviga4on  
    – Human  Language  
    – Unstructured  Data  
    – Rich  Syntax  
    •  Analy4cs  
    –  Real  Time  
    –  Significant  Terms  
    –  En4ty  Oriented  Indexing  
    –  Make  data  available  to  
    the  whole  organiza4on  
    –  Use  ELK  to  figure  out  
    what  ques4ons  to  start  
    asking  

    View full-size slide

  18. Use  cases  
    hmps://www.elas4c.co/elas4con/2015/sf/videos/  

    View full-size slide

  19. Use  Case:  MozDef  
    •  Large  Scale  Security  Challenge  
    –  300  Million  events  per  day  
    •  Amackers  
    –  Innova4ve  
    –  Real  Time  
    –  Adap4ve  
    •  Exis4ng  SIEM  op4ons  
    –  Closed  Systems  
    –  Proprietary  
    –  Lack  of  API’s  &  Endpoints  
    •  Solu4on:  built  their  own  
    –  Mozilla  Defense  Pla€orm  
    –  Open  Source  SIEM  overlay  for  
    Elas4csearch  

    View full-size slide

  20. Use  Case:  Wikimedia  
    •  265+  Languages  
    •  3.1  B  prefix  searches  /  
    month  
    •  870  M  text  searches  /  month  
    •  Real  Time  
    –  Rewards  Contributors  
    –  Fixes  Vandalism  
    •  Plugins  
    –  Contributed  many  
    •  Expressive  Syntax  
    –  Fix  search  without  
    redeployment  of  search  
    infrastructure  
    •  No  Down4me  Management  
    –  Aliases  

    View full-size slide

  21. Use  Case:  Github  Search  
    A  Story  About  Learning  to  Scale  

    View full-size slide

  22. Use  Case:  NASA  JPL  Telemetry  

    View full-size slide

  23. Awesome  Stuff  in  the  road  map  
    •  Marvel  2.0  &  Shield  2.0  
    •  Watcher    (Aler4ng  for  Elas4csearch)  
    •  Logstash  
    –  Cleaner  Plugins  
    –  Bemer  Windows  Support  
    –  More  inputs  (Ka•a,  Avro  ...  etc.)  
    –  Buffering,  Clustering,  Resiliency  
    •  Elas4csearch  
    –  Aggrega4ons  2.0  (name  pending)  
    –  Changes  API,  Re-­‐indexing  API  
    •  Elas4csearch  Hadoop  Connector  
    –  Na4ve  support  for  Spark,  Storm,  Yarn  
    –  Exposing  ES  internals  to  Machine  Learning  
    •  Kibana  
    –  Rapidly  evolving  

    View full-size slide

  24. Elas4csearch  &  .Net  

    View full-size slide

  25. Two  Clients  

    View full-size slide

  26. •  Low  Level:  Elas4csearch.NET  
    –  No  dependencies,  load  balancing,  failover  
    •  High  Level:  NEST  (v1.5  just  released)  
    –  Strongly  Typed  /  Fluent  API  
    –  Dependencies:  Json.NET,  Elas4csearch.NET  
    –  Plain  Old  C#  Objects  (POCOs)  
    –  Annota4on  driven  Mappings  (Don’t  repeat  yourself!)  
    –  Wraps  and  Exposes!  
    •  Directly  Input  Boolean  filters  as  DSL  
    •  Boolean  bitwise  operators  mapped  to  Elas4csearch  DSL  

    View full-size slide

  27. Magic,  when  you  need  it  

    View full-size slide

  28. Key  things  to  make  your  projects  more  
    successful?  
    •  Understand  Distributed  Systems  lingo  /  invarients  
    –  “Master”  =/=  “Primary”  
    –  Don’t  model  your  data  rela4onally  
    •  Read  the  book  
    –  HTML  version  of  O’Reilly  book  is  on  Elas4c.co  >  Learn  
    (Elas4csearch  –  The  Defini4ve  Guide)    …  it’s  really  good  
     
     
     
    •  It’s  not  a  silver  bullet  
    –  No  Such  Thing  As  a  Free  Lunch  

    View full-size slide

  29. THANKS!    QUESTIONS?  
    [email protected]  

    View full-size slide