Massive Data Aggregation Processing, Monitoring and Visualization with Apache Flume, ElasticSearch and D3.js

Strata Conf + Hadoop World 2013

Monday, October 28 2013 09:00AM EST

Transcript

1. Massive  Data  Aggrega-on,   Monitoring,     Processing  and  Visualiza-on  with     Apache  Flume,     Elas-cSearch  and  D3.js   Israel  Ekpo   Walt  Disney  Parks  and  Resorts  Online  (WDPRO)  
2. Massive  Data  Aggrega-on,   Monitoring,     Processing  and  Visualiza-on  with     Open  Source  SoJware   Israel  Ekpo   Walt  Disney  Parks  and  Resorts  Online  (WDPRO)  
3. About  the  Speaker   •  Father  and  Husband   •  Technology  and  Data  Science  Enthusiast   •  Contributor  to  Open  Source  Projects   •  Loves  to  learn  new  technologies   •  So>ware  Engineer  at  Walt  Disney  Company   •  @IsraelEkpo  @MassiveLogData  
4. Open  Posi-ons  at  WDPRO   Loca-ons  in  Glendale,  SeaNle  and  Orlando     •  SoJware  Engineers   •  Web  Developers   •  Mobile  Engineers   •  SoJware  Quality  Assurance  Engineers   •  Automa-on  Engineers   •  Technology  Managers,  Technology  Execu-ves   •  Internship  Opportuni-es     Email  resumes  to  israel.ekpo@disney.com  
5. Pre-­‐Requisites   •  Install  Java  7.0.25   •  Install  Google  Chrome  with  Advanced  REST  Client   •  Install  Apache  Tomcat  7.0.47   •  Install  Apache  Maven  3.x   •  Download  Apache  Flume  1.4.0   •  Download  Logstash  1.2.2  and  Ruby  1.8  or  later   •  Download  Kibana  3.0,  milestone  4   •  Install  Elas-cSearch  0.90.5   •  Install  Apache  HTTP  Server  2.2.25   •  Install  PHP  5.5  with  libcurl  enabled   •  Install  Nginx  1.4.3  
6. Agenda   •  DefiniHon  of  a  log/event   •  Types  of  log/event  data  we  collect   •  Reasons  why  we  collect  log  data   •  Use  cases  for  conHnuously  streaming  data   •  Tools  we  use  to  collect.  How  we  use  the  tools.   •  Tools  for  storing  log  data/events   •  Processing  the  data   •  Visualizing  the  data   •  Monitoring  logs  and  events  
7. What  is  an  Event/Log  ?   •  Structured  Data   •  Semi-­‐Structured  Data   •  Unstructured  Data   •  Usually  associated  with  a  Hmestamp  
8. Examples  of  Con-nuously  Streaming   Events/Logs   •  ApplicaHon  Generated  Data   •  Network  Traffic   •  Social  Media:  LinkedIn,  Google+,  TwiXer   •  Mailing  List  SubscripHons  
9. Some  reasons  why  we  collect  data   •  Business  Intelligence   •  Simplify  TroubleshooHng  Process   •  ApplicaHon  Performance   •  Machine  Learning   •  IdenHfy  Trends  for  Business  OpportuniHes   •  Answer  Difficult  QuesHons  
10. Tools  for  Collec-ng  Data   •  Apache  Flume   •  Logstash  
11. Tools  for  Storing/Processing  Data   •  Apache  Hadoop  (HDFS,  MapReduce)   •  ElasHcSearch  
12. Tools  for  Visualizing  Processed  Data   •  Kibana   •  D3.js,  jQuery,  HTML5  
13. Where  to  Get  Help   Ques-ons  About  How  to  Use  Flume   user@flume.apache.org     Ques-ons  About  How  to  Use  Logstash   logstash-­‐users@googlegroups.com  
14. Where  to  Get  Help   Ques-ons  About  How  to  Use  Elas-cSearch   elasHcsearch@googlegroups.com     Ques-ons  About  How  to  Use  Kibana   logstash-­‐users@googlegroups.com   elasHcsearch@googlegroups.com     Ques-ons  About  How  to  Use  D3.js   d3-­‐js@googlegroups.com    
15. Hands-­‐On   •  Set  up  Graph  Service  on  Tomcat   •  Set  up  ElasHcSearch   •  Set  up  Kibana   •  Configure  Logstash  to  Watch  Logs  in  RealHme   •  Submit  Sample  Node  Data  to  Endpoints   •  Verify  Graph  Events   •  View/Query  Event/Log  data  in  Kibana  
16. AJer-­‐Party  Resources   Code  Samples  and  Slides   github.com/israelekpo/strataconf-­‐ny-­‐2013   speakerdeck.com/israelekpo     Ques-ons  Regarding  Strata  Presenta-on   israel.ekpo@gmail.com   (650)  318-­‐1195