These Logs Were Made for Talking

Logs, logs, logs, as far as the eye can see. Web logs, SQL logs, application logs, firewall and event logs. In today’s world of “log everything”, how can a forensic analyst quickly ingest, index, and visualize the data within their logs to rapidly gain insight and knowledge? Quickly responding to events can be the difference between data about to walk out the door, and data long gone. In a world of big data and large clustered systems, sometimes even the smallest forensic shop can build a powerful, nimble log analytics engine.

In this presentation, we’ll walk through setting up a standalone, powerful log analytics stack in 10 minutes using the free and open-source trio Logstash, Elasticsearch, and Kibana. Attendees will learn how to leverage these tools to generate valuable metadata about their log contents, such as performing on-the-fly geolocation of IP addresses and building analysis dashboards to share with their teams. Attendees will also learn how to use the Kibana visualization tool to quickly transition through their various sets of data without skipping a beat. Lastly, via quick, interchangeable configuration files, attendees will see how to swiftly navigate between various types of logs, regardless of the source.

Are you ready logs? Start talking...

Transcript

1. These  Logs  Were  Made  for   Talking   Ma3  Bromiley  –  @505Forensics  
2. Agenda   ¨  $  whoami   ¨  $  cat  why_are_there_so_many_text2iles   ¨  $  elasticsearch  &  logstash  &  kibana   ¨  $  ./elk_stack  setup   ¨  $  demo  
3. whoami   These  Logs  Were  Made  for  Talking  
4. whoami   ¨  Matt  Bromiley   ¨  Based  in  Dallas,  TX;  formerly  Annapolis,  MD   ¨  Senior  Cybercrime  and  Incident  Response   Consultant   ¨  Cybersecurity  Graduate  Student  (In-­‐Progress)   ¨  http://www.505forensics.com   ¨  @505Forensics  
5. whoami  (obligatory  disclaimer)   I’m  just  going  to  put  this  here:       My  opinions  are  my  own;  not  my  employers.     Also,  this  isn’t  a  product  endorsement.  Some   things  just  work  better  than  others.  
6. cat  why_are_there_so_many_text2iles   These  Logs  Were  Made  for  Talking  
7. why_are_there_so_many_text_files   ¨  Face  it:  We  love  2lat  text   ¨  Artifacts  look  better  in  2lat  text:   ¤  Registry   ¤  $MFT   ¤  Event  Logs   ¤  Web  Logs   ¤  ${insert_your_favorite_log_here}   ¤  Log2timeline/Plaso  (all  of  the  above)  
8. why_are_there_so_many_text_files  (cont.)   ¨  Flat  text  allows  us  to  maintain  our  command   line  kung  fu   ¤  awk  |  sed  |  grep  |  sort  |  split  |  tr  |  join   ¤  Writing  custom  scripts  to  handle  data   ¨  Now  we  have  more  2lat  text  output;  just  smaller   and  re2ined    
9. why_are_there_so_many_text_files  (cont.)   Text  File  Issues   ¨  Structure  is  hard  to  transfer   ¤  Tough  to  join  someone  else  in  The  Matrix   ¨  Little  to  no  enrichment;  text  is  still  just  text   ¤  Keep  your  browser  handy   ¤  Air-­‐gapped  lab?   ¨  Every  try  to  show  off  10  text  2iles  to  management?   ¤  Nope,  nope,  nope   ¨  I  just  want  to  analyze  
10. elasticsearch  &  logstash  &  kibana     The  ELK  Stack   These  Logs  Were  Made  for  Talking  
11. The  ELK  Stack   Logstash  –  Manage  all  those  text  2iles;  bring   any  ingestible  data  into  a  central  repository   for  analytics   Kibana  –  Visualize  your  data  inside   Elasticsearch  with  interactive  pages,  custom   visualization  options   Elasticsearch–  Distributed,  restful  index   engine  based  on  Apache  Lucene  with  free   text  search   More  info  at  http://www.elasticsearch.org  -­‐  more  than  I  could  ever  talk  about  
12. The  ELK  Stack  (cont.)   The  process  2low:    Logstash  -­‐>  Elasticsearch  <-­‐  Kibana     ¤  Elasticsearch:  Schema-­‐less,  JSON  storage   ¤  Kibana:  Angular  webapp  to  interact  with   Elasticsearch   ¤  Logstash:  The  log  shipper;  power  lies  in  the   con;ig  ;iles  
13. The  ELK  Stack  (cont.)   Logstash  con2ig  format:     input {} – Where is it? filter {} – What am I doing with it? output {} – Where do you want it? This  is  where  we  enrich!!  
14. The  ELK  Stack  (cont.)   Logstash  con2ig  2iles  allow  us  to  :   ¤  Handle  a  large  variety  of  data  types   ¤  Enrich  data  our  data  with  geo-­‐location,  DNS  lookups,   etc.   ¤  Mutate,  transform,  combine  data  in-­‐motion;  no  more   “prepping”  before  ingestion   ¤  Input  “non-­‐text”  data,  such  as  net2low,  Twitter,  SQLlite,   syslog    
15. ./elk_stack_setup     These  Logs  Were  Made  for  Talking  
16. ./elk_stack_setup   Two  setups:   ¨  Mature  lab   ¨  On-­‐;ly-­‐analysis,  rapid  triage  
17. Demo     (2inally)     These  Logs  Were  Made  for  Talking  
18. QuesNons?   Matt  Bromiley   @505Forensics   505forensics@gmail.com   http://www.505forensics.com     Thank  you!     Extra  thanks  to:   DFRWS   Elasticsearch  Crew   All  the  forensicators  out  there