Beef Up Your DFIR Toolbox with Elasticsearch

As DFIR investigations grow in complexity, logs have quickly become some of the most fruitful data sources. But logs aren't small. In today's investigations, how can the forensic practitioner quickly turn large amounts of data into actionable knowledge? In this presentation, we'll learn how to incorporate the ELK stack within our forensic toolkit, allowing practitioners to quickly index, analyze, enrich, and visualize even the largest log sets. Watch out haystack - the needle just got bigger.

Transcript

1. 1 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Beef  Up  Your  DFIR  Toolbox  with  Elasticsearch Matt  Bromiley Senior  Consultant,  Mandiant
2. 2 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Agenda • $  whoami • Why  Do  I  Care? • Thinking  as  an  Analyst • Elastic  and  the  ELK  Stack • Bringing  the  Parts  Together • Making  Sense  of  the  Data • Wrapping  Up
3. 3 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       $  whoami
4. 4 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       $  whoami • Currently  a  Senior  Consultant  with  Mandiant • 4+  years  experience  with  a  focus  on  data  breaches,   incident  response,  network  security  monitoring,  and   digital  forensics • Work  with  clients  from  small,  regional  shops  to   multinational  Fortune  50s • Help  out  the  awesome  SANS  team • LOVE  to  develop  &  work  with  open  source  forensic   tools • LOVE  to  share,  learn,  and  help  others  improve  (while   improving  myself!) Tweet/Git/Blog [@]505Forensics[.com]
5. 5 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       WHY  DO  I  CARE?
6. 6 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Why  Do  I  Care? Rate  at  which  I   can  find  needles Rate  at  which  the   haystack  grows <
7. 7 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Why  Do  I  Care?  (cont.) • Per  Minute  (2014): • 277,000  tweets • 4,000,000  Google  searches • 204,000,000  email  messages • 72  hours  of  YouTube  content • 23,300  hours  of  Skype  chats Source:  http://aci.info/2014/07/12/the-­data-­explosion-­in-­2014-­minute-­by-­minute-­infographic/
8. 8 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Why  Do  I  Care?  (cont.) • Humans  aren’t  using  any  less data  in  the  future • The  types of  data  sources  are  now  in  the  hundreds • Hard  drives  now  normally  end  with  TB • The  game  has  changed;;  the  players  (us)  need  to  change  too
9. 9 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Why  Do  I  Care?  (cont.) • Enterprise  sizes  aren’t  getting  smaller  either • Forensic  cases  used  to  be  onesie-­twosies;;  enterprises  now  number  in  the   thousands • Legal  requirements  may  pull  dozens,  if  not  hundreds,  of  systems  into  scope • How  can  we  even  think  about  examining  systems  at  that  scale?
10. 10 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       THINKING  AS  AN  ANALYST
11. 11 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Thinking  as  an  Analyst • I  (hopefully  we)  LOOOOOVE  flat  text  files • Things  that  aren’t flat  text  look  better as  flat  text! • Registry  hives • $MFT • $UsnJrnl • $LogFile • In  fact,  ${insert_NTFS_artifact_here} • Event  Logs • ${insert_your_favorite_log_here} • Time  to  get  serious  with  the  kitchen  sink:  log2timeline/plaso
12. 12 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Thinking  as  an  Analyst  (cont.)
13. 13 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Thinking  as  an  Analyst  (cont.) • Flat  text  helps  us  command  line  kung  fu ninjas  keep  our  skills  sharp • awk |  sed |  grep |  sort  |  split  |  tr |  find  |  join  ALL  THE  THINGS! • How  many  of  us  have  written  custom  scripts  to  parse  through   ${artifactA}  and  ${artifactB}? • Now  all  I  have  is.. • …more  output
14. 14 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Thinking  as  an  Analyst  (cont.)
15. 15 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Thinking  as  an  Analyst  (cont.) • This  is  not  sustainable • Tough  to  share • Tough  to  go  fast • 1000  text  files  =  more  to  analyze • Or  worse:  More  to  report • Ever  try  to  show  off  text  files  in  an  executive  meeting? • Nopeville • As  analysts,  we  typically  have  one  goal  in  mind • To  analyze…fast
16. 16 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       ELASTIC  AND  THE  ELK  STACK
17. 17 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Elastic  and  the  ELK  Stack Elasticsearch is  a  registered   trademark  of  Elasticsearch BV.
18. 18 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Elastic  and  the  ELK  Stack  (cont.) • Elastic  is  the  company  that  sits  behind  the  ELK  stack • Three  open  source  products  consolidated  into  one  name  (+  more!) • Combined  to  bring  one  face  to  a  range  of  products  and  ideas • ELK  Stack: • Elasticsearch • Logstash • Kibana
19. 19 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Elastic  and  the  ELK  Stack  (cont.) • Elasticsearch • Open  source  analytics  engine  with  full  text   search • Based  on  Apache  Lucene (so  super  fast   and  Java-­based) • Nodes  upon  nodes • Schema-­less,  JSON  document  storage • Great  API  support  (scripting  languages,   RESTful,  etc.) Elasticsearch is  a  registered   trademark  of  Elasticsearch BV.
20. 20 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Elastic  and  the  ELK  Stack  (cont.) • Logstash • Data  processing  engine • Allows  you  to  quickly  move  data  into  JSON,   and  into  Elasticsearch • Plugins  on  plugins  for  varying  data  types • Literally  dozens  and  dozens • The  shipper  (gets  data  from  here  -­>  there   (there  being  Elasticsearch)) Elasticsearch is  a  registered   trademark  of  Elasticsearch BV.
21. 21 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Elastic  and  the  ELK  Stack  (cont.) • Kibana • The  web  front-­end  for  all  of  the  above • Written  in  AngularJS;;  now  shipped  as  a   standalone  node.js app • Allows  you  to  visualize  data  inside   Elasticsearch • Currently  at  version  4.0.3;;  many  folks   still  operate  on  3.1.2 • Guidance  learning  curve Elasticsearch is  a  registered   trademark  of  Elasticsearch BV.
22. 22 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       BRINGING  THE  PARTS  TOGETHER
23. 23 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together Logstash Elasticsearch Kibana
24. 24 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.) • How  do  we  set  this  thing  up!?
25. 25 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.) • Mature  Lab • Centralized  Elasticsearch (distributed,  multiple  nodes,   etc.) • Beefy  server(s) • Logstash shippers  on  analyst   workstations • Custom  scripts  for  log  types   allow  for  little  slow-­down
26. 26 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.) • On-­the-­fly • One  machine;;  all  three  tools • Ingestion  scripts  written  as-­ needed • Single  analyst • Great  for  rapid  triage  or   analysis • Not  sustainable  as  data   grows
27. 27 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.) • Remember  all  that  flat  text  we   were  gushing  about? • Now  we  want  to  search  it • Start  with  flat  text;;  need  to  get  it   into  Elasticsearch somehow • Logstash? • Custom  scripts?
28. 28 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – Pros  and  Cons Logstash • Can  be  setup  to  monitor  a  directory • Powerful  data  wrangling  skills  and  enrichment ..but... • Can  be  slow  depending  on  filter • One  more  thing  to  learn/download
29. 29 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – Pros  and  Cons Scripts • May  already  be  in  your  arsenal;;  comfort  in  writing • Nothing  additional  to  download  from  most  systems ..but... • Enrichment  may  not  be  as  powerful/built-­in  (or  it  may  be  better!!) • Re-­engineering  scripts  may  be  more  of  a  hassle
30. 30 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – Pros  and  Cons Which  to  Choose?! • It’s  up  to  you! • Both  have  their  place,  and   both  may  be  equally   capable  in  the  right  hands • Build  a  workflow  that  is   flexible,  makes  sense,  and   allows  your  team  to  adapt
31. 31 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – Writing  Logstash Configs input {} – Where is it? filter {} – What am I doing with it? output {} – Where do you want it?
32. 32 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – Writing  Logstash Configs • Input • Where  is  the  data? • How  to  interpret  it? • Labels • Flat  text,  known-­structure,  or  known-­input  (Twitter,  SQLite,  RabbitMQ,   ZeroMQ,  XMPP,  etc.)  ?
33. 33 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – Writing  Logstash Configs • Input  (sample) input  {   file  {   type  =>  ”weblog" start_position =>  "beginning” path  =>  "/var/log/www/access*.log” sincedb_path =  "/dev/null" }   }
34. 34 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – Writing  Logstash Configs • Filter • What  do  you  want  me  to  do  with  it? • This  is  where  we  enrich! • Perform  data  operations, • Field  exclusions  and  combinations, • Lookups • Timestamp  definitions • Morphs  the  data  into  the  JSON  we  want  AND  the  JSON  we  need
35. 35 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – Writing  Logstash Configs • Input  (sample) filter  {   if  [type]  ==  “weblog”  {   grok {   match  =>  {  "message" =>  "%{COMMONAPACHELOG}" }   }   date  {   match  =>  ["timestamp",  "dd/MMM/yyyy:HH:mm:ss Z”]   }   } }
36. 36 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – Writing  Logstash Configs • Output • Where  do  you  want  me  to  put  it? • Most  common  (for  this  talk)  is  out  to  Elasticsearch • Can  output  to  other  types  as  well!! • Yes,  you  can  use  logstash to  turn  data  into  JSON-­friendly,  with   enrichment,  and  not  go  to  Elasticsearch..but  what’s  the  fun  in  that? • Has  transport/protocol  capabilities  built-­in • All  we  have  to  do  is  provide  the  settings
37. 37 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – Writing  Logstash Configs • Input  (sample) output  { elasticsearch { host  =>  “127.0.0.1” } }
38. 38 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – Writing  Custom  Scripts • Custom  scripts  require  a  bit  more  finesse  of  the  data • Need  to  know  the  structure • How  to  interpret • How  to  parse • Error  handling? • Data  length? • What  if  one is  off?
39. 39 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Bringing  the  Parts  Together  (cont.)  – RESTful API • We  can  also  talk  with  the  server  directly,  requesting  commands • curl localhost:9200/_stats?pretty • We  can  also  use  other  options • XDELETE • XPOST • XPUT
40. 40 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       MAKING  SENSE  OF  THE  DATA
41. 41 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       Making  Sense  of  the  Data • Case  Study • log2timeline  output  from  a  host • Too  much  data  to  analyze  in  Excel • Can  we  use  Elasticsearch to  visualize,  and  get  a  better  grasp?
42. 42 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       MAKING  SENSE  OF  THE  DATA
43. 43 ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       ©    Mandiant,   a  FireEye  Company.    All  rights  reserved.       THANK  YOU Opinions  are  all  mine,  and  not  representative   of  my  employer.