Elastic{ON} 2018 - Sipping from the Firehose: Scalable Endpoint Data for Incident Response

Enterprises have better sources of endpoint telemetry to respond to intrusions than ever before, yet attackers continue to slip through the cracks, often with surprising ease. And security teams still struggle to fully scope or remediate compromises, even after they’ve been detected.

This presentation will examine why it's so difficult to gather and maintain the right mix of endpoint data for effective incident response. It will then demonstrate how a blended approach — combining technologies like Elasticsearch with distributed, on-endpoint analysis — can offer comprehensive, high-speed, and efficient visibility at any scale. Examples from real-world breaches (including a few that inspired hacks in the latest season of Mr. Robot) will illustrate lessons learned from the field.

Ryan Kazanciyan| Chief Security Architect | Tanium

Transcript

1. Tanium
   February 28, 2018
   @ryankaz42
   Sipping from the Firehose:
   Scalable Endpoint Data for Incident Response
   Ryan Kazanciyan, Chief Security Architect
   

   View Slide

2. whoami
   

   View Slide

3. 3
   Alexandria, VA
   

   View Slide

4. 4
   2004 - 2009 2009 - 2015 2015 - Present
   

   View Slide

5. View Slide

6. View Slide

7. Why this topic?
   

   View Slide

8. 8
   • Endpoints are the ultimate security perimeter
   • We’re in a golden age of endpoint security tools and data…
   • …but we still struggle with scale, efficiency, and effectiveness
   

   View Slide

9. WHAT DATA SHOULD I
   PRIORITIZE FOR ENDPOINT
   DETECTION AND RESPONSE… 

   AND WHERE DO I PUT IT?
   

   View Slide

10. Common Challenges
    

    View Slide

11. 11
    You have a much
    wider variety of
    endpoint data than
    you expect
    

    View Slide

12. 12
    …but all you need
    is a best-of-breed
    Endpoint Protection
    product, right?

    

    (wrong)
    

    View Slide

13. • “Black box” flight recorder

    • Limited to the most common event-based
    data (process execution, file changes,
    network connections, etc.)

    • High-volume, high-value
    EDR telemetry
    13
    

    View Slide

14. MITRE ATT&CK Framework
    14
    https://attack.mitre.org/wiki/Technique_Matrix
    

    View Slide

15. • Access Tokens
    • Anti-virus
    • API monitoring
    • Authentication logs
    • Binary file metadata
    • BIOS
    • Browser extensions
    • Data loss prevention
    • Digital Certificate Logs
    • DLL monitoring
    • EFI
    • Environment variable
    • File monitoring
    • Host network interface
    • Kernel drivers
    • Loaded DLLs
    • MBR & VBR
    • Netflow
    • Network device logs
    • Network protocol analysis
    • Packet capture
    • PowerShell logs
    • Process command-line parameters
    • Process monitoring
    • Process use of network
    • Sensor health and status
    • Services
    • SSL/TLS inspection
    • System calls
    • Third-party application logs
    • User interface
    • Windows Error Reporting
    • Windows event logs
    • Windows Registry
    • WMI Objects
    Data sources per MITRE ATT&CK
    15
    

    View Slide

16. • Access Tokens
    • Anti-virus
    • API monitoring
    • Authentication logs
    • Binary file metadata
    • BIOS
    • Browser extensions
    • Data loss prevention
    • Digital Certificate Logs
    • DLL monitoring
    • EFI
    • Environment variable
    • File monitoring
    • Host network interface
    • Kernel drivers
    • Loaded DLLs
    • MBR & VBR
    • Netflow
    • Network device logs
    • Network protocol analysis
    • Packet capture
    • PowerShell logs
    • Process command-line parameters
    • Process monitoring
    • Process use of network
    • Sensor health and status
    • Services
    • SSL/TLS inspection
    • System calls
    • Third-party application logs
    • User interface
    • Windows Error Reporting
    • Windows event logs
    • Windows Registry
    • WMI Objects
    What do most EDR tools focus on?
    16
    

    View Slide

17. View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. View Slide

28. View Slide

29. • What sources of data?
    • What can be centralized?
    • What must be examined on-endpoint?
    • What’s your cadence to collect?
    • What’s your cadence to analyze?
    You cannot capture everything, constantly
    29
    

    View Slide

30. • Typical endpoint sources
    • Alerting tools
    • Telemetry tools
    • Critical logs (limited to select systems)

    • Ideal for correlation with non-endpoint
    sources, aggregate data analysis

    • Resource constrained by event forwarding and
    storage over time
    Centralized approach
    30
    

    View Slide

31. • Broadest set of available data:
    • Volatile / in-memory
    • Files and artifacts on-disk
    • Locally stored telemetry and logs
    • Often difficult to efficiently search and
    collect at-scale
    On-endpoint evidence
    31
    

    View Slide

32. 32
    rule PAS_TOOL_PHP_WEB_KIT
    {
    meta:
    description = "PAS TOOL PHP WEB KIT FOUND"
    strings:
    $php = "$base64decode = /\='base'\.\(\d+\*\d+\)\.'_de'\.'code'/
    $strreplace = "(str_replace("
    $md5 = ".substr(md5(strrev("
    $gzinflate = "gzinflate"
    $cookie = "_COOKIE"
    $isset = “isset"
    condition:
    (filesize > 20KB and filesize < 22KB) and #cookie == 2 and #isset == 3 and all of them
    }
    Searching web server files with Yara
    On-endpoint example #1
    

    View Slide

33. On-endpoint example #2
    33
    Hunting for a unique event in a non-forwarded log
    

    View Slide

34. 34
    Your endpoints are noisier than you might expect…
    

    View Slide

35. • Different OS versions, add-ons, and regional variants
    • User applications
    • Enterprise applications
    • Randomized file paths, GUIDs, and other per-host unique artifacts
    • Churn from updates & patches
    Your software is noisy
    35
    Examining operating system, application, and script usage at-scale
    

    View Slide

36. 5-7 per host
    1-3 per host
    Large networks (>100k endpoints)
    Small networks (<100k endpoints)
    * Measured by total unique instances of installed application versions
    

    View Slide

37. 230,000 systems
    400,000 unique

    application + version pairings
    

    View Slide

38. 38
    “Using Endpoint Telemetry to Accelerate the Baseline”, McCammon,

    https://www.sans.org/summit-archives/file/summit-archive-1492181402.pdf
    

    View Slide

39. What trade-offs do we make?
    

    View Slide

40. TUNNEL VISION
    

    View Slide

41. “Let’s focus on critical systems”
    41
    

    View Slide

42. COGNITIVE LOAD
    

    View Slide

43. 43
    Your analysts are
    overwhelmed
    

    View Slide

44. 44
    Your analysts are
    overwhelmed
    

    View Slide

45. Iterating on an hunting technique
    45
    “How often do
    legitimate Windows
    applications run
    PowerShell encoded
    commands?”
    1 2 3 4 5
    “Oops. 10% of our
    endpoints produce
    1000s of false positives
    per day. Too much
    noise.
    “Let’s apply some
    client-side filters to the
    data and try again.”
    “Eureka!” Now let’s
    collect, centralize, and
    analyze the data over
    time.”
    “Find all the
    evil things!”
    Ask a 

    question
    Get unexpected
    results
    Learn and
    refine
    Add to
    workflow
    Success!
    

    View Slide

46. Common inhibitors
    46
    1 2 3 4 5
    Ask a 

    question
    Get unexpected
    results
    Learn and
    refine
    Add to
    workflow
    Success!
    • Expensive or slow to test at-scale
    • Can only work with pre-selected data
    • Contend for resources with other workflows
    

    “Will this break something?”…“Take too long?”…“I guess I won’t try…”
    

    View Slide

47. Taking a balanced approach
    

    View Slide

48. 48
    Open platform to
    consolidate and
    analyze EDR data
    with other sources of
    evidence
    

    View Slide

49. View Slide

50. View Slide

51. View Slide

52. View Slide

53. Finding a story in the data
    53
    

    View Slide

54. Finding a story in the data (a better way)
    54
    

    View Slide

55. with
    Integrating
    

    View Slide

56. 56
    Real-time visibility
    and control - across
    any number of
    endpoints - from a
    single server
    

    View Slide

57. View Slide

58. Distributed access to endpoint data
    58
    Full-disk index, files at-rest, OS configuration 

    and forensic artifacts
    Volatile memory and short-lived / stateful evidence
    Historical EDR telemetry, OS logs, application logs
    Efficient data aggregation, and a single source of truth
    

    View Slide

59. Search, collect, and analyze at-scale
    59
    1 2 3 4 5
    Ask a 

    question
    Get unexpected
    results
    Learn and
    refine
    Add to
    workflow
    Success!
    Experiment without penalty, with results in seconds
    

    View Slide

60. Demo Video
    

    View Slide

61. View Slide

62. 62
    More Questions?
    Visit me at the AMA
    

    View Slide

63. Tanium
    February 28, 2018
    @ryankaz42
    Sipping from the Firehose:
    Scalable Endpoint Data for Incident Response
    Ryan Kazanciyan, Chief Security Architect
    

    View Slide