osquery: approaching security the hacker way

osquery was released as an open source product by Facebook in October 2014. It is an instrumentation framework for Ubuntu, CentOS, and OS X. osquery makes low-level operating system analytics and monitoring both performant and intuitive.

This talk will walk through why we created osquery, how we use osquery at Facebook to improve our security how other companies currently take advantage of osquery, and how you can too!

We’ll outline some of the challenges and sensitivities we faced when developing osquery and planning its open source release, as well as how we overcame those issues. Additionally, we’ll spend some time talking about why we believe open source is critical to advancing the state of trusted, secure software. Security through obscurity is dead; this is the age of security through transparency.

Transcript

1. mike arpaia / facebook
   osquery: approaching security the hacker way
   @mikearpaia
   

   View Slide

2. why is openness in security important?
   

   View Slide

3. what's osquery?
   why is openness in security important?
   

   View Slide

4. what's osquery?
   why is openness in security important?
   open source security design decisions
   

   View Slide

5. how can we work together?
   what's osquery?
   why is openness in security important?
   open source security design decisions
   

   View Slide

6. being open
   

   View Slide

7. open source makes life easier
   

   View Slide

8. no secrets are required
   we can be open in security
   •secrecy has stifled innovation
   •little software to help defend against modern attacks
   •reimplementing the same solutions, poorly
   •let's do the math
   

   View Slide

9. visualizing attacks and the defenses against them
   attacker math 101
   popularized by Dino Dai Zovi and Dan Guido
   •attacker will take the least cost path through an attack graph
   •the juice has to be worth the squeeze
   https://www.trailofbits.com/resources/attacker_math_101_slides.pdf
   

   View Slide

10. visualizing attacks and the defenses against them
    attack graphs
    start
    overcome
    obstacle
    overcome
    obstacle
    overcome
    obstacle
    accomplish
    objective
    

    View Slide

11. visualizing attacks and the defenses against them
    attack graphs
    start
    overcome
    obstacle
    overcome
    obstacle
    overcome
    obstacle
    accomplish
    objective
    

    View Slide

12. visualizing attacks and the defenses against them
    attack graphs
    start
    overcome
    obstacle
    overcome obstacle
    overcome
    obstacle
    accomplish
    objective
    

    View Slide

13. visualizing attacks and the defenses against them
    attack graphs
    start
    overcome
    obstacle
    overcome
    obstacle
    accomplish
    objective
    overcome
    obstacle
    overcome
    obstacle
    overcome
    obstacle
    

    View Slide

14. we can alter the path of attackers
    by altering our defenses
    

    View Slide

15. the future of security will be written in vim
    it's an engineering problem
    

    View Slide

16. osquery
    

    View Slide

17. an open platform for host instrumentation
    osquery
    if you need to collect low-level information from an operating
    system then you should use osquery
    •small footprint
    •rich capabilities
    •solves real world problems
    

    View Slide

18. SQL for your infrastructure
    osquery
    use SQL queries to explore OS state
    •running processes
    •loaded kernel modules
    •active network connections
    •route table
    •firewall settings
    •installed software
    •much more
    

    View Slide

19. osqueryi
    

    View Slide

20. LaunchDaemons which run a binary at boot
    

    View Slide

21. running processes
    

    View Slide

22. processes listening on ports
    

    View Slide

23. more tables are being written every day
    many tables are available
    •acpi_tables
    •arp_cache
    •crontab
    •file_events
    •kernel_info
    •listening_ports
    •logged_in_users
    •mounts
    •pci_devices
    •processes
    •routes
    •shell_history
    •smbios_tables
    •suid_bin
    •system_controls
    •usb_devices
    •users
    •groups
    •rpm_packages
    •apt_sources
    •deb_packages
    •homebrew_packages
    •kernel_modules
    •memory_map
    •shared_memory
    •browser_plugins
    •startup_items
    

    View Slide

24. osqueryd
    

    View Slide

25. daemon for low-level host monitoring
    osqueryd
    know how the results of a query change over time
    •schedule a query on your hosts via a config

    •the daemon takes care of periodically executing your queries
    •buffers results to disk and generates a log of state changes
    •logs results for aggregation and analytics
    •run your query in differential or snapshot mode
    

    View Slide

26. event-based operating system introspection
    host event pub/sub stream
    subscribe to key OS events to create dynamically growing tables
    •subscribe to “publishers”
    •filesystem changes (inotify, FSEvents)
    •network setting changes (SCNetwork)
    •query the history of your host, as it evolves
    

    View Slide

27. event-based file integrity logging
    file integrity monitoring
    use wildcards to monitor important files on your hosts
    •/bin/*
    •/Users/*/Downloads/**
    

    View Slide

28. for config distribution and data infrastructure
    plugin system
    •simple plugin API
    •specify your plugins at runtime with a command-line flag
    filesystem
    http
    zookeeper
    configuration
    filesystem
    flume
    scribe
    logging
    

    View Slide

29. design
    decisions
    

    View Slide

30. lessons learned from MIDAS
    we knew that we could do better at
    •exposing capabilities safely
    •reducing engineering overhead
    •improving development culture
    

    View Slide

31. the problem
    exposing capabilities
    •sharing a capability required sharing the whole module
    •few example modules were released
    •dramatically limited the utility of MIDAS
    

    View Slide

32. the solution
    exposing capabilities
    SQL allows for simple, flexible analysis
    •tables allow us to give away the answers without giving
    away the questions
    •capabilities can be configured instead of developed
    

    View Slide

33. the problem
    engineering overhead
    in MIDAS, capabilities were written in Python
    •"complex" code to solve simple problems
    •security people are not programmers
    

    View Slide

34. the solution
    engineering overhead
    with SQL, asking a question doesn't require writing complex code
    •low-level operating system analytics with attention to UEX
    SELECT address, mac, count(mac) AS mac_count
    FROM arp_cache GROUP BY mac
    HAVING count(mac) > 1;
    

    View Slide

35. the problem
    development culture
    open source code was not actively used internally
    •codebase began to drift
    •contributions weren't getting pulled in
    

    View Slide

36. the solution
    development culture
    we do 100% of our engineering in the open, on GitHub
    

    View Slide

37. fighting
    together
    

    View Slide

38. and now our watch begins
    working together to stay safe
    many different industries are working together
    •offensive security researchers
    •defensive security professionals
    •entrepreneurs in infrastructure/security
    

    View Slide

39. questions
    https://osquery.io
    

    View Slide