osquery: approaching security the hacker way

osquery was released as an open source product by Facebook in October 2014. It is an instrumentation framework for Ubuntu, CentOS, and OS X. osquery makes low-level operating system analytics and monitoring both performant and intuitive.

This talk will walk through why we created osquery, how we use osquery at Facebook to improve our security how other companies currently take advantage of osquery, and how you can too!

We’ll outline some of the challenges and sensitivities we faced when developing osquery and planning its open source release, as well as how we overcame those issues. Additionally, we’ll spend some time talking about why we believe open source is critical to advancing the state of trusted, secure software. Security through obscurity is dead; this is the age of security through transparency.

Transcript

1. mike arpaia / facebook osquery: approaching security the hacker way @mikearpaia
2. why is openness in security important?
3. what's osquery? why is openness in security important?
4. what's osquery? why is openness in security important? open source security design decisions
5. how can we work together? what's osquery? why is openness in security important? open source security design decisions
6. being open
7. open source makes life easier
8. no secrets are required we can be open in security •secrecy has stifled innovation •little software to help defend against modern attacks •reimplementing the same solutions, poorly •let's do the math
9. visualizing attacks and the defenses against them attacker math 101 popularized by Dino Dai Zovi and Dan Guido •attacker will take the least cost path through an attack graph •the juice has to be worth the squeeze https://www.trailofbits.com/resources/attacker_math_101_slides.pdf
10. visualizing attacks and the defenses against them attack graphs start overcome obstacle overcome obstacle overcome obstacle accomplish objective
11. visualizing attacks and the defenses against them attack graphs start overcome obstacle overcome obstacle overcome obstacle accomplish objective
12. visualizing attacks and the defenses against them attack graphs start overcome obstacle overcome obstacle overcome obstacle accomplish objective
13. visualizing attacks and the defenses against them attack graphs start overcome obstacle overcome obstacle accomplish objective overcome obstacle overcome obstacle overcome obstacle
14. we can alter the path of attackers by altering our defenses
15. the future of security will be written in vim it's an engineering problem
16. osquery
17. an open platform for host instrumentation osquery if you need to collect low-level information from an operating system then you should use osquery •small footprint •rich capabilities •solves real world problems
18. SQL for your infrastructure osquery use SQL queries to explore OS state •running processes •loaded kernel modules •active network connections •route table •firewall settings •installed software •much more
19. osqueryi
20. LaunchDaemons which run a binary at boot
21. running processes
22. processes listening on ports
23. more tables are being written every day many tables are available •acpi_tables •arp_cache •crontab •file_events •kernel_info •listening_ports •logged_in_users •mounts •pci_devices •processes •routes •shell_history •smbios_tables •suid_bin •system_controls •usb_devices •users •groups •rpm_packages •apt_sources •deb_packages •homebrew_packages •kernel_modules •memory_map •shared_memory •browser_plugins •startup_items
24. osqueryd
25. daemon for low-level host monitoring osqueryd know how the results of a query change over time •schedule a query on your hosts via a config
 •the daemon takes care of periodically executing your queries •buffers results to disk and generates a log of state changes •logs results for aggregation and analytics •run your query in differential or snapshot mode
26. event-based operating system introspection host event pub/sub stream subscribe to key OS events to create dynamically growing tables •subscribe to “publishers” •filesystem changes (inotify, FSEvents) •network setting changes (SCNetwork) •query the history of your host, as it evolves
27. event-based file integrity logging file integrity monitoring use wildcards to monitor important files on your hosts •/bin/* •/Users/*/Downloads/**
28. for config distribution and data infrastructure plugin system •simple plugin API •specify your plugins at runtime with a command-line flag filesystem http zookeeper configuration filesystem flume scribe logging
29. design decisions
30. lessons learned from MIDAS we knew that we could do better at •exposing capabilities safely •reducing engineering overhead •improving development culture
31. the problem exposing capabilities •sharing a capability required sharing the whole module •few example modules were released •dramatically limited the utility of MIDAS
32. the solution exposing capabilities SQL allows for simple, flexible analysis •tables allow us to give away the answers without giving away the questions •capabilities can be configured instead of developed
33. the problem engineering overhead in MIDAS, capabilities were written in Python •"complex" code to solve simple problems •security people are not programmers
34. the solution engineering overhead with SQL, asking a question doesn't require writing complex code •low-level operating system analytics with attention to UEX SELECT address, mac, count(mac) AS mac_count FROM arp_cache GROUP BY mac HAVING count(mac) > 1;
35. the problem development culture open source code was not actively used internally •codebase began to drift •contributions weren't getting pulled in
36. the solution development culture we do 100% of our engineering in the open, on GitHub
37. fighting together
38. and now our watch begins working together to stay safe many different industries are working together •offensive security researchers •defensive security professionals •entrepreneurs in infrastructure/security
39. questions https://osquery.io