Osquery, He Knows Me

This talk, aimed at everyone, highlights the journey from before Osquery in my time with the Etsy security team, and some of the tooling and problems we faced. The wrong decisions I've made, so you can learn from my foolish hubris. To a number of thousand deployment of Osquery (and fleet and some not so fleet) at Stripe.

Transcript

1. @benjammingh for QueryCon 2018 1
   

   View Slide

2. Who's this clown? 2
   • Security Engineer at Stripe.
   • Infrastructure security at Etsy.
   • Now has a commit in osquery, be afraid.
   • Once wore Mike Arpaia's pants to work, because he leC them in
   the office.
   2 h$ps:/
   /twi$er.com/skullmandible/status/411281851131523072
   @benjammingh for QueryCon 2018 2
   

   View Slide

3. What have the organisers unleashed?!
   • A lot of Genesis / Phil Collins references.
   • Some talk of osquery, probably.
   • Endpoint visibility And you may ask yourself, "Well... how did I
   get here?"
   @benjammingh for QueryCon 2018 3
   

   View Slide

4. Actually Mac visibility
   @benjammingh for QueryCon 2018 4
   

   View Slide

5. Enter BigMac
   2012 Facebook talk of Big Mac
   "Checks most basic persistence op2ons"
   @benjammingh for QueryCon 2018 5
   

   View Slide

6. View Slide

7. ENHANCE!
   Huh? This looks familiar...
   @benjammingh for QueryCon 2018 7
   

   View Slide

8. Meanwhile, back at stately Wayne Manor
   @benjammingh for QueryCon 2018 8
   

   View Slide

9. Etsy security team
   • Roll on early 2013
   • Etsy looking to make our own version
   • Standard development prac=ces apply
   @benjammingh for QueryCon 2018 9
   

   View Slide

10. View Slide

11. is born
    • Python based (system python)
    • modular
    • persistent datastore (sqlite)
    • logs to disk, which then goes to splunk
    @benjammingh for QueryCon 2018 11
    

    View Slide

12. Scroll forward to December 2013
    @benjammingh for QueryCon 2018 12
    

    View Slide

13. Which then got released publicly (and /ever so slightly/ nerfed) as
    MIDAS, to rave reviews on HackerNews
    Also, a(er looking at the code, it's barely useful.
    — [deleted] 22 points 4 years ago
    @benjammingh for QueryCon 2018 13
    

    View Slide

14. Mike -> Facebook
    Zane -> Signal Sciences
    @benjammingh for QueryCon 2018 14
    

    View Slide

15. 2014
    • Rich Smith adds a proper build system...
    • "Stealth mode" of no binaries on disk, by using pyinstaller (yes I
    know they're s@ll on disk)
    • I became the overprotec@ve maintainer of it.
    @benjammingh for QueryCon 2018 15
    

    View Slide

16. Etsy security
    ❤
    Facebook security
    @benjammingh for QueryCon 2018 16
    

    View Slide

17. Mike Arpaia
    @benjammingh for QueryCon 2018 17
    

    View Slide

18. View Slide

19. "640K ought to be because of architectural
    limita6on of the IBM XT"
    • "Facebook has a whole floor of analysts, we have none, so
    Python is be9er than SQL for us."
    • "I want to be alerted when someone compromises something,
    not when I go looking for it."
    • "We already have something that works, lets just keep
    maintaining that."
    @benjammingh for QueryCon 2018 19
    

    View Slide

20. So what happened?
    @benjammingh for QueryCon 2018 20
    

    View Slide

21. @benjammingh for QueryCon 2018 21
    

    View Slide

22. View Slide

23. "I was completely and u2erly wrong
    on every level"
    — Me
    @benjammingh for QueryCon 2018 23
    

    View Slide

24. So why are we even
    listening to you again?
    @benjammingh for QueryCon 2018 24
    

    View Slide

25. You don't have to always be right,
    but it's helpful to admit when you're wrong
    @benjammingh for QueryCon 2018 25
    

    View Slide

26. Don't Get A*ached To Your Code
    @benjammingh for QueryCon 2018 26
    

    View Slide

27. being proud of code you write is
    different to being beholden to it
    @benjammingh for QueryCon 2018 27
    

    View Slide

28. and that was the only catharsis that they could find
    without violence...
    @benjammingh for QueryCon 2018 28
    

    View Slide

29. @benjammingh for QueryCon 2018 29
    

    View Slide

30. Osquery
    @benjammingh for QueryCon 2018 30
    

    View Slide

31. If leaving me is Etsy
    @benjammingh for QueryCon 2018 31
    

    View Slide

32. We had osquery
    % osqueryi --version
    osqueryi version 2.2.3
    @benjammingh for QueryCon 2018 32
    

    View Slide

33. @benjammingh for QueryCon 2018 33
    

    View Slide

34. Doorman
    • rad, useful, easy to get going!
    • has a backing persistent storage, so queries get hunted down.
    • from looking at it, looked a solid architecture and in python
    @benjammingh for QueryCon 2018 34
    

    View Slide

35. Doorman cont.
    • from looking at it, argh my eyes, burning... (okay, its very
    func9onal but not pre;y)
    • like everything at Stripe, customised forked version
    • which you could only access over SSH port forwarding
    @benjammingh for QueryCon 2018 35
    

    View Slide

36. kolide/fleet
    Was just kolide back then, commercial offering
    @benjammingh for QueryCon 2018 36
    

    View Slide

37. How does fleet work?
    @benjammingh for QueryCon 2018 37
    

    View Slide

38. View Slide

39. View Slide

40. logs
    

    View Slide

41. {
    "cake": "eccles",
    "coffee": "long black",
    "serialisation": "ASN1"
    }
    @benjammingh for QueryCon 2018 41
    

    View Slide

42. ELK
    @benjammingh for QueryCon 2018 42
    

    View Slide

43. @benjammingh for QueryCon 2018 43
    

    View Slide

44. @benjammingh for QueryCon 2018 44
    

    View Slide

45. Fleet at Stripe
    • 1000s of endpoints.
    • mul2ple pla4orms.
    • phased roll out thanks to Munki
    • lots of exci2ng interes2ng queries!
    @benjammingh for QueryCon 2018 45
    

    View Slide

46. Ben, it's lunch soon, wrap this up!
    — everyone
    @benjammingh for QueryCon 2018 46
    

    View Slide

47. Fine, lets Neal Stephenson this slide
    show!
    — me
    @benjammingh for QueryCon 2018 47
    

    View Slide

48. Security and opera/ons and everything
    in between, be careful with that pride
    @benjammingh for QueryCon 2018 48
    

    View Slide

49. Just because you own the code,
    don't let the code own you
    @benjammingh for QueryCon 2018 49
    

    View Slide

50. Community
    @benjammingh for QueryCon 2018 50
    

    View Slide

51. Lunch!
    @benjammingh for QueryCon 2018 51
    

    View Slide

52. • Twidder: @benjammingh
    • LinkedIn: lnkdin.me/p/benyeah
    • SpeakerDeck: speakerdeck.com/barnbarn
    • Stripe: Careers <--- Engineering blog
    @benjammingh for QueryCon 2018 52
    

    View Slide