A Look Into HTTP.rb And why you shouldn’t use Net::HTTP janko-m @jankomarohnic

Implementation Net::HTTP variants Net::HTTP pure ruby REST Client Net::HTTP HTTParty Net::HTTP open-uri Net::HTTP libcurl variants Typhoeus libcurl Curb libcurl Patron libcurl wrapper Faraday wrapper pure ruby HTTP.rb pure ruby HTTPClient pure ruby https://www.slideshare.net/HiroshiNakamura/rubyhttp-clients-comparison

Let’s just use Net::HTTP What could possibly go wrong?

Net::HTTP.get(URI("https://example.com")) #=> "…" Net::HTTP.get_response(URI("https://example.com")) #=> # Net::HTTP.post(URI("https://example.com")) #=> # Net::HTTP.post_form(URI("https://example.com"), {}) #=> # Net::HTTP.put(URI("https://example.com")) # NoMethodError: undefined method `put’ Net::HTTP.delete(URI("https://example.com")) # NoMethodError: undefined method `delete’

No content

uri = URI.parse("https://example.com/path") use_ssl = uri.is_a?(URI::HTTPS) Net::HTTP.start(uri.host, uri.port, use_ssl: use_ssl) do |http| http.post(uri.path, URI.encode_www_form(params)) end 1. parse the URL string 2. determine whether we need to use SSL 3. open the TCP connection 4. encode the post parameters 5. send the request

uri = URI.parse("http://example.com/path") begin Net::HTTP.start(uri.host, uri.port) do |http| http.post(uri.path, URI.encode_www_form(params)) end rescue SocketError, EOFError, IOError, Errno::ECONNABORTED, Errno::ECONNRESET, Errno::EINVAL, Errno::ETIMEDOUT, Errno::EHOSTUNREACH, Errno::ENETUNREACH, Errno::ECONNREFUSED, Errno::EPIPE retry end

uri = URI.parse("http://example.com/path") begin Net::HTTP.start(uri.host, uri.port) do |http| http.post(uri.path, URI.encode_www_form(params)) end rescue SocketError, EOFError, IOError, SystemCallError retry end SocketError EOFError IOError SystemCallError = ConnectionError?

Downsides of Net::HTTP • Wide and verbose interface • e.g. 3 mutually inconsistent ways of making requests • Poor OO design • Exposes low-level exceptions • Ugly codebase (it’s in stdlib)

Implementation Net::HTTP variants Net::HTTP pure ruby REST Client Net::HTTP HTTParty Net::HTTP open-uri Net::HTTP libcurl variants Typhoeus libcurl Curb libcurl Patron libcurl wrapper Faraday wrapper pure ruby HTTP.rb pure ruby HTTPClient pure ruby https://www.slideshare.net/HiroshiNakamura/rubyhttp-clients-comparison

Implementation Net::HTTP variants Net::HTTP pure ruby REST Client Net::HTTP HTTParty Net::HTTP open-uri Net::HTTP libcurl variants Typhoeus libcurl Curb libcurl Patron libcurl wrapper Faraday wrapper pure ruby HTTP.rb pure ruby HTTPClient pure ruby https://www.slideshare.net/HiroshiNakamura/rubyhttp-clients-comparison

HTTP.rb gem "http"

uri = URI.parse("https://example.com/path") use_ssl = uri.is_a?(URI::HTTPS) Net::HTTP.start(uri.host, uri.port, use_ssl: use_ssl) do |http| http.post(uri.path, URI.encode_www_form(params)) end HTTP.post("https://example.com/path", form: params) Net::HTTP HTTP.rb

• Pure ruby implementation • Clean and Chainable API • Correct URL parsing • Native timeouts • Persistent connections • Streaming bodies • Compressing and decompressing bodies

• Pure ruby implementation • Clean and Chainable API • Correct URL parsing • Native timeouts • Persistent connections • Streaming bodies • Compressing and decompressing bodies

• Pure ruby implementation • Clean and Chainable API • Correct URL parsing • Native timeouts • Persistent connections • Streaming bodies • Compressing and decompressing bodies

response = HTTP.get("https://example.com") response # => # response.status # => # response.status.code # => 200 response.status.ok? # => true response.status.success? # => true response.headers # => # response.headers.to_h # => {"Content-Type"=>"text/html", …} response.body # => # response.body.to_s # => "…"

HTTP.headers("Accept" => "application/json") .basic_auth("janko", "password") .follow(max_hops: 2) .get("http://example.com") http = HTTP .headers("Accept" => "application/json") .basic_auth("janko", "password") .follow(max_hops: 2) http #=> # http.get("http://example.com/posts") http.get("http://example.com/posts/1/comments") http.post("http//example.com/posts/1/comments", json: {…})

begin response = HTTP.get("https://example.com") rescue HTTP::ConnectionError retry end HTTP::Error !"" HTTP::ConnectionError !"" HTTP::RequestError !"" HTTP::ResponseError # $"" HTTP::StateError !"" HTTP::TimeoutError $"" HTTP::HeaderError

• Pure ruby implementation • Clean and Chainable API • Correct URL parsing • Native timeouts • Persistent connections • Streaming bodies • Compressing and decompressing bodies

url = "https://movies.com/matrix[1999].mp4" Net::HTTP.get_response(URI(url)) url = "https://movies.com/matrix[1999].mp4" url = URI.encode(url) Net::HTTP.get_response(URI(url)) url = "https://movies.com/matrix[1999].mp4" url = URI.decode(url) url = URI.encode(url) Net::HTTP.get_response(URI(url)) HTTP.get("https://movies.com/matrix[1999].mp4") #=> URI::InvalidURIError #=> URI::InvalidURIError #=> URI::InvalidURIError

• Pure ruby implementation • Clean and Chainable API • Correct URL parsing • Native timeouts • Persistent connections • Streaming bodies • Compressing and decompressing bodies

require "timeout" Timeout.timeout(5) do HTTP.get("https://example.com") end "Ensure" blocks might not get executed HTTP.timeout(connect: 3) .get("http://example.com") HTTP.timeout(connect: 3, write: 3) .get("http://example.com") HTTP.timeout(connect: 3, write: 3, read: 3) .get("http://example.com") # 5 seconds allowed for entire call HTTP.timeout(:global, connect: 1, write: 2, read: 2) .get("http://example.com")

• Pure ruby implementation • Clean and Chainable API • Correct URL parsing • Native timeouts • Persistent connections • Streaming bodies • Compressing and decompressing bodies

HTTP.get("https://example.com") # connect + write + read + close HTTP.get("https://example.com") # connect + write + read + close HTTP.get("https://example.com") # connect + write + read + close HTTP.persistent("https://example.com") do |http| http.get("/") # connect + write + read http.get("/") # write + read http.get("/") # write + read end # close Typhoeus.get("https://example.com") # connect + write + read Typhoeus.get("https://example.com") # write + read Typhoeus.get("https://example.com") # write + read

• Pure ruby implementation • Clean and Chainable API • Correct URL parsing • Native timeouts • Persistent connections • Streaming bodies • Compressing and decompressing bodies

source = Transcoder.call("matrix.mp4") source.size #=> 500 MB destination = File.open("matrix-transcoded.mp4", "w") while (chunk = source.read(16*1024, buffer ||= "")) destination.write(chunk) end destination.write(source.read) IO.copy_stream(source, destination)

require "socket" socket = TCPSocket.open("example.com", 80) socket.write "GET / HTTP/1.1" + "\r\n" + "Host: example.com" + "\r\n" + "Content-Length: 0" + "\r\n" + "Connection: close" + "\r\n" + "\r\n" socket.read #=> "HTTP/1.1 200 OK" + "\r\n" + # "Content-Type: text/html" + "\r\n" + # "Content-Length: 1270" + "\r\n" + # "Connection: close" + "\r\n" + # "\r\n" + # " …" socket.close

require "socket" socket = TCPSocket.open("example.com", 80) socket.write "GET / HTTP/1.1" + "\r\n" + "Host: example.com" + "\r\n" + "Content-Length: #{body.size}" + "\r\n" + "Connection: close" + "\r\n" + "\r\n" socket.write body.read socket.read #=> "HTTP/1.1 200 OK" + "\r\n" + # "Content-Type: text/html" + "\r\n" + # "Content-Length: 1270" + "\r\n" + # "Connection: close" + "\r\n" + # "\r\n" + # " …" socket.close

require "socket" socket = TCPSocket.open("example.com", 80) socket.write "GET / HTTP/1.1" + "\r\n" + "Host: example.com" + "\r\n" + "Content-Length: #{body.size}" + "\r\n" + "Connection: close" + "\r\n" + "\r\n" IO.copy_stream(body, socket) # streaming! socket.read #=> "HTTP/1.1 200 OK" + "\r\n" + # "Content-Type: text/html" + "\r\n" + # "Content-Length: 1270" + "\r\n" + # "Connection: close" + "\r\n" + # "\r\n" + # " …" socket.close

require "socket" socket = TCPSocket.open("example.com", 80) socket.write "GET / HTTP/1.1" + "\r\n" + "Host: example.com" + "\r\n" + "Content-Length: #{body.size}" + "\r\n" + "Connection: close" + "\r\n" + "\r\n" IO.copy_stream(body, socket) # streaming! while (chunk = socket.readpartial(16*1024)) # streaming! # parse response end socket.close

HTTP.post(url, body: "this is my body") # string HTTP.post(url, body: enumerable) # #each HTTP.post(url, body: io) # #read & #size # File streaming HTTP.post(url, body: File.open("path/to/file.txt")) # StringIO streaming HTTP.post(url, body: StringIO.new("content")) # Pipe streaming HTTP.post(url, body: IO.popen("shell command")) # Multipart form data streaming HTTP.post(url, form: { file: HTTP::FormData::File.new("path/to/file.txt") })

response = HTTP.get("http://example.com/export.csv") response.body # nothing has been read yet response.body.to_s # reads whole response body # or response.body.readpartial # reads first chunk response.body.readpartial # reads next chunk # or response.body.each { |chunk| … } # yields chunks response = HTTP.get("http://example.com/export.csv") # reading headers before download fail "too large" if response.content_length > max_size # streaming download to disk File.open("export.csv", "w") do |file| response.body.each do |chunk| file.write(chunk) end end

Streaming bodies ↓ Constant memory usage Less disk I/O

• Pure ruby implementation • Clean and Chainable API • Correct URL parsing • Native timeouts • Persistent connections • Streaming bodies • Compressing and decompressing bodies

HTTP.post(url, body: File.open("file.txt")) # as is # POST /path HTTP/1.1 # Content-Length: 423847673 # Content-Encoding: identity # # [raw content] # HTTP/1.1 200 OK Request body

HTTP.use(:auto_deflate) post(url, body: File.open("file.txt")) # compression # POST /path HTTP/1.1 # Content-Length: 214328782 # Content-Encoding: gzip # # [compressed content] # HTTP/1.1 200 OK Request body

HTTP.get("http://example.com/file.txt") # GET /file.txt HTTP/1.1 # HTTP/1.1 200 OK # Content-Length: 423847673 # Content-Encoding: identity # # [raw content] response.body.each { |chunk| … } # as is Response body

HTTP.use(:auto_inflate) get("http://example.com/file.txt") # GET /file.txt HTTP/1.1 # HTTP/1.1 200 OK # Content-Length: 214328782 # Content-Encoding: gzip # # [compressed content] response.body.each { |chunk| … } # decompression Response body

Compressed bodies ↓ Faster upload/download Less network resources

Celluloid, Reel, Socketry (Nio4r), …

https://github.com/httprb/http