A look into HTTP.rb (And why you shouldn't use Net::HTTP)

Transcript

1. A Look Into HTTP.rb
   And why you shouldn’t use Net::HTTP
   janko-m @jankomarohnic
   

   View Slide

2. Implementation
   Net::HTTP
   variants
   Net::HTTP pure ruby
   REST Client Net::HTTP
   HTTParty Net::HTTP
   open-uri Net::HTTP
   libcurl
   variants
   Typhoeus libcurl
   Curb libcurl
   Patron libcurl
   wrapper Faraday wrapper
   pure ruby HTTP.rb pure ruby
   HTTPClient pure ruby
   https://www.slideshare.net/HiroshiNakamura/rubyhttp-clients-comparison
   

   View Slide

3. Let’s just use Net::HTTP
   What could possibly go wrong?
   

   View Slide

4. Net::HTTP.get(URI("https://example.com"))
   #=> "…"
   Net::HTTP.get_response(URI("https://example.com"))
   #=> #
   Net::HTTP.post(URI("https://example.com"))
   #=> #
   Net::HTTP.post_form(URI("https://example.com"), {})
   #=> #
   Net::HTTP.put(URI("https://example.com"))
   # NoMethodError: undefined method `put’
   Net::HTTP.delete(URI("https://example.com"))
   # NoMethodError: undefined method `delete’
   

   View Slide

5. View Slide

6. uri = URI.parse("https://example.com/path")
   use_ssl = uri.is_a?(URI::HTTPS)
   Net::HTTP.start(uri.host, uri.port, use_ssl: use_ssl) do |http|
   http.post(uri.path, URI.encode_www_form(params))
   end
   1. parse the URL string
   2. determine whether we need to use SSL
   3. open the TCP connection
   4. encode the post parameters
   5. send the request
   
   

   View Slide

7. uri = URI.parse("http://example.com/path")
   begin
   Net::HTTP.start(uri.host, uri.port) do |http|
   http.post(uri.path, URI.encode_www_form(params))
   end
   rescue SocketError, EOFError, IOError,
   Errno::ECONNABORTED,
   Errno::ECONNRESET,
   Errno::EINVAL,
   Errno::ETIMEDOUT,
   Errno::EHOSTUNREACH,
   Errno::ENETUNREACH,
   Errno::ECONNREFUSED,
   Errno::EPIPE
   retry
   end
   

   View Slide

8. uri = URI.parse("http://example.com/path")
   begin
   Net::HTTP.start(uri.host, uri.port) do |http|
   http.post(uri.path, URI.encode_www_form(params))
   end
   rescue SocketError, EOFError, IOError, SystemCallError
   retry
   end
   SocketError
   EOFError
   IOError
   SystemCallError
   = ConnectionError?
   
   

   View Slide

9. Downsides of Net::HTTP
   • Wide and verbose interface
   • e.g. 3 mutually inconsistent ways of making requests
   • Poor OO design
   • Exposes low-level exceptions
   • Ugly codebase (it’s in stdlib)
   

   View Slide

10. Implementation
    Net::HTTP
    variants
    Net::HTTP pure ruby
    REST Client Net::HTTP
    HTTParty Net::HTTP
    open-uri Net::HTTP
    libcurl
    variants
    Typhoeus libcurl
    Curb libcurl
    Patron libcurl
    wrapper Faraday wrapper
    pure ruby HTTP.rb pure ruby
    HTTPClient pure ruby
    https://www.slideshare.net/HiroshiNakamura/rubyhttp-clients-comparison
    

    View Slide

11. Implementation
    Net::HTTP
    variants
    Net::HTTP pure ruby
    REST Client Net::HTTP
    HTTParty Net::HTTP
    open-uri Net::HTTP
    libcurl
    variants
    Typhoeus libcurl
    Curb libcurl
    Patron libcurl
    wrapper Faraday wrapper
    pure ruby HTTP.rb pure ruby
    HTTPClient pure ruby
    https://www.slideshare.net/HiroshiNakamura/rubyhttp-clients-comparison
    

    View Slide

12. HTTP.rb
    gem "http"
    

    View Slide

13. uri = URI.parse("https://example.com/path")
    use_ssl = uri.is_a?(URI::HTTPS)
    Net::HTTP.start(uri.host, uri.port, use_ssl: use_ssl) do |http|
    http.post(uri.path, URI.encode_www_form(params))
    end
    HTTP.post("https://example.com/path", form: params)
    Net::HTTP
    HTTP.rb
    

    View Slide

14. • Pure ruby implementation
    • Clean and Chainable API
    • Correct URL parsing
    • Native timeouts
    • Persistent connections
    • Streaming bodies
    • Compressing and decompressing bodies
    

    View Slide

15. • Pure ruby implementation
    • Clean and Chainable API
    • Correct URL parsing
    • Native timeouts
    • Persistent connections
    • Streaming bodies
    • Compressing and decompressing bodies
    

    View Slide

16. • Pure ruby implementation
    • Clean and Chainable API
    • Correct URL parsing
    • Native timeouts
    • Persistent connections
    • Streaming bodies
    • Compressing and decompressing bodies
    

    View Slide

17. response = HTTP.get("https://example.com")
    response # => #
    response.status # => #
    response.status.code # => 200
    response.status.ok? # => true
    response.status.success? # => true
    response.headers # => #
    response.headers.to_h # => {"Content-Type"=>"text/html", …}
    response.body # => #
    response.body.to_s # => "…"
    

    View Slide

18. HTTP.headers("Accept" => "application/json")
    .basic_auth("janko", "password")
    .follow(max_hops: 2)
    .get("http://example.com")
    http = HTTP
    .headers("Accept" => "application/json")
    .basic_auth("janko", "password")
    .follow(max_hops: 2)
    http #=> #
    http.get("http://example.com/posts")
    http.get("http://example.com/posts/1/comments")
    http.post("http//example.com/posts/1/comments", json: {…})
    

    View Slide

19. begin
    response = HTTP.get("https://example.com")
    rescue HTTP::ConnectionError
    retry
    end
    HTTP::Error
    !"" HTTP::ConnectionError
    !"" HTTP::RequestError
    !"" HTTP::ResponseError
    # $"" HTTP::StateError
    !"" HTTP::TimeoutError
    $"" HTTP::HeaderError
    

    View Slide

20. • Pure ruby implementation
    • Clean and Chainable API
    • Correct URL parsing
    • Native timeouts
    • Persistent connections
    • Streaming bodies
    • Compressing and decompressing bodies
    

    View Slide

21. url = "https://movies.com/matrix[1999].mp4"
    Net::HTTP.get_response(URI(url))
    url = "https://movies.com/matrix[1999].mp4"
    url = URI.encode(url)
    Net::HTTP.get_response(URI(url))
    url = "https://movies.com/matrix[1999].mp4"
    url = URI.decode(url)
    url = URI.encode(url)
    Net::HTTP.get_response(URI(url))
    HTTP.get("https://movies.com/matrix[1999].mp4")
    #=> URI::InvalidURIError
    #=> URI::InvalidURIError
    #=> URI::InvalidURIError
    
    

    View Slide

22. • Pure ruby implementation
    • Clean and Chainable API
    • Correct URL parsing
    • Native timeouts
    • Persistent connections
    • Streaming bodies
    • Compressing and decompressing bodies
    

    View Slide

23. require "timeout"
    Timeout.timeout(5) do
    HTTP.get("https://example.com")
    end
    "Ensure" blocks might not get executed
    HTTP.timeout(connect: 3)
    .get("http://example.com")
    HTTP.timeout(connect: 3, write: 3)
    .get("http://example.com")
    HTTP.timeout(connect: 3, write: 3, read: 3)
    .get("http://example.com")
    # 5 seconds allowed for entire call
    HTTP.timeout(:global, connect: 1, write: 2, read: 2)
    .get("http://example.com")
    

    View Slide

24. • Pure ruby implementation
    • Clean and Chainable API
    • Correct URL parsing
    • Native timeouts
    • Persistent connections
    • Streaming bodies
    • Compressing and decompressing bodies
    

    View Slide

25. HTTP.get("https://example.com") # connect + write + read + close
    HTTP.get("https://example.com") # connect + write + read + close
    HTTP.get("https://example.com") # connect + write + read + close
    HTTP.persistent("https://example.com") do |http|
    http.get("/") # connect + write + read
    http.get("/") # write + read
    http.get("/") # write + read
    end # close
    Typhoeus.get("https://example.com") # connect + write + read
    Typhoeus.get("https://example.com") # write + read
    Typhoeus.get("https://example.com") # write + read
    

    View Slide

26. • Pure ruby implementation
    • Clean and Chainable API
    • Correct URL parsing
    • Native timeouts
    • Persistent connections
    • Streaming bodies
    • Compressing and decompressing bodies
    

    View Slide

27. source = Transcoder.call("matrix.mp4")
    source.size #=> 500 MB
    destination = File.open("matrix-transcoded.mp4", "w")
    while (chunk = source.read(16*1024, buffer ||= ""))
    destination.write(chunk)
    end
    destination.write(source.read)
    IO.copy_stream(source, destination)
    
    
    
    
    

    View Slide

28. require "socket"
    socket = TCPSocket.open("example.com", 80)
    socket.write "GET / HTTP/1.1" + "\r\n" +
    "Host: example.com" + "\r\n" +
    "Content-Length: 0" + "\r\n" +
    "Connection: close" + "\r\n" +
    "\r\n"
    socket.read #=> "HTTP/1.1 200 OK" + "\r\n" +
    # "Content-Type: text/html" + "\r\n" +
    # "Content-Length: 1270" + "\r\n" +
    # "Connection: close" + "\r\n" +
    # "\r\n" +
    # " …"
    socket.close
    

    View Slide

29. require "socket"
    socket = TCPSocket.open("example.com", 80)
    socket.write "GET / HTTP/1.1" + "\r\n" +
    "Host: example.com" + "\r\n" +
    "Content-Length: #{body.size}" + "\r\n" +
    "Connection: close" + "\r\n" +
    "\r\n"
    socket.write body.read
    socket.read #=> "HTTP/1.1 200 OK" + "\r\n" +
    # "Content-Type: text/html" + "\r\n" +
    # "Content-Length: 1270" + "\r\n" +
    # "Connection: close" + "\r\n" +
    # "\r\n" +
    # " …"
    socket.close
    

    View Slide

30. require "socket"
    socket = TCPSocket.open("example.com", 80)
    socket.write "GET / HTTP/1.1" + "\r\n" +
    "Host: example.com" + "\r\n" +
    "Content-Length: #{body.size}" + "\r\n" +
    "Connection: close" + "\r\n" +
    "\r\n"
    IO.copy_stream(body, socket) # streaming!
    socket.read #=> "HTTP/1.1 200 OK" + "\r\n" +
    # "Content-Type: text/html" + "\r\n" +
    # "Content-Length: 1270" + "\r\n" +
    # "Connection: close" + "\r\n" +
    # "\r\n" +
    # " …"
    socket.close
    

    View Slide

31. require "socket"
    socket = TCPSocket.open("example.com", 80)
    socket.write "GET / HTTP/1.1" + "\r\n" +
    "Host: example.com" + "\r\n" +
    "Content-Length: #{body.size}" + "\r\n" +
    "Connection: close" + "\r\n" +
    "\r\n"
    IO.copy_stream(body, socket) # streaming!
    while (chunk = socket.readpartial(16*1024)) # streaming!
    # parse response
    end
    socket.close
    

    View Slide

32. HTTP.post(url, body: "this is my body") # string
    HTTP.post(url, body: enumerable) # #each
    HTTP.post(url, body: io) # #read & #size
    # File streaming
    HTTP.post(url, body: File.open("path/to/file.txt"))
    # StringIO streaming
    HTTP.post(url, body: StringIO.new("content"))
    # Pipe streaming
    HTTP.post(url, body: IO.popen("shell command"))
    # Multipart form data streaming
    HTTP.post(url, form: {
    file: HTTP::FormData::File.new("path/to/file.txt")
    })
    

    View Slide

33. response = HTTP.get("http://example.com/export.csv")
    response.body # nothing has been read yet
    response.body.to_s # reads whole response body
    # or
    response.body.readpartial # reads first chunk
    response.body.readpartial # reads next chunk
    # or
    response.body.each { |chunk| … } # yields chunks
    response = HTTP.get("http://example.com/export.csv")
    # reading headers before download
    fail "too large" if response.content_length > max_size
    # streaming download to disk
    File.open("export.csv", "w") do |file|
    response.body.each do |chunk|
    file.write(chunk)
    end
    end
    

    View Slide

34. Streaming bodies
    ↓
    Constant memory usage
    Less disk I/O
    

    View Slide

35. • Pure ruby implementation
    • Clean and Chainable API
    • Correct URL parsing
    • Native timeouts
    • Persistent connections
    • Streaming bodies
    • Compressing and decompressing bodies
    

    View Slide

36. HTTP.post(url, body: File.open("file.txt")) # as is
    # POST /path HTTP/1.1
    # Content-Length: 423847673
    # Content-Encoding: identity
    #
    # [raw content]
    # HTTP/1.1 200 OK
    Request body
    

    View Slide

37. HTTP.use(:auto_deflate)
    post(url, body: File.open("file.txt")) # compression
    # POST /path HTTP/1.1
    # Content-Length: 214328782
    # Content-Encoding: gzip
    #
    # [compressed content]
    # HTTP/1.1 200 OK
    Request body
    

    View Slide

38. HTTP.get("http://example.com/file.txt")
    # GET /file.txt HTTP/1.1
    # HTTP/1.1 200 OK
    # Content-Length: 423847673
    # Content-Encoding: identity
    #
    # [raw content]
    response.body.each { |chunk| … } # as is
    Response body
    

    View Slide

39. HTTP.use(:auto_inflate)
    get("http://example.com/file.txt")
    # GET /file.txt HTTP/1.1
    # HTTP/1.1 200 OK
    # Content-Length: 214328782
    # Content-Encoding: gzip
    #
    # [compressed content]
    response.body.each { |chunk| … } # decompression
    Response body
    

    View Slide

40. Compressed bodies
    ↓
    Faster upload/download
    Less network resources
    

    View Slide

41. Celluloid, Reel, Socketry (Nio4r), …
    

    View Slide

42. https://github.com/httprb/http
    

    View Slide