Ruby HTTP clients comparison

Ruby HTTP clients comparison

RubyConf 2012 presentation slides

6c06915d9aa00cd5c7d4acfb27cdd4e9?s=128

Hiroshi Nakamura

November 02, 2012
Tweet

Transcript

  1. Ruby HTTP clients comparison Hiroshi Nakamura nahi at Twitter, github

    Technical Architect at Appirio Japan CRuby and JRuby committer Asakusa.rb: http://qwik.jp/asakusarb/
  2. Ruby HTTP Clients Matrix advantages-and-disadvantages comparison of HTTP client libraries

    http://bit.ly/RubyHTTPClients2012 Disclaimer: I'm the author of "httpclient"
  3. Agenda net/http 16 libraries I picked Ruby HTTP Clients Matrix

    API style Compatibility Supported features Performance Comparisons My Recommendations
  4. net/http Net::HTTP::Proxy? net/https?

  5. Ruby HTTP Client libraries

  6. HTTP client libraries I didn’t evaluate Cannot evaluate • activeresource

    (Rails specific) • http (under development) • http_request.rb (test doesn't pass) • nestful (no test) • typhoeus (under heavy rewrite) Obsolete • eventmachine (built-in client is obsolete) • right_http_connection (no update) • simplehttp (no update) • rfuzz (no update)
  7. Evaluation Axis Project Stats API style Compatibility: CRuby, JRuby, Rubinius

    Supported features Connection features Basic HTTP features Development support Advanced features http://bit.ly/RubyHTTPClientsFeatureTest (test/unit scripts)
  8. Project Stats

  9. None
  10. None
  11. None
  12. Project Stats

  13. API style

  14. None
  15. sync API (1/5) - Client instance net/http, mechanize, httpclient, patron,

    curb, faraday client = Net::HTTP.new(host, port) p client.get(path).body client = Mechanize.new client = HTTPClient.new client = Patron::Session.new p client.get(url).body curl = Curl::Easy.new(url) curl.http_get p curl.body_str client = Faraday.new(:url => baseurl) p client.get(path).body
  16. sync API (2/5) - Client class restfulie, excon, httpi p

    Restfulie.at(url).get!.body p Excon.get(url).body p HTTPI.get(url).body
  17. sync API (3/5) - Resource rest-client, rufus-verbs rs = RestClient::Resource.new('http://example.com')

    rs['posts/1/comments'].post 'Good article.', :content_type => 'text/plain' ep = EndPoint.new( :host => "resta.farian.host", :port => 80, :resource => "inventory/tools") res = ep.get :id => 1 res = ep.get :id => 2
  18. sync API (4/5) - Include & Customize httparty, weary client

    = Class.new { include HTTParty } p client.get(url) class WearyClient < Weary::Client domain 'http://api.target.org/' get :retrieve, '{path}' end p WearyClient.new.fetch(:path => path).perform.body
  19. sync API (5/5) - Others open-uri.rb, wrest # open-uri.rb p

    open(url) { |f| f.read } # wrest p 'http://www.google.co.jp/'.to_uri.get.body
  20. async API(1/2) - Callback em-http-request body = nil EM.run do

    req = EM::HttpRequest.new( 'http://www.google.com/').get req.callback do body = req.response EM.stop end req.errback do body = nil end end p body
  21. async API(2/2) - Polling httpclient, weary, wrest client = HTTPClient.new

    conn = client.get_async(url) conn.finished? # => false # ... io = conn.pop.content while str = io.read(4096) p str end
  22. parallel API - curb responses = [] m = Curl::Multi.new

    urls.each do |url| responses[url] = '' m.add(Curl::Easy.new(url) { |curl| curl.on_body { |data| responses[url] << data data.bytesize } }) end m.perform p responses.map { |e| e.bytesize }
  23. None
  24. None
  25. API style

  26. Compatibility

  27. None
  28. Compatibility

  29. Connection features

  30. None
  31. 3 types of HTTP connections

  32. None
  33. Keep-Alive in em-http-request body = [] EM.run do conn =

    EventMachine::HttpRequest.new(server.url) req1 = conn.get(:keepalive => true) req1.callback { body << req1.response req2 = conn.get(:keepalive => true) req2.callback { body << req2.response req3 = conn.get(:keepalive => true) req3.callback { body << req3.response req4 = conn.get(:keepalive => true) req4.callback { body << req4.response EM.stop req4.errback { ... }} req3.errback { ... }} req2.errback { ... }} req1.errback { ... } end
  34. Pipelining in em-http-request body = [] EM.run do conn =

    EventMachine::HttpRequest.new(server.url) req1 = conn.get(:keepalive => true) req2 = conn.get(:keepalive => true) req3 = conn.get(:keepalive => true) req4 = conn.get() req1.callback { body << req1.response } req2.callback { body << req2.response } req3.callback { body << req3.response } req4.callback { body << req4.response; EM.stop } req1.errback { ... } req2.errback { ... } req3.errback { ... } req4.errback { ... } end
  35. None
  36. NO verification by default?! options[:ssl_ca_file] == nil => VERIFY_NONE if

    http.use_ssl? http.verify_mode = OpenSSL::SSL::VERIFY_NONE if options[:ssl_ca_file] http.ca_file = options[:ssl_ca_file] http.verify_mode = OpenSSL::SSL::VERIFY_PEER end end
  37. None
  38. None
  39. None
  40. Connection features

  41. Basic HTTP features

  42. None
  43. None
  44. IRI: Internationalized Resource Identifier uri.rb doesn't support IRI addressable/uri does

    client.get("http://www.ebooks.com/797059/some-kind- of-peace/grebe-camilla-träff-åsa-norlen-paul/")
  45. None
  46. "Cross-site Cooking" bug in httpclient httpclient eats this cookie and

    send it to all *.com site Mechanize handles it properly like browsers http://en.wikipedia.org/wiki/Cross-site_cooking Set-Cookie: TEST=test; path=/; domain=.com
  47. None
  48. None
  49. None
  50. None
  51. Streaming upload/download # Chunked upload with Patron client = Patron::Session.new

    res = client.request(:post, url, {}, :file => path_to_upload) # Chunked download client.get_file(url, path_to_write) # Chunked upload with em-http-request req = EM::HttpRequest.new(url).post :file => path_to_upload # Chunked download req = EM::HttpRequest.new(url).get req.stream do |chunk| p chunk end
  52. None
  53. None
  54. Basic HTTP features

  55. Development Support

  56. None
  57. Response stubbing # Stubbing response body client = HTTPClient.new client.test_loopback_response

    << 'Hello!' client.get('http://www.example.com/hello').body #=> "Hello!" # Stubbing HTTP response client.test_loopback_http_response << "HTTP/1.0 302 Found¥r¥nLocation: http://foo/¥r¥n¥r¥n" << "HTTP/1.0 200 OK¥r¥n¥r¥nHello!" client.post('http://www.example.com/todo', :follow_redirect => true, :body => '{}').body #=> "Hello!"
  58. None
  59. None
  60. IRB like shell rest-client, httpclient, wrest % restclient https://example.com user

    pass >> delete '/private/resource' % httpclient >> get "https://www.google.com", :q => :ruby % wrest >> 'http://www.google.com?q=ruby'.to_uri.get
  61. Replayable log rest-client % RESTCLIENT_LOG=/tmp/restclient.log restclient >> RestClient.get "https://www.google.com/" ...

    % cat /tmp/restclient.log RestClient.get "https://www.google.com/", "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate" # => 200 OK | text/html 13354 bytes
  62. None
  63. Development Support

  64. Advanced features

  65. None
  66. None
  67. None
  68. None
  69. HTML form handling of Mechanize agent = Mechanize.new page =

    agent.get(url) form = page.form('login') form.email = 'nahi@ruby-lang.org' form.password = 'jKH.P945wruV*qh3' page = agent.submit(form, form.button('submit'))
  70. Advanced features

  71. Testing your client webmock by Bartosz Blimke (bblimke) Library for

    stubbing and setting expectations on HTTP requests in Ruby. vcr by Myron Marston (myronmarston) Record your test suite's HTTP interactions and replay them during future test runs for fast, deterministic, accurate tests.
  72. Performance Comparisons Server Linode Xen VPS (Linode 512) at Fremont,

    CA Ubuntu 10.10 Apache 2.2, KeepAlive On Client AWS EC2 (m1.small) at North Virginia (us-east-1b) Ubuntu 12.04 HTTP clients w/ CRuby 1.9.3p286 Multiple downloads of 177B.html and 24MB.zip Don't take it serious! http://bit.ly/RubyHTTPClientsBenchmarkScript
  73. Multiple 177B downloads [sec]

  74. [sec]

  75. [sec]

  76. My Recommendations • Speed is the king => em-http-request, curb

    w/ multi • HTML operation, Cookies => Mechanize • API client => Faraday and adapter based impls • SSL, Connectivity => httpclient Check the matrix before you use the libraries Please let me know when you find incorrect cell
  77. Development Timeline