HTTP in Ruby: Tips, Tricks and Techniques Andrey Deryabin 

Andrey Deryabin 

No content

EVIL MARTIANS 

EVIL MARTIANS 

No content

By Roman Shamin: https://twitter.com/romanshamin 

¡Vamos allá! 

HTTP Hypertext Transfer Protocol 

History – HTTP/1.1 (v1) - 1999 (RFCs 2616, 7230, 7231, 7232, 7233, 7234 and 7235) 

HTTP 1.1 – most popular protocol in WEB – a stateless – request => response 

curl -v https: //google.com > GET / HTTP/1.1 > Host: google.com > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 302 Found < Cache-Control: private < Content-Type: text/html; charset=UTF-8 < Location: https: // www.google.es/?gfe_rd=cr&dcr=0&ei=XIINWr2XIuis8wfFipPwDQ < Content-Length: 269 < 302 Moved

302 Moved

The document has moved here . Method Resource Protocol Request Response Body Headers 

History – HTTP/1.1 (v1) - 1999 (RFCs 2616, 7230, 7231, 7232, 7233, 7234 and 7235) – HTTP/2.0 -2015 (RFC 7540) 

HTTP/2 – Binary framing parsing and encoding – Query multiplexing – Headers Compression – Connection and stream management – And more and more 

HTTP Evolution http://crowdhailer.me/talks/2017-08-17/working-with-http2-in-elixir/slides.html#37 

What is about ? 

How many Ruby HTTP clients do you know? 

Ruby HTTP Clients Net::HTTP RestClient Faraday Ethcon Curb Typhoeus HTTP.rb HTTPClient HTTParty Patron EM-HTTP-Request Excon 

What are the differences? 

Engine? 

Ruby HTTP Clients Faraday Ethcon Curb Typhoeus Patron Libcurl EM-HTTP-Request Excon Event machine HTTParty Net::HTTP RestClient HTTP.rb HTTPClient TCPSocket Net::HTTP 

DSL? 

Type of connection? – No Keep Alive – Keep Alive – Pipelining 

What else? 

Feature (Bugs)!!! 

Net::HTTP # request uri = URI.parse('http: //localhost:4567/json') http = Net ::HTTP.new(uri.host, uri.port) request = Net ::HTTP ::Post.new(uri.request_uri, 'Content-Type' => 'text/json') request.body = { ‘first_name' => 'Leo', ‘last_name' => 'Messi' }.to_json http.request(request) # response [201, { "Сontent-length" => "7", 'Connection' => "Barca" }, "Created"] 

Net::HTTP { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block", "x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT? 

Net::HTTP # [201, { "Сontent-length" => "7", 'Connection' => "Barca" }, "Created"] { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block", "x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT? 

RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1", Section 4.2, "Message Headers" Each header field consists of a name followed by a colon (":") and the field value. Field names are case- insensitive 

HTTPClient Documented? NO! 

Typhoeus Typhoeus ::Request.new('localhost:4567/?lang=ruby&author=matz', method: :get).run 

Typhoeus # request headers {"User-Agent" =>"Typhoeus - https: //github.com/typhoeus/typhoeus”} WHAT? 

Typhoeus Default User-Agent? 

Typhoeus 

Typhoeus 

Fail story 1 

Groupon 

# Simplifed ERB <%= Ipgeobase.lookup(current_ip).city %> 

# https: //github.com/mokevnin/ipgeobase/blob/master/lib/ipgeobase.rb require 'uri' require 'open-uri' module Ipgeobase URL = 'http: //ipgeobase.ru:7020/geo' autoload 'IpMetaData', 'ipgeobase/ip_meta_data' def self.lookup(ip, params = {}) uri = URI.parse(URL) uri.query = URI.encode_ www_form :ip => ip resp = open(uri, params).read() IpMetaData.parse(resp.to_s) end end 

Use timeouts 

Fail story 2 

Useless box 

TopSecretProjectMlApi ::AdaptiveFeed .perform({id:"15234", date: "2017-09-07"}) 

What is it? API call? What JSON do you send? 

Epic thinking face 

Control HTTP traffic 

Command line tools 

Tcpdump 

GUI applications 

Charles (Web debug proxy application) 

Developer friendly? 

puts "no" 

No content

Sniffer – Log all requests – Debug requests – Understand what is under the hood 

gem install sniffer 

Sniffer supports most popular HTTP clients: Net::HTTP HTTP HTTPClient Patron Curb Ethan Typhoeus 

Disabled by default 

Sniffer.enable! 

Easy configuration 

Sniffer.config do logger: Logger.new($stdout), severity: Logger ::Severity ::DEBUG, # HTTP options to log log: { request_url: true, request_headers: true, request_body: true, request_method: true, response_status: true, response_headers: true, response_body: true, timing: true }, store: true, # save requests/responses to Sniffer.data enabled: false # Sniffer disabled by default end 

Time to play 

pry(main)> Sniffer.enable!; pry(main)> client = Elasticsearch ::Client.new; pry(main)> client.search q: 'Barcelona on Rails'; D, [2017-11-16T01:19:45.455444 #45358] DEBUG -- : {"port": 9200,"host":"localhost","query":"/_search? q=Barcelona+on+Rails","rq_content_type":"application/ json","rq_user_agent":"Faraday v0.11.0","rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity ;q=0.3","rq_accept":"*/ *","rq_connection":"close","method":"GET","request_body":"","sta tus":200,"rs_content_type":"application/json; charset=UTF-8","rs_content_length":"124","timing": 1.6817439999431372,"response_body":"{\"took\": 989,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\": 5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null, \"hits\":[]}}"} Example pry(main)> Sniffer.enable!; pry(main)> client = Elasticsearch ::Client.new; pry(main)> client.search q: 'Barcelona on Rails'; D, [2017-11-16T01:19:45.455444 #45358] DEBUG -- : {"port": 9200,"host":"localhost","query":"/_search? q=Barcelona+on+Rails","rq_content_type":"application/ json","rq_user_agent":"Faraday v0.11.0","rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity ;q=0.3","rq_accept":"*/ *","rq_connection":"close","method":"GET","request_body":"","sta tus":200,"rs_content_type":"application/json; charset=UTF-8","rs_content_length":"124","timing": 1.6817439999431372,"response_body":"{\"took\": 989,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\": 5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null, \"hits\":[]}}"} 

{"port":9200,"host":"localhost","query":"/_search? q=Barcelona+on+Rails","rq_content_type":"application/ json","rq_user_agent":"Faraday v0.11.0","rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity ;q=0.3","rq_accept":"*/ *","rq_connection":"close","method":"GET","request_body":"","sta tus":200,"rs_content_type":"application/json; charset=UTF-8","rs_content_length":"124","timing": 1.6817439999431372,"response_body":"{\"took\": 989,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\": 5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null, \"hits\":[]}}"} Logging 

Sniffer.data.first.request => #"application/json", "user-agent" =>"Faraday v0.11.0", "accept- encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept" =>"*/*", "connection" =>"close"}, @host="localhost", @method="GET", @port=9200, @query="/_search?q=Barcelona+on+Rails"> Debug request 

Sniffer.data.first.response => #"application/json; charset=UTF-8", "content-length" =>"124"}, @status=200, @timing=1.6817439999431372> Debug response 

No content

Resume 

Read a source code 

Write open source 

Use Sniffer! 

Thanks! aderyabin @aderyabin @evilmartians evilmartians.com [email protected]