Working with HTTP in Ruby: Tips, Tricks, and Techniques.

HTTP in Ruby: Tips, Tricks and Techniques Andrey Deryabin

Andrey Deryabin

EVIL MARTIANS

By Roman Shamin: https://twitter.com/romanshamin

¡Vamos allá!

HTTP Hypertext Transfer Protocol

History – HTTP/1.1 (v1) - 1999 (RFCs 2616, 7230, 7231,
7232, 7233, 7234 and 7235)

HTTP 1.1 – most popular protocol in WEB – a
stateless – request => response

curl -v https: //google.com > GET / HTTP/1.1 > Host:
google.com > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 302 Found < Cache-Control: private < Content-Type: text/html; charset=UTF-8 < Location: https: // www.google.es/?gfe_rd=cr&dcr=0&ei=XIINWr2XIuis8wfFipPwDQ < Content-Length: 269 < <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>302 Moved </TITLE> </HEAD><BODY> <H1>302 Moved </H1> The document has moved <A HREF="https: // www.google.es/? gfe_rd=cr&dcr=0&ei=XIINWr2XIuis8wfFipPwDQ">here </A>. </BODY> </HTML> Method Resource Protocol Request Response Body Headers

History – HTTP/1.1 (v1) - 1999 (RFCs 2616, 7230, 7231,
7232, 7233, 7234 and 7235) – HTTP/2.0 -2015 (RFC 7540)

HTTP/2 – Binary framing parsing and encoding – Query multiplexing
– Headers Compression – Connection and stream management – And more and more

HTTP Evolution http://crowdhailer.me/talks/2017-08-17/working-with-http2-in-elixir/slides.html#37

What is about ?

How many Ruby HTTP clients do you know?

Ruby HTTP Clients Net::HTTP RestClient Faraday Ethcon Curb Typhoeus HTTP.rb
HTTPClient HTTParty Patron EM-HTTP-Request Excon

What are the differences?

Engine?

Ruby HTTP Clients Faraday Ethcon Curb Typhoeus Patron Libcurl EM-HTTP-Request
Excon Event machine HTTParty Net::HTTP RestClient HTTP.rb HTTPClient TCPSocket Net::HTTP

Type of connection? – No Keep Alive – Keep Alive
– Pipelining

What else?

Feature (Bugs)!!!

Net::HTTP # request uri = URI.parse('http: //localhost:4567/json') http = Net
::HTTP.new(uri.host, uri.port) request = Net ::HTTP ::Post.new(uri.request_uri, 'Content-Type' => 'text/json') request.body = { ‘first_name' => 'Leo', ‘last_name' => 'Messi' }.to_json http.request(request) # response [201, { "Сontent-length" => "7", 'Connection' => "Barca" }, "Created"]

Net::HTTP { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block",
"x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?

Net::HTTP # [201, { "Сontent-length" => "7", 'Connection' => "Barca"
}, "Created"] { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block", "x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?

RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1", Section 4.2,
"Message Headers" Each header ﬁeld consists of a name followed by a colon (":") and the ﬁeld value. Field names are case- insensitive

HTTPClient Documented? NO!

Typhoeus Typhoeus ::Request.new('localhost:4567/?lang=ruby&author=matz', method: :get).run

Typhoeus # request headers {"User-Agent" =>"Typhoeus - https: //github.com/typhoeus/typhoeus”} WHAT?

Typhoeus Default User-Agent?

Typhoeus

Fail story 1

Groupon

# Simplifed ERB <%= Ipgeobase.lookup(current_ip).city %>

# https: //github.com/mokevnin/ipgeobase/blob/master/lib/ipgeobase.rb require 'uri' require 'open-uri' module Ipgeobase URL
= 'http: //ipgeobase.ru:7020/geo' autoload 'IpMetaData', 'ipgeobase/ip_meta_data' def self.lookup(ip, params = {}) uri = URI.parse(URL) uri.query = URI.encode_ www_form :ip => ip resp = open(uri, params).read() IpMetaData.parse(resp.to_s) end end

Use timeouts

Fail story 2

Useless box

TopSecretProjectMlApi ::AdaptiveFeed .perform({id:"15234", date: "2017-09-07"})

What is it? API call? What JSON do you send?

Epic thinking face

Control HTTP trafﬁc

Command line tools

Tcpdump

GUI applications

Charles (Web debug proxy application)

Developer friendly?

puts "no"

Sniffer – Log all requests – Debug requests – Understand
what is under the hood

gem install sniffer

Sniffer supports most popular HTTP clients: Net::HTTP HTTP HTTPClient Patron
Curb Ethan Typhoeus

Disabled by default

Sniffer.enable!

Easy conﬁguration

Sniffer.config do logger: Logger.new($stdout), severity: Logger ::Severity ::DEBUG, # HTTP
options to log log: { request_url: true, request_headers: true, request_body: true, request_method: true, response_status: true, response_headers: true, response_body: true, timing: true }, store: true, # save requests/responses to Sniffer.data enabled: false # Sniffer disabled by default end

Time to play

pry(main)> Sniffer.enable!; pry(main)> client = Elasticsearch ::Client.new; pry(main)> client.search q:
'Barcelona on Rails'; D, [2017-11-16T01:19:45.455444 #45358] DEBUG -- : {"port": 9200,"host":"localhost","query":"/_search? q=Barcelona+on+Rails","rq_content_type":"application/ json","rq_user_agent":"Faraday v0.11.0","rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity ;q=0.3","rq_accept":"*/ *","rq_connection":"close","method":"GET","request_body":"","sta tus":200,"rs_content_type":"application/json; charset=UTF-8","rs_content_length":"124","timing": 1.6817439999431372,"response_body":"{\"took\": 989,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\": 5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null, \"hits\":[]}}"} Example pry(main)> Sniffer.enable!; pry(main)> client = Elasticsearch ::Client.new; pry(main)> client.search q: 'Barcelona on Rails'; D, [2017-11-16T01:19:45.455444 #45358] DEBUG -- : {"port": 9200,"host":"localhost","query":"/_search? q=Barcelona+on+Rails","rq_content_type":"application/ json","rq_user_agent":"Faraday v0.11.0","rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity ;q=0.3","rq_accept":"*/ *","rq_connection":"close","method":"GET","request_body":"","sta tus":200,"rs_content_type":"application/json; charset=UTF-8","rs_content_length":"124","timing": 1.6817439999431372,"response_body":"{\"took\": 989,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\": 5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null, \"hits\":[]}}"}

{"port":9200,"host":"localhost","query":"/_search? q=Barcelona+on+Rails","rq_content_type":"application/ json","rq_user_agent":"Faraday v0.11.0","rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity ;q=0.3","rq_accept":"*/ *","rq_connection":"close","method":"GET","request_body":"","sta tus":200,"rs_content_type":"application/json; charset=UTF-8","rs_content_length":"124","timing": 1.6817439999431372,"response_body":"{\"took\": 989,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\":
5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null, \"hits\":[]}}"} Logging

Sniffer.data.first.request => #<Sniffer ::DataItem ::Request:0x00007fbe880cd700 @body="", @headers= {"content-type" =>"application/json", "user-agent"
=>"Faraday v0.11.0", "accept- encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept" =>"*/*", "connection" =>"close"}, @host="localhost", @method="GET", @port=9200, @query="/_search?q=Barcelona+on+Rails"> Debug request

Sniffer.data.first.response => #<Sniffer ::DataItem ::Response:0x00007fbe86078658 @body= "{\"took\":989,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\": 0,\"max_score\":null,\"hits\":[]}}", @headers=
{"content-type" =>"application/json; charset=UTF-8", "content-length" =>"124"}, @status=200, @timing=1.6817439999431372> Debug response

Resume

Read a source code

Write open source

Use Sniffer!

Thanks! aderyabin @aderyabin @evilmartians evilmartians.com [email protected]

Working with HTTP in Ruby: Tips, Tricks, and Techniques.

Working with HTTP in Ruby: Tips, Tricks, and Techniques.

More Decks by Andrey Deryabin

Other Decks in Programming

Featured

Transcript