Working with HTTP in Ruby: Tips, Tricks, and Techniques.

HTTP in Ruby: Tips, Tricks and Techniques Andrey Deryabin

Andrey Deryabin

EVIL MARTIANS

Промедление смерти подобно

HTTP Hypertext Transfer Protocol

History – HTTP/1.1 (v1) - 1999 (RFCs 2616, 7230, 7231,
7232, 7233, 7234 and 7235) – HTTP/2.0 -2015 (RFC 7540)

HTTP 1.1 – most popular protocol in WEB – a
stateless – request => response

HTTP/2 – Binary framing parsing and encoding – Query multiplexing
– Headers Compression – Connection and stream management – And more and more

curl -v https: //google.com > GET / HTTP/1.1 > Host:
google.com > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 302 Found < Cache-Control: private < Content-Type: text/html; charset=UTF-8 < Location: https: // www.google.es/?gfe_rd=cr&dcr=0&ei=XIINWr2XIuis8wfFipPwDQ < Content-Length: 269 < <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>302 Moved </TITLE> </HEAD><BODY> <H1>302 Moved </H1> The document has moved <A HREF="https: // www.google.es/? gfe_rd=cr&dcr=0&ei=XIINWr2XIuis8wfFipPwDQ">here </A>. </BODY> </HTML> Method Resource Protocol Request Response Body Headers

What is about ?

How many Ruby HTTP clients do you know?

Ruby HTTP Clients Net::HTTP RestClient Faraday Ethcon Curb Typhoeus HTTP.rb
HTTPClient HTTParty Patron EM-HTTP-Request Excon

Identical

Ruby HTTP Clients – Support GET/POST/PUT/DELETE methods – Follow redirect
– Proxy – HTTP Auth – Compression – etc

What are the differences?

Engine?

Ruby HTTP Clients Faraday Ethcon Curb Typhoeus Patron Libcurl EM-HTTP-Request
Excon Event machine HTTParty Net::HTTP RestClient HTTP.rb HTTPClient TCPSocket Net::HTTP

What else?

Feature (Bugs)!!!

Net::HTTP # request uri = URI.parse('http: //localhost:4567/json') http = Net
::HTTP.new(uri.host, uri.port) request = Net ::HTTP ::Post.new(uri.request_uri, 'Content-Type' => 'text/json') request.body = { ‘first_name' => 'Leo', ‘last_name' => 'Messi' }.to_json http.request(request) # response [201, { "Сontent-length" => "7", 'Connection' => "Barca" }, "Created"]

Net::HTTP { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block",
"x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?

Net::HTTP # [201, { "Сontent-length" => "7", 'Connection' => "Barca"
}, "Created"] { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block", "x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?

RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1", Section 4.2,
"Message Headers" Each header ﬁeld consists of a name followed by a colon (":") and the ﬁeld value. Field names are case- insensitive

HTTPClient Documented? NO!

Typhoeus Typhoeus ::Request.new('localhost:4567/?lang=ruby&author=matz', method: :get).run

Typhoeus # request headers {"User-Agent" =>"Typhoeus - https: //github.com/typhoeus/typhoeus”} WHAT?

Typhoeus Default User-Agent?

Typhoeus

Fail story 1

Groupon

# Simplifed ERB <%= Ipgeobase.lookup(current_ip).city %>

# https: //github.com/mokevnin/ipgeobase/blob/master/lib/ipgeobase.rb require 'uri' require 'open-uri' module Ipgeobase URL
= 'http: //ipgeobase.ru:7020/geo' autoload 'IpMetaData', 'ipgeobase/ip_meta_data' def self.lookup(ip, params = {}) uri = URI.parse(URL) uri.query = URI.encode_ www_form :ip => ip resp = open(uri, params).read() IpMetaData.parse(resp.to_s) end end

Use timeouts

Fail story 2

Useless box

TopSecretProjectMlApi ::AdaptiveFeed .perform({id:"15234", date: “2017-11-23"})

What is it? API call? What JSON do you send?

Epic thinking face

Control HTTP trafﬁc

Command line tools

Tcpdump

GUI applications

Charles (Web debug proxy application)

Does it developer friendly?

puts "no"

Requirements – Log all requests – Analyse requests – Understand
what is under the hood

gem install sniffer

Sniffer supports most popular HTTP clients: Net::HTTP HTTP HTTPClient Patron
Curb Ethan Typhoeus

Disabled by default

Sniffer.enable!

Easy conﬁguration

Sniffer.config do logger: Logger.new($stdout), severity: Logger ::Severity ::DEBUG, # HTTP
options to log log: { request_url: true, request_headers: true, request_body: true, request_method: true, response_status: true, response_headers: true, response_body: true, timing: true }, store: true, # save requests/responses to Sniffer.data enabled: false # Sniffer disabled by default end

Time to play

pry(main)> Sniffer.enable!; pry(main)> client = Elasticsearch ::Client.new; pry(main)> client.search q:
'Saint P Ruby'; D, [2017-11-21T21:13:25.120892 #87793] DEBUG -- : {"port": 9200,"host":"localhost","query":"/_search? q=Saint+P+Ruby","rq_content_type":"application/ json","rq_user_agent":"Faraday v0.9.2","rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0 .3","rq_accept":"*/ *","rq_connection":"close","method":"GET","request_body":"","status ":200,"rs_content_type":"application/json; charset=UTF-8","rs_content_length":"124","timing": 0.49408299988135695,"response_body":"{\"took\": 224,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\": 5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null,\"hits\": []}}"} Example

{ "port":9200, "host":"localhost", "query":"/_search?q=Saint+P+Ruby", "rq_content_type":"application/json", "rq_user_agent":"Faraday v0.9.2", "rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "rq_accept":"*/*", "rq_connection":"close",
"method":"GET", "request_body":"", "status":200, "rs_content_type":"application/json; charset=UTF-8", "rs_content_length":"124", "timing":0.49408299988135695, "response_body":"{\"took\":224,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null, \"hits\":[]}}" } Logging

Sniffer.data.first.request => #<Sniffer ::DataItem ::Request:0x00007fbe880cd700 @body="", @headers= {"content-type" =>"application/json", "user-agent"
=>"Faraday v0.11.0", "accept- encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept" =>"*/*", "connection" =>"close"}, @host="localhost", @method="GET", @port=9200, @query="/_search?q=Saint+P+Ruby"> Analyze request

Sniffer.data.first.request.headers => {"content-type" =>"application/json", "user-agent" =>"Faraday v0.9.2", "accept-encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept"
=>"*/*", “connection" =>"close"} Sniffer.data.first.request.method => "GET" Analyze request

Sniffer.data.first.response => #<Sniffer ::DataItem ::Response:0x00007fbe86078658 @body= "{\"took\":989,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\": 0,\"max_score\":null,\"hits\":[]}}", @headers=
{"content-type" =>"application/json; charset=UTF-8", "content-length" =>"124"}, @status=200, @timing=1.6817439999431372> Analyze response

Sniffer.data.first.response.body => “{\"took\":224,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\": 0,\"max_score\":null,\"hits\":[]}}" Sniffer.data.first.response.headers => {"content-type" =>"application/json; charset=UTF-8",
“content- length" =>"124"} Sniffer.data.first.response.timing => 0.49408299988135695 Analyze response

Log Management

require 'elasticsearch' require 'sniffer' Sniffer.config.logger = Logger.new('sniffer.log') Sniffer.enable! client =
Elasticsearch ::Client.new; client.cluster.health client.search q: 'Saint P Ruby'; sudo remote_syslog -p 300 \n -d logs.papertrailapp.com \n sniffer.log

require 'elasticsearch' require 'sniffer' require 'le' Sniffer.config.logger = Le.new('<token>') Sniffer.enable!
client = Elasticsearch ::Client.new; client.cluster.health client.search q: 'Saint P Ruby';

Resume

Read a source code

Write open source

Use Sniffer!

Thanks! aderyabin @aderyabin @evilmartians evilmartians.com [email protected]

Working with HTTP in Ruby: Tips, Tricks, and Te...

Working with HTTP in Ruby: Tips, Tricks, and Techniques.

More Decks by Andrey Deryabin

Other Decks in Technology

Featured

Transcript