Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Working with HTTP in Ruby: Tips, Tricks, and Te...
Search
Andrey Deryabin
November 23, 2017
Technology
2
600
Working with HTTP in Ruby: Tips, Tricks, and Techniques.
Slides from "Fall Saint P Ruby Meetup" meetup (23.11.2017)
Andrey Deryabin
November 23, 2017
Tweet
Share
More Decks by Andrey Deryabin
See All by Andrey Deryabin
Working with HTTP in Ruby: Tips, Tricks, and Techniques.
aderyabin
0
31
Микросервисы. Мифы и суровая реальность
aderyabin
0
60
Microservices in Gett
aderyabin
1
380
Rove
aderyabin
0
140
Manage environment with Vagrant
aderyabin
0
78
Other Decks in Technology
See All in Technology
GeminiとNotebookLMによる金融実務の業務革新
abenben
0
240
監視のこれまでとこれから/sakura monitoring seminar 2025
fujiwara3
11
4k
AIとともに進化するエンジニアリング / Engineering-Evolving-with-AI_final.pdf
lycorptech_jp
PRO
0
110
より良いプロダクトの開発を目指して - 情報を中心としたプロダクト開発 #phpcon #phpcon2025
bengo4com
1
3.1k
Witchcraft for Memory
pocke
1
470
米国国防総省のDevSecOpsライフサイクルをAWSのセキュリティサービスとOSSで実現
syoshie
2
1.2k
BrainPadプログラミングコンテスト記念LT会2025_社内イベント&問題解説
brainpadpr
1
170
Lambda Web Adapterについて自分なりに理解してみた
smt7174
5
130
CI/CD/IaC 久々に0から環境を作ったらこうなりました
kaz29
1
190
フィンテック養成勉強会#54
finengine
0
180
Prox Industries株式会社 会社紹介資料
proxindustries
0
330
SalesforceArchitectGroupOsaka#20_CNX'25_Report
atomica7sei
0
200
Featured
See All Featured
The Pragmatic Product Professional
lauravandoore
35
6.7k
Code Reviewing Like a Champion
maltzj
524
40k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
5
230
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Facilitating Awesome Meetings
lara
54
6.4k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
20
1.3k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Visualization
eitanlees
146
16k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
The Straight Up "How To Draw Better" Workshop
denniskardys
234
140k
Automating Front-end Workflow
addyosmani
1370
200k
Transcript
HTTP in Ruby: Tips, Tricks and Techniques Andrey Deryabin
Andrey Deryabin
None
None
EVIL MARTIANS
EVIL MARTIANS
Промедление смерти подобно
HTTP Hypertext Transfer Protocol
History – HTTP/1.1 (v1) - 1999 (RFCs 2616, 7230, 7231,
7232, 7233, 7234 and 7235) – HTTP/2.0 -2015 (RFC 7540)
HTTP 1.1 – most popular protocol in WEB – a
stateless – request => response
HTTP/2 – Binary framing parsing and encoding – Query multiplexing
– Headers Compression – Connection and stream management – And more and more
curl -v https: //google.com > GET / HTTP/1.1 > Host:
google.com > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 302 Found < Cache-Control: private < Content-Type: text/html; charset=UTF-8 < Location: https: // www.google.es/?gfe_rd=cr&dcr=0&ei=XIINWr2XIuis8wfFipPwDQ < Content-Length: 269 < <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>302 Moved </TITLE> </HEAD><BODY> <H1>302 Moved </H1> The document has moved <A HREF="https: // www.google.es/? gfe_rd=cr&dcr=0&ei=XIINWr2XIuis8wfFipPwDQ">here </A>. </BODY> </HTML> Method Resource Protocol Request Response Body Headers
What is about ?
How many Ruby HTTP clients do you know?
Ruby HTTP Clients Net::HTTP RestClient Faraday Ethcon Curb Typhoeus HTTP.rb
HTTPClient HTTParty Patron EM-HTTP-Request Excon
Identical
Ruby HTTP Clients – Support GET/POST/PUT/DELETE methods – Follow redirect
– Proxy – HTTP Auth – Compression – etc
What are the differences?
Engine?
Ruby HTTP Clients Faraday Ethcon Curb Typhoeus Patron Libcurl EM-HTTP-Request
Excon Event machine HTTParty Net::HTTP RestClient HTTP.rb HTTPClient TCPSocket Net::HTTP
DSL?
What else?
Feature (Bugs)!!!
Net::HTTP # request uri = URI.parse('http: //localhost:4567/json') http = Net
::HTTP.new(uri.host, uri.port) request = Net ::HTTP ::Post.new(uri.request_uri, 'Content-Type' => 'text/json') request.body = { ‘first_name' => 'Leo', ‘last_name' => 'Messi' }.to_json http.request(request) # response [201, { "Сontent-length" => "7", 'Connection' => "Barca" }, "Created"]
Net::HTTP { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block",
"x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?
Net::HTTP # [201, { "Сontent-length" => "7", 'Connection' => "Barca"
}, "Created"] { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block", "x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?
RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1", Section 4.2,
"Message Headers" Each header field consists of a name followed by a colon (":") and the field value. Field names are case- insensitive
HTTPClient Documented? NO!
Typhoeus Typhoeus ::Request.new('localhost:4567/?lang=ruby&author=matz', method: :get).run
Typhoeus # request headers {"User-Agent" =>"Typhoeus - https: //github.com/typhoeus/typhoeus”} WHAT?
Typhoeus Default User-Agent?
Typhoeus
Typhoeus
Fail story 1
Groupon
# Simplifed ERB <%= Ipgeobase.lookup(current_ip).city %>
# https: //github.com/mokevnin/ipgeobase/blob/master/lib/ipgeobase.rb require 'uri' require 'open-uri' module Ipgeobase URL
= 'http: //ipgeobase.ru:7020/geo' autoload 'IpMetaData', 'ipgeobase/ip_meta_data' def self.lookup(ip, params = {}) uri = URI.parse(URL) uri.query = URI.encode_ www_form :ip => ip resp = open(uri, params).read() IpMetaData.parse(resp.to_s) end end
Use timeouts
Fail story 2
Useless box
TopSecretProjectMlApi ::AdaptiveFeed .perform({id:"15234", date: “2017-11-23"})
What is it? API call? What JSON do you send?
Epic thinking face
Control HTTP traffic
Command line tools
Tcpdump
GUI applications
Charles (Web debug proxy application)
Does it developer friendly?
puts "no"
Requirements – Log all requests – Analyse requests – Understand
what is under the hood
gem install sniffer
Sniffer supports most popular HTTP clients: Net::HTTP HTTP HTTPClient Patron
Curb Ethan Typhoeus
Disabled by default
Sniffer.enable!
Easy configuration
Sniffer.config do logger: Logger.new($stdout), severity: Logger ::Severity ::DEBUG, # HTTP
options to log log: { request_url: true, request_headers: true, request_body: true, request_method: true, response_status: true, response_headers: true, response_body: true, timing: true }, store: true, # save requests/responses to Sniffer.data enabled: false # Sniffer disabled by default end
Time to play
pry(main)> Sniffer.enable!; pry(main)> client = Elasticsearch ::Client.new; pry(main)> client.search q:
'Saint P Ruby'; D, [2017-11-21T21:13:25.120892 #87793] DEBUG -- : {"port": 9200,"host":"localhost","query":"/_search? q=Saint+P+Ruby","rq_content_type":"application/ json","rq_user_agent":"Faraday v0.9.2","rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0 .3","rq_accept":"*/ *","rq_connection":"close","method":"GET","request_body":"","status ":200,"rs_content_type":"application/json; charset=UTF-8","rs_content_length":"124","timing": 0.49408299988135695,"response_body":"{\"took\": 224,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\": 5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null,\"hits\": []}}"} Example
{ "port":9200, "host":"localhost", "query":"/_search?q=Saint+P+Ruby", "rq_content_type":"application/json", "rq_user_agent":"Faraday v0.9.2", "rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "rq_accept":"*/*", "rq_connection":"close",
"method":"GET", "request_body":"", "status":200, "rs_content_type":"application/json; charset=UTF-8", "rs_content_length":"124", "timing":0.49408299988135695, "response_body":"{\"took\":224,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null, \"hits\":[]}}" } Logging
Sniffer.data.first.request => #<Sniffer ::DataItem ::Request:0x00007fbe880cd700 @body="", @headers= {"content-type" =>"application/json", "user-agent"
=>"Faraday v0.11.0", "accept- encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept" =>"*/*", "connection" =>"close"}, @host="localhost", @method="GET", @port=9200, @query="/_search?q=Saint+P+Ruby"> Analyze request
Sniffer.data.first.request.headers => {"content-type" =>"application/json", "user-agent" =>"Faraday v0.9.2", "accept-encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept"
=>"*/*", “connection" =>"close"} Sniffer.data.first.request.method => "GET" Analyze request
Sniffer.data.first.response => #<Sniffer ::DataItem ::Response:0x00007fbe86078658 @body= "{\"took\":989,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\": 0,\"max_score\":null,\"hits\":[]}}", @headers=
{"content-type" =>"application/json; charset=UTF-8", "content-length" =>"124"}, @status=200, @timing=1.6817439999431372> Analyze response
Sniffer.data.first.response.body => “{\"took\":224,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\": 0,\"max_score\":null,\"hits\":[]}}" Sniffer.data.first.response.headers => {"content-type" =>"application/json; charset=UTF-8",
“content- length" =>"124"} Sniffer.data.first.response.timing => 0.49408299988135695 Analyze response
Log Management
require 'elasticsearch' require 'sniffer' Sniffer.config.logger = Logger.new('sniffer.log') Sniffer.enable! client =
Elasticsearch ::Client.new; client.cluster.health client.search q: 'Saint P Ruby'; sudo remote_syslog -p 300 \n -d logs.papertrailapp.com \n sniffer.log
None
require 'elasticsearch' require 'sniffer' require 'le' Sniffer.config.logger = Le.new('<token>') Sniffer.enable!
client = Elasticsearch ::Client.new; client.cluster.health client.search q: 'Saint P Ruby';
None
Resume
Read a source code
Write open source
Use Sniffer!
None
Thanks! aderyabin @aderyabin @evilmartians evilmartians.com
[email protected]