Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Working with HTTP in Ruby: Tips, Tricks, and Techniques.
Search
Andrey Deryabin
November 23, 2017
Technology
2
410
Working with HTTP in Ruby: Tips, Tricks, and Techniques.
Slides from "Fall Saint P Ruby Meetup" meetup (23.11.2017)
Andrey Deryabin
November 23, 2017
Tweet
Share
More Decks by Andrey Deryabin
See All by Andrey Deryabin
Working with HTTP in Ruby: Tips, Tricks, and Techniques.
aderyabin
0
25
Микросервисы. Мифы и суровая реальность
aderyabin
0
51
Microservices in Gett
aderyabin
1
370
Rove
aderyabin
0
110
Manage environment with Vagrant
aderyabin
0
75
Other Decks in Technology
See All in Technology
[NIKKEI Tech Talk] KDDI/KAG Scrum & Community for Engineering Training
curanosuke
2
220
How to Think Like a Performance Engineer
csswizardry
4
590
AWS IAMのアンチパターン/AWSが考える最低権限実現へのアプローチ概略(JAWS-UG朝会#59資料改修20分版)
htan
0
330
セキュリティ研修 Day1【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
160
公共領域から学ぶ クラウド移行についてエンジニアが意識していること
kawakawa2222
0
140
データベース研修 DB基礎【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
210
Git 研修 Basic【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
310
楽しくGoを学び合う、LayerXの勉強会文化 / LayerX's study culture of having fun and learning Go together
ar_tama
2
350
Docker互換のセキュアなコンテナ実行環境「Podman」超入門
devops_vtj
6
3.2k
RAGのサービスをリリースして1年3ヶ月が経ちました
segavvy
4
950
[I/O Extended Android 2024] What`s new in Android 2024
kyeongwan
0
220
ペパボのオブザーバビリティ研修2024 説明資料
kesompochy
0
1.1k
Featured
See All Featured
The Brand Is Dead. Long Live the Brand.
mthomps
52
36k
Ruby is Unlike a Banana
tanoku
96
10k
Fontdeck: Realign not Redesign
paulrobertlloyd
79
5.1k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
18
1.2k
Statistics for Hackers
jakevdp
792
220k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
155
14k
It's Worth the Effort
3n
181
27k
BBQ
matthewcrist
82
9k
For a Future-Friendly Web
brad_frost
173
9.2k
Adopting Sorbet at Scale
ufuk
71
8.8k
Designing the Hi-DPI Web
ddemaree
276
34k
WebSockets: Embracing the real-time Web
robhawkes
59
7.2k
Transcript
HTTP in Ruby: Tips, Tricks and Techniques Andrey Deryabin
Andrey Deryabin
None
None
EVIL MARTIANS
EVIL MARTIANS
Промедление смерти подобно
HTTP Hypertext Transfer Protocol
History – HTTP/1.1 (v1) - 1999 (RFCs 2616, 7230, 7231,
7232, 7233, 7234 and 7235) – HTTP/2.0 -2015 (RFC 7540)
HTTP 1.1 – most popular protocol in WEB – a
stateless – request => response
HTTP/2 – Binary framing parsing and encoding – Query multiplexing
– Headers Compression – Connection and stream management – And more and more
curl -v https: //google.com > GET / HTTP/1.1 > Host:
google.com > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 302 Found < Cache-Control: private < Content-Type: text/html; charset=UTF-8 < Location: https: // www.google.es/?gfe_rd=cr&dcr=0&ei=XIINWr2XIuis8wfFipPwDQ < Content-Length: 269 < <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>302 Moved </TITLE> </HEAD><BODY> <H1>302 Moved </H1> The document has moved <A HREF="https: // www.google.es/? gfe_rd=cr&dcr=0&ei=XIINWr2XIuis8wfFipPwDQ">here </A>. </BODY> </HTML> Method Resource Protocol Request Response Body Headers
What is about ?
How many Ruby HTTP clients do you know?
Ruby HTTP Clients Net::HTTP RestClient Faraday Ethcon Curb Typhoeus HTTP.rb
HTTPClient HTTParty Patron EM-HTTP-Request Excon
Identical
Ruby HTTP Clients – Support GET/POST/PUT/DELETE methods – Follow redirect
– Proxy – HTTP Auth – Compression – etc
What are the differences?
Engine?
Ruby HTTP Clients Faraday Ethcon Curb Typhoeus Patron Libcurl EM-HTTP-Request
Excon Event machine HTTParty Net::HTTP RestClient HTTP.rb HTTPClient TCPSocket Net::HTTP
DSL?
What else?
Feature (Bugs)!!!
Net::HTTP # request uri = URI.parse('http: //localhost:4567/json') http = Net
::HTTP.new(uri.host, uri.port) request = Net ::HTTP ::Post.new(uri.request_uri, 'Content-Type' => 'text/json') request.body = { ‘first_name' => 'Leo', ‘last_name' => 'Messi' }.to_json http.request(request) # response [201, { "Сontent-length" => "7", 'Connection' => "Barca" }, "Created"]
Net::HTTP { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block",
"x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?
Net::HTTP # [201, { "Сontent-length" => "7", 'Connection' => "Barca"
}, "Created"] { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block", "x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?
RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1", Section 4.2,
"Message Headers" Each header field consists of a name followed by a colon (":") and the field value. Field names are case- insensitive
HTTPClient Documented? NO!
Typhoeus Typhoeus ::Request.new('localhost:4567/?lang=ruby&author=matz', method: :get).run
Typhoeus # request headers {"User-Agent" =>"Typhoeus - https: //github.com/typhoeus/typhoeus”} WHAT?
Typhoeus Default User-Agent?
Typhoeus
Typhoeus
Fail story 1
Groupon
# Simplifed ERB <%= Ipgeobase.lookup(current_ip).city %>
# https: //github.com/mokevnin/ipgeobase/blob/master/lib/ipgeobase.rb require 'uri' require 'open-uri' module Ipgeobase URL
= 'http: //ipgeobase.ru:7020/geo' autoload 'IpMetaData', 'ipgeobase/ip_meta_data' def self.lookup(ip, params = {}) uri = URI.parse(URL) uri.query = URI.encode_ www_form :ip => ip resp = open(uri, params).read() IpMetaData.parse(resp.to_s) end end
Use timeouts
Fail story 2
Useless box
TopSecretProjectMlApi ::AdaptiveFeed .perform({id:"15234", date: “2017-11-23"})
What is it? API call? What JSON do you send?
Epic thinking face
Control HTTP traffic
Command line tools
Tcpdump
GUI applications
Charles (Web debug proxy application)
Does it developer friendly?
puts "no"
Requirements – Log all requests – Analyse requests – Understand
what is under the hood
gem install sniffer
Sniffer supports most popular HTTP clients: Net::HTTP HTTP HTTPClient Patron
Curb Ethan Typhoeus
Disabled by default
Sniffer.enable!
Easy configuration
Sniffer.config do logger: Logger.new($stdout), severity: Logger ::Severity ::DEBUG, # HTTP
options to log log: { request_url: true, request_headers: true, request_body: true, request_method: true, response_status: true, response_headers: true, response_body: true, timing: true }, store: true, # save requests/responses to Sniffer.data enabled: false # Sniffer disabled by default end
Time to play
pry(main)> Sniffer.enable!; pry(main)> client = Elasticsearch ::Client.new; pry(main)> client.search q:
'Saint P Ruby'; D, [2017-11-21T21:13:25.120892 #87793] DEBUG -- : {"port": 9200,"host":"localhost","query":"/_search? q=Saint+P+Ruby","rq_content_type":"application/ json","rq_user_agent":"Faraday v0.9.2","rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0 .3","rq_accept":"*/ *","rq_connection":"close","method":"GET","request_body":"","status ":200,"rs_content_type":"application/json; charset=UTF-8","rs_content_length":"124","timing": 0.49408299988135695,"response_body":"{\"took\": 224,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\": 5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null,\"hits\": []}}"} Example
{ "port":9200, "host":"localhost", "query":"/_search?q=Saint+P+Ruby", "rq_content_type":"application/json", "rq_user_agent":"Faraday v0.9.2", "rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "rq_accept":"*/*", "rq_connection":"close",
"method":"GET", "request_body":"", "status":200, "rs_content_type":"application/json; charset=UTF-8", "rs_content_length":"124", "timing":0.49408299988135695, "response_body":"{\"took\":224,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null, \"hits\":[]}}" } Logging
Sniffer.data.first.request => #<Sniffer ::DataItem ::Request:0x00007fbe880cd700 @body="", @headers= {"content-type" =>"application/json", "user-agent"
=>"Faraday v0.11.0", "accept- encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept" =>"*/*", "connection" =>"close"}, @host="localhost", @method="GET", @port=9200, @query="/_search?q=Saint+P+Ruby"> Analyze request
Sniffer.data.first.request.headers => {"content-type" =>"application/json", "user-agent" =>"Faraday v0.9.2", "accept-encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept"
=>"*/*", “connection" =>"close"} Sniffer.data.first.request.method => "GET" Analyze request
Sniffer.data.first.response => #<Sniffer ::DataItem ::Response:0x00007fbe86078658 @body= "{\"took\":989,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\": 0,\"max_score\":null,\"hits\":[]}}", @headers=
{"content-type" =>"application/json; charset=UTF-8", "content-length" =>"124"}, @status=200, @timing=1.6817439999431372> Analyze response
Sniffer.data.first.response.body => “{\"took\":224,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\": 0,\"max_score\":null,\"hits\":[]}}" Sniffer.data.first.response.headers => {"content-type" =>"application/json; charset=UTF-8",
“content- length" =>"124"} Sniffer.data.first.response.timing => 0.49408299988135695 Analyze response
Log Management
require 'elasticsearch' require 'sniffer' Sniffer.config.logger = Logger.new('sniffer.log') Sniffer.enable! client =
Elasticsearch ::Client.new; client.cluster.health client.search q: 'Saint P Ruby'; sudo remote_syslog -p 300 \n -d logs.papertrailapp.com \n sniffer.log
None
require 'elasticsearch' require 'sniffer' require 'le' Sniffer.config.logger = Le.new('<token>') Sniffer.enable!
client = Elasticsearch ::Client.new; client.cluster.health client.search q: 'Saint P Ruby';
None
Resume
Read a source code
Write open source
Use Sniffer!
None
Thanks! aderyabin @aderyabin @evilmartians evilmartians.com
[email protected]