Working with HTTP in Ruby: Tips, Tricks, and Techniques.

Slide 1

Slide 1 text

HTTP in Ruby: Tips, Tricks and Techniques Andrey Deryabin

Slide 2

Slide 2 text

Andrey Deryabin

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

EVIL MARTIANS

Slide 6

Slide 6 text

EVIL MARTIANS

Slide 7

Slide 7 text

Промедление смерти подобно

Slide 8

Slide 8 text

HTTP Hypertext Transfer Protocol

Slide 9

Slide 9 text

History – HTTP/1.1 (v1) - 1999 (RFCs 2616, 7230, 7231, 7232, 7233, 7234 and 7235) – HTTP/2.0 -2015 (RFC 7540)

Slide 10

Slide 10 text

HTTP 1.1 – most popular protocol in WEB – a stateless – request => response

Slide 11

Slide 11 text

HTTP/2 – Binary framing parsing and encoding – Query multiplexing – Headers Compression – Connection and stream management – And more and more

Slide 12

Slide 12 text

curl -v https: //google.com > GET / HTTP/1.1 > Host: google.com > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 302 Found < Cache-Control: private < Content-Type: text/html; charset=UTF-8 < Location: https: // www.google.es/?gfe_rd=cr&dcr=0&ei=XIINWr2XIuis8wfFipPwDQ < Content-Length: 269 < 302 Moved

302 Moved

The document has moved here . Method Resource Protocol Request Response Body Headers

Slide 13

Slide 13 text

What is about ?

Slide 14

Slide 14 text

How many Ruby HTTP clients do you know?

Slide 15

Slide 15 text

Ruby HTTP Clients Net::HTTP RestClient Faraday Ethcon Curb Typhoeus HTTP.rb HTTPClient HTTParty Patron EM-HTTP-Request Excon

Slide 16

Slide 16 text

Identical

Slide 17

Slide 17 text

Ruby HTTP Clients – Support GET/POST/PUT/DELETE methods – Follow redirect – Proxy – HTTP Auth – Compression – etc

Slide 18

Slide 18 text

What are the differences?

Slide 19

Slide 19 text

Engine?

Slide 20

Slide 20 text

Ruby HTTP Clients Faraday Ethcon Curb Typhoeus Patron Libcurl EM-HTTP-Request Excon Event machine HTTParty Net::HTTP RestClient HTTP.rb HTTPClient TCPSocket Net::HTTP

Slide 21

Slide 21 text

DSL?

Slide 22

Slide 22 text

What else?

Slide 23

Slide 23 text

Feature (Bugs)!!!

Slide 24

Slide 24 text

Net::HTTP # request uri = URI.parse('http: //localhost:4567/json') http = Net ::HTTP.new(uri.host, uri.port) request = Net ::HTTP ::Post.new(uri.request_uri, 'Content-Type' => 'text/json') request.body = { ‘first_name' => 'Leo', ‘last_name' => 'Messi' }.to_json http.request(request) # response [201, { "Сontent-length" => "7", 'Connection' => "Barca" }, "Created"]

Slide 25

Slide 25 text

Net::HTTP { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block", "x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?

Slide 26

Slide 26 text

Net::HTTP # [201, { "Сontent-length" => "7", 'Connection' => "Barca" }, "Created"] { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block", "x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?

Slide 27

Slide 27 text

RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1", Section 4.2, "Message Headers" Each header ﬁeld consists of a name followed by a colon (":") and the ﬁeld value. Field names are case- insensitive

Slide 28

Slide 28 text

HTTPClient Documented? NO!

Slide 29

Slide 29 text

Typhoeus Typhoeus ::Request.new('localhost:4567/?lang=ruby&author=matz', method: :get).run

Slide 30

Slide 30 text

Typhoeus # request headers {"User-Agent" =>"Typhoeus - https: //github.com/typhoeus/typhoeus”} WHAT?

Slide 31

Slide 31 text

Typhoeus Default User-Agent?

Slide 32

Slide 32 text

Typhoeus

Slide 33

Slide 33 text

Typhoeus

Slide 34

Slide 34 text

Fail story 1

Slide 35

Slide 35 text

Groupon

Slide 36

Slide 36 text

# Simplifed ERB <%= Ipgeobase.lookup(current_ip).city %>

Slide 37

Slide 37 text

# https: //github.com/mokevnin/ipgeobase/blob/master/lib/ipgeobase.rb require 'uri' require 'open-uri' module Ipgeobase URL = 'http: //ipgeobase.ru:7020/geo' autoload 'IpMetaData', 'ipgeobase/ip_meta_data' def self.lookup(ip, params = {}) uri = URI.parse(URL) uri.query = URI.encode_ www_form :ip => ip resp = open(uri, params).read() IpMetaData.parse(resp.to_s) end end

Slide 38

Slide 38 text

Use timeouts

Slide 39

Slide 39 text

Fail story 2

Slide 40

Slide 40 text

Useless box

Slide 41

Slide 41 text

TopSecretProjectMlApi ::AdaptiveFeed .perform({id:"15234", date: “2017-11-23"})

Slide 42

Slide 42 text

What is it? API call? What JSON do you send?

Slide 43

Slide 43 text

Epic thinking face

Slide 44

Slide 44 text

Control HTTP trafﬁc

Slide 45

Slide 45 text

Command line tools

Slide 46

Slide 46 text

Tcpdump

Slide 47

Slide 47 text

GUI applications

Slide 48

Slide 48 text

Charles (Web debug proxy application)

Slide 49

Slide 49 text

Does it developer friendly?

Slide 50

Slide 50 text

puts "no"

Slide 51

Slide 51 text

Requirements – Log all requests – Analyse requests – Understand what is under the hood

Slide 52

Slide 52 text

gem install sniffer

Slide 53

Slide 53 text

Sniffer supports most popular HTTP clients: Net::HTTP HTTP HTTPClient Patron Curb Ethan Typhoeus

Slide 54

Slide 54 text

Disabled by default

Slide 55

Slide 55 text

Sniffer.enable!

Slide 56

Slide 56 text

Easy conﬁguration

Slide 57

Slide 57 text

Sniffer.config do logger: Logger.new($stdout), severity: Logger ::Severity ::DEBUG, # HTTP options to log log: { request_url: true, request_headers: true, request_body: true, request_method: true, response_status: true, response_headers: true, response_body: true, timing: true }, store: true, # save requests/responses to Sniffer.data enabled: false # Sniffer disabled by default end

Slide 58

Slide 58 text

Time to play

Slide 59

Slide 59 text

pry(main)> Sniffer.enable!; pry(main)> client = Elasticsearch ::Client.new; pry(main)> client.search q: 'Saint P Ruby'; D, [2017-11-21T21:13:25.120892 #87793] DEBUG -- : {"port": 9200,"host":"localhost","query":"/_search? q=Saint+P+Ruby","rq_content_type":"application/ json","rq_user_agent":"Faraday v0.9.2","rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0 .3","rq_accept":"*/ *","rq_connection":"close","method":"GET","request_body":"","status ":200,"rs_content_type":"application/json; charset=UTF-8","rs_content_length":"124","timing": 0.49408299988135695,"response_body":"{\"took\": 224,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\": 5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null,\"hits\": []}}"} Example

Slide 60

Slide 60 text

{ "port":9200, "host":"localhost", "query":"/_search?q=Saint+P+Ruby", "rq_content_type":"application/json", "rq_user_agent":"Faraday v0.9.2", "rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "rq_accept":"*/*", "rq_connection":"close", "method":"GET", "request_body":"", "status":200, "rs_content_type":"application/json; charset=UTF-8", "rs_content_length":"124", "timing":0.49408299988135695, "response_body":"{\"took\":224,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null, \"hits\":[]}}" } Logging

Slide 61

Slide 61 text

Sniffer.data.first.request => #"application/json", "user-agent" =>"Faraday v0.11.0", "accept- encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept" =>"*/*", "connection" =>"close"}, @host="localhost", @method="GET", @port=9200, @query="/_search?q=Saint+P+Ruby"> Analyze request

Slide 62

Slide 62 text

Sniffer.data.first.request.headers => {"content-type" =>"application/json", "user-agent" =>"Faraday v0.9.2", "accept-encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept" =>"*/*", “connection" =>"close"} Sniffer.data.first.request.method => "GET" Analyze request

Slide 63

Slide 63 text

Sniffer.data.first.response => #"application/json; charset=UTF-8", "content-length" =>"124"}, @status=200, @timing=1.6817439999431372> Analyze response

Slide 64

Slide 64 text

Sniffer.data.first.response.body => “{\"took\":224,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\": 0,\"max_score\":null,\"hits\":[]}}" Sniffer.data.first.response.headers => {"content-type" =>"application/json; charset=UTF-8", “content- length" =>"124"} Sniffer.data.first.response.timing => 0.49408299988135695 Analyze response

Slide 65

Slide 65 text

Log Management

Slide 66

Slide 66 text

require 'elasticsearch' require 'sniffer' Sniffer.config.logger = Logger.new('sniffer.log') Sniffer.enable! client = Elasticsearch ::Client.new; client.cluster.health client.search q: 'Saint P Ruby'; sudo remote_syslog -p 300 \n -d logs.papertrailapp.com \n sniffer.log

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

require 'elasticsearch' require 'sniffer' require 'le' Sniffer.config.logger = Le.new('') Sniffer.enable! client = Elasticsearch ::Client.new; client.cluster.health client.search q: 'Saint P Ruby';