Working with HTTP in Ruby: Tips, Tricks, and Techniques.

Slides from "Fall Saint P Ruby Meetup" meetup (23.11.2017)

Transcript

1. HTTP in Ruby: Tips, Tricks and Techniques Andrey Deryabin
2. Andrey Deryabin
5. EVIL MARTIANS
6. EVIL MARTIANS
7. Промедление смерти подобно
8. HTTP Hypertext Transfer Protocol
9. History – HTTP/1.1 (v1) - 1999 (RFCs 2616, 7230, 7231, 7232, 7233, 7234 and 7235) – HTTP/2.0 -2015 (RFC 7540)
10. HTTP 1.1 – most popular protocol in WEB – a stateless – request => response
11. HTTP/2 – Binary framing parsing and encoding – Query multiplexing – Headers Compression – Connection and stream management – And more and more
12. curl -v https: //google.com > GET / HTTP/1.1 > Host: google.com > User-Agent: curl/7.51.0 > Accept: */* > < HTTP/1.1 302 Found < Cache-Control: private < Content-Type: text/html; charset=UTF-8 < Location: https: // www.google.es/?gfe_rd=cr&dcr=0&ei=XIINWr2XIuis8wfFipPwDQ < Content-Length: 269 < <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8"> <TITLE>302 Moved </TITLE> </HEAD><BODY> <H1>302 Moved </H1> The document has moved <A HREF="https: // www.google.es/? gfe_rd=cr&amp;dcr=0&amp;ei=XIINWr2XIuis8wfFipPwDQ">here </A>. </BODY> </HTML> Method Resource Protocol Request Response Body Headers
13. What is about ?
14. How many Ruby HTTP clients do you know?
15. Ruby HTTP Clients Net::HTTP RestClient Faraday Ethcon Curb Typhoeus HTTP.rb HTTPClient HTTParty Patron EM-HTTP-Request Excon
16. Identical
17. Ruby HTTP Clients – Support GET/POST/PUT/DELETE methods – Follow redirect – Proxy – HTTP Auth – Compression – etc
18. What are the differences?
19. Engine?
20. Ruby HTTP Clients Faraday Ethcon Curb Typhoeus Patron Libcurl EM-HTTP-Request Excon Event machine HTTParty Net::HTTP RestClient HTTP.rb HTTPClient TCPSocket Net::HTTP
21. DSL?
22. What else?
23. Feature (Bugs)!!!
24. Net::HTTP # request uri = URI.parse('http: //localhost:4567/json') http = Net ::HTTP.new(uri.host, uri.port) request = Net ::HTTP ::Post.new(uri.request_uri, 'Content-Type' => 'text/json') request.body = { ‘first_name' => 'Leo', ‘last_name' => 'Messi' }.to_json http.request(request) # response [201, { "Сontent-length" => "7", 'Connection' => "Barca" }, "Created"]
25. Net::HTTP { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block", "x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?
26. Net::HTTP # [201, { "Сontent-length" => "7", 'Connection' => "Barca" }, "Created"] { "content-type" =>"text/html;charset=utf-8", "connection" =>"Barca, close", "x-xss-protection" =>"1; mode=block", "x-content-type-options" =>"nosniff", "x-frame-options" =>"SAMEORIGIN", "content-length" =>"7" } WHAT?
27. RFC 2616 - "Hypertext Transfer Protocol -- HTTP/1.1", Section 4.2, "Message Headers" Each header field consists of a name followed by a colon (":") and the field value. Field names are case- insensitive
28. HTTPClient Documented? NO!
29. Typhoeus Typhoeus ::Request.new('localhost:4567/?lang=ruby&author=matz', method: :get).run
30. Typhoeus # request headers {"User-Agent" =>"Typhoeus - https: //github.com/typhoeus/typhoeus”} WHAT?
31. Typhoeus Default User-Agent?
32. Typhoeus
33. Typhoeus
34. Fail story 1
35. Groupon
36. # Simplifed ERB <%= Ipgeobase.lookup(current_ip).city %>
37. # https: //github.com/mokevnin/ipgeobase/blob/master/lib/ipgeobase.rb require 'uri' require 'open-uri' module Ipgeobase URL = 'http: //ipgeobase.ru:7020/geo' autoload 'IpMetaData', 'ipgeobase/ip_meta_data' def self.lookup(ip, params = {}) uri = URI.parse(URL) uri.query = URI.encode_ www_form :ip => ip resp = open(uri, params).read() IpMetaData.parse(resp.to_s) end end
38. Use timeouts
39. Fail story 2
40. Useless box
41. TopSecretProjectMlApi ::AdaptiveFeed .perform({id:"15234", date: “2017-11-23"})
42. What is it? API call? What JSON do you send?
43. Epic thinking face
44. Control HTTP traffic
45. Command line tools
46. Tcpdump
47. GUI applications
48. Charles (Web debug proxy application)
49. Does it developer friendly?
50. puts "no"
51. Requirements – Log all requests – Analyse requests – Understand what is under the hood
52. gem install sniffer
53. Sniffer supports most popular HTTP clients: Net::HTTP HTTP HTTPClient Patron Curb Ethan Typhoeus
54. Disabled by default
55. Sniffer.enable!
56. Easy configuration
57. Sniffer.config do logger: Logger.new($stdout), severity: Logger ::Severity ::DEBUG, # HTTP options to log log: { request_url: true, request_headers: true, request_body: true, request_method: true, response_status: true, response_headers: true, response_body: true, timing: true }, store: true, # save requests/responses to Sniffer.data enabled: false # Sniffer disabled by default end
58. Time to play
59. pry(main)> Sniffer.enable!; pry(main)> client = Elasticsearch ::Client.new; pry(main)> client.search q: 'Saint P Ruby'; D, [2017-11-21T21:13:25.120892 #87793] DEBUG -- : {"port": 9200,"host":"localhost","query":"/_search? q=Saint+P+Ruby","rq_content_type":"application/ json","rq_user_agent":"Faraday v0.9.2","rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0 .3","rq_accept":"*/ *","rq_connection":"close","method":"GET","request_body":"","status ":200,"rs_content_type":"application/json; charset=UTF-8","rs_content_length":"124","timing": 0.49408299988135695,"response_body":"{\"took\": 224,\"timed_out\":false,\"_shards\":{\"total\":5,\"successful\": 5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null,\"hits\": []}}"} Example
60. { "port":9200, "host":"localhost", "query":"/_search?q=Saint+P+Ruby", "rq_content_type":"application/json", "rq_user_agent":"Faraday v0.9.2", "rq_accept_encoding":"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "rq_accept":"*/*", "rq_connection":"close", "method":"GET", "request_body":"", "status":200, "rs_content_type":"application/json; charset=UTF-8", "rs_content_length":"124", "timing":0.49408299988135695, "response_body":"{\"took\":224,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\":0,\"max_score\":null, \"hits\":[]}}" } Logging
61. Sniffer.data.first.request => #<Sniffer ::DataItem ::Request:0x00007fbe880cd700 @body="", @headers= {"content-type" =>"application/json", "user-agent" =>"Faraday v0.11.0", "accept- encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept" =>"*/*", "connection" =>"close"}, @host="localhost", @method="GET", @port=9200, @query="/_search?q=Saint+P+Ruby"> Analyze request
62. Sniffer.data.first.request.headers => {"content-type" =>"application/json", "user-agent" =>"Faraday v0.9.2", "accept-encoding" =>"gzip;q=1.0,deflate;q=0.6,identity;q=0.3", "accept" =>"*/*", “connection" =>"close"} Sniffer.data.first.request.method => "GET" Analyze request
63. Sniffer.data.first.response => #<Sniffer ::DataItem ::Response:0x00007fbe86078658 @body= "{\"took\":989,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\": 0,\"max_score\":null,\"hits\":[]}}", @headers= {"content-type" =>"application/json; charset=UTF-8", "content-length" =>"124"}, @status=200, @timing=1.6817439999431372> Analyze response
64. Sniffer.data.first.response.body => “{\"took\":224,\"timed_out\":false,\"_shards\":{\"total\": 5,\"successful\":5,\"failed\":0},\"hits\":{\"total\": 0,\"max_score\":null,\"hits\":[]}}" Sniffer.data.first.response.headers => {"content-type" =>"application/json; charset=UTF-8", “content- length" =>"124"} Sniffer.data.first.response.timing => 0.49408299988135695 Analyze response
65. Log Management
66. require 'elasticsearch' require 'sniffer' Sniffer.config.logger = Logger.new('sniffer.log') Sniffer.enable! client = Elasticsearch ::Client.new; client.cluster.health client.search q: 'Saint P Ruby'; sudo remote_syslog -p 300 \n -d logs.papertrailapp.com \n sniffer.log
68. require 'elasticsearch' require 'sniffer' require 'le' Sniffer.config.logger = Le.new('<token>') Sniffer.enable! client = Elasticsearch ::Client.new; client.cluster.health client.search q: 'Saint P Ruby';
70. Resume
71. Read a source code
72. Write open source
73. Use Sniffer!
75. Thanks! aderyabin @aderyabin @evilmartians evilmartians.com ad@evl.ms