Mind the gap - filling holes in other people's SaaS with MongoDB

Transcript

1. Mind the gap filling holes in other people's SaaS with Mongo
2. { Movember
3. IE
4. BULLETPROOF
5. BULLETPROOF I work here!
6. I'm Lindsay, @auxesis
7. Engineering Manager @ BULLETPROOF
8. Movember IE BULLETPROOF
9. PHP + Zend + MySQL + Redis + Memcache + Sphinx + Nginx + apache
10. IBM + HDS + VMware + Cisco + F5 + Linux + Windows
11. 2011 Campaign
12. Full stack Performance issues
13. DAtabase Performance Incorrect mounts Noisy neighbours MAXING OUT IOPS SAN FIRMWARE BUG
14. NGINX CACHing Proxy BEHAVIOUR
15. DDOS
17. 2012 Campaign
18. Started planning in December 2011
19. Resolutions:
20. More testing
21. More Benchmarks (automated where possible)
22. Benchmark driven development
24. What tools are available?
25. Load impact
26. Browser Mob (now Neustar Load Testing)
27. Blitz
28. Cheap + Flexible pricing
29. Runs on AWS
30. Great documentation
31. website
32. Browser plugin
33. API clients Ruby, Python, Java, Node, Perl
34. Sign up @ http://blitz.io/
35. $ gem install blitz $ blitz help Usage: blitz <command> <options> help - Display this help account:about - Show information about your account api:init - Validate and login with your API key couch:fuzz - Auto generate blitz tests from CouchDB curl - Run a sprint or a rush curl:help - Show help on sprint and rushing traceroute - Run traceroute remotely traceroute:help - Show help on traceroute version - Show the version of this Ruby gem $ blitz api:init # writes credentials to ~/.blitz/credentials
36. $ blitz curl:help Usage: blitz curl <options> <url> --user-agent -A <string> User-Agent to send to server --cookie -b name=<string> Cookie to send to the server (multiple) --data -d <string> Data to send in a PUT or POST request --dump-header -D <file> Print the request/response headers --referer -e <string> Referer URL --help -h Help on command line options --header -H <string> Custom header to pass to server --pattern -p <s>-<e>:<d> Ramp from s to e concurrent requests in d secs --region -r <string> california|oregon|virginia|singapore|ireland| japan --status -s <number> Assert on the HTTP response status code --timeout -T <ms> Wait time for both connect and responses --user -u <user[:pass]> User and password for authentication --request -X <string> Request method to use (GET, HEAD, PUT, etc.) --variable -v <string> Define a variable to use --verbose -V Print the request/response headers --tlsv1 -1 Use TLSv1 (SSL) --sslv2 -2 Use SSLv2 (SSL) --sslv3 -3 Use SSLv3 (SSL)
38. $ blitz curl --region australia --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/
39. $ blitz curl --region australia --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/ user growth / time
40. $ blitz curl --region australia --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/ user growth / time for AWS lag
41. $ blitz curl --region australia --pattern 1-250:20 \ --timeout 10000 --verbose http://example.org/ rushing from australia... Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27 user growth / time for AWS lag
42. --pattern 1-250:20
43. --pattern 1-250:20 blitz calculates the growth automatically
45. Consistent User experience
46. Consistent User experience
47. Consistent User experience CLI
48. Consistent User experience CLI website
49. Consistent User experience CLI website browser plugin
50. Consistent User experience CLI website browser plugin API clients
51. Consistent User experience CLI website browser plugin API clients benchmark results
52. benchmark results
53. benchmark results
54. benchmark results not stored :-(
55. comparative analysis
57. • Make change
58. • Make change • Run Benchmark
59. • Make change • Run Benchmark • Analyse results
60. • Make change • Run Benchmark • Analyse results { scientific Method
61. “Are we getting slower over time?”
62. “When was the last benchmark run?”
64. Can we poke at the Results? CLI website browser plugin API clients
65. Let's emulate `blitz curl`
66. #!/usr/bin/env ruby # load_test.rb require 'blitz' arguments = "--region australia --pattern 1-250:60" url = "http://example.org/" command = "#{arguments} #{url}" job = Blitz::Curl.parse(command) curl = Blitz::Command::Curl.new curl.rush(job)
67. # ruby load_test.rb rushing from australia... Time Users Response Hits Timeouts Errors Hits/s Mbps 2.5s 31 -1.000s 0 0 0 8.3s 103 1.075s 146 0 9 50.53 0.88 11.2s 140 0.406s 376 0 9 79.64 1.41 14.1s 176 0.409s 682 0 9 105.24 1.86 22.8s 0 0.417s 1552 0 9 15.57 0.27
68. #!/usr/bin/env ruby # load_test.rb require 'blitz' arguments = "--region australia --pattern 1-250:60" url = "http://example.org/" command = "#{arguments} #{url}" job = Blitz::Curl.parse(command) curl = Blitz::Command::Curl.new curl.rush(job)
69. #!/usr/bin/env ruby # load_test.rb require 'blitz' arguments = "--region australia --pattern 1-250:60" url = "http://example.org/" command = "#{arguments} #{url}" job = Blitz::Curl.parse(command) curl = Blitz::Command::Curl.new curl.rush(job)
70. Blitz::Curl.parse(command) returns Blitz::Curl::Rush
71. #!/usr/bin/env ruby # load_test.rb require 'blitz' arguments = "--region australia --pattern 1-250:60" url = "http://example.org/" command = "#{arguments} #{url}" job = Blitz::Curl.parse(command) curl = Blitz::Command::Curl.new curl.rush(job)
72. Blitz::Command::Curl#rush
73. Blitz::Command::Curl#rush initiates the test + outputs the test results
74. #!/usr/bin/env ruby # load_test.rb require 'blitz' arguments = "--region australia --pattern 1-250:60" url = "http://example.org/" command = "#{arguments} #{url}" job = Blitz::Curl.parse(command) curl = Blitz::Command::Curl.new curl.rush(job) puts job.result.timeline.to_json
75. Produces muck
76. duck punch our way to success
77. class Blitz::Curl::Rush class Point def to_json(*args) { 'timestamp' => @timestamp, 'duration' => @duration, 'total' => @total, 'hits' => @hits, 'errors' => @errors, 'timeouts' => @timeouts, 'volume' => @volume, 'txbytes' => @txbytes, 'rxbytes' => @rxbytes, 'steps' => @steps, }.to_json end end # ...
78. # ... class Step def to_json(*args) { 'duration' => @duration, 'connect' => @connect, 'errors' => @errors, 'timeouts' => @timeouts, 'asserts' => @asserts, }.to_json end end end
79. #!/usr/bin/env ruby # load_test.rb require 'blitz' arguments = "--region australia --pattern 1-250:60" url = "http://example.org/" command = "#{arguments} #{url}" job = Blitz::Curl.parse(command) curl = Blitz::Command::Curl.new curl.rush(job) puts job.result.timeline.to_json
80. {"start":1335694346,"finish":1335694367,"results":[{"timestamp": 2.502626,"duration":1.13912,"total":1,"hits":1,"errors":0,"timeouts": 0,"volume":2,"txbytes":258.0,"rxbytes":17810.0,"steps":[{"duration": 2.13912,"connect":0.260004,"errors":0,"timeouts":0,"asserts":0}]}, {"timestamp":5.016102,"duration":0.640047,"total":5,"hits":5,"errors": 0,"timeouts":0,"volume":4,"txbytes":1548.0,"rxbytes":90882.0,"steps": [{"duration":1.640047,"connect":0.158944,"errors":0,"timeouts": 0,"asserts":0}]},{"timestamp":7.524594,"duration":0.639987,"total": 10,"hits":10,"errors":0,"timeouts":0,"volume":5,"txbytes": 3354.0,"rxbytes":182222.0,"steps":[{"duration":1.639987,"connect": 0.158999,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp": 10.03388,"duration":0.63961,"total":20,"hits":20,"errors":0,"timeouts": 0,"volume":7,"txbytes":5418.0,"rxbytes":364902.0,"steps":[{"duration": 1.63961,"connect":0.158847,"errors":0,"timeouts":0,"asserts":0}]}, {"timestamp":12.543211,"duration":0.640421,"total":31,"hits": 31,"errors":0,"timeouts":0,"volume":9,"txbytes":9030.0,"rxbytes": 565850.0,"steps":[{"duration":1.640422,"connect":0.158767,"errors": 0,"timeouts":0,"asserts":0}]},{"timestamp":15.051705,"duration": 0.639894,"total":44,"hits":44,"errors":0,"timeouts":0,"volume": 10,"txbytes":12384.0,"rxbytes":803334.0,"steps":[{"duration": 1.639894,"connect":0.158794,"errors":0,"timeouts":0,"asserts":0}]}, {"timestamp":17.560136,"duration":-1.0,"total":44,"hits":44,"errors": 0,"timeouts":0,"volume":0,"txbytes":12384.0,"rxbytes":803334.0,"steps": [{"duration":0.0,"connect":0.0,"errors":0,"timeouts":0,"asserts":0}]}]}
81. {"start":1335694346,"finish":1335694367,"results":[{"timestamp": 2.502626,"duration":1.13912,"total":1,"hits":1,"errors":0,"timeouts": 0,"volume":2,"txbytes":258.0,"rxbytes":17810.0,"steps":[{"duration": 2.13912,"connect":0.260004,"errors":0,"timeouts":0,"asserts":0}]}, {"timestamp":5.016102,"duration":0.640047,"total":5,"hits":5,"errors": 0,"timeouts":0,"volume":4,"txbytes":1548.0,"rxbytes":90882.0,"steps": [{"duration":1.640047,"connect":0.158944,"errors":0,"timeouts": 0,"asserts":0}]},{"timestamp":7.524594,"duration":0.639987,"total": 10,"hits":10,"errors":0,"timeouts":0,"volume":5,"txbytes": 3354.0,"rxbytes":182222.0,"steps":[{"duration":1.639987,"connect": 0.158999,"errors":0,"timeouts":0,"asserts":0}]},{"timestamp": 10.03388,"duration":0.63961,"total":20,"hits":20,"errors":0,"timeouts": 0,"volume":7,"txbytes":5418.0,"rxbytes":364902.0,"steps":[{"duration": 1.63961,"connect":0.158847,"errors":0,"timeouts":0,"asserts":0}]}, {"timestamp":12.543211,"duration":0.640421,"total":31,"hits": 31,"errors":0,"timeouts":0,"volume":9,"txbytes":9030.0,"rxbytes": 565850.0,"steps":[{"duration":1.640422,"connect":0.158767,"errors": 0,"timeouts":0,"asserts":0}]},{"timestamp":15.051705,"duration": 0.639894,"total":44,"hits":44,"errors":0,"timeouts":0,"volume": 10,"txbytes":12384.0,"rxbytes":803334.0,"steps":[{"duration": 1.639894,"connect":0.158794,"errors":0,"timeouts":0,"asserts":0}]}, {"timestamp":17.560136,"duration":-1.0,"total":44,"hits":44,"errors": 0,"timeouts":0,"volume":0,"txbytes":12384.0,"rxbytes":803334.0,"steps": [{"duration":0.0,"connect":0.0,"errors":0,"timeouts":0,"asserts":0}]}]} protip!
82. cat json | jsonpp
83. { "start": 1335694346, "finish": 1335694367, "results": [ { "timestamp": 2.502626, "duration": 1.13912, "total": 1, "hits": 1, "errors": 0, "timeouts": 0, "volume": 2, "txbytes": 258.0, "rxbytes": 17810.0, "steps": [ { "duration": 2.13912, "connect": 0.260004, "errors": 0, "timeouts": 0, "asserts": 0 } ] } # ... ] }
84. Let's store it!
86. Capture.controllers :load_tests do post :index do parser = Yajl::Parser.new data = parser.parse(params[:data]) @load_test = LoadTest.create(data) @load_test.id end end
87. Capture.controllers :load_tests do post :index do parser = Yajl::Parser.new data = parser.parse(params[:data]) @load_test = LoadTest.create(data) @load_test.id end end
88. Capture.controllers :load_tests do post :index do parser = Yajl::Parser.new data = parser.parse(params[:data]) @load_test = LoadTest.create(data) @load_test.id end end
89. Where did LoadTest come from?
90. class LoadTest include MongoMapper::Document key :start, Time key :finish, Time key :user, String key :email, String key :login, String key :region, String key :environment, String has_many :results timestamps! end
91. class LoadTest include MongoMapper::Document key :start, Time key :finish, Time key :user, String key :email, String key :login, String key :region, String key :environment, String has_many :results timestamps! end 1-to-many relationship
92. class Result include MongoMapper::EmbeddedDocument key :timestamp, Float key :duration, Float key :total, Integer key :hits, Integer key :errors, Integer key :timeouts, Integer key :volume, Integer key :txbytes, Float key :rxbytes, Float has_many :steps end
93. class Result include MongoMapper::EmbeddedDocument key :timestamp, Float key :duration, Float key :total, Integer key :hits, Integer key :errors, Integer key :timeouts, Integer key :volume, Integer key :txbytes, Float key :rxbytes, Float has_many :steps end 1-to-many relationship
94. class Step include MongoMapper::EmbeddedDocument key :duration, Float key :connect, Float key :errors, Integer key :timeouts, Integer key :asserts, Integer end
95. class Step include MongoMapper::EmbeddedDocument key :duration, Float key :connect, Float key :errors, Integer key :timeouts, Integer key :asserts, Integer end what is this?
96. Embedded documents are almost identical to Documents with one exception: they are saved inside of another document instead of in their own collection. -- MongoMapper Documentation
97. class Order include MongoMapper::Document many :line_items timestamps! end class LineItem include MongoMapper::EmbeddedDocument key :name, String key :quantity, Integer end Order.create(:line_items => [ LineItem.new(:name => 'Undershirt', :quantity => 5), LineItem.new(:name => 'Underwear', :quantity => 5), LineItem.new(:name => 'Socks', :quantity => 3), ])
98. { "_id"=>BSON::ObjectId('4d39d708bcd1b368fc000004'), "created_at"=>Fri Jan 21 18:57:12 UTC 2011, "updated_at"=>Fri Jan 21 18:57:12 UTC 2011, "line_items"=> [ {"name"=>"Undershirt", "quantity"=>5, "_id"=>BSON::ObjectId('4d39d7fc000001')}, {"name"=>"Underwear", "quantity"=>5, "_id"=>BSON::ObjectId('4d39d78fc000002')}, {"name"=>"Socks", "quantity"=>3, "_id"=>BSON::ObjectId('4d39d708d1b8fc000003')} ] }
100. Clean abstraction
101. Capture.controllers :load_tests do post :index do parser = Yajl::Parser.new data = parser.parse(params[:data]) @load_test = LoadTest.create(data) @load_test.id end end
102. Creates
103. LoadTest # => has_many :results Result # => has_many :steps Steps
106. Let's make a UI!
109. Graphs with NVd3.js
110. Why Mongo?
111. We don't know the future data format
112. We could pull in data from other sources
113. It's really quick to build apps with
114. Don't accept limitations of SaaS you use
115. You can plug the gaps!
116. Thank you! Questions?
117. Thank you! Questions? Like the talk? Let @auxesis know!