Testing the Untestable

Given in 2014. Good tests are isolated, they’re repeatable, they’re deterministic. Good tests don’t touch the network and are flexible when it comes to change. Bad tests are all of the above and more. Bad tests are no tests at all: which is where I found myself with a 5 year legacy codebase running in production and touching millions of customers with minimal use-case documentation. We’ll cover this experience and several like it while digging into how to go from zero to total test coverage as painlessly as possible. You will learn how to stay sane in the face of insane testing conditions, and how to use these tests to deconstruct a monolith app. When life gives you a big ball of mud, write a big ball of tests

Transcript

1. 2014 Testing the Untestable @schneems
2. @schneems
3. Ruby Schneems
4. Ruby Python
7. Studied ME
8. Thermo Dynamics
9. In school there are answer keys
10. Have you ever flipped the sign?
11. -1000
12. +1000
13. Huge difference
14. What about the real world?
15. Co-Op
16. How does GE calculate designs for a Frige?
17. Freaking
18. Freaking Spreadsheets
19. What if the spreadsheet is wrong?
20. Introducing:
21. Testing!
22. Thermocouples
23. Programmers are lucky
24. Our inputs are known, our outputs are known
25. Our product is a program
26. Our tests are programs
27. Others aren’t so lucky
28. 1960’s
29. The US will go to the moon
30. Outer Space is a lifeless vacuum
34. How do you trust the calculations?
35. How do you test…
36. The untestable?
37. Unit Test components
38. At the end of the day
39. You launch it
41. Integration Tests
42. “SAFE” Tests
43. Back To Software
45. Ruby Task
 Force
46. Ruby Task
 Force Member
47. Name That year:
51. Heroku Aspen Stack
52. What is a buildpack?
53. credit @zeke
54. 4,573 lines of edge cases of edge cases
55. 0 tests (Jan 2013)
56. Not to say it wasn’t tested
57. it was just
58. w Manual Testing
59. *
60. We have platform tests (they’re not buildpack specific)
62. MVP
63. Minimum Viable
 Patch
64. The smallest possible code change
65. Rarely the most Maintainable
66. Rarely the most Flexible
67. Rarely the Fastest
68. Too many MVPs and your code becomes difficult
69. What’s the cure for the MVP?
70. Tests! (& Refactoring)
72. Black box testing
73. Given a set of inputs
74. Expect a known output
75. Ignore the how
76. So how do we actually test a buildpack?
77. Real Ruby apps, real deploys
78. Let’s start with the the framework:
79. Hatchet
80. Hacks tests github.com/heroku/hatchet
81. clone repo create heroku app deploy
82. git repo is input
83. Output is deploy log
84. Output is `heroku run`
85. Who has used? $ heroku run bash
86. $ heroku run bash Running `bash` attached to terminal… ~ $ ruby -v ruby 2.1.0p0 ~ $ rails -v Rails 4.0.2 Heroku run bash
87. Anyone not using Ruby 2.1.1 right now?
88. PHP?
89. github.com/ schneems/ repl_runner
90. Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'") do |r| expect(r).to match('helloworld') end end end repl_runner
91. Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'") do |r| expect(r).to match('helloworld') end end end repl_runner
92. Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'") do |r| expect(r).to match('helloworld') end end end repl_runner
93. Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'") do |r| expect(r).to match('helloworld') end end end repl_runner
94. Looks simple, but took me days
95. Process deadlock is no joke
96. Where did “rails3” come from?
98. github.com/ sharpstone
99. 47 Repos of edge cases (and counting)
100. Sidenote!
101. Threads in Ruby are easy + awesome
102. Threaded
103. Threaded.later do require “YAML" url = “https://s3-external-1.amazonaws.com/” #... YAML.load `curl #{url} 2>/dev/null` end Threaded. later
104. $ bundle exec hatchet install Installing repos for hatchet == pulling 'git://github.com/sharpstone/ asset_precompile_pass.git' == pulling ‘git://github.com/sharpstone/ asset_precompile_not_found.git' # … hatchet install ~ 45 seconds
105. $ bundle exec hatchet install Installing repos for hatchet == pulling 'git://github.com/sharpstone/ asset_precompile_pass.git' == pulling ‘git://github.com/sharpstone/ asset_precompile_not_found.git' # … hatchet install ~ 2 seconds
106. Copies repo to tmp dir
107. Creates new app through Heroku API
108. Deploys
109. Assertions done in the “deploy” block
110. So we have input & output
111. Tests are done, right?
112. What if S3 Goes down
113. What if Rubygems Goes down
114. What if Heroku API Goes down
115. What if Network Goes down
116. What if Github Goes down
117. Your tests FAIL
119. Seems untestable
120. What do we do when we fall off the horse?
121. We rrrretry
123. All deploys are retried
124. $ cat .travis.yml | grep HATCHET_RETRIES - HATCHET_RETRIES=3
125. Sidebar: Bundler 1.5+ automatically retries
126. Deploys are idempotent
127. What about non-deploy network hiccups?
128. RSpec.configure do |config| config.default_retry_count = 2 # … end
129. Failed assertions cause a test re-run
130. Fool me once, shame on you
131. Fool me six times sequentially, then there’s an error in my tests
132. When life gives you non-deterministic code, use probability to approximate determinism
133. Test Speed
134. First test: ~5 minutes
135. ~1000 travis runs later
136. 44 test cases in ~12 minutes (when passing)
137. How?
138. $ bundle exec parallel_rspec -n 11 spec/ Parallel Rspec Runner
139. Also, the buildpack got faster
141. Coincidence?
143. Tests allow us to be aggressive in refactoring
144. Big changes in architecture == big changes in speed
145. Did testing make the buildpack faster?
146. Tests beget tests
147. With black box tests in place: refactor to modular components
148. Unit test those components
149. Unit tests are faster
150. rake = LanguagePack::Helpers::RakeRunner.new .load_rake_tasks! task = rake.task("assets:precompile") task.invoke expect(task.status).to eq(:pass) expect(task.output).to match("success!") expect(task.time).not_to be_nil Finished in 1.63 seconds
151. Faster tests means quicker iteration
152. Some real world “untestable” scenarios for your web apps
153. codetriage .com
155. External network dependency: github.com
156. What does a black box Rails test look like?
157. Click buttons, observe page changes
158. Capybara to the rescue!
159. What about all that network retry stuff?
160. Mock for determinism
161. webmock
162. VCR
163. Record “real” web traffic, play back in tests
164. --- http_interactions: - request: method: get uri: https://api.github.com/repos/bemurphy/ issue_triage_bogus_repo/issues? direction=desc&page=1&sort=comments body: encoding: US-ASCII string: '' headers: Accept: - application/vnd.github.3.raw+json cassette.yml
165. Libraries
166. Puma Auto Tune
167. Optimizes Puma “workers” based on RAM
168. How do we test it?
169. PumaRemote. new. spawn
170. Process.spawn("exec env PUMA_FREQUENCY=#{frequency} PUMA_RAM=#{ram} bundle exec puma #{path} -C #{config} > #{log}") Actually Run Puma
171. require 'rack' require 'rack/server' run Proc.new {|env| [200, {}, ['Hello World']] } Stub App
172. @puma = PumaRemote.new.spawn assert_match "cannot have less than one worker", @puma.log.read Assert against logs
173. That was a lot to digest, I just want to write tests
174. Nothing is untestable
175. Start with integration tests
176. Test the things that will hurt if they break
177. Use your tests to refactor and write smaller, faster tests
178. Don’t get paged at 3am about trivial failures
179. Avoid the MVP
180. Be maintainable
181. Be flexible
182. Be fast(able)
183. Be tested
184. @schneems
185. Sextant Gem
186. Wicked ‘ ‘ Gem
188. Questions? @schneems