Testing the Untestable

Given in 2014. Good tests are isolated, they’re repeatable, they’re deterministic. Good tests don’t touch the network and are flexible when it comes to change. Bad tests are all of the above and more. Bad tests are no tests at all: which is where I found myself with a 5 year legacy codebase running in production and touching millions of customers with minimal use-case documentation. We’ll cover this experience and several like it while digging into how to go from zero to total test coverage as painlessly as possible. You will learn how to stay sane in the face of insane testing conditions, and how to use these tests to deconstruct a monolith app. When life gives you a big ball of mud, write a big ball of tests

Transcript

1. 2014
   Testing
   the
   Untestable
   @schneems
   

   View Slide

2. @schneems
   

   View Slide

3. Ruby
   Schneems
   

   View Slide

4. Ruby
   Python
   

   View Slide

5. View Slide

6. View Slide

7. Studied
   ME
   

   View Slide

8. Thermo
   Dynamics
   

   View Slide

9. In school
   there are
   answer keys
   

   View Slide

10. Have you
    ever flipped
    the sign?
    

    View Slide

11. -1000
    

    View Slide

12. +1000
    

    View Slide

13. Huge
    difference
    

    View Slide

14. What about
    the real
    world?
    

    View Slide

15. Co-Op
    

    View Slide

16. How does GE
    calculate
    designs for a
    Frige?
    

    View Slide

17. Freaking
    

    View Slide

18. Freaking
    Spreadsheets
    

    View Slide

19. What if the
    spreadsheet
    is wrong?
    

    View Slide

20. Introducing:
    

    View Slide

21. Testing!
    

    View Slide

22. Thermocouples
    

    View Slide

23. Programmers
    are lucky
    

    View Slide

24. Our inputs
    are known,
    our outputs
    are known
    

    View Slide

25. Our product is
    a program
    

    View Slide

26. Our tests are
    programs
    

    View Slide

27. Others aren’t
    so lucky
    

    View Slide

28. 1960’s
    

    View Slide

29. The US will
    go to the
    moon
    

    View Slide

30. Outer Space
    is a lifeless
    vacuum
    

    View Slide

31. View Slide

32. View Slide

33. View Slide

34. How do you
    trust the
    calculations?
    

    View Slide

35. How do you
    test…
    

    View Slide

36. The
    untestable?
    

    View Slide

37. Unit Test
    components
    

    View Slide

38. At the end of
    the day
    

    View Slide

39. You launch it
    

    View Slide

40. View Slide

41. Integration
    Tests
    

    View Slide

42. “SAFE”
    Tests
    

    View Slide

43. Back To
    Software
    

    View Slide

44. View Slide

45. Ruby
    Task

    Force
    

    View Slide

46. Ruby
    Task

    Force
    Member
    

    View Slide

47. Name That
    year:
    

    View Slide

48. View Slide

49. View Slide

50. View Slide

51. Heroku
    Aspen Stack
    

    View Slide

52. What is a
    buildpack?
    

    View Slide

53. credit @zeke
    

    View Slide

54. 4,573 lines of
    edge cases of
    edge cases
    

    View Slide

55. 0 tests
    (Jan 2013)
    

    View Slide

56. Not to say it
    wasn’t tested
    

    View Slide

57. it was just
    

    View Slide

58. w
    Manual
    Testing
    

    View Slide

59. *
    

    View Slide

60. We have
    platform tests
    (they’re not
    buildpack
    specific)
    

    View Slide

61. View Slide

62. MVP
    

    View Slide

63. Minimum
    Viable

    Patch
    

    View Slide

64. The smallest
    possible code
    change
    

    View Slide

65. Rarely
    the most
    Maintainable
    

    View Slide

66. Rarely
    the most
    Flexible
    

    View Slide

67. Rarely
    the
    Fastest
    

    View Slide

68. Too many
    MVPs and your
    code becomes
    difficult
    

    View Slide

69. What’s the
    cure for the
    MVP?
    

    View Slide

70. Tests!
    (& Refactoring)
    

    View Slide

71. View Slide

72. Black box
    testing
    

    View Slide

73. Given a set of
    inputs
    

    View Slide

74. Expect a
    known output
    

    View Slide

75. Ignore the
    how
    

    View Slide

76. So how do we
    actually test a
    buildpack?
    

    View Slide

77. Real Ruby
    apps, real
    deploys
    

    View Slide

78. Let’s start
    with the the
    framework:
    

    View Slide

79. Hatchet
    

    View Slide

80. Hacks tests
    github.com/heroku/hatchet
    

    View Slide

81. clone repo
    create heroku app
    deploy
    

    View Slide

82. git repo is
    input
    

    View Slide

83. Output is
    deploy log
    

    View Slide

84. Output is
    `heroku run`
    

    View Slide

85. Who has
    used?
    $ heroku run bash
    

    View Slide

86. $ heroku run bash
    Running `bash` attached to terminal…
    ~ $ ruby -v
    ruby 2.1.0p0
    ~ $ rails -v
    Rails 4.0.2
    Heroku run
    bash
    

    View Slide

87. Anyone not
    using Ruby
    2.1.1 right
    now?
    

    View Slide

88. PHP?
    

    View Slide

89. github.com/
    schneems/
    repl_runner
    

    View Slide

90. Hatchet::Runner.new("rails3").deploy do |app|
    app.run("rails console") do |console|
    console.run("'hello' + 'world'") do |r|
    expect(r).to match('helloworld')
    end
    end
    end
    repl_runner
    

    View Slide

91. Hatchet::Runner.new("rails3").deploy do |app|
    app.run("rails console") do |console|
    console.run("'hello' + 'world'") do |r|
    expect(r).to match('helloworld')
    end
    end
    end
    repl_runner
    

    View Slide

92. Hatchet::Runner.new("rails3").deploy do |app|
    app.run("rails console") do |console|
    console.run("'hello' + 'world'") do |r|
    expect(r).to match('helloworld')
    end
    end
    end
    repl_runner
    

    View Slide

93. Hatchet::Runner.new("rails3").deploy do |app|
    app.run("rails console") do |console|
    console.run("'hello' + 'world'") do |r|
    expect(r).to match('helloworld')
    end
    end
    end
    repl_runner
    

    View Slide

94. Looks simple,
    but took me
    days
    

    View Slide

95. Process
    deadlock is
    no joke
    

    View Slide

96. Where did
    “rails3” come
    from?
    

    View Slide

97. View Slide

98. github.com/
    sharpstone
    

    View Slide

99. 47 Repos
    of edge cases
    (and counting)
    

    View Slide

100. Sidenote!
     

     View Slide

101. Threads in
     Ruby are easy
     + awesome
     

     View Slide

102. Threaded
     

     View Slide

103. Threaded.later do
     require “YAML"
     url = “https://s3-external-1.amazonaws.com/” #...
     YAML.load `curl #{url} 2>/dev/null`
     end
     Threaded.
     later
     

     View Slide

104. $ bundle exec hatchet install
     Installing repos for hatchet
     == pulling 'git://github.com/sharpstone/
     asset_precompile_pass.git'
     == pulling ‘git://github.com/sharpstone/
     asset_precompile_not_found.git'
     # … hatchet
     install
     ~ 45 seconds
     

     View Slide

105. $ bundle exec hatchet install
     Installing repos for hatchet
     == pulling 'git://github.com/sharpstone/
     asset_precompile_pass.git'
     == pulling ‘git://github.com/sharpstone/
     asset_precompile_not_found.git'
     # … hatchet
     install
     ~ 2 seconds
     

     View Slide

106. Copies repo to
     tmp dir
     

     View Slide

107. Creates new
     app through
     Heroku API
     

     View Slide

108. Deploys
     

     View Slide

109. Assertions
     done in the
     “deploy”
     block
     

     View Slide

110. So we have
     input &
     output
     

     View Slide

111. Tests are
     done, right?
     

     View Slide

112. What if
     S3
     Goes down
     

     View Slide

113. What if
     Rubygems
     Goes down
     

     View Slide

114. What if
     Heroku API
     Goes down
     

     View Slide

115. What if
     Network
     Goes down
     

     View Slide

116. What if
     Github
     Goes down
     

     View Slide

117. Your
     tests
     FAIL
     

     View Slide

118. View Slide

119. Seems
     untestable
     

     View Slide

120. What do we
     do when we
     fall off the
     horse?
     

     View Slide

121. We rrrretry
     

     View Slide

122. View Slide

123. All deploys
     are retried
     

     View Slide

124. $ cat .travis.yml | grep HATCHET_RETRIES
     - HATCHET_RETRIES=3
     

     View Slide

125. Sidebar:
     Bundler 1.5+
     automatically
     retries
     

     View Slide

126. Deploys are
     idempotent
     

     View Slide

127. What about
     non-deploy
     network
     hiccups?
     

     View Slide

128. RSpec.configure do |config|
     config.default_retry_count = 2
     # …
     end
     

     View Slide

129. Failed
     assertions
     cause a test
     re-run
     

     View Slide

130. Fool me once,
     shame on you
     

     View Slide

131. Fool me six
     times
     sequentially,
     then there’s an
     error in my tests
     

     View Slide

132. When life gives you
     non-deterministic
     code, use
     probability to
     approximate
     determinism
     

     View Slide

133. Test Speed
     

     View Slide

134. First test: ~5
     minutes
     

     View Slide

135. ~1000 travis
     runs later
     

     View Slide

136. 44 test cases
     in ~12 minutes
     (when passing)
     

     View Slide

137. How?
     

     View Slide

138. $ bundle exec parallel_rspec -n 11 spec/
     Parallel
     Rspec
     Runner
     

     View Slide

139. Also, the
     buildpack got
     faster
     

     View Slide

140. View Slide

141. Coincidence?
     

     View Slide

142. View Slide

143. Tests allow
     us to be
     aggressive in
     refactoring
     

     View Slide

144. Big changes in
     architecture
     ==
     big changes in
     speed
     

     View Slide

145. Did testing
     make the
     buildpack
     faster?
     

     View Slide

146. Tests beget
     tests
     

     View Slide

147. With black box
     tests in place:
     refactor to
     modular
     components
     

     View Slide

148. Unit test
     those
     components
     

     View Slide

149. Unit tests are
     faster
     

     View Slide

150. rake = LanguagePack::Helpers::RakeRunner.new
     .load_rake_tasks!
     task = rake.task("assets:precompile")
     task.invoke
     expect(task.status).to eq(:pass)
     expect(task.output).to match("success!")
     expect(task.time).not_to be_nil
     Finished in
     1.63 seconds
     

     View Slide

151. Faster tests
     means
     quicker
     iteration
     

     View Slide

152. Some real world
     “untestable”
     scenarios for
     your web apps
     

     View Slide

153. codetriage
     .com
     

     View Slide

154. View Slide

155. External
     network
     dependency:
     github.com
     

     View Slide

156. What does a
     black box
     Rails test
     look like?
     

     View Slide

157. Click buttons,
     observe page
     changes
     

     View Slide

158. Capybara to
     the rescue!
     

     View Slide

159. What about
     all that
     network retry
     stuff?
     

     View Slide

160. Mock
     for
     determinism
     

     View Slide

161. webmock
     

     View Slide

162. VCR
     

     View Slide

163. Record “real”
     web traffic,
     play back in
     tests
     

     View Slide

164. ---
     http_interactions:
     - request:
     method: get
     uri: https://api.github.com/repos/bemurphy/
     issue_triage_bogus_repo/issues?
     direction=desc&page=1&sort=comments
     body:
     encoding: US-ASCII
     string: ''
     headers:
     Accept:
     - application/vnd.github.3.raw+json
     cassette.yml
     

     View Slide

165. Libraries
     

     View Slide

166. Puma
     Auto
     Tune
     

     View Slide

167. Optimizes
     Puma
     “workers”
     based on RAM
     

     View Slide

168. How do we
     test it?
     

     View Slide

169. PumaRemote.
     new.
     spawn
     

     View Slide

170. Process.spawn("exec env
     PUMA_FREQUENCY=#{frequency}
     PUMA_RAM=#{ram}
     bundle exec puma #{path}
     -C #{config} > #{log}")
     Actually
     Run
     Puma
     

     View Slide

171. require 'rack'
     require 'rack/server'
     run Proc.new {|env| [200, {}, ['Hello World']] }
     Stub
     App
     

     View Slide

172. @puma = PumaRemote.new.spawn
     assert_match "cannot have less than one worker",
     @puma.log.read
     Assert
     against
     logs
     

     View Slide

173. That was a
     lot to digest, I
     just want to
     write tests
     

     View Slide

174. Nothing is
     untestable
     

     View Slide

175. Start with
     integration
     tests
     

     View Slide

176. Test the
     things that
     will hurt if
     they break
     

     View Slide

177. Use your tests
     to refactor and
     write smaller,
     faster tests
     

     View Slide

178. Don’t get
     paged at 3am
     about trivial
     failures
     

     View Slide

179. Avoid the
     MVP
     

     View Slide

180. Be
     maintainable
     

     View Slide

181. Be
     flexible
     

     View Slide

182. Be
     fast(able)
     

     View Slide

183. Be
     tested
     

     View Slide

184. @schneems
     

     View Slide

185. Sextant
     Gem
     

     View Slide

186. Wicked
     ‘
     ‘
     Gem
     

     View Slide

187. View Slide

188. Questions?
     @schneems
     

     View Slide