Testing the Untestable

Testing the Untestable

Given in 2014. Good tests are isolated, they’re repeatable, they’re deterministic. Good tests don’t touch the network and are flexible when it comes to change. Bad tests are all of the above and more. Bad tests are no tests at all: which is where I found myself with a 5 year legacy codebase running in production and touching millions of customers with minimal use-case documentation. We’ll cover this experience and several like it while digging into how to go from zero to total test coverage as painlessly as possible. You will learn how to stay sane in the face of insane testing conditions, and how to use these tests to deconstruct a monolith app. When life gives you a big ball of mud, write a big ball of tests

Db953d125f5cc49756edb6149f1b813e?s=128

Richard Schneeman

November 21, 2016
Tweet

Transcript

  1. 2014 Testing the Untestable @schneems

  2. @schneems

  3. Ruby Schneems

  4. Ruby Python

  5. None
  6. None
  7. Studied ME

  8. Thermo Dynamics

  9. In school there are answer keys

  10. Have you ever flipped the sign?

  11. -1000

  12. +1000

  13. Huge difference

  14. What about the real world?

  15. Co-Op

  16. How does GE calculate designs for a Frige?

  17. Freaking

  18. Freaking Spreadsheets

  19. What if the spreadsheet is wrong?

  20. Introducing:

  21. Testing!

  22. Thermocouples

  23. Programmers are lucky

  24. Our inputs are known, our outputs are known

  25. Our product is a program

  26. Our tests are programs

  27. Others aren’t so lucky

  28. 1960’s

  29. The US will go to the moon

  30. Outer Space is a lifeless vacuum

  31. None
  32. None
  33. None
  34. How do you trust the calculations?

  35. How do you test…

  36. The untestable?

  37. Unit Test components

  38. At the end of the day

  39. You launch it

  40. None
  41. Integration Tests

  42. “SAFE” Tests

  43. Back To Software

  44. None
  45. Ruby Task
 Force

  46. Ruby Task
 Force Member

  47. Name That year:

  48. None
  49. None
  50. None
  51. Heroku Aspen Stack

  52. What is a buildpack?

  53. credit @zeke

  54. 4,573 lines of edge cases of edge cases

  55. 0 tests (Jan 2013)

  56. Not to say it wasn’t tested

  57. it was just

  58. w Manual Testing

  59. *

  60. We have platform tests (they’re not buildpack specific)

  61. None
  62. MVP

  63. Minimum Viable
 Patch

  64. The smallest possible code change

  65. Rarely the most Maintainable

  66. Rarely the most Flexible

  67. Rarely the Fastest

  68. Too many MVPs and your code becomes difficult

  69. What’s the cure for the MVP?

  70. Tests! (& Refactoring)

  71. None
  72. Black box testing

  73. Given a set of inputs

  74. Expect a known output

  75. Ignore the how

  76. So how do we actually test a buildpack?

  77. Real Ruby apps, real deploys

  78. Let’s start with the the framework:

  79. Hatchet

  80. Hacks tests github.com/heroku/hatchet

  81. clone repo create heroku app deploy

  82. git repo is input

  83. Output is deploy log

  84. Output is `heroku run`

  85. Who has used? $ heroku run bash

  86. $ heroku run bash Running `bash` attached to terminal… ~

    $ ruby -v ruby 2.1.0p0 ~ $ rails -v Rails 4.0.2 Heroku run bash
  87. Anyone not using Ruby 2.1.1 right now?

  88. PHP?

  89. github.com/ schneems/ repl_runner

  90. Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'")

    do |r| expect(r).to match('helloworld') end end end repl_runner
  91. Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'")

    do |r| expect(r).to match('helloworld') end end end repl_runner
  92. Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'")

    do |r| expect(r).to match('helloworld') end end end repl_runner
  93. Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'")

    do |r| expect(r).to match('helloworld') end end end repl_runner
  94. Looks simple, but took me days

  95. Process deadlock is no joke

  96. Where did “rails3” come from?

  97. None
  98. github.com/ sharpstone

  99. 47 Repos of edge cases (and counting)

  100. Sidenote!

  101. Threads in Ruby are easy + awesome

  102. Threaded

  103. Threaded.later do require “YAML" url = “https://s3-external-1.amazonaws.com/” #... YAML.load `curl

    #{url} 2>/dev/null` end Threaded. later
  104. $ bundle exec hatchet install Installing repos for hatchet ==

    pulling 'git://github.com/sharpstone/ asset_precompile_pass.git' == pulling ‘git://github.com/sharpstone/ asset_precompile_not_found.git' # … hatchet install ~ 45 seconds
  105. $ bundle exec hatchet install Installing repos for hatchet ==

    pulling 'git://github.com/sharpstone/ asset_precompile_pass.git' == pulling ‘git://github.com/sharpstone/ asset_precompile_not_found.git' # … hatchet install ~ 2 seconds
  106. Copies repo to tmp dir

  107. Creates new app through Heroku API

  108. Deploys

  109. Assertions done in the “deploy” block

  110. So we have input & output

  111. Tests are done, right?

  112. What if S3 Goes down

  113. What if Rubygems Goes down

  114. What if Heroku API Goes down

  115. What if Network Goes down

  116. What if Github Goes down

  117. Your tests FAIL

  118. None
  119. Seems untestable

  120. What do we do when we fall off the horse?

  121. We rrrretry

  122. None
  123. All deploys are retried

  124. $ cat .travis.yml | grep HATCHET_RETRIES - HATCHET_RETRIES=3

  125. Sidebar: Bundler 1.5+ automatically retries

  126. Deploys are idempotent

  127. What about non-deploy network hiccups?

  128. RSpec.configure do |config| config.default_retry_count = 2 # … end

  129. Failed assertions cause a test re-run

  130. Fool me once, shame on you

  131. Fool me six times sequentially, then there’s an error in

    my tests
  132. When life gives you non-deterministic code, use probability to approximate

    determinism
  133. Test Speed

  134. First test: ~5 minutes

  135. ~1000 travis runs later

  136. 44 test cases in ~12 minutes (when passing)

  137. How?

  138. $ bundle exec parallel_rspec -n 11 spec/ Parallel Rspec Runner

  139. Also, the buildpack got faster

  140. None
  141. Coincidence?

  142. None
  143. Tests allow us to be aggressive in refactoring

  144. Big changes in architecture == big changes in speed

  145. Did testing make the buildpack faster?

  146. Tests beget tests

  147. With black box tests in place: refactor to modular components

  148. Unit test those components

  149. Unit tests are faster

  150. rake = LanguagePack::Helpers::RakeRunner.new .load_rake_tasks! task = rake.task("assets:precompile") task.invoke expect(task.status).to eq(:pass)

    expect(task.output).to match("success!") expect(task.time).not_to be_nil Finished in 1.63 seconds
  151. Faster tests means quicker iteration

  152. Some real world “untestable” scenarios for your web apps

  153. codetriage .com

  154. None
  155. External network dependency: github.com

  156. What does a black box Rails test look like?

  157. Click buttons, observe page changes

  158. Capybara to the rescue!

  159. What about all that network retry stuff?

  160. Mock for determinism

  161. webmock

  162. VCR

  163. Record “real” web traffic, play back in tests

  164. --- http_interactions: - request: method: get uri: https://api.github.com/repos/bemurphy/ issue_triage_bogus_repo/issues? direction=desc&page=1&sort=comments

    body: encoding: US-ASCII string: '' headers: Accept: - application/vnd.github.3.raw+json cassette.yml
  165. Libraries

  166. Puma Auto Tune

  167. Optimizes Puma “workers” based on RAM

  168. How do we test it?

  169. PumaRemote. new. spawn

  170. Process.spawn("exec env PUMA_FREQUENCY=#{frequency} PUMA_RAM=#{ram} bundle exec puma #{path} -C #{config}

    > #{log}") Actually Run Puma
  171. require 'rack' require 'rack/server' run Proc.new {|env| [200, {}, ['Hello

    World']] } Stub App
  172. @puma = PumaRemote.new.spawn assert_match "cannot have less than one worker",

    @puma.log.read Assert against logs
  173. That was a lot to digest, I just want to

    write tests
  174. Nothing is untestable

  175. Start with integration tests

  176. Test the things that will hurt if they break

  177. Use your tests to refactor and write smaller, faster tests

  178. Don’t get paged at 3am about trivial failures

  179. Avoid the MVP

  180. Be maintainable

  181. Be flexible

  182. Be fast(able)

  183. Be tested

  184. @schneems

  185. Sextant Gem

  186. Wicked ‘ ‘ Gem

  187. None
  188. Questions? @schneems