Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing the Untestable

Testing the Untestable

Given in 2014. Good tests are isolated, they’re repeatable, they’re deterministic. Good tests don’t touch the network and are flexible when it comes to change. Bad tests are all of the above and more. Bad tests are no tests at all: which is where I found myself with a 5 year legacy codebase running in production and touching millions of customers with minimal use-case documentation. We’ll cover this experience and several like it while digging into how to go from zero to total test coverage as painlessly as possible. You will learn how to stay sane in the face of insane testing conditions, and how to use these tests to deconstruct a monolith app. When life gives you a big ball of mud, write a big ball of tests

Richard Schneeman

November 21, 2016
Tweet

More Decks by Richard Schneeman

Other Decks in Programming

Transcript

  1. 2014
    Testing
    the
    Untestable
    @schneems

    View Slide

  2. @schneems

    View Slide

  3. Ruby
    Schneems

    View Slide

  4. Ruby
    Python

    View Slide

  5. View Slide

  6. View Slide

  7. Studied
    ME

    View Slide

  8. Thermo
    Dynamics

    View Slide

  9. In school
    there are
    answer keys

    View Slide

  10. Have you
    ever flipped
    the sign?

    View Slide

  11. -1000

    View Slide

  12. +1000

    View Slide

  13. Huge
    difference

    View Slide

  14. What about
    the real
    world?

    View Slide

  15. Co-Op

    View Slide

  16. How does GE
    calculate
    designs for a
    Frige?

    View Slide

  17. Freaking

    View Slide

  18. Freaking
    Spreadsheets

    View Slide

  19. What if the
    spreadsheet
    is wrong?

    View Slide

  20. Introducing:

    View Slide

  21. Testing!

    View Slide

  22. Thermocouples

    View Slide

  23. Programmers
    are lucky

    View Slide

  24. Our inputs
    are known,
    our outputs
    are known

    View Slide

  25. Our product is
    a program

    View Slide

  26. Our tests are
    programs

    View Slide

  27. Others aren’t
    so lucky

    View Slide

  28. 1960’s

    View Slide

  29. The US will
    go to the
    moon

    View Slide

  30. Outer Space
    is a lifeless
    vacuum

    View Slide

  31. View Slide

  32. View Slide

  33. View Slide

  34. How do you
    trust the
    calculations?

    View Slide

  35. How do you
    test…

    View Slide

  36. The
    untestable?

    View Slide

  37. Unit Test
    components

    View Slide

  38. At the end of
    the day

    View Slide

  39. You launch it

    View Slide

  40. View Slide

  41. Integration
    Tests

    View Slide

  42. “SAFE”
    Tests

    View Slide

  43. Back To
    Software

    View Slide

  44. View Slide

  45. Ruby
    Task

    Force

    View Slide

  46. Ruby
    Task

    Force
    Member

    View Slide

  47. Name That
    year:

    View Slide

  48. View Slide

  49. View Slide

  50. View Slide

  51. Heroku
    Aspen Stack

    View Slide

  52. What is a
    buildpack?

    View Slide

  53. credit @zeke

    View Slide

  54. 4,573 lines of
    edge cases of
    edge cases

    View Slide

  55. 0 tests
    (Jan 2013)

    View Slide

  56. Not to say it
    wasn’t tested

    View Slide

  57. it was just

    View Slide

  58. w
    Manual
    Testing

    View Slide

  59. *

    View Slide

  60. We have
    platform tests
    (they’re not
    buildpack
    specific)

    View Slide

  61. View Slide

  62. MVP

    View Slide

  63. Minimum
    Viable

    Patch

    View Slide

  64. The smallest
    possible code
    change

    View Slide

  65. Rarely
    the most
    Maintainable

    View Slide

  66. Rarely
    the most
    Flexible

    View Slide

  67. Rarely
    the
    Fastest

    View Slide

  68. Too many
    MVPs and your
    code becomes
    difficult

    View Slide

  69. What’s the
    cure for the
    MVP?

    View Slide

  70. Tests!
    (& Refactoring)

    View Slide

  71. View Slide

  72. Black box
    testing

    View Slide

  73. Given a set of
    inputs

    View Slide

  74. Expect a
    known output

    View Slide

  75. Ignore the
    how

    View Slide

  76. So how do we
    actually test a
    buildpack?

    View Slide

  77. Real Ruby
    apps, real
    deploys

    View Slide

  78. Let’s start
    with the the
    framework:

    View Slide

  79. Hatchet

    View Slide

  80. Hacks tests
    github.com/heroku/hatchet

    View Slide

  81. clone repo
    create heroku app
    deploy

    View Slide

  82. git repo is
    input

    View Slide

  83. Output is
    deploy log

    View Slide

  84. Output is
    `heroku run`

    View Slide

  85. Who has
    used?
    $ heroku run bash

    View Slide

  86. $ heroku run bash
    Running `bash` attached to terminal…
    ~ $ ruby -v
    ruby 2.1.0p0
    ~ $ rails -v
    Rails 4.0.2
    Heroku run
    bash

    View Slide

  87. Anyone not
    using Ruby
    2.1.1 right
    now?

    View Slide

  88. PHP?

    View Slide

  89. github.com/
    schneems/
    repl_runner

    View Slide

  90. Hatchet::Runner.new("rails3").deploy do |app|
    app.run("rails console") do |console|
    console.run("'hello' + 'world'") do |r|
    expect(r).to match('helloworld')
    end
    end
    end
    repl_runner

    View Slide

  91. Hatchet::Runner.new("rails3").deploy do |app|
    app.run("rails console") do |console|
    console.run("'hello' + 'world'") do |r|
    expect(r).to match('helloworld')
    end
    end
    end
    repl_runner

    View Slide

  92. Hatchet::Runner.new("rails3").deploy do |app|
    app.run("rails console") do |console|
    console.run("'hello' + 'world'") do |r|
    expect(r).to match('helloworld')
    end
    end
    end
    repl_runner

    View Slide

  93. Hatchet::Runner.new("rails3").deploy do |app|
    app.run("rails console") do |console|
    console.run("'hello' + 'world'") do |r|
    expect(r).to match('helloworld')
    end
    end
    end
    repl_runner

    View Slide

  94. Looks simple,
    but took me
    days

    View Slide

  95. Process
    deadlock is
    no joke

    View Slide

  96. Where did
    “rails3” come
    from?

    View Slide

  97. View Slide

  98. github.com/
    sharpstone

    View Slide

  99. 47 Repos
    of edge cases
    (and counting)

    View Slide

  100. Sidenote!

    View Slide

  101. Threads in
    Ruby are easy
    + awesome

    View Slide

  102. Threaded

    View Slide

  103. Threaded.later do
    require “YAML"
    url = “https://s3-external-1.amazonaws.com/” #...
    YAML.load `curl #{url} 2>/dev/null`
    end
    Threaded.
    later

    View Slide

  104. $ bundle exec hatchet install
    Installing repos for hatchet
    == pulling 'git://github.com/sharpstone/
    asset_precompile_pass.git'
    == pulling ‘git://github.com/sharpstone/
    asset_precompile_not_found.git'
    # … hatchet
    install
    ~ 45 seconds

    View Slide

  105. $ bundle exec hatchet install
    Installing repos for hatchet
    == pulling 'git://github.com/sharpstone/
    asset_precompile_pass.git'
    == pulling ‘git://github.com/sharpstone/
    asset_precompile_not_found.git'
    # … hatchet
    install
    ~ 2 seconds

    View Slide

  106. Copies repo to
    tmp dir

    View Slide

  107. Creates new
    app through
    Heroku API

    View Slide

  108. Deploys

    View Slide

  109. Assertions
    done in the
    “deploy”
    block

    View Slide

  110. So we have
    input &
    output

    View Slide

  111. Tests are
    done, right?

    View Slide

  112. What if
    S3
    Goes down

    View Slide

  113. What if
    Rubygems
    Goes down

    View Slide

  114. What if
    Heroku API
    Goes down

    View Slide

  115. What if
    Network
    Goes down

    View Slide

  116. What if
    Github
    Goes down

    View Slide

  117. Your
    tests
    FAIL

    View Slide

  118. View Slide

  119. Seems
    untestable

    View Slide

  120. What do we
    do when we
    fall off the
    horse?

    View Slide

  121. We rrrretry

    View Slide

  122. View Slide

  123. All deploys
    are retried

    View Slide

  124. $ cat .travis.yml | grep HATCHET_RETRIES
    - HATCHET_RETRIES=3

    View Slide

  125. Sidebar:
    Bundler 1.5+
    automatically
    retries

    View Slide

  126. Deploys are
    idempotent

    View Slide

  127. What about
    non-deploy
    network
    hiccups?

    View Slide

  128. RSpec.configure do |config|
    config.default_retry_count = 2
    # …
    end

    View Slide

  129. Failed
    assertions
    cause a test
    re-run

    View Slide

  130. Fool me once,
    shame on you

    View Slide

  131. Fool me six
    times
    sequentially,
    then there’s an
    error in my tests

    View Slide

  132. When life gives you
    non-deterministic
    code, use
    probability to
    approximate
    determinism

    View Slide

  133. Test Speed

    View Slide

  134. First test: ~5
    minutes

    View Slide

  135. ~1000 travis
    runs later

    View Slide

  136. 44 test cases
    in ~12 minutes
    (when passing)

    View Slide

  137. How?

    View Slide

  138. $ bundle exec parallel_rspec -n 11 spec/
    Parallel
    Rspec
    Runner

    View Slide

  139. Also, the
    buildpack got
    faster

    View Slide

  140. View Slide

  141. Coincidence?

    View Slide

  142. View Slide

  143. Tests allow
    us to be
    aggressive in
    refactoring

    View Slide

  144. Big changes in
    architecture
    ==
    big changes in
    speed

    View Slide

  145. Did testing
    make the
    buildpack
    faster?

    View Slide

  146. Tests beget
    tests

    View Slide

  147. With black box
    tests in place:
    refactor to
    modular
    components

    View Slide

  148. Unit test
    those
    components

    View Slide

  149. Unit tests are
    faster

    View Slide

  150. rake = LanguagePack::Helpers::RakeRunner.new
    .load_rake_tasks!
    task = rake.task("assets:precompile")
    task.invoke
    expect(task.status).to eq(:pass)
    expect(task.output).to match("success!")
    expect(task.time).not_to be_nil
    Finished in
    1.63 seconds

    View Slide

  151. Faster tests
    means
    quicker
    iteration

    View Slide

  152. Some real world
    “untestable”
    scenarios for
    your web apps

    View Slide

  153. codetriage
    .com

    View Slide

  154. View Slide

  155. External
    network
    dependency:
    github.com

    View Slide

  156. What does a
    black box
    Rails test
    look like?

    View Slide

  157. Click buttons,
    observe page
    changes

    View Slide

  158. Capybara to
    the rescue!

    View Slide

  159. What about
    all that
    network retry
    stuff?

    View Slide

  160. Mock
    for
    determinism

    View Slide

  161. webmock

    View Slide

  162. VCR

    View Slide

  163. Record “real”
    web traffic,
    play back in
    tests

    View Slide

  164. ---
    http_interactions:
    - request:
    method: get
    uri: https://api.github.com/repos/bemurphy/
    issue_triage_bogus_repo/issues?
    direction=desc&page=1&sort=comments
    body:
    encoding: US-ASCII
    string: ''
    headers:
    Accept:
    - application/vnd.github.3.raw+json
    cassette.yml

    View Slide

  165. Libraries

    View Slide

  166. Puma
    Auto
    Tune

    View Slide

  167. Optimizes
    Puma
    “workers”
    based on RAM

    View Slide

  168. How do we
    test it?

    View Slide

  169. PumaRemote.
    new.
    spawn

    View Slide

  170. Process.spawn("exec env
    PUMA_FREQUENCY=#{frequency}
    PUMA_RAM=#{ram}
    bundle exec puma #{path}
    -C #{config} > #{log}")
    Actually
    Run
    Puma

    View Slide

  171. require 'rack'
    require 'rack/server'
    run Proc.new {|env| [200, {}, ['Hello World']] }
    Stub
    App

    View Slide

  172. @puma = PumaRemote.new.spawn
    assert_match "cannot have less than one worker",
    @puma.log.read
    Assert
    against
    logs

    View Slide

  173. That was a
    lot to digest, I
    just want to
    write tests

    View Slide

  174. Nothing is
    untestable

    View Slide

  175. Start with
    integration
    tests

    View Slide

  176. Test the
    things that
    will hurt if
    they break

    View Slide

  177. Use your tests
    to refactor and
    write smaller,
    faster tests

    View Slide

  178. Don’t get
    paged at 3am
    about trivial
    failures

    View Slide

  179. Avoid the
    MVP

    View Slide

  180. Be
    maintainable

    View Slide

  181. Be
    flexible

    View Slide

  182. Be
    fast(able)

    View Slide

  183. Be
    tested

    View Slide

  184. @schneems

    View Slide

  185. Sextant
    Gem

    View Slide

  186. Wicked


    Gem

    View Slide

  187. View Slide

  188. Questions?
    @schneems

    View Slide