Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Breaking up (with) your test suite

Breaking up (with) your test suite

As presented at Ancient City Ruby 2014.

Please send questions & feedback to [email protected]!

Justin Searls

April 03, 2014
Tweet

More Decks by Justin Searls

Other Decks in Programming

Transcript

  1. Breaking up
    (with) your
    test suite
    poa asd

    View Slide

  2. My name is Justin Searls
    Please tweet me @searls &
    Say [email protected]

    View Slide

  3. Any questions?
    Tweet @searls during the talk
    Favorite question gets a book!

    View Slide

  4. Any questions?
    Tweet @searls during the talk
    Favorite question gets a book!

    View Slide

  5. This talk is about
    z aPPOOIU

    View Slide

  6. This talk is about
    z aPPOOIU
    relationships

    View Slide

  7. I my
    customers
    ias al

    View Slide

  8. I coding
    zffrttyu z

    View Slide

  9. I testing
    aske zc

    View Slide

  10. We don't test for
    testing's sake
    rRth asd

    View Slide

  11. I get paid for code that
    works, not for tests, so
    my philosophy is to test
    as little as possible to
    reach a given level of
    confidence.
    - Kent Beck
    asdz ty

    View Slide

  12. We don't code for
    code's sake, either
    mmmmznxdfr lr

    View Slide

  13. Writing code is a costly
    way to solve problems
    oiz eu

    View Slide

  14. Too often, we cheat on our
    customers with our code
    c d aoiey

    View Slide

  15. How to cheat on
    your customers:
    ae ts

    View Slide

  16. How to cheat on
    your customers:
    ae ts
    1. Uncritically assume your
    favorite way to write code
    applies to every situation

    View Slide

  17. How to cheat on
    your customers:
    ae ts
    1. Uncritically assume your
    favorite way to write code
    applies to every situation
    2. There is no #2

    View Slide

  18. evh zc
    Cargo-culting
    as a service

    View Slide

  19. We must challenge our biases
    in each new context to grow
    fh ywe

    View Slide

  20. iO ZOe
    infatuation

    View Slide

  21. xmt iRosXc
    Remember
    TATFT?

    View Slide

  22. TDD literally
    rescued me
    from the pit
    of despair.

    View Slide

  23. lzxcm zxcmlzcl lzxckz zxczc zxjcozcjoe AOO asodjaosd encz caosdjojo jojo
    6 months later
    ~ ~

    View Slide

  24. Why does
    every refactor
    break 6 tests?!

    View Slide

  25. Why does the
    build take nine
    hours to run?!

    View Slide

  26. Why are people
    mocking methods
    on the object
    they're testing?!

    View Slide

  27. Why can't we
    get anything
    done anymore?

    View Slide

  28. asodo zoxjcz aoo ueue yyyz cxjk zxcczx
    and eventually
    ~ ~

    View Slide

  29. the break-up

    View Slide

  30. the break-up

    View Slide

  31. $
    the break-up

    View Slide

  32. $
    the break-up
    rm -rf spec features

    View Slide

  33. jn poi
    introspection

    View Slide

  34. qwer bvt
    Why test?

    View Slide

  35. View Slide

  36. !

    View Slide

  37. !


    !

    View Slide

  38. !


    !



    View Slide

  39. !


    !



    !


    View Slide

  40. !


    !



    !






    View Slide

  41. !


    !



    !








    View Slide

  42. !


    !



    !








    !


    View Slide

  43. !


    !



    !








    !



    !

    View Slide

  44. !


    !



    !








    !



    !



    View Slide

  45. !


    !



    !








    !



    !



    !!

    View Slide

  46. !


    !



    !



    !


    !









    !!

    !



    !
    !




    !

    View Slide

  47. !


    !



    !



    !


    !









    !!

    !



    !
    !




    !

    Confidence

    View Slide

  48. !


    !



    !



    !


    !









    !!



    !









    !!

    View Slide

  49. !


    !



    !



    !


    !









    !!

    Understanding


    !









    !!

    View Slide

  50. mni pl
    Single
    Responsibility
    Testing

    View Slide

  51. describe "Panda" do
    end

    View Slide

  52. describe "Panda" do
    let(:bamboo) { double("bamboo").as_null_object }
    end

    View Slide

  53. describe "Panda" do
    let(:bamboo) { double("bamboo").as_null_object }
    subject { Panda.new(bamboo) }
    end

    View Slide

  54. describe "Panda" do
    let(:bamboo) { double("bamboo").as_null_object }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    end
    end

    View Slide

  55. describe "Panda" do
    let(:bamboo) { double("bamboo").as_null_object }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    subject.eat!
    end
    end

    View Slide

  56. describe "Panda" do
    let(:bamboo) { double("bamboo").as_null_object }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    end

    View Slide

  57. describe "Panda" do
    let(:bamboo) { double("bamboo").as_null_object }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    end

    View Slide

  58. asdj zoxc mmr qu psoi rnzc vnkl la
    later, fixing a bug
    ~ ~

    View Slide

  59. describe "Panda" do
    let(:bamboo) { double("bamboo").as_null_object }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    end

    View Slide

  60. describe "Panda" do
    let(:bamboo) { Bamboo.find(34) }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    end

    View Slide

  61. describe "Panda" do
    let(:bamboo) { Bamboo.find(34) }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    allow(bamboo).to receive(:munch)
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    end

    View Slide

  62. describe "Panda" do
    let(:bamboo) { Bamboo.find(34) }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    allow(bamboo).to receive(:munch)
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    !
    it "returns bamboo id" do
    end
    end

    View Slide

  63. describe "Panda" do
    let(:bamboo) { Bamboo.find(34) }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    allow(bamboo).to receive(:munch)
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    !
    it "returns bamboo id" do
    expect(subject.eat!).to eq(34)
    end
    end

    View Slide

  64. describe "Panda" do
    let(:bamboo) { Bamboo.find(34) }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    allow(bamboo).to receive(:munch)
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    !
    it "returns bamboo id" do
    expect(subject.eat!).to eq(34)
    end
    end

    View Slide

  65. asdj zoxc mmr qu psoi rnzc vnkl la
    one day, I find it
    ~ ~

    View Slide

  66. describe "Panda" do
    let(:bamboo) { Bamboo.find(34) }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    allow(bamboo).to receive(:munch)
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    !
    it "returns bamboo id" do
    expect(subject.eat!).to eq(34)
    end
    end

    View Slide

  67. describe "Panda" do
    let(:bamboo) { Bamboo.find(34) }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    allow(bamboo).to receive(:munch)
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    !
    it "returns bamboo id" do
    expect(subject.eat!).to eq(34)
    end
    end
    ?
    ? ?

    View Slide

  68. describe "Panda" do
    let(:bamboo) { Bamboo.find(34) }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    allow(bamboo).to receive(:munch)
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    !
    it "returns bamboo id" do
    expect(subject.eat!).to eq(34)
    end
    end
    ?
    ? ?

    View Slide

  69. describe "Panda" do
    let(:bamboo) { Bamboo.find(34) }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    allow(bamboo).to receive(:munch)
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    !
    it "returns bamboo id" do
    expect(subject.eat!).to eq(34)
    end
    end
    ?
    ? ?
    Why does this test exist?

    View Slide

  70. describe "Panda" do
    let(:bamboo) { Bamboo.find(34) }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    allow(bamboo).to receive(:munch)
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    !
    it "returns bamboo id" do
    expect(subject.eat!).to eq(34)
    end
    end
    ?
    ? ?
    Why does this test exist?
    What type of test is this?

    View Slide

  71. describe "Panda" do
    let(:bamboo) { Bamboo.find(34) }
    subject { Panda.new(bamboo) }
    !
    it "munches bamboo" do
    allow(bamboo).to receive(:munch)
    subject.eat!
    expect(bamboo).to have_received(:munch)
    end
    !
    it "returns bamboo id" do
    expect(subject.eat!).to eq(34)
    end
    end
    ?
    ? ?
    Why does this test exist?
    How should tests be written?
    What type of test is this?

    View Slide

  72. nini lkre
    We should know
    these answers before
    opening the test!

    View Slide

  73. drt byby
    The purpose, rules, and
    structure of each test should
    be immediately clear

    View Slide

  74. kpo qw
    A single test can't address
    all of our motivations

    View Slide

  75. zxc eqr
    Such tests are unclear:

    View Slide

  76. zxc eqr
    • purpose is hazy
    Such tests are unclear:

    View Slide

  77. zxc eqr
    • purpose is hazy
    • rules are debatable
    Such tests are unclear:

    View Slide

  78. zxc eqr
    • purpose is hazy
    • rules are debatable
    • structure is ad hoc
    Such tests are unclear:

    View Slide

  79. tyu fd
    Unclear tests
    cost money

    View Slide

  80. awe voOa
    Unclear tests must
    constantly have their:

    View Slide

  81. awe voOa
    Unclear tests must
    constantly have their:
    • purpose rediscovered

    View Slide

  82. awe voOa
    Unclear tests must
    constantly have their:
    • purpose rediscovered
    • rules renegotiated

    View Slide

  83. awe voOa
    Unclear tests must
    constantly have their:
    • purpose rediscovered
    • rules renegotiated
    • structure reorganized

    View Slide

  84. xmcu peo
    None of those
    activities have value!

    View Slide

  85. ae r
    Every suite should
    promote at most
    type of
    confidence

    View Slide

  86. ae r
    Every suite should
    promote at most
    type of
    confidence

    View Slide

  87. c z
    Every suite should
    promote at most
    type of
    understanding

    View Slide

  88. c z
    Every suite should
    promote at most
    type of
    understanding

    View Slide

  89. suite design
    zxcm nz

    View Slide

  90. oeiz jdk
    Omakase Rails

    View Slide

  91. oeiz jdk
    Omakase Rails
    MiniTest

    View Slide

  92. oeiz jdk
    Omakase Rails
    Views MiniTest

    View Slide

  93. oeiz jdk
    Omakase Rails
    Views MiniTest
    MiniTest

    View Slide

  94. oeiz jdk
    Omakase Rails
    Controllers
    Views MiniTest
    MiniTest

    View Slide

  95. oeiz jdk
    Omakase Rails
    Controllers
    Views MiniTest
    MiniTest
    MiniTest

    View Slide

  96. oeiz jdk
    Omakase Rails
    Controllers
    Views
    Models
    MiniTest
    MiniTest
    MiniTest

    View Slide

  97. oeiz jdk
    Omakase Rails
    Controllers
    Views
    Models
    MiniTest
    MiniTest
    MiniTest

    View Slide

  98. ri xc
    Prime Rails

    View Slide

  99. ri xc
    Prime Rails
    Cucumber

    View Slide

  100. ri xc
    Prime Rails
    UI Cucumber

    View Slide

  101. ri xc
    Prime Rails
    RSpec
    UI Cucumber

    View Slide

  102. ri xc
    Prime Rails
    Views RSpec
    UI Cucumber

    View Slide

  103. ri xc
    Prime Rails
    Views RSpec
    RSpec
    UI Cucumber

    View Slide

  104. ri xc
    Prime Rails
    Controllers
    Views RSpec
    RSpec
    UI Cucumber

    View Slide

  105. ri xc
    Prime Rails
    Controllers
    Views RSpec
    RSpec
    RSpec
    UI Cucumber

    View Slide

  106. ri xc
    Prime Rails
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber

    View Slide

  107. ri xc
    Prime Rails
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber

    View Slide

  108. zpo qw
    "Fast Specs" Rails
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber

    View Slide

  109. zpo qw
    "Fast Specs" Rails
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber
    RSpec

    View Slide

  110. POROs
    zpo qw
    "Fast Specs" Rails
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber
    RSpec

    View Slide

  111. What's the
    problem?
    rRth asd

    View Slide

  112. Redundant test
    coverage
    rRth asd

    View Slide

  113. ri xc
    One model change
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber

    View Slide

  114. ri xc
    One model change
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber

    View Slide

  115. ri xc
    One model change
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber

    View Slide

  116. ri xc
    One model change
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber

    View Slide

  117. ri xc
    One model change
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber

    View Slide

  118. ri xc
    One model change
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber

    View Slide

  119. ri xc
    One model change
    Controllers
    Views
    Models
    RSpec
    RSpec
    RSpec
    UI Cucumber

    View Slide

  120. Which leads to:
    rRth asd

    View Slide

  121. ji cnzm
    Controllers
    Views
    Models
    UI RSpec
    (›°□°ʣ›ớ ᵲᴸᵲ Rails

    View Slide

  122. View Slide

  123. architecture
    jd qr

    View Slide

  124. +
    -

    View Slide

  125. +
    -
    app

    View Slide

  126. +
    -
    router
    web
    app
    #1
    web
    app
    #2
    walrus
    data
    panda
    data
    cage
    control
    service

    View Slide

  127. +
    -
    router
    walrus
    data
    panda
    data
    web
    front-
    end
    web
    back-
    end
    web app #1
    cage
    control
    service

    View Slide

  128. +
    -
    walrus
    data
    web backend
    panda
    data
    router
    Feed Zoo
    Animals
    Build Feed
    Plan
    Provide
    Meal
    Fetch
    Schedule
    Plan Meal
    Animal
    Adapter
    Compute
    Portions
    cage
    control
    service
    Release
    Food
    Cage
    Adapter

    View Slide

  129. +
    -
    class ProvidesMeal
    def initialize...
    !
    def provide(plan)
    @releases_food.release(
    @computes_portion.compute(plan)
    )
    end
    end
    provides_meal.rb

    View Slide

  130. suite design
    slp er

    View Slide

  131. +
    -
    class ProvidesMeal
    def initialize...
    !
    def provide(plan)
    @releases_food.release(
    @computes_portion.compute(plan)
    )
    end
    end
    provides_meal.rb

    View Slide

  132. +
    -
    walrus
    data
    web backend
    panda
    data
    router
    Feed Zoo
    Animals
    Build Feed
    Plan
    Provide
    Meal
    Fetch
    Schedule
    Plan Meal
    Animal
    Adapter
    Compute
    Portions
    cage
    control
    service
    Release
    Food
    Cage
    Adapter

    View Slide

  133. +
    -
    router
    walrus
    data
    panda
    data
    web
    front-
    end
    web
    back-
    end
    web app #1
    cage
    control
    service

    View Slide

  134. +
    -
    router
    web
    app
    #1
    web
    app
    #2
    walrus
    data
    panda
    data
    cage
    control
    service

    View Slide

  135. +
    -
    app

    View Slide

  136. +
    -

    View Slide

  137. The greatest
    test of all:
    dk rrt

    View Slide

  138. Does our software
    serve its purpose?
    qwe rtb

    View Slide

  139. Wait… what
    is its purpose?
    zl dfr

    View Slide

  140. Generating
    revenue?
    asd er

    View Slide

  141. Cutting
    costs?
    eit dfp

    View Slide

  142. Limiting our
    liability?
    xcg ke

    View Slide

  143. Funding
    non-profits?
    xd asdasdrg

    View Slide

  144. Can we write a test
    for those outcomes?
    zc er

    View Slide

  145. Maybe!*
    zc er

    View Slide

  146. Maybe!*
    zc er
    *But not in this talk.

    View Slide

  147. +
    -

    View Slide

  148. +
    -
    app

    View Slide

  149. Smoke tests
    oi fg
    app

    View Slide

  150. Acceptance tests
    zxc cxoije
    app

    View Slide

  151. Feature tests
    rv cm
    app

    View Slide

  152. End-to-end tests
    zoc qwe
    app

    View Slide

  153. Integration tests
    xu vll
    app

    View Slide

  154. Integration tests
    xu vll
    app

    View Slide

  155. as cjhf
    app
    Smoke
    Acceptance
    Feature
    End-to-end

    View Slide

  156. a cv
    app
    Smoke
    Acceptance
    Feature
    End-to-end

    View Slide

  157. ed cx
    app
    S
    A
    F
    E

    View Slide

  158. zfh ie
    app
    SAFE tests

    View Slide

  159. The test's "user"

    View Slide

  160. Confidence sought

    View Slide

  161. Understanding gained

    View Slide

  162. Guidelines

    View Slide

  163. Warning Signs

    View Slide

  164. app
    ec ze

    View Slide

  165. SAFE

    View Slide

  166. A real-world user
    SAFE

    View Slide

  167. web apps
    app
    SAFE

    View Slide

  168. http services
    app
    $ curl http://
    SAFE

    View Slide

  169. SAFE

    View Slide

  170. Application works when
    it's all glued together
    SAFE

    View Slide

  171. SAFE

    View Slide

  172. "How simple is
    our product?"
    SAFE

    View Slide

  173. Can't fit your tests
    in 30 minutes?
    SAFE

    View Slide

  174. Can't fit your tests
    in 30 minutes?
    SAFE
    The app is complex

    View Slide

  175. SAFE
    Can't write a test
    in 30 minutes?

    View Slide

  176. SAFE
    Can't write a test
    in 30 minutes?
    The app is complicated

    View Slide

  177. SAFE

    View Slide

  178. SAFE tests shouldn't
    know internal APIs
    SAFE

    View Slide

  179. Web tests should bind to
    user-visible text, not markup
    SAFE

    View Slide

  180. Enforce a fixed-time budget
    SAFE

    View Slide

  181. Time-box the suite
    5 minutes
    5 minutes
    SAFE

    View Slide

  182. Time-box the suite
    5 minutes
    5 minutes
    monthly
    financials
    SAFE

    View Slide

  183. Time-box the suite
    5 minutes
    5 minutes
    monthly
    financials
    search
    SAFE

    View Slide

  184. Time-box the suite
    5 minutes
    5 minutes
    monthly
    financials
    process
    order
    search
    SAFE

    View Slide

  185. Time-box the suite
    5 minutes
    5 minutes
    monthly
    financials
    process
    order ad
    words
    search
    SAFE

    View Slide

  186. Time-box the suite
    5 minutes
    5 minutes
    sign-up
    monthly
    financials
    process
    order ad
    words
    search
    SAFE

    View Slide

  187. Time-box the suite
    5 minutes
    5 minutes
    sign-up
    monthly
    financials
    process
    order ad
    words
    search
    daily
    deals
    SAFE

    View Slide

  188. Time-box the suite
    5 minutes
    5 minutes
    sign-up
    monthly
    financials
    process
    order ad
    words
    search
    daily
    deals
    ?
    SAFE

    View Slide

  189. SAFE

    View Slide

  190. SAFE
    Failures due to refactors
    are false negatives

    View Slide

  191. SAFE
    Beware: human intuition
    overvalues "realistic" tests

    View Slide

  192. SAFE
    Building multiple releases
    & branches is expensive

    View Slide

  193. SAFE
    Superlinear Slowdown

    View Slide

  194. SAFE
    Superlinear Slowdown
    More
    tests
    Slower
    build

    View Slide

  195. SAFE
    Superlinear Slowdown

    View Slide

  196. SAFE
    Superlinear Slowdown
    Bigger
    System
    Slower
    build

    View Slide

  197. !


    !



    !








    !



    !



    !!

    SAFE

    View Slide

  198. !


    !



    !








    !



    !



    !!

    SAFE

    View Slide

  199. !


    !



    !








    !



    !



    !!

    SAFE

    View Slide

  200. +
    -
    app

    View Slide

  201. +
    -
    router
    web
    app
    #1
    web
    app
    #2
    walrus
    data
    panda
    data
    cage
    control
    service

    View Slide

  202. +
    -
    router
    walrus
    data
    panda
    data
    web
    front-
    end
    web
    back-
    end
    web app #1
    cage
    control
    service

    View Slide

  203. Good SAFE tests
    are accurate
    ef zc

    View Slide

  204. But they're
    also expensive
    asd tg

    View Slide

  205. And they don't inform
    our public API's design
    ef gi

    View Slide

  206. Every library or service you write
    is consumed by somebody
    tg gz

    View Slide

  207. Verify behavior & demonstrate
    usage with Consumption tests
    rRth asd

    View Slide

  208. Consumption

    View Slide

  209. Your 's consumer
    Consumption

    View Slide

  210. Your 's consumer
    Consumption
    service

    View Slide

  211. Your 's consumer
    Consumption
    library

    View Slide

  212. Your 's consumer
    Consumption
    user interface

    View Slide

  213. Your 's consumer
    Consumption
    repository

    View Slide

  214. Consumption

    View Slide

  215. Verifies behavior you're
    directly responsible for
    Consumption

    View Slide

  216. Consumption

    View Slide

  217. Answers:
    "Is it usable?"
    Consumption

    View Slide

  218. If it's hard to test,
    then it's hard to use
    Consumption

    View Slide

  219. Consumption

    View Slide

  220. Module boundaries should be
    meaningful beyond testing
    Consumption

    View Slide

  221. Consumption
    Fakes all external dependencies

    View Slide

  222. Consumption
    Fakes all external dependencies
    (i.e. can run without
    access to other services)

    View Slide

  223. Consumption
    Fakes all external dependencies
    (i.e. can run without
    access to other services)
    (e.g. in Travis CI)

    View Slide

  224. Exercises public,
    not private APIs
    Consumption

    View Slide

  225. Organized by consumers'
    desired outcomes
    Consumption

    View Slide

  226. Consumption

    View Slide

  227. Keep consumption
    tests really fast
    Consumption

    View Slide

  228. faked
    services
    Consumption
    no good
    reason to
    be slow

    View Slide

  229. The only suite that says
    you just broke something
    Consumption

    View Slide

  230. !


    !



    !








    !



    !



    !!

    Consumption

    View Slide

  231. !


    !



    !








    !



    !



    !!

    Consumption

    View Slide

  232. !


    !



    !








    !



    !



    !!

    Consumption

    View Slide

  233. +
    -
    router
    walrus
    data
    panda
    data
    web
    front-
    end
    web
    back-
    end
    web app #1
    cage
    control
    service

    View Slide

  234. +
    -
    router
    web
    app
    #1
    web
    app
    #2
    walrus
    data
    panda
    data
    cage
    control
    service

    View Slide

  235. "If my tests fake out
    WalrusService
    how can I trust that
    GET /walruses
    still returns walruses?"
    xc r

    View Slide

  236. Why not test across
    service boundaries?
    q od

    View Slide

  237. Inter-service tests are easy!
    cd fe

    View Slide

  238. Inter-service tests are easy!
    cd fe
    1. Start by not writing them

    View Slide

  239. Inter-service tests are easy!
    cd fe
    1. Start by not writing them
    2. Continue to not write them

    View Slide

  240. Inter-service tests are easy!
    cd fe
    1. Start by not writing them
    2. Continue to not write them
    3. You're already done. Easy!

    View Slide

  241. qwe sd
    Inter-service tests are tempting,
    but they're .

    View Slide

  242. Inter-service tests are tempting,
    but they're .
    hard to set up
    qwe sd

    View Slide

  243. Inter-service tests are tempting,
    but they're .
    slow to run
    qwe sd

    View Slide

  244. Inter-service tests are tempting,
    but they're .
    redundant coverage
    qwe sd

    View Slide

  245. The root question
    indicates a lack of trust
    aie wq

    View Slide

  246. Default to trusting your
    depended-on services
    zxc see

    View Slide

  247. ... but sometimes trust
    isn't appropriate!
    poae fg

    View Slide

  248. Sometimes our usage is
    abnormal or unsupported
    xcg asd

    View Slide

  249. Sometimes other teams
    demonstrate blind-spots
    iocv hgt

    View Slide

  250. When trust isn't enough,
    try writing Contract tests
    zx rrr

    View Slide

  251. Contract

    View Slide

  252. The "user" is us!
    Contract

    View Slide

  253. Test is written to
    represent your interests
    in somebody else's repo
    Contract

    View Slide

  254. Your needs become part
    of their Consumption suite
    Contract

    View Slide

  255. If they break your test,
    they know to call you!
    Contract

    View Slide

  256. Speedier Feedback
    Contract
    fast
    slow

    View Slide

  257. Speedier Feedback
    Contract
    production
    release
    fast
    slow

    View Slide

  258. Speedier Feedback
    Contract
    production
    release
    git push
    fast
    slow

    View Slide

  259. Contract

    View Slide

  260. Dependencies that we own
    behave how we need them to
    Contract

    View Slide

  261. Contract

    View Slide

  262. We learn: "is this service
    a good fit for us?"
    Contract

    View Slide

  263. Frequent failures may
    reveal differing priorities
    Contract

    View Slide

  264. The maintainer learns:
    "are people using our service
    as we expect (or desire)?"
    Contract

    View Slide

  265. Maybe this isn't
    the best service
    for them
    Contract

    View Slide

  266. Maybe this isn't
    the best service
    period.
    Contract
    ,

    View Slide

  267. Contract

    View Slide

  268. Written just like
    consumption tests
    Contract

    View Slide

  269. Committed into the repo
    of the depended-on service
    Contract

    View Slide

  270. Maintainers can lower barrier of
    entry by providing conveniences
    Contract

    View Slide

  271. Contract

    View Slide

  272. Not a replacement
    for human interaction
    Contract

    View Slide

  273. !


    !



    !








    !



    !



    !!

    Contract

    View Slide

  274. !


    !



    !








    !



    !



    !!

    Contract

    View Slide

  275. !


    !



    !








    !



    !



    !!

    Contract

    View Slide

  276. +
    -
    router
    web
    app
    #1
    web
    app
    #2
    walrus
    data
    panda
    data
    cage
    control
    service

    View Slide

  277. +
    -
    router
    walrus
    data
    panda
    data
    web
    front-
    end
    web
    back-
    end
    web app #1
    cage
    control
    service

    View Slide

  278. +
    -
    walrus
    data
    web backend
    panda
    data
    router
    Feed Zoo
    Animals
    Build Feed
    Plan
    Provide
    Meal
    Fetch
    Schedule
    Plan Meal
    Animal
    Adapter
    Compute
    Portions
    cage
    control
    service
    Release
    Food
    Cage
    Adapter

    View Slide

  279. +
    -
    provides_meal.rb
    class ProvidesMeal
    def initialize...
    !
    def provide(plan)
    @releases_food.release(
    @computes_portion.compute(plan)
    )
    end
    end

    View Slide

  280. What about TDD?
    zxc ei

    View Slide

  281. "What's the point?"
    eq rf

    View Slide

  282. "We already have
    lots of tests!"
    xc fe

    View Slide

  283. So far, no feedback
    about code's design
    ei kt

    View Slide

  284. You already have a
    definition for "unit" test
    eq xc

    View Slide

  285. I wrote about this
    a few months ago
    xi x

    View Slide

  286. I wrote about this
    a few months ago
    xi x
    (just Google "tdd failure")

    View Slide

  287. View Slide

  288. It even elicited a
    response from Uncle Bob!
    qw rf

    View Slide

  289. I will refute his arguments
    and utterly destroy his
    conclusions. And then, once
    I'm done salting the ground
    where he used to live, ...l tell
    you why I completely agree
    with him.
    - Uncle Bob
    asdz ty

    View Slide

  290. I will refute his arguments
    and utterly destroy his
    conclusions. And then, once
    I'm done salting the ground
    where he used to live, I'll tell
    you why I completely agree
    with him.
    - Uncle Bob
    asdz ty

    View Slide

  291. There's no
    "one true TDD"
    iz rgr

    View Slide

  292. The principal value of TDD
    in this model is discovery
    vj sdf

    View Slide

  293. Discovery of tiny, boring,
    consistent units that break
    big, scary problems down
    into small, manageable ones
    ri gh

    View Slide

  294. Discovery

    View Slide

  295. The first caller of
    a new method
    Discovery

    View Slide

  296. Concerned primarily
    with inputs and output
    (or side effect)
    Discovery

    View Slide

  297. Also concerned with
    basic code design details
    Discovery

    View Slide

  298. Inputs & Outcome
    Discovery
    Feed Zoo
    Animals

    View Slide

  299. Inputs & Outcome
    Discovery
    Feed Zoo
    Animals
    (walrus, cheese)

    View Slide

  300. Inputs & Outcome
    Discovery
    Feed Zoo
    Animals
    (walrus, cheese) Walrus has
    cheese

    View Slide

  301. Discovery
    Now what?!

    View Slide

  302. Discovery
    describe FeedsAnimals do
    end

    View Slide

  303. Discovery
    describe FeedsAnimals do
    describe "#feed" do
    end
    end

    View Slide

  304. Discovery
    describe FeedsAnimals do
    describe "#feed" do
    expect(my_face).to look_like(ಠ_ಠ)
    end
    end

    View Slide

  305. Discovery
    Feed Zoo
    Animals
    Code by wishful thinking!

    View Slide

  306. Discovery
    Build Feed
    Plan
    Provide
    Meal
    Feed Zoo
    Animals
    Code by wishful thinking!

    View Slide

  307. Discovery
    The test is our sounding board
    Build Feed
    Plan
    Provide
    Meal
    Feed Zoo
    Animals

    View Slide

  308. Discovery
    describe FeedsAnimals do
    !
    !
    !
    end

    View Slide

  309. Discovery
    describe FeedsAnimals do
    Given(:builds_feed_plan) { gimme(BuildsFeedPlan) }
    !
    !
    end

    View Slide

  310. Discovery
    describe FeedsAnimals do
    Given(:builds_feed_plan) { gimme(BuildsFeedPlan) }
    Given(:provides_meal) { gimme(ProvidesMeal) }
    !
    !
    end

    View Slide

  311. Discovery
    describe FeedsAnimals do
    Given(:builds_feed_plan) { gimme(BuildsFeedPlan) }
    Given(:provides_meal) { gimme(ProvidesMeal) }
    !
    subject { FeedsAnimals.new(builds_feed_plan, provides_meal) }
    !
    end

    View Slide

  312. Discovery
    describe FeedsAnimals do
    Given(:builds_feed_plan) { gimme(BuildsFeedPlan) }
    Given(:provides_meal) { gimme(ProvidesMeal) }
    !
    subject { FeedsAnimals.new(builds_feed_plan, provides_meal) }
    !
    describe "#feed" do
    end
    end

    View Slide

  313. Discovery
    describe FeedsAnimals do
    Given(:builds_feed_plan) { gimme(BuildsFeedPlan) }
    Given(:provides_meal) { gimme(ProvidesMeal) }
    !
    subject { FeedsAnimals.new(builds_feed_plan, provides_meal) }
    !
    describe "#feed" do
    When(:result) { subject.feed(:walrus, :cheese) }
    end
    end

    View Slide

  314. Discovery
    describe FeedsAnimals do
    Given(:builds_feed_plan) { gimme(BuildsFeedPlan) }
    Given(:provides_meal) { gimme(ProvidesMeal) }
    !
    subject { FeedsAnimals.new(builds_feed_plan, provides_meal) }
    !
    describe "#feed" do
    When(:result) { subject.feed(:walrus, :cheese) }
    Then { verify(provides_meal).provide(:cheese, :plan) }
    end
    end

    View Slide

  315. Discovery
    describe FeedsAnimals do
    Given(:builds_feed_plan) { gimme(BuildsFeedPlan) }
    Given(:provides_meal) { gimme(ProvidesMeal) }
    !
    subject { FeedsAnimals.new(builds_feed_plan, provides_meal) }
    !
    describe "#feed" do
    Given { give(builds_feed_plan).build(:walrus) {:plan}}
    When(:result) { subject.feed(:walrus, :cheese) }
    Then { verify(provides_meal).provide(:cheese, :plan) }
    end
    end

    View Slide

  316. Discovery
    describe FeedsAnimals do
    Given(:builds_feed_plan) { gimme(BuildsFeedPlan) }
    Given(:provides_meal) { gimme(ProvidesMeal) }
    !
    subject { FeedsAnimals.new(builds_feed_plan, provides_meal) }
    !
    describe "#feed" do
    Given { give(builds_feed_plan).build(:walrus) {:plan}}
    When(:result) { subject.feed(:walrus, :cheese) }
    Then { verify(provides_meal).provide(:cheese, :plan) }
    end
    end

    View Slide

  317. Discovery
    Now I have two problems!
    Build Feed
    Plan
    Provide
    Meal

    View Slide

  318. Discovery
    Now I have two problems!
    Build Feed
    Plan
    Provide
    Meal


    View Slide

  319. Discovery

    View Slide

  320. Correct behavior of
    logical leaf nodes
    Discovery

    View Slide

  321. (a.k.a. pure, first-
    order functions)
    Discovery

    View Slide

  322. Discovery
    Feed Zoo
    Animals
    Build Feed
    Plan
    Provide
    Meal

    View Slide

  323. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    Build Feed
    Plan
    Provide
    Meal

    View Slide

  324. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    Compute
    Portions
    Release
    Food
    Build Feed
    Plan
    Provide
    Meal

    View Slide

  325. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    Compute
    Portions
    Release
    Food
    Build Feed
    Plan
    Provide
    Meal

    View Slide

  326. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    ✔️
    Compute
    Portions
    Release
    Food
    Build Feed
    Plan
    Provide
    Meal

    View Slide

  327. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    ✔️
    Compute
    Portions
    Release
    Food
    ✔️
    Build Feed
    Plan
    Provide
    Meal

    View Slide

  328. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    ✔️
    Compute
    Portions
    Release
    Food
    ✔️
    Build Feed
    Plan
    Provide
    Meal

    View Slide

  329. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    ✔️
    Compute
    Portions
    Release
    Food
    ✔️
    Build Feed
    Plan
    Provide
    Meal

    View Slide

  330. Discovery

    View Slide

  331. Small things
    Discovery

    View Slide

  332. Small things
    Discovery
    (Free SRP!)

    View Slide

  333. Separation of Roles
    Discovery

    View Slide

  334. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    Compute
    Portions
    Release
    Food
    Build Feed
    Plan
    Provide
    Meal

    View Slide

  335. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    Compute
    Portions
    Release
    Food
    Build Feed
    Plan
    Provide
    Meal
    Command

    View Slide

  336. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    Compute
    Portions
    Release
    Food
    Build Feed
    Plan
    Provide
    Meal
    Command
    Query

    View Slide

  337. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    Compute
    Portions
    Release
    Food
    Build Feed
    Plan
    Provide
    Meal
    Command
    Query Command

    View Slide

  338. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    Compute
    Portions
    Release
    Food
    Build Feed
    Plan
    Provide
    Meal
    Command
    Query
    Query
    Command

    View Slide

  339. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    Compute
    Portions
    Release
    Food
    Build Feed
    Plan
    Provide
    Meal
    Command
    Query
    Query Logic
    Command

    View Slide

  340. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    Compute
    Portions
    Release
    Food
    Build Feed
    Plan
    Provide
    Meal
    Command
    Query
    Query Logic Logic
    Command

    View Slide

  341. Discovery
    Feed Zoo
    Animals
    Fetch
    Schedule
    Plan Meal
    Compute
    Portions
    Release
    Food
    Build Feed
    Plan
    Provide
    Meal
    Command
    Query
    Query Logic Logic Command
    Command

    View Slide

  342. Discovery

    View Slide

  343. Command & Query tests discover
    dependencies with test doubles
    Discovery

    View Slide

  344. Discovery
    Logic tests discover
    implementation by usage

    View Slide

  345. Discovery
    (i.e. no test doubles)
    Logic tests discover
    implementation by usage

    View Slide

  346. Commands and Queries
    contain very little logic
    Discovery

    View Slide

  347. Pain is good!
    Discovery

    View Slide

  348. Discovery

    View Slide

  349. Discovery tests yield
    small, disposable units
    Discovery

    View Slide

  350. Discovery tests yield
    small, disposable units
    Discovery
    ...so don't be afraid
    to throw them away!

    View Slide

  351. If requirements change, trash
    a minimal sub-tree of units
    and drive out a new solution
    Discovery

    View Slide

  352. Re-use is
    overrated
    Discovery

    View Slide

  353. Extract refactors
    are a smell
    Discovery

    View Slide

  354. Frameworks will
    fight you!
    Discovery

    View Slide

  355. Frameworks & TDD try to
    solve the same problems
    Discovery

    View Slide

  356. Frameworks provide
    an orderly structure
    Discovery

    View Slide

  357. TDD is an approach
    for discovering an
    orderly structure
    Discovery

    View Slide

  358. Discovery testing is at odds
    with most app frameworks
    Discovery

    View Slide

  359. !


    !



    !








    !



    !



    !!

    Discovery

    View Slide

  360. !


    !



    !








    !



    !



    !!

    Discovery

    View Slide

  361. !


    !



    !








    !



    !



    !!

    Discovery

    View Slide

  362. +
    -
    provides_meal.rb
    class ProvidesMeal
    def initialize...
    !
    def provide(plan)
    @releases_food.release(
    @computes_portion.compute(plan)
    )
    end
    end

    View Slide

  363. +
    -
    walrus
    data
    web backend
    panda
    data
    router
    Feed Zoo
    Animals
    Build Feed
    Plan
    Provide
    Meal
    Fetch
    Schedule
    Plan Meal
    Animal
    Adapter
    Compute
    Portions
    cage
    control
    service
    Release
    Food
    Cage
    Adapter

    View Slide

  364. "Wow, that's a lot
    of !"
    zxc tg
    mock objects

    View Slide

  365. "Wow, that's a lot
    of !"
    fake stuff
    zxc tg

    View Slide

  366. "Wow, that's a lot
    of !"
    test doubles
    zxc tg

    View Slide

  367. "Aren't those many
    test doubles a pain?"
    ad er

    View Slide

  368. They can be!
    asd xc

    View Slide

  369. Note: only use doubles
    when testing collaborators
    g rg

    View Slide

  370. Note: better test double
    libraries are better
    gr cz

    View Slide

  371. kdasd fk
    Discovery tests try to
    improve dependencies' APIs

    View Slide

  372. kg er
    But we can't change
    3rd party APIs

    View Slide

  373. er zxc
    Mocking what you don't
    own leads to useless pain

    View Slide

  374. Wrap 3rd party API calls in
    adapters that you do own
    uc vt

    View Slide

  375. Wrap 3rd party API calls in
    adapters that you do own
    uc vt
    (And mock those adapters instead)

    View Slide

  376. Typically, adapters
    shouldn't need tests
    zx bt

    View Slide

  377. Most tests exercise features
    that make use 3rd party APIs
    jd hn

    View Slide

  378. It sometimes make sense to test
    code we don't own separately
    jd hn

    View Slide

  379. When you want to respond to 3rd party
    failures differently, write Adapter tests
    jd hn

    View Slide

  380. Adapter

    View Slide

  381. Your application, pondering its
    relationship with a 3rd party API
    Adapter

    View Slide

  382. Exercises your adapter API under
    realistic-enough circumstances
    Adapter

    View Slide

  383. Adapter

    View Slide

  384. Adapter
    Tests of libraries warn
    of unsafe upgrades

    View Slide

  385. Adapter
    Tests of network services warn
    of outages & breaking changes

    View Slide

  386. Adapter

    View Slide

  387. Adapters (& tests) specify
    how you depend on a thing
    Adapter

    View Slide

  388. Establishes boundaries, prevents
    3rd party API references from
    seeping throughout your app
    Adapter

    View Slide

  389. Drastically reduces the cost of
    replacing dependencies later
    Adapter

    View Slide

  390. Adapter

    View Slide

  391. Only test adapters
    with good cause
    Adapter

    View Slide

  392. "Don't test the
    framework."
    Adapter

    View Slide

  393. Adapter

    View Slide

  394. Adapter test suites of
    network code can be
    tricky to run in CI
    Adapter

    View Slide

  395. It's easier to test 3rd party
    services from your SAFE suite
    Adapter

    View Slide

  396. All adapter tests are
    likely to be slow in ways
    outside your control
    Adapter

    View Slide

  397. !


    !



    !








    !



    !



    !!

    Adapter

    View Slide

  398. !


    !



    !








    !



    !



    !!

    Adapter

    View Slide

  399. !


    !



    !








    !



    !



    !!

    Adapter

    View Slide

  400. And that's just
    way to break
    up your test suite
    asd

    View Slide

  401. And that's just
    way to break
    up your test suite
    asd

    View Slide

  402. My name is Justin Searls
    Please tweet me @searls &
    Say [email protected]

    View Slide

  403. Like everyone, we're hiring!
    Just [email protected]

    View Slide

  404. Please say hello if your team
    could use our team's help #

    View Slide

  405. Quotation Marks designed by Horacio Bella from the thenounproject.com
    !
    Happy designed by Michael Hourigan from the thenounproject.com
    !
    Tired designed by Aha-Soft from the thenounproject.com
    !
    Indifference designed by Paul F. from the thenounproject.com
    !
    Mad designed by Aha-Soft from the thenounproject.com
    !
    Sad designed by Simple Icons from the thenounproject.com
    !
    Marker designed by Jeff Seevers from the thenounproject.com
    !
    Database designed by Grant Fisher from the thenounproject.com
    !
    Cloud designed by Thomas Hirter from the thenounproject.com
    !
    Safety Net designed by irene hoffman from the thenounproject.com
    !
    Light Bulb designed by Olivier Guin from the thenounproject.com
    !
    Scale designed by Stephanie Wauters from the thenounproject.com
    !
    Warning designed by Lorena Salagre from the thenounproject.com
    !
    Reload designed by Mateo Zlatar from the thenounproject.com
    Noun project attribution

    View Slide