Breaking up (with) your test suite

Breaking up (with) your test suite poa asd

My name is Justin Searls Please tweet me @searls &
Say [email protected]

Any questions? Tweet @searls during the talk Favorite question
gets a book!

This talk is about z aPPOOIU

This talk is about z aPPOOIU relationships

I my customers ias al

I coding zffrttyu z

I testing aske zc

We don't test for testing's sake rRth asd

I get paid for code that works, not for tests,
so my philosophy is to test as little as possible to reach a given level of confidence. - Kent Beck asdz ty

We don't code for code's sake, either mmmmznxdfr lr

Writing code is a costly way to solve problems oiz
eu

Too often, we cheat on our customers with our code
c d aoiey

How to cheat on your customers: ae ts

How to cheat on your customers: ae ts 1. Uncritically
assume your favorite way to write code applies to every situation

How to cheat on your customers: ae ts 1. Uncritically
assume your favorite way to write code applies to every situation 2. There is no #2

evh zc Cargo-culting as a service

We must challenge our biases in each new context to
grow fh ywe

iO ZOe infatuation

xmt iRosXc Remember TATFT?

TDD literally rescued me from the pit of despair.

lzxcm zxcmlzcl lzxckz zxczc zxjcozcjoe AOO asodjaosd encz caosdjojo jojo
6 months later ~ ~

Why does every refactor break 6 tests?!

Why does the build take nine hours to run?!

Why are people mocking methods on the object they're testing?!

Why can't we get anything done anymore?

asodo zoxjcz aoo ueue yyyz cxjk zxcczx and eventually ~
~

the break-up

$ the break-up

$ the break-up rm -rf spec features

jn poi introspection

qwer bvt Why test?

! !
! !

! !
! ! !

! !
! ! ! !!

! !
! ! ! !! ! ! ! !

! !
! ! ! !! ! ! ! ! Confidence

! !
! ! ! !! ! !!

! !
! ! ! !! Understanding ! !!

mni pl Single Responsibility Testing

describe "Panda" do end

describe "Panda" do let(:bamboo) { double("bamboo").as_null_object } end

describe "Panda" do let(:bamboo) { double("bamboo").as_null_object } subject { Panda.new(bamboo)
} end

} ! it "munches bamboo" do end end

} ! it "munches bamboo" do subject.eat! end end

} ! it "munches bamboo" do subject.eat! expect(bamboo).to have_received(:munch) end end

asdj zoxc mmr qu psoi rnzc vnkl la later, fixing
a bug ~ ~

describe "Panda" do let(:bamboo) { Bamboo.find(34) } subject { Panda.new(bamboo)

} ! it "munches bamboo" do allow(bamboo).to receive(:munch) subject.eat! expect(bamboo).to have_received(:munch) end end

} ! it "munches bamboo" do allow(bamboo).to receive(:munch) subject.eat! expect(bamboo).to have_received(:munch) end ! it "returns bamboo id" do end end

} ! it "munches bamboo" do allow(bamboo).to receive(:munch) subject.eat! expect(bamboo).to have_received(:munch) end ! it "returns bamboo id" do expect(subject.eat!).to eq(34) end end

asdj zoxc mmr qu psoi rnzc vnkl la one day,
I find it ~ ~

} ! it "munches bamboo" do allow(bamboo).to receive(:munch) subject.eat! expect(bamboo).to have_received(:munch) end ! it "returns bamboo id" do expect(subject.eat!).to eq(34) end end

} ! it "munches bamboo" do allow(bamboo).to receive(:munch) subject.eat! expect(bamboo).to have_received(:munch) end ! it "returns bamboo id" do expect(subject.eat!).to eq(34) end end ? ? ?

} ! it "munches bamboo" do allow(bamboo).to receive(:munch) subject.eat! expect(bamboo).to have_received(:munch) end ! it "returns bamboo id" do expect(subject.eat!).to eq(34) end end ? ? ? Why does this test exist?

} ! it "munches bamboo" do allow(bamboo).to receive(:munch) subject.eat! expect(bamboo).to have_received(:munch) end ! it "returns bamboo id" do expect(subject.eat!).to eq(34) end end ? ? ? Why does this test exist? What type of test is this?

} ! it "munches bamboo" do allow(bamboo).to receive(:munch) subject.eat! expect(bamboo).to have_received(:munch) end ! it "returns bamboo id" do expect(subject.eat!).to eq(34) end end ? ? ? Why does this test exist? How should tests be written? What type of test is this?

nini lkre We should know these answers before opening the
test!

drt byby The purpose, rules, and structure of each test
should be immediately clear

kpo qw A single test can't address all of our
motivations

zxc eqr Such tests are unclear:

zxc eqr • purpose is hazy Such tests are unclear:

zxc eqr • purpose is hazy • rules are debatable
Such tests are unclear:

zxc eqr • purpose is hazy • rules are debatable
• structure is ad hoc Such tests are unclear:

tyu fd Unclear tests cost money

awe voOa Unclear tests must constantly have their:

awe voOa Unclear tests must constantly have their: • purpose
rediscovered

rediscovered • rules renegotiated

rediscovered • rules renegotiated • structure reorganized

xmcu peo None of those activities have value!

ae r Every suite should promote at most type of
confidence

c z Every suite should promote at most type of
understanding

suite design zxcm nz

oeiz jdk Omakase Rails

oeiz jdk Omakase Rails MiniTest

oeiz jdk Omakase Rails Views MiniTest

oeiz jdk Omakase Rails Views MiniTest MiniTest

oeiz jdk Omakase Rails Controllers Views MiniTest MiniTest

oeiz jdk Omakase Rails Controllers Views MiniTest MiniTest MiniTest

oeiz jdk Omakase Rails Controllers Views Models MiniTest MiniTest MiniTest

ri xc Prime Rails

ri xc Prime Rails Cucumber

ri xc Prime Rails UI Cucumber

ri xc Prime Rails RSpec UI Cucumber

ri xc Prime Rails Views RSpec UI Cucumber

ri xc Prime Rails Views RSpec RSpec UI Cucumber

ri xc Prime Rails Controllers Views RSpec RSpec UI Cucumber

ri xc Prime Rails Controllers Views RSpec RSpec RSpec UI
Cucumber

ri xc Prime Rails Controllers Views Models RSpec RSpec RSpec
UI Cucumber

zpo qw "Fast Specs" Rails Controllers Views Models RSpec RSpec
RSpec UI Cucumber

zpo qw "Fast Specs" Rails Controllers Views Models RSpec RSpec
RSpec UI Cucumber RSpec

POROs zpo qw "Fast Specs" Rails Controllers Views Models RSpec
RSpec RSpec UI Cucumber RSpec

What's the problem? rRth asd

Redundant test coverage rRth asd

ri xc One model change Controllers Views Models RSpec RSpec
RSpec UI Cucumber

Which leads to: rRth asd

ji cnzm Controllers Views Models UI RSpec (›°□°ʣ›ớ ᵲᴸᵲ Rails

architecture jd qr

+ - app

+ - router web app #1 web app #2 walrus
data panda data cage control service

+ - router walrus data panda data web front- end
web backend web app #1 cage control service

+ - walrus data web backend panda data router Feed
Zoo Animals Build Feed Plan Provide Meal Fetch Schedule Plan Meal Animal Adapter Compute Portions cage control service Release Food Cage Adapter

+ - class ProvidesMeal def initialize... ! def provide(plan) @releases_food.release(
@computes_portion.compute(plan) ) end end provides_meal.rb

suite design slp er

+ - class ProvidesMeal def initialize... ! def provide(plan) @releases_food.release(
@computes_portion.compute(plan) ) end end provides_meal.rb

+ - app

The greatest test of all: dk rrt

Does our software serve its purpose? qwe rtb

Wait… what is its purpose? zl dfr

Generating revenue? asd er

Cutting costs? eit dfp

Limiting our liability? xcg ke

Funding non-profits? xd asdasdrg

Can we write a test for those outcomes? zc er

Maybe!* zc er

Maybe!* zc er *But not in this talk.

+ - app

Smoke tests oi fg app

Acceptance tests zxc cxoije app

Feature tests rv cm app

End-to-end tests zoc qwe app

Integration tests xu vll app

as cjhf app Smoke Acceptance Feature End-to-end

a cv app Smoke Acceptance Feature End-to-end

ed cx app S A F E

zfh ie app SAFE tests

The test's "user"

Confidence sought

Understanding gained

Guidelines

Warning Signs

app ec ze

A real-world user SAFE

web apps app SAFE

http services app $ curl http:// SAFE

Application works when it's all glued together SAFE

"How simple is our product?" SAFE

Can't fit your tests in 30 minutes? SAFE

Can't fit your tests in 30 minutes? SAFE The app
is complex

SAFE Can't write a test in 30 minutes?

SAFE Can't write a test in 30 minutes? The app
is complicated

SAFE tests shouldn't know internal APIs SAFE

Web tests should bind to user-visible text, not markup SAFE

Enforce a fixed-time budget SAFE

Time-box the suite 5 minutes 5 minutes SAFE

Time-box the suite 5 minutes 5 minutes monthly financials SAFE

Time-box the suite 5 minutes 5 minutes monthly financials search
SAFE

Time-box the suite 5 minutes 5 minutes monthly financials process
order search SAFE

Time-box the suite 5 minutes 5 minutes monthly financials process
order ad words search SAFE

Time-box the suite 5 minutes 5 minutes sign-up monthly financials
process order ad words search SAFE

process order ad words search daily deals SAFE

process order ad words search daily deals ? SAFE

SAFE Failures due to refactors are false negatives

SAFE Beware: human intuition overvalues "realistic" tests

SAFE Building multiple releases & branches is expensive

SAFE Superlinear Slowdown

SAFE Superlinear Slowdown More tests Slower build

SAFE Superlinear Slowdown

SAFE Superlinear Slowdown Bigger System Slower build

! !
! ! ! !! SAFE

+ - app

Good SAFE tests are accurate ef zc

But they're also expensive asd tg

And they don't inform our public API's design ef gi

Every library or service you write is consumed by somebody
tg gz

Verify behavior & demonstrate usage with Consumption tests rRth asd

Consumption

Your 's consumer Consumption

Your 's consumer Consumption service

Your 's consumer Consumption library

Your 's consumer Consumption user interface

Your 's consumer Consumption repository

Consumption

Verifies behavior you're directly responsible for Consumption

Consumption

Answers: "Is it usable?" Consumption

If it's hard to test, then it's hard to use
Consumption

Consumption

Module boundaries should be meaningful beyond testing Consumption

Consumption Fakes all external dependencies

Consumption Fakes all external dependencies (i.e. can run without access
to other services)

Consumption Fakes all external dependencies (i.e. can run without access
to other services) (e.g. in Travis CI)

Exercises public, not private APIs Consumption

Organized by consumers' desired outcomes Consumption

Consumption

Keep consumption tests really fast Consumption

faked services Consumption no good reason to be slow

The only suite that says you just broke something Consumption

! !
! ! ! !! Consumption

"If my tests fake out WalrusService how can I trust
that GET /walruses still returns walruses?" xc r

Why not test across service boundaries? q od

Inter-service tests are easy! cd fe

Inter-service tests are easy! cd fe 1. Start by not
writing them

writing them 2. Continue to not write them

writing them 2. Continue to not write them 3. You're already done. Easy!

qwe sd Inter-service tests are tempting, but they're .

Inter-service tests are tempting, but they're . hard to set
up qwe sd

Inter-service tests are tempting, but they're . slow to run
qwe sd

Inter-service tests are tempting, but they're . redundant coverage qwe
sd

The root question indicates a lack of trust aie wq

Default to trusting your depended-on services zxc see

... but sometimes trust isn't appropriate! poae fg

Sometimes our usage is abnormal or unsupported xcg asd

Sometimes other teams demonstrate blind-spots iocv hgt

When trust isn't enough, try writing Contract tests zx rrr

Contract

The "user" is us! Contract

Test is written to represent your interests in somebody else's
repo Contract

Your needs become part of their Consumption suite Contract

If they break your test, they know to call you!
Contract

Speedier Feedback Contract fast slow

Speedier Feedback Contract production release fast slow

Speedier Feedback Contract production release git push fast slow

Contract

Dependencies that we own behave how we need them to
Contract

Contract

We learn: "is this service a good fit for us?"
Contract

Frequent failures may reveal differing priorities Contract

The maintainer learns: "are people using our service as we
expect (or desire)?" Contract

Maybe this isn't the best service for them Contract

Maybe this isn't the best service period. Contract ,

Contract

Written just like consumption tests Contract

Committed into the repo of the depended-on service Contract

Maintainers can lower barrier of entry by providing conveniences Contract

Contract

Not a replacement for human interaction Contract

! !
! ! ! !! Contract

+ - provides_meal.rb class ProvidesMeal def initialize... ! def provide(plan)
@releases_food.release( @computes_portion.compute(plan) ) end end

What about TDD? zxc ei

"What's the point?" eq rf

"We already have lots of tests!" xc fe

So far, no feedback about code's design ei kt

You already have a definition for "unit" test eq xc

I wrote about this a few months ago xi x

I wrote about this a few months ago xi x
(just Google "tdd failure")

It even elicited a response from Uncle Bob! qw rf

I will refute his arguments and utterly destroy his conclusions.
And then, once I'm done salting the ground where he used to live, ...l tell you why I completely agree with him. - Uncle Bob asdz ty

I will refute his arguments and utterly destroy his conclusions.
And then, once I'm done salting the ground where he used to live, I'll tell you why I completely agree with him. - Uncle Bob asdz ty

There's no "one true TDD" iz rgr

The principal value of TDD in this model is discovery
vj sdf

Discovery of tiny, boring, consistent units that break big, scary
problems down into small, manageable ones ri gh

Discovery

The first caller of a new method Discovery

Concerned primarily with inputs and output (or side effect) Discovery

Also concerned with basic code design details Discovery

Inputs & Outcome Discovery Feed Zoo Animals

Inputs & Outcome Discovery Feed Zoo Animals (walrus, cheese)

Inputs & Outcome Discovery Feed Zoo Animals (walrus, cheese) Walrus
has cheese

Discovery Now what?!

Discovery describe FeedsAnimals do end

Discovery describe FeedsAnimals do describe "#feed" do end end

Discovery describe FeedsAnimals do describe "#feed" do expect(my_face).to look_like(ಠ_ಠ) end
end

Discovery Feed Zoo Animals Code by wishful thinking!

Discovery Build Feed Plan Provide Meal Feed Zoo Animals Code
by wishful thinking!

Discovery The test is our sounding board Build Feed Plan
Provide Meal Feed Zoo Animals

Discovery describe FeedsAnimals do ! ! ! end

Discovery describe FeedsAnimals do Given(:builds_feed_plan) { gimme(BuildsFeedPlan) } ! !
end

Discovery describe FeedsAnimals do Given(:builds_feed_plan) { gimme(BuildsFeedPlan) } Given(:provides_meal) {
gimme(ProvidesMeal) } ! ! end

gimme(ProvidesMeal) } ! subject { FeedsAnimals.new(builds_feed_plan, provides_meal) } ! end

gimme(ProvidesMeal) } ! subject { FeedsAnimals.new(builds_feed_plan, provides_meal) } ! describe "#feed" do end end

gimme(ProvidesMeal) } ! subject { FeedsAnimals.new(builds_feed_plan, provides_meal) } ! describe "#feed" do When(:result) { subject.feed(:walrus, :cheese) } end end

gimme(ProvidesMeal) } ! subject { FeedsAnimals.new(builds_feed_plan, provides_meal) } ! describe "#feed" do When(:result) { subject.feed(:walrus, :cheese) } Then { verify(provides_meal).provide(:cheese, :plan) } end end

gimme(ProvidesMeal) } ! subject { FeedsAnimals.new(builds_feed_plan, provides_meal) } ! describe "#feed" do Given { give(builds_feed_plan).build(:walrus) {:plan}} When(:result) { subject.feed(:walrus, :cheese) } Then { verify(provides_meal).provide(:cheese, :plan) } end end

Discovery Now I have two problems! Build Feed Plan Provide
Meal

Discovery

Correct behavior of logical leaf nodes Discovery

(a.k.a. pure, first- order functions) Discovery

Discovery Feed Zoo Animals Build Feed Plan Provide Meal

Discovery Feed Zoo Animals Fetch Schedule Plan Meal Build Feed
Plan Provide Meal

Discovery Feed Zoo Animals Fetch Schedule Plan Meal Compute Portions
Release Food Build Feed Plan Provide Meal

Discovery Feed Zoo Animals Fetch Schedule Plan Meal ✔️ Compute
Portions Release Food Build Feed Plan Provide Meal

Discovery Feed Zoo Animals Fetch Schedule Plan Meal ✔️ Compute
Portions Release Food ✔️ Build Feed Plan Provide Meal

Discovery

Small things Discovery

Small things Discovery (Free SRP!)

Separation of Roles Discovery

Release Food Build Feed Plan Provide Meal

Release Food Build Feed Plan Provide Meal Command

Release Food Build Feed Plan Provide Meal Command Query

Release Food Build Feed Plan Provide Meal Command Query Command

Release Food Build Feed Plan Provide Meal Command Query Query Command

Release Food Build Feed Plan Provide Meal Command Query Query Logic Command

Release Food Build Feed Plan Provide Meal Command Query Query Logic Logic Command

Release Food Build Feed Plan Provide Meal Command Query Query Logic Logic Command Command

Discovery

Command & Query tests discover dependencies with test doubles Discovery

Discovery Logic tests discover implementation by usage

Discovery (i.e. no test doubles) Logic tests discover implementation by
usage

Commands and Queries contain very little logic Discovery

Pain is good! Discovery

Discovery

Discovery tests yield small, disposable units Discovery

Discovery tests yield small, disposable units Discovery ...so don't be
afraid to throw them away!

If requirements change, trash a minimal sub-tree of units and
drive out a new solution Discovery

Re-use is overrated Discovery

Extract refactors are a smell Discovery

Frameworks will fight you! Discovery

Frameworks & TDD try to solve the same problems Discovery

Frameworks provide an orderly structure Discovery

TDD is an approach for discovering an orderly structure Discovery

Discovery testing is at odds with most app frameworks Discovery

! !
! ! ! !! Discovery

+ - provides_meal.rb class ProvidesMeal def initialize... ! def provide(plan)
@releases_food.release( @computes_portion.compute(plan) ) end end

"Wow, that's a lot of !" zxc tg mock objects

"Wow, that's a lot of !" fake stuff zxc tg

"Wow, that's a lot of !" test doubles zxc tg

"Aren't those many test doubles a pain?" ad er

They can be! asd xc

Note: only use doubles when testing collaborators g rg

Note: better test double libraries are better gr cz

kdasd fk Discovery tests try to improve dependencies' APIs

kg er But we can't change 3rd party APIs

er zxc Mocking what you don't own leads to useless
pain

Wrap 3rd party API calls in adapters that you do
own uc vt

Wrap 3rd party API calls in adapters that you do
own uc vt (And mock those adapters instead)

Typically, adapters shouldn't need tests zx bt

Most tests exercise features that make use 3rd party APIs
jd hn

It sometimes make sense to test code we don't own
separately jd hn

When you want to respond to 3rd party failures differently,
write Adapter tests jd hn

Adapter

Your application, pondering its relationship with a 3rd party API
Adapter

Exercises your adapter API under realistic-enough circumstances Adapter

Adapter

Adapter Tests of libraries warn of unsafe upgrades

Adapter Tests of network services warn of outages & breaking
changes

Adapter

Adapters (& tests) specify how you depend on a thing
Adapter

Establishes boundaries, prevents 3rd party API references from seeping throughout
your app Adapter

Drastically reduces the cost of replacing dependencies later Adapter

Adapter

Only test adapters with good cause Adapter

"Don't test the framework." Adapter

Adapter

Adapter test suites of network code can be tricky to
run in CI Adapter

It's easier to test 3rd party services from your SAFE
suite Adapter

All adapter tests are likely to be slow in ways
outside your control Adapter

! !
! ! ! !! Adapter

And that's just way to break up your test suite
asd

My name is Justin Searls Please tweet me @searls &
Say [email protected]

Like everyone, we're hiring! Just [email protected]

Please say hello if your team could use our team's
help #

Quotation Marks designed by Horacio Bella from the thenounproject.com !
Happy designed by Michael Hourigan from the thenounproject.com ! Tired designed by Aha-Soft from the thenounproject.com ! Indifference designed by Paul F. from the thenounproject.com ! Mad designed by Aha-Soft from the thenounproject.com ! Sad designed by Simple Icons from the thenounproject.com ! Marker designed by Jeff Seevers from the thenounproject.com ! Database designed by Grant Fisher from the thenounproject.com ! Cloud designed by Thomas Hirter from the thenounproject.com ! Safety Net designed by irene hoffman from the thenounproject.com ! Light Bulb designed by Olivier Guin from the thenounproject.com ! Scale designed by Stephanie Wauters from the thenounproject.com ! Warning designed by Lorena Salagre from the thenounproject.com ! Reload designed by Mateo Zlatar from the thenounproject.com Noun project attribution

Breaking up (with) your test suite

Breaking up (with) your test suite

More Decks by Justin Searls

Other Decks in Programming

Featured

Transcript