Slide 1

Slide 1 text

2014 Testing the Untestable @schneems

Slide 2

Slide 2 text

@schneems

Slide 3

Slide 3 text

Ruby Schneems

Slide 4

Slide 4 text

Ruby Python

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Studied ME

Slide 8

Slide 8 text

Thermo Dynamics

Slide 9

Slide 9 text

In school there are answer keys

Slide 10

Slide 10 text

Have you ever flipped the sign?

Slide 11

Slide 11 text

-1000

Slide 12

Slide 12 text

+1000

Slide 13

Slide 13 text

Huge difference

Slide 14

Slide 14 text

What about the real world?

Slide 15

Slide 15 text

Co-Op

Slide 16

Slide 16 text

How does GE calculate designs for a Frige?

Slide 17

Slide 17 text

Freaking

Slide 18

Slide 18 text

Freaking Spreadsheets

Slide 19

Slide 19 text

What if the spreadsheet is wrong?

Slide 20

Slide 20 text

Introducing:

Slide 21

Slide 21 text

Testing!

Slide 22

Slide 22 text

Thermocouples

Slide 23

Slide 23 text

Programmers are lucky

Slide 24

Slide 24 text

Our inputs are known, our outputs are known

Slide 25

Slide 25 text

Our product is a program

Slide 26

Slide 26 text

Our tests are programs

Slide 27

Slide 27 text

Others aren’t so lucky

Slide 28

Slide 28 text

1960’s

Slide 29

Slide 29 text

The US will go to the moon

Slide 30

Slide 30 text

Outer Space is a lifeless vacuum

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

How do you trust the calculations?

Slide 35

Slide 35 text

How do you test…

Slide 36

Slide 36 text

The untestable?

Slide 37

Slide 37 text

Unit Test components

Slide 38

Slide 38 text

At the end of the day

Slide 39

Slide 39 text

You launch it

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Integration Tests

Slide 42

Slide 42 text

“SAFE” Tests

Slide 43

Slide 43 text

Back To Software

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

Ruby Task
 Force

Slide 46

Slide 46 text

Ruby Task
 Force Member

Slide 47

Slide 47 text

Name That year:

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

Heroku Aspen Stack

Slide 52

Slide 52 text

What is a buildpack?

Slide 53

Slide 53 text

credit @zeke

Slide 54

Slide 54 text

4,573 lines of edge cases of edge cases

Slide 55

Slide 55 text

0 tests (Jan 2013)

Slide 56

Slide 56 text

Not to say it wasn’t tested

Slide 57

Slide 57 text

it was just

Slide 58

Slide 58 text

w Manual Testing

Slide 59

Slide 59 text

*

Slide 60

Slide 60 text

We have platform tests (they’re not buildpack specific)

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

MVP

Slide 63

Slide 63 text

Minimum Viable
 Patch

Slide 64

Slide 64 text

The smallest possible code change

Slide 65

Slide 65 text

Rarely the most Maintainable

Slide 66

Slide 66 text

Rarely the most Flexible

Slide 67

Slide 67 text

Rarely the Fastest

Slide 68

Slide 68 text

Too many MVPs and your code becomes difficult

Slide 69

Slide 69 text

What’s the cure for the MVP?

Slide 70

Slide 70 text

Tests! (& Refactoring)

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

Black box testing

Slide 73

Slide 73 text

Given a set of inputs

Slide 74

Slide 74 text

Expect a known output

Slide 75

Slide 75 text

Ignore the how

Slide 76

Slide 76 text

So how do we actually test a buildpack?

Slide 77

Slide 77 text

Real Ruby apps, real deploys

Slide 78

Slide 78 text

Let’s start with the the framework:

Slide 79

Slide 79 text

Hatchet

Slide 80

Slide 80 text

Hacks tests github.com/heroku/hatchet

Slide 81

Slide 81 text

clone repo create heroku app deploy

Slide 82

Slide 82 text

git repo is input

Slide 83

Slide 83 text

Output is deploy log

Slide 84

Slide 84 text

Output is `heroku run`

Slide 85

Slide 85 text

Who has used? $ heroku run bash

Slide 86

Slide 86 text

$ heroku run bash Running `bash` attached to terminal… ~ $ ruby -v ruby 2.1.0p0 ~ $ rails -v Rails 4.0.2 Heroku run bash

Slide 87

Slide 87 text

Anyone not using Ruby 2.1.1 right now?

Slide 88

Slide 88 text

PHP?

Slide 89

Slide 89 text

github.com/ schneems/ repl_runner

Slide 90

Slide 90 text

Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'") do |r| expect(r).to match('helloworld') end end end repl_runner

Slide 91

Slide 91 text

Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'") do |r| expect(r).to match('helloworld') end end end repl_runner

Slide 92

Slide 92 text

Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'") do |r| expect(r).to match('helloworld') end end end repl_runner

Slide 93

Slide 93 text

Hatchet::Runner.new("rails3").deploy do |app| app.run("rails console") do |console| console.run("'hello' + 'world'") do |r| expect(r).to match('helloworld') end end end repl_runner

Slide 94

Slide 94 text

Looks simple, but took me days

Slide 95

Slide 95 text

Process deadlock is no joke

Slide 96

Slide 96 text

Where did “rails3” come from?

Slide 97

Slide 97 text

No content

Slide 98

Slide 98 text

github.com/ sharpstone

Slide 99

Slide 99 text

47 Repos of edge cases (and counting)

Slide 100

Slide 100 text

Sidenote!

Slide 101

Slide 101 text

Threads in Ruby are easy + awesome

Slide 102

Slide 102 text

Threaded

Slide 103

Slide 103 text

Threaded.later do require “YAML" url = “https://s3-external-1.amazonaws.com/” #... YAML.load `curl #{url} 2>/dev/null` end Threaded. later

Slide 104

Slide 104 text

$ bundle exec hatchet install Installing repos for hatchet == pulling 'git://github.com/sharpstone/ asset_precompile_pass.git' == pulling ‘git://github.com/sharpstone/ asset_precompile_not_found.git' # … hatchet install ~ 45 seconds

Slide 105

Slide 105 text

$ bundle exec hatchet install Installing repos for hatchet == pulling 'git://github.com/sharpstone/ asset_precompile_pass.git' == pulling ‘git://github.com/sharpstone/ asset_precompile_not_found.git' # … hatchet install ~ 2 seconds

Slide 106

Slide 106 text

Copies repo to tmp dir

Slide 107

Slide 107 text

Creates new app through Heroku API

Slide 108

Slide 108 text

Deploys

Slide 109

Slide 109 text

Assertions done in the “deploy” block

Slide 110

Slide 110 text

So we have input & output

Slide 111

Slide 111 text

Tests are done, right?

Slide 112

Slide 112 text

What if S3 Goes down

Slide 113

Slide 113 text

What if Rubygems Goes down

Slide 114

Slide 114 text

What if Heroku API Goes down

Slide 115

Slide 115 text

What if Network Goes down

Slide 116

Slide 116 text

What if Github Goes down

Slide 117

Slide 117 text

Your tests FAIL

Slide 118

Slide 118 text

No content

Slide 119

Slide 119 text

Seems untestable

Slide 120

Slide 120 text

What do we do when we fall off the horse?

Slide 121

Slide 121 text

We rrrretry

Slide 122

Slide 122 text

No content

Slide 123

Slide 123 text

All deploys are retried

Slide 124

Slide 124 text

$ cat .travis.yml | grep HATCHET_RETRIES - HATCHET_RETRIES=3

Slide 125

Slide 125 text

Sidebar: Bundler 1.5+ automatically retries

Slide 126

Slide 126 text

Deploys are idempotent

Slide 127

Slide 127 text

What about non-deploy network hiccups?

Slide 128

Slide 128 text

RSpec.configure do |config| config.default_retry_count = 2 # … end

Slide 129

Slide 129 text

Failed assertions cause a test re-run

Slide 130

Slide 130 text

Fool me once, shame on you

Slide 131

Slide 131 text

Fool me six times sequentially, then there’s an error in my tests

Slide 132

Slide 132 text

When life gives you non-deterministic code, use probability to approximate determinism

Slide 133

Slide 133 text

Test Speed

Slide 134

Slide 134 text

First test: ~5 minutes

Slide 135

Slide 135 text

~1000 travis runs later

Slide 136

Slide 136 text

44 test cases in ~12 minutes (when passing)

Slide 137

Slide 137 text

How?

Slide 138

Slide 138 text

$ bundle exec parallel_rspec -n 11 spec/ Parallel Rspec Runner

Slide 139

Slide 139 text

Also, the buildpack got faster

Slide 140

Slide 140 text

No content

Slide 141

Slide 141 text

Coincidence?

Slide 142

Slide 142 text

No content

Slide 143

Slide 143 text

Tests allow us to be aggressive in refactoring

Slide 144

Slide 144 text

Big changes in architecture == big changes in speed

Slide 145

Slide 145 text

Did testing make the buildpack faster?

Slide 146

Slide 146 text

Tests beget tests

Slide 147

Slide 147 text

With black box tests in place: refactor to modular components

Slide 148

Slide 148 text

Unit test those components

Slide 149

Slide 149 text

Unit tests are faster

Slide 150

Slide 150 text

rake = LanguagePack::Helpers::RakeRunner.new .load_rake_tasks! task = rake.task("assets:precompile") task.invoke expect(task.status).to eq(:pass) expect(task.output).to match("success!") expect(task.time).not_to be_nil Finished in 1.63 seconds

Slide 151

Slide 151 text

Faster tests means quicker iteration

Slide 152

Slide 152 text

Some real world “untestable” scenarios for your web apps

Slide 153

Slide 153 text

codetriage .com

Slide 154

Slide 154 text

No content

Slide 155

Slide 155 text

External network dependency: github.com

Slide 156

Slide 156 text

What does a black box Rails test look like?

Slide 157

Slide 157 text

Click buttons, observe page changes

Slide 158

Slide 158 text

Capybara to the rescue!

Slide 159

Slide 159 text

What about all that network retry stuff?

Slide 160

Slide 160 text

Mock for determinism

Slide 161

Slide 161 text

webmock

Slide 162

Slide 162 text

VCR

Slide 163

Slide 163 text

Record “real” web traffic, play back in tests

Slide 164

Slide 164 text

--- http_interactions: - request: method: get uri: https://api.github.com/repos/bemurphy/ issue_triage_bogus_repo/issues? direction=desc&page=1&sort=comments body: encoding: US-ASCII string: '' headers: Accept: - application/vnd.github.3.raw+json cassette.yml

Slide 165

Slide 165 text

Libraries

Slide 166

Slide 166 text

Puma Auto Tune

Slide 167

Slide 167 text

Optimizes Puma “workers” based on RAM

Slide 168

Slide 168 text

How do we test it?

Slide 169

Slide 169 text

PumaRemote. new. spawn

Slide 170

Slide 170 text

Process.spawn("exec env PUMA_FREQUENCY=#{frequency} PUMA_RAM=#{ram} bundle exec puma #{path} -C #{config} > #{log}") Actually Run Puma

Slide 171

Slide 171 text

require 'rack' require 'rack/server' run Proc.new {|env| [200, {}, ['Hello World']] } Stub App

Slide 172

Slide 172 text

@puma = PumaRemote.new.spawn assert_match "cannot have less than one worker", @puma.log.read Assert against logs

Slide 173

Slide 173 text

That was a lot to digest, I just want to write tests

Slide 174

Slide 174 text

Nothing is untestable

Slide 175

Slide 175 text

Start with integration tests

Slide 176

Slide 176 text

Test the things that will hurt if they break

Slide 177

Slide 177 text

Use your tests to refactor and write smaller, faster tests

Slide 178

Slide 178 text

Don’t get paged at 3am about trivial failures

Slide 179

Slide 179 text

Avoid the MVP

Slide 180

Slide 180 text

Be maintainable

Slide 181

Slide 181 text

Be flexible

Slide 182

Slide 182 text

Be fast(able)

Slide 183

Slide 183 text

Be tested

Slide 184

Slide 184 text

@schneems

Slide 185

Slide 185 text

Sextant Gem

Slide 186

Slide 186 text

Wicked ‘ ‘ Gem

Slide 187

Slide 187 text

No content

Slide 188

Slide 188 text

Questions? @schneems