Tweet
Tweet

Slide 1

Slide 1 text

Testing & Integration THE REMIX!

Slide 2

Slide 2 text

Ines Sombra ines@fastly.com @Randommood ! " #

Slide 3

Slide 3 text

Globally distributed & highly available

Slide 4

Slide 4 text

DS Testing Continuous Integration Bring Home " ! $ Our Agenda DS = Distributed Systems

Slide 5

Slide 5 text

Visual Guide ^ Distributed Systems Testing " $ !

Slide 6

Slide 6 text

Why do we test? We test to gain confidence that our system is doing the right thing 
 (now & later) " $ !

Slide 7

Slide 7 text

MEDIUM Unit Integration System Acceptance SMALL Unit Integration (maybe) COMPLEX SYSTEM Fault injection Stress / performance Canary Regression Unit Integration System Acceptance Compatibility Many Types of Tests " $ !

Slide 8

Slide 8 text

Nondeterminism Timing Failures Unbounded inputs Ordering Many states Concurrency No centralized view Challenges of DS Testing " $ !

Slide 9

Slide 9 text

Behavior is aggregate Components tested in isolation also need to be tested together ? ! A ! B " $ ! Challenges of DS Testing

Slide 10

Slide 10 text

Reliability of a linear system is the product of the reliability of each of the system’s components " $ ! Challenges of DS Testing

Slide 11

Slide 11 text

.95 * .95 * .95 = .85 Your system is only 85% reliable Component A 95% reliable ! Component B 95% reliable ! Component C 95% reliable ! " $ ! Challenges of DS Testing

Slide 12

Slide 12 text

Hierarchy of Errors* BYZANTINE FAILURES (fail by doing whatever I want) OMISSION FAILURES (fail by dropping messages) CRASH FAILURES (fail by stopping) * Stolen from Henry Robinson’s PWLSF talk Deadlocks Livelock / starvation Under specification Over specification " $ !

Slide 13

Slide 13 text

Distributed Systems Testing Difficult to approach & many factors in play Behavior is aggregate Goal is to gain confidence of proper system behavior now & later " $ !

Slide 14

Slide 14 text

Continuous Integration CI for short " $ !

Slide 15

Slide 15 text

Waiting till the end of a project to integrate leads to hell (a special kind) " $ !

Slide 16

Slide 16 text

What is CI? Development practice Integrate code to master branch daily Each integration is automatically verified " $ !

Slide 17

Slide 17 text

Why CI Today? Often overlooked/ Taken for granted “Eat your veggies” Impacts your risk & quality " $ !

Slide 18

Slide 18 text

Commit frequently 100% test must pass Fix broken tests Automation is CI Practices " $ !

Slide 19

Slide 19 text

CI gives us Rapid feedback Rapid deployment Repeatable automated testing Reduce risk " $ !

Slide 20

Slide 20 text

Revisiting Tests Differentiated by required setup which affects running time Unit Integration System Functional 
 (also Smoke /Acceptance) " $ !

Slide 21

Slide 21 text

CI Challenges Complex systems / dependencies Multiple architectures Type of applications & types of tests " $ !

Slide 22

Slide 22 text

CI Challenges " $ ! Overfitted/ brittle / slow tests Writing good tests is hard Maintenance Consider Regulations (PCI)

Slide 23

Slide 23 text

“Manage the CI System 
 or be controlled by manual processes” Is it worth it? " $ !

Slide 24

Slide 24 text

You integrate to master very often (daily) All of your tests pass & you fix broken builds! Success indicators: # commits / build status Am I doing CI right? " $ !

Slide 25

Slide 25 text

CI In the Wild " $ !

Slide 26

Slide 26 text

Tools & Services How to pick? Security, Cost Language support Services & integrations Scalability " $ !

Slide 27

Slide 27 text

" $ !

Slide 28

Slide 28 text

CI In the Wild Branches & pull requests. Merges & deploys Consolidated view of build statuses across all codebases " $ !

Slide 29

Slide 29 text

Master PRs Kick off a suite Stacks OS + Our Image Scenario Live setup + assertions Suite Collection of scenarios My Assertions: Installing, backing up/restoring, setting up, replicating, promoting, etc Any prod deploy Kicks off a suite " $ ! Dredd Tests Systems, boundaries, & integration CI In the Wild

Slide 30

Slide 30 text

CI In the Wild " $ ! * Stolen from Google’s talk on their Continuous Build System

Slide 31

Slide 31 text

" $ ! CI In the Wild

Slide 32

Slide 32 text

* Slide stolen from Seth Falcon’s ChefConf 2015 talk CI In the Wild: CD tools* " $ !

Slide 33

Slide 33 text

CI Matters Predictor of professional maturity at the organizational & individual level Adds visibility & allows us to manage risk and quality " $ !

Slide 34

Slide 34 text

How are we doing?

Slide 35

Slide 35 text

Bringing it home " $ !

Slide 36

Slide 36 text

" $ ! Getting Started Test the full distributed system. This means testing the client, system, and provisioning code! HAVE DECENT TESTS ADD VISIBILITY VERIFY & 
 ENHANCE

Slide 37

Slide 37 text

CI & Code Quality Automated inspection as part of the build Customizable by Org Test coverage Code duplication " $ !

Slide 38

Slide 38 text

CI & Quality " $ ! Languages help you: go cov, race detector,
 go lint Infrastructure testing matters But beware of misguided processes

Slide 39

Slide 39 text

CI & Quality Cyclomatic Complexity Number (CNN) Afferent/Efferent Coupling Software instability as “quantifiable” " $ !

Slide 40

Slide 40 text

And more! " $ ! Use it for docs generation Architectural/design diagrams too ! Accessibility testing Used to test mobile architectures too

Slide 41

Slide 41 text

Pro-Tips " $ ! Add CI early Different machine for integrating & put everything you need to built in git Parallelize builds to scale up or down

Slide 42

Slide 42 text

Rapid feedback is critical. Slow is Staggered builds: run fast tests first, then second phase kicks in The more likely a test is to fail the sooner you should execute it " $ ! Pro-Tips

Slide 43

Slide 43 text

Tl;DR DS TESTING Different types of tests Can be tricky Behavior is aggregate CI PRACTICES Everyone commits to baseline daily Automate build & make it self- testing Keep builds fast Everyone can see builds & status ON QUALITY CI can be used to drive good coding standards Some metrics available to gauge stability / quality Keep track of them

Slide 44

Slide 44 text

Rantifesto: Build a Culture of Quality ✨The technology is just a tool but your culture is really the key ✨ Lower the risk of change with tools & culture " $ !

Slide 45

Slide 45 text

“Quality means doing it right when no one is looking”
 Henry Ford

Slide 46

Slide 46 text

Thank You! Special thanks to: Caitie McCaffrey, Jordan West, Jon Hyman, Bruce Spang, Devon O’Dell, Kelsey Gilmore-Innis, Aysylu Greenberg, Alan Kasindorf, Paul Reed, André Arko, Mike O’Neill, Thomas Mahoney, Eric Kustarz, Ian Fung, Neha Narula, Karl Smith, and Greg Bako. ines@fastly.com | @Randommood | github.com/Randommood/CraftConf2015

Slide 47

Slide 47 text

Questions? ines@fastly.com | @Randommood | github.com/Randommood/CraftConf2015