Testing & Integration (The Remix)

Given at CraftConf 2015: http://craft-conf.com/2015
Accompanying talk repo: https://github.com/randommood/Craftconf2015
Video: http://www.ustream.tv/recorded/61449003

Let's face it, testing distributed systems is hard. Large number of inputs, partial failures, and asynchrony make these systems seemingly impossible to verify. Formal testing methods are onerous and system correctness continues to elude us. Despite the difficulty, we need to have a way to change our systems with confidence. Integration and unit tests are vital to our sanity and we know it's good for us to use continuous integration to ensure functional consistency across our products.

Come to this talk for a fresh and practical look into the various aspects of distributed systems testing and integration. We will cover CI pipelines: their strengths, weaknesses, and ways to improvement them. Get ready to rediscover the untapped power of your integration practices and develop a burning desire to make them more awesome!

Transcript

1. Testing & Integration THE REMIX!
2. Ines Sombra ines@fastly.com @Randommood ! " #
3. Globally distributed & highly available
4. DS Testing Continuous Integration Bring Home " ! $ Our Agenda DS = Distributed Systems
5. Visual Guide ^ Distributed Systems Testing " $ !
6. Why do we test? We test to gain confidence that our system is doing the right thing 
 (now & later) " $ !
7. MEDIUM Unit Integration System Acceptance SMALL Unit Integration (maybe) COMPLEX SYSTEM Fault injection Stress / performance Canary Regression Unit Integration System Acceptance Compatibility Many Types of Tests " $ !
8. Nondeterminism Timing Failures Unbounded inputs Ordering Many states Concurrency No centralized view Challenges of DS Testing " $ !
9. Behavior is aggregate Components tested in isolation also need to be tested together ? ! A ! B " $ ! Challenges of DS Testing
10. Reliability of a linear system is the product of the reliability of each of the system’s components " $ ! Challenges of DS Testing
11. .95 * .95 * .95 = .85 Your system is only 85% reliable Component A 95% reliable ! Component B 95% reliable ! Component C 95% reliable ! " $ ! Challenges of DS Testing
12. Hierarchy of Errors* BYZANTINE FAILURES (fail by doing whatever I want) OMISSION FAILURES (fail by dropping messages) CRASH FAILURES (fail by stopping) * Stolen from Henry Robinson’s PWLSF talk Deadlocks Livelock / starvation Under specification Over specification " $ !
13. Distributed Systems Testing Difficult to approach & many factors in play Behavior is aggregate Goal is to gain confidence of proper system behavior now & later " $ !
14. Continuous Integration CI for short " $ !
15. Waiting till the end of a project to integrate leads to hell (a special kind) " $ !
16. What is CI? Development practice Integrate code to master branch daily Each integration is automatically verified " $ !
17. Why CI Today? Often overlooked/ Taken for granted “Eat your veggies” Impacts your risk & quality " $ !
18. Commit frequently 100% test must pass Fix broken tests Automation is CI Practices " $ !
19. CI gives us Rapid feedback Rapid deployment Repeatable automated testing Reduce risk " $ !
20. Revisiting Tests Differentiated by required setup which affects running time Unit Integration System Functional 
 (also Smoke /Acceptance) " $ !
21. CI Challenges Complex systems / dependencies Multiple architectures Type of applications & types of tests " $ !
22. CI Challenges " $ ! Overfitted/ brittle / slow tests Writing good tests is hard Maintenance Consider Regulations (PCI)
23. “Manage the CI System 
 or be controlled by manual processes” Is it worth it? " $ !
24. You integrate to master very often (daily) All of your tests pass & you fix broken builds! Success indicators: # commits / build status Am I doing CI right? " $ !
25. CI In the Wild " $ !
26. Tools & Services How to pick? Security, Cost Language support Services & integrations Scalability " $ !
27. " $ !
28. CI In the Wild Branches & pull requests. Merges & deploys Consolidated view of build statuses across all codebases " $ !
29. Master PRs Kick off a suite Stacks OS + Our Image Scenario Live setup + assertions Suite Collection of scenarios My Assertions: Installing, backing up/restoring, setting up, replicating, promoting, etc Any prod deploy Kicks off a suite " $ ! Dredd Tests Systems, boundaries, & integration CI In the Wild
30. CI In the Wild " $ ! * Stolen from Google’s talk on their Continuous Build System
31. " $ ! CI In the Wild
32. * Slide stolen from Seth Falcon’s ChefConf 2015 talk CI In the Wild: CD tools* " $ !
33. CI Matters Predictor of professional maturity at the organizational & individual level Adds visibility & allows us to manage risk and quality " $ !
34. How are we doing?
35. Bringing it home " $ !
36. " $ ! Getting Started Test the full distributed system. This means testing the client, system, and provisioning code! HAVE DECENT TESTS ADD VISIBILITY VERIFY & 
 ENHANCE
37. CI & Code Quality Automated inspection as part of the build Customizable by Org Test coverage Code duplication " $ !
38. CI & Quality " $ ! Languages help you: go cov, race detector,
 go lint Infrastructure testing matters But beware of misguided processes
39. CI & Quality Cyclomatic Complexity Number (CNN) Afferent/Efferent Coupling Software instability as “quantifiable” " $ !
40. And more! " $ ! Use it for docs generation Architectural/design diagrams too ! Accessibility testing Used to test mobile architectures too
41. Pro-Tips " $ ! Add CI early Different machine for integrating & put everything you need to built in git Parallelize builds to scale up or down
42. Rapid feedback is critical. Slow is Staggered builds: run fast tests first, then second phase kicks in The more likely a test is to fail the sooner you should execute it " $ ! Pro-Tips
43. Tl;DR DS TESTING Different types of tests Can be tricky Behavior is aggregate CI PRACTICES Everyone commits to baseline daily Automate build & make it self- testing Keep builds fast Everyone can see builds & status ON QUALITY CI can be used to drive good coding standards Some metrics available to gauge stability / quality Keep track of them
44. Rantifesto: Build a Culture of Quality ✨The technology is just a tool but your culture is really the key ✨ Lower the risk of change with tools & culture " $ !
45. “Quality means doing it right when no one is looking”
 Henry Ford
46. Thank You! Special thanks to: Caitie McCaffrey, Jordan West, Jon Hyman, Bruce Spang, Devon O’Dell, Kelsey Gilmore-Innis, Aysylu Greenberg, Alan Kasindorf, Paul Reed, André Arko, Mike O’Neill, Thomas Mahoney, Eric Kustarz, Ian Fung, Neha Narula, Karl Smith, and Greg Bako. ines@fastly.com | @Randommood | github.com/Randommood/CraftConf2015
47. Questions? ines@fastly.com | @Randommood | github.com/Randommood/CraftConf2015