Testing & Automation in the Era of Containers

Testing distributed systems shouldn't follow the same rules as old systems; focusing all efforts on unit testing creates incomplete scenarios or another mock-based similar system that needs maintenance of its own.
Familiarization with the systems is crucial to create better e2e tests that might catch failures that integration and unit tests no longer can.

Transcript

1. Verónica López Sr. Software Engineer @ CoreOS @maria_fibonacci FOSDEM 2018 Testing and Automation in the era of Containers
2. • Sr. Software Engineer @ CoreOS • Automating RHEL • Now: Distributed Systems Before: Scientific Computing, Mobile • Common path: Linux whoami
3. - How we do it @ CoreOS -Strategy - What I’ve learned - Change of Paradigm - Go is great for tooling - Closing thoughts Agenda
4. - Every team/dev takes care of their own tests - Don’t merge ‘till all tests pass/get fixed - Strict Release Automation guidelines - Cool for accountability, also because products are different How we do it
5. - Some teams need/want a tougher strategy, like Etcd - +100k lines of code. +60k are tests. - Unit, integration, migration, e2e, regression, stress, functional, benchmarks. - We also use Bazel... How we do it
6. - Testing and Automation teams works on building targeted tools, not on QA. - Relatively new team: still evolving and learning (LOL) - Framework: no more. How we do it
7. - No formal experience in testing, but experience with many distributed systems: fresh look - Focus on test coverage: necessary for baseline, not enough - Incomplete e2e scenarios. Empathy necessary. - Engineers’ backgrounds make a difference What I’ve learnt
8. - My experience: Large systems in Latin America. All of them live or die by ruthless optimization of all the parts. - Privileged/bleeding edge cultures: can afford overengineering and rewrites (skilled people) - Would be great to have the best of both worlds What I’ve learnt
9. - Testing distributed systems is hard. New considerations necessary. - Formal verification of a distsys: hard and expensive. - All the perspectives at once: combinatorial possibilities. - Containers, service meshes, K8s: solve many problems, are fault tolerant, but need different levels of specialized skills. Change of Paradigm
10. 1. Using containers for testing 2. How to test containers Testing & Containers
11. Package your test suite, and make your system run against it. Benefits: practical, neat, fast, secure, scalability, portability. Sets a standard for distributed components. One of the developing strategies @CoreOS. Using containers for testing
12. - K8s e2e test suite: “It is not uncommon that a minor change may pass all unit and integration tests, but cause unforeseen changes at system level” Testing Containers...
13. - E2e: by definition, as comprehensive as possible - In order to do that = deep knowledge of the system - The fact that we don’t have to worry about fixing things anymore doesn’t mean that we don’t have to know how they work, and how they fail. - Familiarity with clusters, pods, OS, etc… lifecycles. Testing Containers...
14. - Unit testing: always important - Two outcomes: either incomplete, too complicated - Massive integration tests: antipattern - Based on mocks: only 3 nodes to fail, but what matters is the number of inputs Testing Containers...
15. Monitoring and Support teams must not act as your systems’ nanny! Testing Containers...
16. Writing this talk: scary similarities. Many people experiencing the same problems and perspectives. Working on different teams. Cindy @copyconstruct likes to write about it http://bit.ly/2meWzaF Testing Containers...
17. “Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems” - 92% of the catastrophic system failures are the result of incorrect handling of non-fatal errors - Go: error handling, cover Testing Containers...
18. - Go was created with tooling in mind. There’s an article about it https://talks.golang.org/2012/splash.article#TOC_17. Go Tooling
19. - Go cover https://blog.golang.org/cover - Go test - Go error handling https://blog.golang.org/errors-are-values - Errors are Values https://blog.golang.org/errors-are-values I recommend those blog posts. I also recommend Kelsey’s Kubernetes the Hard Way Go Tooling
20. Thank you! Verónica López @maria_fibonacci We’re hiring!