Verónica López Sr. Software Engineer @ CoreOS @maria_fibonacci FOSDEM 2018 Testing and Automation in the era of Containers

● Sr. Software Engineer @ CoreOS ● Automating RHEL ● Now: Distributed Systems Before: Scientific Computing, Mobile ● Common path: Linux whoami

- How we do it @ CoreOS -Strategy - What I’ve learned - Change of Paradigm - Go is great for tooling - Closing thoughts Agenda

- Every team/dev takes care of their own tests - Don’t merge ‘till all tests pass/get fixed - Strict Release Automation guidelines - Cool for accountability, also because products are different How we do it

- Some teams need/want a tougher strategy, like Etcd - +100k lines of code. +60k are tests. - Unit, integration, migration, e2e, regression, stress, functional, benchmarks. - We also use Bazel... How we do it

- Testing and Automation teams works on building targeted tools, not on QA. - Relatively new team: still evolving and learning (LOL) - Framework: no more. How we do it

- No formal experience in testing, but experience with many distributed systems: fresh look - Focus on test coverage: necessary for baseline, not enough - Incomplete e2e scenarios. Empathy necessary. - Engineers’ backgrounds make a difference What I’ve learnt

- My experience: Large systems in Latin America. All of them live or die by ruthless optimization of all the parts. - Privileged/bleeding edge cultures: can afford overengineering and rewrites (skilled people) - Would be great to have the best of both worlds What I’ve learnt

- Testing distributed systems is hard. New considerations necessary. - Formal verification of a distsys: hard and expensive. - All the perspectives at once: combinatorial possibilities. - Containers, service meshes, K8s: solve many problems, are fault tolerant, but need different levels of specialized skills. Change of Paradigm

1. Using containers for testing 2. How to test containers Testing & Containers

Package your test suite, and make your system run against it. Benefits: practical, neat, fast, secure, scalability, portability. Sets a standard for distributed components. One of the developing strategies @CoreOS. Using containers for testing

- K8s e2e test suite: “It is not uncommon that a minor change may pass all unit and integration tests, but cause unforeseen changes at system level” Testing Containers...

- E2e: by definition, as comprehensive as possible - In order to do that = deep knowledge of the system - The fact that we don’t have to worry about fixing things anymore doesn’t mean that we don’t have to know how they work, and how they fail. - Familiarity with clusters, pods, OS, etc… lifecycles. Testing Containers...

- Unit testing: always important - Two outcomes: either incomplete, too complicated - Massive integration tests: antipattern - Based on mocks: only 3 nodes to fail, but what matters is the number of inputs Testing Containers...

Monitoring and Support teams must not act as your systems’ nanny! Testing Containers...

Writing this talk: scary similarities. Many people experiencing the same problems and perspectives. Working on different teams. Cindy @copyconstruct likes to write about it http://bit.ly/2meWzaF Testing Containers...

“Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems” - 92% of the catastrophic system failures are the result of incorrect handling of non-fatal errors - Go: error handling, cover Testing Containers...

- Go was created with tooling in mind. There’s an article about it https://talks.golang.org/2012/splash.article#TOC_17. Go Tooling

- Go cover https://blog.golang.org/cover - Go test - Go error handling https://blog.golang.org/errors-are-values - Errors are Values https://blog.golang.org/errors-are-values I recommend those blog posts. I also recommend Kelsey’s Kubernetes the Hard Way Go Tooling

Thank you! Verónica López @maria_fibonacci We’re hiring!