Reliability Tests All the Way Down!

Modern application development has become more complex with many potential points of failure. As desires for an Internal Developer Platform (IDP) grow within many organizations, let's review some of the tooling and processes available to ensure we create reliable software from planning to production by including reliability tests. Afterward, let's collectively discuss ways we can bring these ideas back to experiment with.

This talk was presented and livestreamed to the MO Reliability Meetup group in St. Louis, Missouri on April 10, 2024. The video is available at https://www.youtube.com/watch?v=Jz1DXsynVGU.

Transcript

1. Reliability Tests All the Way Down! Paul Balogh, @javaducky
2. Disclaimer Paul is not here to sway you to one product over another, nor am I here to persuade you into a specific paradigm over another. He is simply interested in Continuous Improvement of the software we create. Forward complaints to
3. Act I When things were simple…or were they?
4. The Monolith “My Cool App”
5. The Monolith - Development handed-off “finished” work to QA - Developers are introduced to Test Driven Development (TDD) - Deployments were infrequent, unless there were bugs - Scaling was done vertically - We loved our servers!
6. Domain-Driven Development “Bounded Context” “My Cool(er?) App”
7. Domain-Driven Development - Monoliths became complex and inflexible - Breaking up things made us more agile - Our testing practices stayed the same - More things to test; fortunately we now have tools like Selenium, JMeter, and Postman - APIs are becoming important - We learn to love VMs and Cloud!
8. Microservices “A Netflix of our Own”
9. Microservices - Finer-grained services and even serverless functions - More teams + more services + more APIs = more complex - Hyper scalability based upon metrics and requests - Best practices outlined by the 12-factor app - Shift-left is necessary, testers are overwhelmed - We love our containers and Kubernetes!
10. To those of you here at NISC…I’m sorry.
11. Act II A continuous process to deal…
12. Our Actors Developer Tester
13. A Continuous Lifecycle Dev Ops Plan Code Build Test Release Deploy Operate Monitor
14. Our Actors Developer Tester Site Reliability Engineer (SRE) Oh, Hai!
15. Disclaimer I’m using somewhat generic terms for ease. In reality, these roles have many names and a single person may assume the role of multiple roles at any given time. - SRE: anyone actively working on upkeep of infrastructure and watching for alerts. - Developer: anyone actively working on the creation of code to be deployed as an application. - Tester: anyone actively creating automation or directly ensuring functionality. Forward complaints to
16. Plan phase Dev Ops Plan Code Build Test Release Deploy Operate Monitor
17. Plan phase The beginning. - Establish performance baselines given past experience. - SREs and Developers agree upon stack and features.
18. Plan phase The beginning. - Establish performance baselines given past experience. - SREs and Developers agree upon stack and features. Don’t forget Product!!!
19. Code phase Dev Ops Plan Code Build Test Release Deploy Operate Monitor
20. Code phase Develops develop. Testers…prepare. - Developers begin creating the features, with unit tests and integration tests. - Developers include telemetry data for use in later phases. - Developers ensure performant code using profiling tools.
21. Code phase (cont’d) Develops develop. Testers…prepare. - Testers create automation tests as Developers are creating based upon agreed specifications. These can be Contract tests and even Load tests.
22. Build phase Dev Ops Plan Code Build Test Release Deploy Operate Monitor
23. Build phase SREs buildout infrastructure. - SREs create resources using tools like Terraform while creating tests to validate IaC. - If new tooling was agreed upon, they build expertise in operating such resources.
24. Test phase Dev Ops Plan Code Build Test Release Deploy Operate Monitor
25. Test phase Testers test. - Create hybrid-testing with chaos-style disruptions to dependent systems. - Validate user experience. - Attempt to raise security issues. - Verify observability metrics are available.
26. Release phase Dev Ops Plan Code Build Test Release Deploy Operate Monitor
27. Release phase Check quality gates and readiness. - Developers, Testers, and SREs agree on ready state for application. - Testers should confirm End-to-End testing (E2E). - Perform load testing.
28. Deploy phase Dev Ops Plan Code Build Test Release Deploy Operate Monitor
29. Deploy phase Thrusters engaged. - SREs utilize metrics-based quality gates with Canary deployments, or use Blue/Green deployment.
30. Operate phase Dev Ops Plan Code Build Test Release Deploy Operate Monitor
31. Operate phase Keeping the wheels moving. - SREs and Testers conduct Chaos experiments. - SREs watch costs for compute.
32. Monitor phase Dev Ops Plan Code Build Test Release Deploy Operate Monitor
33. Monitor phase Keeping the peace. - SREs create alerts and on-call schedules. - SREs provide continuous profiling for applications. - SREs install kernel-monitoring tools like eBPF
34. Plan phase…again Dev Ops Plan Code Build Test Release Deploy Operate Monitor
35. Plan phase Back to the beginning. - SREs provide feedback to refine baselines.
36. Act III Choosing the right tools…
37. Navigating the CNCF To be continued…
38. Thank you! Connect with Paul as @javaducky or linkedin/in/pabalogh
39. THIS SLIDE INTENTIONALLY LEFT BLANK