DevOpsDaysPortugal 2019 - Ken Mugrage - Modern Continuous Delivery - A story of 3 real projects

At ThoughtWorks we pride ourselves in learning from our mistakes. To make this easier, we’ve made some big ones!

This talk will tell the stories of 3 different real projects which have transformed not only the way we work, but large portions of our industry as well.

In the first story, we learned the hard way about the importance of deploying to a production like environment as often as possible. We were proud of our development practices, but had not learned yet about the importance of the “last mile”. The learnings from this project (and others) became the book “Continuous Delivery”.

In the second story, we learned a lot from a project which was widely considered a success. In this project we were able to regularly release to production, but not as efficiently as we would have liked. The software was a fairly large monolith and could only be deployed as a single large package. This project taught us a lot about loosely coupled architectures and led to much of our work in Microservices and related technologies.

The third story isn’t over yet, and will be most of the talk. We’re building a highly distributed application handling 10 million transactions a day. To do this we’ve had to take a hard look at everything from how services communicate, to how we test such a complicated application, to the design of the pipeline itself.

At the end of the talk people should have a much better understanding of the real world issues that led to Continuous Delivery and DevOps, as well as the importance of learning from both failure and success. They will also see real world examples of how their own distributed applications can be deployed and tested.

Transcript

1. @kmugrage https://www.gocd.org/ Modern Continuous Delivery A JOURNEY IN 4 ACTS
2. @kmugrage https://www.gocd.org/ Act I UNITED KINGDOM – 2005
3. @kmugrage https://www.gocd.org/ We knew everything Signed Agile Manifesto Defined Continuous Integration Created the first (or second) CI server Created Selenium
4. @kmugrage https://www.gocd.org/ Java – Write once, run anywhere Developed a system on Windows laptops to be deployed to a Solaris cluster Did all of the right Continuous Integration things One small issue…
5. @kmugrage https://www.gocd.org/ It didn’t work in production
6. @kmugrage https://www.gocd.org/ Conan The Deployer Massive shell script Automated deployment to a cluster after every successful CI run Deployment became a non-issue
7. @kmugrage https://www.gocd.org/ The Output Jez Humble, Dan North and Chris Read write “The Deployment Product Line” Jez Humble creates GoCD Jez Humble and David Farley release the book “Continuous Delivery"
8. @kmugrage https://www.gocd.org/ Act II UNITED KINGDOM – 2000
9. @kmugrage https://www.gocd.org/ Fairly typical architecture Edge Load Balancer Web Application Web Application Customer Service Application Web Application Customer Service Application Frontend Application Servers Web Application Web Application Web Application Business Servers Business Servers Business Servers MSMQ Biztalk Distributed Cache Session DB Transactional DB Biztalk DB
10. @kmugrage https://www.gocd.org/ Traditional Continuous Delivery git pull Version Build Unit Test Package Artifact Repository Build Test Artifact Repository Get Package Deploy Test Environment Component Service UI Deploy Artifact Repository Get Package Deploy Environment Smoke Tests
11. @kmugrage https://www.gocd.org/ Traditional Continuous Delivery Build Functional Tests Regression Tests Deploy to Stage Deploy to Production Artifact Repository Hand rolled environments Performance Tests
12. @kmugrage https://www.gocd.org/ The good Reproducible builds Generate package once High level of automation Safety net with automated test stages
13. @kmugrage https://www.gocd.org/ The Bad Entire system deployed at once Releases were large Browser tests were flaky Rollbacks were hard
14. @kmugrage https://www.gocd.org/ The Output Sam Newman releases the book “Building Microservices” Kief Morris releases the book “Infrastructure as Code” Created Gauge open source testing framework Created Taiko to combat flaky browser tests
15. @kmugrage https://www.gocd.org/ Act III UNITED STATES – 2019
16. @kmugrage https://www.gocd.org/ Financial Services Platform Edge Load Balancer Service Repository Apply Domain Event Store Command Handlers Event Handlers Service Repository Account Domain Event Store Command Handlers Event Handlers Service Repository Product Domain Event Store Command Handlers Event Handlers Service Repository Customer Domain Event Store Command Handlers Event Handlers E V E N T B U S
17. @kmugrage https://www.gocd.org/ Modern Pipelines Build Functional Tests Regression Tests Deploy to Stage Deploy to Production Service A Service B Build Functional Tests Regression Tests Deploy to Stage Deploy to Production
18. @kmugrage https://www.gocd.org/ Another option Build Functional Tests Regression Tests Deploy to Stage Deploy to Production Service A Build Functional Tests Regression Tests Service B
19. @kmugrage https://www.gocd.org/ Act IV THE DETAILS
20. @kmugrage https://www.gocd.org/ Traditional Structure Development Teams QA Team Operations Team
21. @kmugrage https://www.gocd.org/ Products over Projects (do the DevOps) Service 1 Service 2 Service 5 Service 6 Service 3 Service 4
22. @kmugrage https://www.gocd.org/ Develop Build Test Deploy Monitor
23. @kmugrage https://www.gocd.org/ The New Build Artifact git pull Version Build Unit Test Package Docker File thoughtworks/gocd-server:v18.10
24. @kmugrage https://www.gocd.org/ A modern Continuous delivery pipeline
25. @kmugrage https://www.gocd.org/ Trunk Based Development TRUNK RELEASE BRANCHES RELEASE 1.1.x HOTFIX CHERRYPICK CHERRYPICK COMMIT DEVELOPERS SHORT-LIVED DEVELOPMENT BRANCHES COMMIT
26. @kmugrage https://www.gocd.org/ Feature Toggles
27. @kmugrage https://www.gocd.org/ Develop Build Test Deploy Monitor
28. @kmugrage https://www.gocd.org/ The Test Pyramid UNIT TESTS Slow, Expensive Fast, Cheap SERVICE TESTS UI TESTS
29. @kmugrage https://www.gocd.org/ The Test Pyramid In Context Build Test Deploy to Stage Deploy to Production UNIT TESTS Slow, Expensive Fast, Cheap Monitoring Distributed Tracing Fault-injection Testing Canary Deployments Blue-Green Deployments A/B Testing INTEGRATION TESTS COMPONENT TESTS CONTRACT TESTS E2E TESTS Build Test Deploy to Stage Deploy to Stage Deploy to Stage Deploy to Stage Deploy to Stage Deploy to Stage
30. @kmugrage https://www.gocd.org/ Develop Build Test Deploy Monitor
31. @kmugrage https://www.gocd.org/ Declarative Deployments services db volumes load balancer YAML deployment descriptor Ingress Service Node Pod Deployment Pod Node Pod Pod Replica Set
32. @kmugrage https://www.gocd.org/ Deployment Strategies Load Balancer V1 V1 V2 Load Balancer V1 V2 V2 Load Balancer V2 V2 V2 Rolling Update Load Balancer V1 V1 Load Balancer V2 V2 V1 V1 V2 V2 Blue Green Deployment Load Balancer V1 V1 V2 Canary Deployment 25% 75%
33. @kmugrage https://www.gocd.org/ Release DB Changes Out Of Band D B V 1 A P P V 1 D B V 2 A P P V 2 R O L L B A C K A P P V 2 Time DB migrated to V1 App uses DB V1 DB migrated to V2 App uses DB V2 App rolled back to V1
34. @kmugrage https://www.gocd.org/ Dynamic Environments Build Build Image Build Pipeline Test Deploy Test Destroy Deploy Stage Deploy Test Deploy Prod Docker Registry Push Image Pull Image namespace: Stage namespace: Prod Kubernetes Image metadata Image metadata
35. @kmugrage https://www.gocd.org/ Develop Build Test Deploy Monitor
36. @kmugrage https://www.gocd.org/ Monitoring and observability Hardware metrics (CPU, Memory, IO) App/Business metrics (CPU, Memory, IO) Logs Metrics Collector Metrics Collect & Ship Aggregate & Store Visualize Typical Monitoring Setup
37. @kmugrage https://www.gocd.org/ Monitoring and observability High cardinality events honeycomb.io Observability
38. @kmugrage https://www.gocd.org/ Develop Build Test Deploy Monitor Security
39. @kmugrage https://www.gocd.org/ Develop Build Test Deploy Monitor Security
40. @kmugrage https://www.gocd.org/ Vulnerability Planning Build Functional Tests Regression Tests Deploy to Stage Deploy to Production Static CVE Scan Scan Images Monitor Runtime Continuers Monitor Runtime Continuers
41. @kmugrage https://www.gocd.org/ Secrets Management Continuous Delivery Workflow ORCHESTRATOR (Chef / Terraform /Kubernetes) Application Server / Container Fetch RoleID Write RoleID Machine/Container Image Deliver Secret ID (Auth Token) to the Application Use Auth Token To Access Secrets Trusted Entity
42. @kmugrage https://www.gocd.org/ Epilogue
43. @kmugrage https://www.gocd.org/ In the future Teams have to be structured to own small pieces Docker and Kubernetes are here to stay… until they aren’t You have to test in production. Own it and architect for it Security is everybody’s job
44. @kmugrage https://www.gocd.org/ Resources & Citations http://agilemanifesto.org/ https://martinfowler.com/articles/originalContinuousIntegration.html https://continuousdelivery.com/wp-content/uploads/2011/04/deployment_production_line.pdf https://martinfowler.com/articles/microservice-trade-offs.html https://martinfowler.com/articles/practical-test-pyramid.html https://trunkbaseddevelopment.com/ https://martinfowler.com/articles/feature-toggles.html https://docs.honeycomb.io/learning-about-observability/intro-to-observability/
45. @kmugrage https://www.gocd.org/ Thank You