Valor por encima de código: el poder del Despliegue Continuo

Slides de la charla dada el 27 de enero de 2024 en la BilboStack: https://bilbostack.com/isidro-lopez

¿Por qué y para qué desarrollamos software o tecnología en general? ¿Qué es realmente la “Integración Continua”? ¿Qué valor aporta la práctica del Despliegue Continuo? ¿En qué principios y prácticas sociotécnicas podemos apoyarnos para sacarle todo el jugo?

Te invito a acompañarme en éstas y algunas otras reflexiones, a compartir humildemente mi experiencia “en el mundo real” sobre el Despliegue Continuo, con varios ejemplos reales y algunas propuestas sobre cómo abordarlo.

Hablaré de por qué la simplicidad (en tecnología y negocio), trabajando en equipo, con pasos seguros extremadamente pequeños y feedback rápido marcan la diferencia. Olvídate del palabrerío técnico y centrémonos en cómo el desarrollo de software Lean y Extreme Programming pueden ser fantásticos aliados para conseguir lo que realmente importa: una entrega de valor continua y a un ritmo sostenible… ¡y disfrutando del proceso!

Esta charla no es solo para programadoras: es para todas aquellas personas que deseen acercarse a “otras maneras de hacer las cosas” más allá del mainstream (sin ánimo de proselitismo sino únicamente de compartir experiencias 😉).

Transcript

1. 27 de enero de 2024 Valor por encima de código: el poder del Despliegue Continuo Isidro López
3. Ego slide Hemos venido a jugar... Product developer/engineer (and Product Engineering Coach and…) @islomar islomar@gmail.com <curioso> Teleco, Psicología, Antropología Social y Cultural, Executive MBA… </curioso> 3
7. Photo: https://www.indiatimes.com/health/tips-tricks/this-harvard-psychologist-reveals-the-first-couple-of-criteria-people-judge-you-on-351236 .html NO JUDGEMENT !! 7
9. #NoTools #NoCode 9
10. Agenda • Continuous X • “Why” and “what for”: some principles of software development • Which are some “best” practices for Continuous Deployment? • Real life™ examples: Continuous Deployment pipelines • When NOT to do Continuous Deployment • How to start (a humble proposal) • Final recap • Recommended people, books and resources • Q&A and discussions 10
11. Continuous X 11
12. Continuous X • Continuous Integration • Continuous Delivery • Continuous Deployment 12
13. What is Continuous Integration? 13
14. Continuous Delivery Every commit to the main branch leaves your code continuously ready to be safely and quickly deployed to Production (with one single and simple manual step). Our software is always in a deployable state. • Continuous Integration is a prerequisite • Probably you will need feature flags (among other techniques) • Deployments need to be automated 14
15. Continuous Deployment Every commit done to the main branch goes through the pipeline and automatically gets put into Production. • Continuous Delivery is a prerequisite • No human intervention You need many things, for example: • Great safety net (different kind of automated tests, statics analysis, monitoring, etc.) • You might certainly need feature flags (or/plus other parallel changes techniques) • Easy way to rollback 15
16. https://faun.pub/ci-cd-in-basic-terms-8cf739914d20 It is NOT Continuous Integration if you build and test in a branch different to the main branch 16
17. Deployment != Release • Decouple “Deployment” and “Release” when needed (e.g. with feature flags) ◦ Deployed: the code is “put” in the Production systems, maybe not visible. ▪ With a feature toggle/flag, or an unknown URL or a hidden HTML element, etc. ◦ Released: visible to the users • Deploying is a technical decision • Releasing is a business decision 17
18. Continuous Deployment/Release • Deploying to Production (without releasing) has value in itself: ◦ The whole pipeline works fine (e.g. tests, infrastructure, etc.) ◦ We check that nothing breaks (even not being active). ◦ We might want to partially activate it (e.g. with feature toggles/flags). E.g. canary releases. ◦ We learn about the HOW. • Release: we get continuous feedback from our users/customers/business (are we building the right thing?). We learn about the WHAT. 18
19. “Why” and “what for”: some principles of software development 19
20. We want a feedback cycle “as short as possible” Longer delay between integrations/ deployments https://www.goodreads.com/book/show/34695798-release-it Higher risk of bugs and downtime More changes in each integration/ deployment Review processes A great vicious cycle! We aim for: • ↓ Lead time • ↓ Product delivery lead time 20
21. MMMSS: Many More Much Smaller Steps The smaller the chunks/pieces to integrate/deploy/release… • Less probability of individual errors (an error in one piece) • Less probability of combined errors (errors because of the interaction of the pieces) • Easier to know what failed and why • Easier to fix it • Less traumatic to revert/undo the last failing chunk/piece/step (if needed) • Less human stress, less burnout. Less drama. • Faster learning, much faster feedback (from business/users and from systems) • More focus on the important things: thinking, improving and solving business problems • Everything gets easier: bigger motivation and reinforcement to take even smaller steps https://www.geepawhill.org/series/many-more-much-smaller-steps/ 21
22. But wait… let’s take several steps back... 22
23. Why and what for do we develop software professionally? To solve problems which can not be solved in a more optimal/efficient way without software 23
24. • The best line of code is the one not written. ◦ “Fewest elements”: https://martinfowler.com/bliki/BeckDesignRules.html • Code needs to be maintained. • Code needs to be tested. • Code needs to be understood. • A line of code is one more possible point of failure. • It is inventory with unknown bugs until it is deployed and released. • Software has a COST (and it delivers no value until it is in hands of the users). https://www.eferro.net/2021/02/basal-cost-of-software.html https://wiki.c2.com/?SoftwareAsLiability 24 Code is not an asset, it is liability!
25. Which are some “best” practices for Continuous Deployment? 25
26. Branching strategies (over-simplified) • Feature branching ◦ One branch per feature (sometimes it means per developer) • GitFlow • GitHub Flow • Other combined strategies ◦ Feature + Main + Release branches ◦ Release trains • Ship, show, ask https://martinfowler.com/articles/branching-patterns.html https://nvie.com/posts/a-successful-git-branching-model/ 26
27. • For feeling safer ◦ Peer review: Someone else can take a look to it and detect problems ◦ Manual testing • For rising the code quality: code reviews • For sharing/spreading knowledge • For getting the control (lack of trust): ◦ "Safe" way to accept contributions from other people/teams ◦ Final review centralized in a few people (e.g. QA, PM, committee, etc.) Why do we work with branches, code reviews and Pull Requests? 27
28. • It completely breaks the flow ◦ Increased lead time: no Continuous Delivery/Deployment ▪ Blocks, dependencies, waiting times… ◦ The cost of context switching • Increased risk ◦ It’s too late: someone already worked several hours (days?) on it…. Waste. ◦ Big batch: more probability of things breaking or not fitting… or not being needed anymore. ◦ Sometimes false feeling of “safety” because of superficial code reviews (e.g. tenths/hundreds of lines changed: are you kidding? LGTM 👌). 28 Problems with branches and Pull Requests
29. Problems with branches and Pull Requests • Async blocking code reviews: ◦ you don’t have all the context of the person for every decision made. (USUALLY) A SOURCE OF INEFFICIENCIES AND HUMAN CONFLICTS 29 https://twitter.com/d_stepanovic/status/1379451260638785536
30. (IMO) When do branches or pull/merge requests might make sense? • OSS: very async, worldwide distributed unknown people • Contributions between teams inside a company (“InnerSource”) ◦ Though, even better: pair with someone from the other team • For some “exceptional” reason, someone working on their own in something “complex” or with lots of new decisions ◦ “Ship / Show / Ask” branching strategy 30
31. This is what it REALLY happens when branching... https://martinfowler.com/articles/branching-patterns.html#continuous-integration https://twitter.com/jahnnie/status/937917022247120898 31
32. Which branching strategy would “better” fit with reducing the risk and optimizing the continuous learning and the continuous and sustainable delivery of value? (and having less people suffering!!) 32
34. Trunk-Based Development for the win!! https://unsplash.com/photos/abkEAOjnY0s 34
35. 35 Trunk-Based Development == Continuous Integration https://martinfowler.com/bliki/SemanticDiffusion.html
36. Keeping one single branch “by default” and pushing your code there every few minutes (… living together with many other practices, e.g. pairing + TDD + micro-commits + strong/safe/fast CD pipeline +...) How to get the most out of Trunk-Based Development? 36
37. Some practices towards Continuous Deployment Trunk-Based Development Parallel changes techniques (e.g. feature toggles, branch by abstraction) Baby commits (Many More Much Smaller Steps) A “Lean based” product management vision Fast, automated and easy rollbacks Continuous Deployment pipeline (fast, safe and automated) “Loosely-coupled” and autonomous teams/services An engineering culture that supports it pair/mob/ensemble programming Technical “excellence”: code easy to understand/change/test Monitoring + Logging + Traces + o11y Test-Driven Development (TDD / BDD) 37
38. A “Lean based” product management vision • Vertical slicing: ◦ Simplified and small end-to-end chunks (“maximize the amount of work not done”) ◦ Business (user stories): GREAT (release) > we learn the WHAT ◦ Technical: good (deployment) > we learn the HOW • Continuous prioritization • Focus on value/impact (aligned with “higher company goals”) ◦ Flow efficiency, not resource efficiency ◦ #NoEstimates: small chunks continuously prioritized and delivered to our users • Pull system (keep it flowing!!) ◦ Limit WIP (Work In Progress): stop starting, start finishing! ◦ Guided by the system/team capacity + JIT (last responsible moment - “Defer commitment”) 38
39. • https://martinfowler.com/bliki/ParallelChange.html • Postel’s law: “Be conservative in what you do, be liberal in what you accept from others” • Feature toggles (aka “feature flags”) ◦ Remember to clean them up!! • Branch by abstraction • Strangler Fig pattern ◦ Strangler Fig Application ◦ Refactoring Legacy Code with the Strangler Fig Pattern • Anti-Corruption Layer (ACL, one of the DDD patterns) Parallel changes 101 39
40. Deployment techniques • Canary releases ◦ Technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure and making it available to everybody. • Blue/green deployment ◦ Blue green deployment is an application release model that gradually transfers user traffic from a previous version of an app or microservice to a nearly identical new release—both of which are running in production. • Dark Launching ◦ Taking a new or changed back-end behavior and calling it from existing users without the users being able to tell it's being called. 40
41. What if I can not do ALL of that? • Keep calm and do not jump into deploying immediately to Production (yet) • It is sometimes a sign of deeper inefficiencies and improvement opportunities, it just makes them “explicit” (if we have enough awareness, self-criticism and humility) Congratulations! We have a great opportunity of improvement ahead! 🥳 41
42. • Risk management: ◦ smaller batches/chunks more frequently integrated/deployed ◦ One-piece flow (“flow efficiency” vs “resource efficiency”) • Learn “fast” (not “fail fast”): about the “WHAT” and the “HOW” ◦ That’s where the BOTTLENECK resides in SW development • “Fast” feedback: from the users/business and from the system • REALLY continuous and sustainable delivery of value What is Continuous Deployment “really” about? 42
43. Real life™ examples: Continuous Deployment pipelines 43
44. Example 1: Continuous Deployment pipeline https://martinfowler.com/bliki/StranglerFigApplication.html 44
45. Example 2: Continuous Deployment pipeline (DevEx team) 45 Drama-free experience: you even forget you’re continuously deploying to Production!! 🥳
46. • Having a lot of DISCIPLINE ◦ MMMSS: Many More Much Smaller Steps ◦ E.g. merciless/opportunistic refactoring • Simplicity and FOCUS (stop starting, start finishing!) • Collective continuous improvement mindset, EMPATHY and HUMILITY • Slack time: not being fully occupied. Optimize “flow efficiency”. • Support from the C-level • Systemic vision and leveraging effort/impact What else matters? 46
47. When NOT to do Continuous Deployment 47
48. When I might NOT to do Continuous Deployment? • Spikes ◦ After the spike, start from scratch in the main branch • Most OSS projects (super-async, no pairing, distributed timezones…) ◦ Lack of trust (but inside a company, lack of trust is unacceptable) • When someone worked on their own on something complex/critical/new • Complex infrastructure changes not possible to do in “baby steps” • When the team doesn’t feel “comfortable enough” ◦ Really hard to achieve if nobody in the team has previous experience. 48
49. When you have other “basic problems” to be solved, e.g.: • Lack of team empowerment and psychological safety • Safety net not robust enough (e.g. automated testing: acceptance, e2e, integration) • No simple/easy rollback mechanism • High incidents/bugs/”change failure” rate in Production (first stabilize it…) When NOT to jump into Continuous Deployment 49
50. What about highly regulated environments? Case Studies: Modern Development Practices In Highly Regulated Environments (by Charity Majors) 50
51. How to start? (a humble proposal with a few ideas) 51
52. • Get support (hire/contract people with experience in these principles and practices) • Understand why we work the way we work: observe, ask, do not judge! • Get a clear high level vision about the value streams (the e2e processes) • Explain and share the goals, the current pains, the improvements expected. • Start working in pairs (even better: try ensemble/mob!!) ◦ Collaborate more and more, build trust, break silos… • Write automated tests (of “all needed kinds”) and include them in the pipeline • Create deliberated learning spaces: e.g. katas, book club, etc. • Set up a continuous integration workflow/process that can monitor your branches and run the tests automatically for every new commit merged. How to start? (I) 52
53. How to start? (II) •Simplify your branching strategy: ◦ Ship, show, ask ◦ e.g. start with GitHub Flow • Limit the work in progress a.k.a. WIP (e.g. half the number of developers) •Automate your deployment to Dev/Staging/Production (even if it has several steps) ◦ Configure a release/deployment pipeline which breaks if needed (Andon a.k.a. “Stop the line”). A pipeline break should be a top priority (only below a production incident) • [only if needed] Get a “pre-production server” where you improve the safety net ◦ So similar to production as possible (Infrastructure as Code for the win!) ◦ Run there tests like Acceptance, End to end, Load tests?, etc. 53
54. How to start? (III) • Make your branches ages visible and limit it to 24 hours • Start using feature toggles/flags (e.g. for changes which will take more than 2-3 days) • Improve your baby steps and baby commits skills ◦ Learn about parallel changes techniques ◦ Learn to take many much smaller pieces to be deployed • Improve your vertical slicing and business simplification skills • Reduce the branches lifetime to 2 hours. • Automate your rollbacks (or at least have a “one-click” operation) 54
55. How to start? (IV) • Try Test-Driven Development (again: get support if nobody has deep experience) • Speed up your tests and pipeline if needed (ideally whole process under 15 min.) • Avoid PRs/MRs by default, work with one single main remote branch by default. • Pick up a non-critical service (e.g. a simple API) and start trying Continuous Delivery (a push deploys your code to the staging environment). • Remove the manual step: deploy to Production after Dev/Staging if everything works! In the “worst case scenario”, if you don’t finish with a Continuous Deployment pipeline, you will have learnt a lot and improved in many other things!!! 🥳 55
56. Final Recap 56
57. • Continuous Deployment is not about “creating and configuring a pipeline”, but about: ◦ Risk management (both business and technical risk) ◦ Reduced lead time ▪ Continuous and sustainable delivery of value to our USERS/CUSTOMERS/BUSINESS ▪ Fast feedback loops (users and systems) for learning • Continuous Deployment is a great excuse to improve and detect many inefficiencies: ◦ What is stopping us to be able to start working like that tomorrow? ◦ What should be “changed” to achieve it? ◦ Everything is “connected”!! (systems thinking) Final Recap 57
58. The challenge here is not technological but sociotechnical Keep calm and start in baby steps… but start! (experimentation mindset for the win!) Final Recap 58
59. Recommended people, books and resources 59
60. Recommended books 60
61. Recommended books 61
62. Recommended books 62
63. People to follow • Eduardo Ferro ◦ https://twitter.com/eferro ◦ https://www.eferro.net/ • Jessica Kerr ◦ https://twitter.com/jessitron ◦ https://jessitron.com/ • Dave Farley ◦ https://twitter.com/davefarley77 ◦ https://www.youtube.com/c/ContinuousDelivery • Charity Majors ◦ https://twitter.com/mipsytipsy ◦ https://charity.wtf/ • Jez Humble ◦ https://twitter.com/jezhumble ◦ https://continuousdelivery.com/ • Geepaw Hill ◦ https://mastodon.social/@GeePawHill ◦ https://www.geepawhill.org/ • Clare Sudbery ◦ https://twitter.com/ClareSudbery ◦ https://insimpleterms.blog/ • Kent Beck ◦ https://twitter.com/KentBeck ◦ https://www.kentbeck.com/ • Emily Bache ◦ https://twitter.com/emilybache ◦ https://www.youtube.com/@EmilyBache-tech-coach • Martin Fowler ◦ https://twitter.com/martinfowler ◦ https://martinfowler.com/ 63
64. Resources (I) • Continuous Integration, Delivery and Deployment ◦ “A major revision of Continuous Integration” (by Martin Fowler) ◦ Continuous Integration: That’s Not What They Meant • Clare Sudbery • YOW! 2023 (video) ◦ Minimum Viable Continuous Delivery (minimumcd.org) ◦ CI/CD: the what, why and how ◦ Continuous Delivery vs Continuous Deployment (video by Dave Farley) ◦ https://martinfowler.com/articles/continuousIntegration.html ◦ https://www.infoq.com/articles/continuous-delivery-teamplay/ ◦ https://martinfowler.com/bliki/ContinuousDelivery.html ◦ Card game to design Continuous Delivery pipelines (by Emily Bache) ◦ Continuous integration vs. delivery vs. deployment ◦ https://charity.wtf/2021/08/27/software-deploys-and-cognitive-biases/ ◦ https://martinfowler.com/bliki/FrequencyReducesDifficulty.html 64
65. Resources (II) • Lean and Flow ◦ https://abrahamvallez.medium.com/one-piece-flow-7d5ade46d203 ◦ https://jessitron.com/2021/01/18/when-costs-are-nonlinear-keep-it-small/ ◦ https://www.geepawhill.org/2021/05/18/big-batch-releases ◦ https://www.eferro.net/2021/01/small-batches-for-win-continuous.html ◦ https://refactoring.fm/p/how-shipping-fast-changes-your-life ◦ https://innolution.com/blog/limiting-work-in-process-wip-and-having-small-batches-is-antifragile ◦ Henrik Kniberg : Multiple WIP vs One Piece Flow Example [Video] 65
66. Resources (III) • Parallel changes ◦ https://martinfowler.com/bliki/ParallelChange.html ◦ Feature toggles (aka “feature flags”) ◦ Branch by abstraction ◦ Strangler Fig pattern ▪ Strangler Fig Application ▪ Refactoring Legacy Code with the Strangler Fig Pattern ◦ Anti-Corruption Layer ◦ The Limited Red Society 66
67. Resources (IV) • Workshops about parallel changes: ◦ Small Safe Steps 3s workshop - Agile (slides), by Eduardo Ferro ◦ https://paucls.wordpress.com/2020/02/26/ci-cd-non-breaking-changes-exercise/ • Canary releases and Blue-green deployment ◦ https://martinfowler.com/bliki/CanaryRelease.html ◦ https://martinfowler.com/bliki/BlueGreenDeployment.html ◦ https://www.redhat.com/en/topics/devops/what-is-blue-green-deployment ◦ https://learn.hashicorp.com/tutorials/terraform/blue-green-canary-tests-deployments ◦ https://aws.amazon.com/es/blogs/compute/bluegreen-deployments-with-amazon-ecs 67
68. Resources (V) • Trunk-Based Development ◦ Lightning talk: “Pusheando a master, que es gerundio” (in Spanish) ◦ Why I prefer trunk-based development (Trisha Gee) ◦ Perceived Barriers to Trunk-Based Development (Dave Farley) ◦ Neil Crawdford - Trunk based development, continuous deployment and why you should adopt them (video) ◦ Of Death and Dying: A Journey to Trunk Based Development (video) ◦ What is Trunk-Based Development? (Clare Sudbery) ◦ https://nelis.boucke.be/post/trunk-based-development/ 68
69. Resources (VI) • Branching, Pull requests and async code reviews ◦ How to stop branching ◦ https://thinkinglabs.io/articles/2022/05/30/on-the-evilness-of-feature-branching-the-problems.html ◦ Async Code Reviews Are Killing Your Company's Throughput [Video] ◦ https://martinfowler.com/articles/branching-patterns.html ◦ Other experiences: ▪ https://twitter.com/jlangr/status/1487135760012627973 ▪ https://twitter.com/aahoogendoorn/status/1531709454701453312 69
70. Resources (VII) • General ◦ https://wiki.c2.com/?SoftwareAsLiability ◦ What is Value? by Jez Humble (video) ◦ Estimar es timar: Example Mapping ◦ Evolutionary Database Design ◦ https://twitter.com/tottinge/status/1748277312200540523 • (Vertical) Slicing: ◦ Agile Delivery - Lean Coffee - Slicing (video) ◦ https://gojko.net/2012/01/23/splitting-user-stories-the-hamburger-method/ ◦ https://xp123.com/articles/slicing-functionality-alternate-paths/ ◦ How to split a user story? (PDF flowchart) ◦ Elephant Carpaccio facilitation guide 70
72. ¿Me ayudas a mejorar? 🙏 https://forms.gle/4omyq7PWoVbkWEJr5 72 @islomar