Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Valor por encima de código: el poder del Despliegue Continuo

Valor por encima de código: el poder del Despliegue Continuo

Slides de la charla dada el 27 de enero de 2024 en la BilboStack: https://bilbostack.com/isidro-lopez

¿Por qué y para qué desarrollamos software o tecnología en general? ¿Qué es realmente la “Integración Continua”? ¿Qué valor aporta la práctica del Despliegue Continuo? ¿En qué principios y prácticas sociotécnicas podemos apoyarnos para sacarle todo el jugo?

Te invito a acompañarme en éstas y algunas otras reflexiones, a compartir humildemente mi experiencia “en el mundo real” sobre el Despliegue Continuo, con varios ejemplos reales y algunas propuestas sobre cómo abordarlo.

Hablaré de por qué la simplicidad (en tecnología y negocio), trabajando en equipo, con pasos seguros extremadamente pequeños y feedback rápido marcan la diferencia. Olvídate del palabrerío técnico y centrémonos en cómo el desarrollo de software Lean y Extreme Programming pueden ser fantásticos aliados para conseguir lo que realmente importa: una entrega de valor continua y a un ritmo sostenible… ¡y disfrutando del proceso!

Esta charla no es solo para programadoras: es para todas aquellas personas que deseen acercarse a “otras maneras de hacer las cosas” más allá del mainstream (sin ánimo de proselitismo sino únicamente de compartir experiencias 😉).

Isidro López

January 28, 2024

More Decks by Isidro López

Other Decks in Programming


  1. 27 de enero de 2024 Valor por encima de código:

    el poder del Despliegue Continuo Isidro López
  2. Ego slide Hemos venido a jugar... Product developer/engineer (and Product

    Engineering Coach and…) @islomar [email protected] <curioso> Teleco, Psicología, Antropología Social y Cultural, Executive MBA… </curioso> 3
  3. 4

  4. 5

  5. 6

  6. 8

  7. Agenda • Continuous X • “Why” and “what for”: some

    principles of software development • Which are some “best” practices for Continuous Deployment? • Real life™ examples: Continuous Deployment pipelines • When NOT to do Continuous Deployment • How to start (a humble proposal) • Final recap • Recommended people, books and resources • Q&A and discussions 10
  8. Continuous Delivery Every commit to the main branch leaves your

    code continuously ready to be safely and quickly deployed to Production (with one single and simple manual step). Our software is always in a deployable state. • Continuous Integration is a prerequisite • Probably you will need feature flags (among other techniques) • Deployments need to be automated 14
  9. Continuous Deployment Every commit done to the main branch goes

    through the pipeline and automatically gets put into Production. • Continuous Delivery is a prerequisite • No human intervention You need many things, for example: • Great safety net (different kind of automated tests, statics analysis, monitoring, etc.) • You might certainly need feature flags (or/plus other parallel changes techniques) • Easy way to rollback 15
  10. Deployment != Release • Decouple “Deployment” and “Release” when needed

    (e.g. with feature flags) ◦ Deployed: the code is “put” in the Production systems, maybe not visible. ▪ With a feature toggle/flag, or an unknown URL or a hidden HTML element, etc. ◦ Released: visible to the users • Deploying is a technical decision • Releasing is a business decision 17
  11. Continuous Deployment/Release • Deploying to Production (without releasing) has value

    in itself: ◦ The whole pipeline works fine (e.g. tests, infrastructure, etc.) ◦ We check that nothing breaks (even not being active). ◦ We might want to partially activate it (e.g. with feature toggles/flags). E.g. canary releases. ◦ We learn about the HOW. • Release: we get continuous feedback from our users/customers/business (are we building the right thing?). We learn about the WHAT. 18
  12. We want a feedback cycle “as short as possible” Longer

    delay between integrations/ deployments https://www.goodreads.com/book/show/34695798-release-it Higher risk of bugs and downtime More changes in each integration/ deployment Review processes A great vicious cycle! We aim for: • ↓ Lead time • ↓ Product delivery lead time 20
  13. MMMSS: Many More Much Smaller Steps The smaller the chunks/pieces

    to integrate/deploy/release… • Less probability of individual errors (an error in one piece) • Less probability of combined errors (errors because of the interaction of the pieces) • Easier to know what failed and why • Easier to fix it • Less traumatic to revert/undo the last failing chunk/piece/step (if needed) • Less human stress, less burnout. Less drama. • Faster learning, much faster feedback (from business/users and from systems) • More focus on the important things: thinking, improving and solving business problems • Everything gets easier: bigger motivation and reinforcement to take even smaller steps https://www.geepawhill.org/series/many-more-much-smaller-steps/ 21
  14. Why and what for do we develop software professionally? To

    solve problems which can not be solved in a more optimal/efficient way without software 23
  15. • The best line of code is the one not

    written. ◦ “Fewest elements”: https://martinfowler.com/bliki/BeckDesignRules.html • Code needs to be maintained. • Code needs to be tested. • Code needs to be understood. • A line of code is one more possible point of failure. • It is inventory with unknown bugs until it is deployed and released. • Software has a COST (and it delivers no value until it is in hands of the users). https://www.eferro.net/2021/02/basal-cost-of-software.html https://wiki.c2.com/?SoftwareAsLiability 24 Code is not an asset, it is liability!
  16. Branching strategies (over-simplified) • Feature branching ◦ One branch per

    feature (sometimes it means per developer) • GitFlow • GitHub Flow • Other combined strategies ◦ Feature + Main + Release branches ◦ Release trains • Ship, show, ask https://martinfowler.com/articles/branching-patterns.html https://nvie.com/posts/a-successful-git-branching-model/ 26
  17. • For feeling safer ◦ Peer review: Someone else can

    take a look to it and detect problems ◦ Manual testing • For rising the code quality: code reviews • For sharing/spreading knowledge • For getting the control (lack of trust): ◦ "Safe" way to accept contributions from other people/teams ◦ Final review centralized in a few people (e.g. QA, PM, committee, etc.) Why do we work with branches, code reviews and Pull Requests? 27
  18. • It completely breaks the flow ◦ Increased lead time:

    no Continuous Delivery/Deployment ▪ Blocks, dependencies, waiting times… ◦ The cost of context switching • Increased risk ◦ It’s too late: someone already worked several hours (days?) on it…. Waste. ◦ Big batch: more probability of things breaking or not fitting… or not being needed anymore. ◦ Sometimes false feeling of “safety” because of superficial code reviews (e.g. tenths/hundreds of lines changed: are you kidding? LGTM 👌). 28 Problems with branches and Pull Requests
  19. Problems with branches and Pull Requests • Async blocking code

    reviews: ◦ you don’t have all the context of the person for every decision made. (USUALLY) A SOURCE OF INEFFICIENCIES AND HUMAN CONFLICTS 29 https://twitter.com/d_stepanovic/status/1379451260638785536
  20. (IMO) When do branches or pull/merge requests might make sense?

    • OSS: very async, worldwide distributed unknown people • Contributions between teams inside a company (“InnerSource”) ◦ Though, even better: pair with someone from the other team • For some “exceptional” reason, someone working on their own in something “complex” or with lots of new decisions ◦ “Ship / Show / Ask” branching strategy 30
  21. Which branching strategy would “better” fit with reducing the risk

    and optimizing the continuous learning and the continuous and sustainable delivery of value? (and having less people suffering!!) 32
  22. 33

  23. Keeping one single branch “by default” and pushing your code

    there every few minutes (… living together with many other practices, e.g. pairing + TDD + micro-commits + strong/safe/fast CD pipeline +...) How to get the most out of Trunk-Based Development? 36
  24. Some practices towards Continuous Deployment Trunk-Based Development Parallel changes techniques

    (e.g. feature toggles, branch by abstraction) Baby commits (Many More Much Smaller Steps) A “Lean based” product management vision Fast, automated and easy rollbacks Continuous Deployment pipeline (fast, safe and automated) “Loosely-coupled” and autonomous teams/services An engineering culture that supports it pair/mob/ensemble programming Technical “excellence”: code easy to understand/change/test Monitoring + Logging + Traces + o11y Test-Driven Development (TDD / BDD) 37
  25. A “Lean based” product management vision • Vertical slicing: ◦

    Simplified and small end-to-end chunks (“maximize the amount of work not done”) ◦ Business (user stories): GREAT (release) > we learn the WHAT ◦ Technical: good (deployment) > we learn the HOW • Continuous prioritization • Focus on value/impact (aligned with “higher company goals”) ◦ Flow efficiency, not resource efficiency ◦ #NoEstimates: small chunks continuously prioritized and delivered to our users • Pull system (keep it flowing!!) ◦ Limit WIP (Work In Progress): stop starting, start finishing! ◦ Guided by the system/team capacity + JIT (last responsible moment - “Defer commitment”) 38
  26. • https://martinfowler.com/bliki/ParallelChange.html • Postel’s law: “Be conservative in what you

    do, be liberal in what you accept from others” • Feature toggles (aka “feature flags”) ◦ Remember to clean them up!! • Branch by abstraction • Strangler Fig pattern ◦ Strangler Fig Application ◦ Refactoring Legacy Code with the Strangler Fig Pattern • Anti-Corruption Layer (ACL, one of the DDD patterns) Parallel changes 101 39
  27. Deployment techniques • Canary releases ◦ Technique to reduce the

    risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure and making it available to everybody. • Blue/green deployment ◦ Blue green deployment is an application release model that gradually transfers user traffic from a previous version of an app or microservice to a nearly identical new release—both of which are running in production. • Dark Launching ◦ Taking a new or changed back-end behavior and calling it from existing users without the users being able to tell it's being called. 40
  28. What if I can not do ALL of that? •

    Keep calm and do not jump into deploying immediately to Production (yet) • It is sometimes a sign of deeper inefficiencies and improvement opportunities, it just makes them “explicit” (if we have enough awareness, self-criticism and humility) Congratulations! We have a great opportunity of improvement ahead! 🥳 41
  29. • Risk management: ◦ smaller batches/chunks more frequently integrated/deployed ◦

    One-piece flow (“flow efficiency” vs “resource efficiency”) • Learn “fast” (not “fail fast”): about the “WHAT” and the “HOW” ◦ That’s where the BOTTLENECK resides in SW development • “Fast” feedback: from the users/business and from the system • REALLY continuous and sustainable delivery of value What is Continuous Deployment “really” about? 42
  30. Example 2: Continuous Deployment pipeline (DevEx team) 45 Drama-free experience:

    you even forget you’re continuously deploying to Production!! 🥳
  31. • Having a lot of DISCIPLINE ◦ MMMSS: Many More

    Much Smaller Steps ◦ E.g. merciless/opportunistic refactoring • Simplicity and FOCUS (stop starting, start finishing!) • Collective continuous improvement mindset, EMPATHY and HUMILITY • Slack time: not being fully occupied. Optimize “flow efficiency”. • Support from the C-level • Systemic vision and leveraging effort/impact What else matters? 46
  32. When I might NOT to do Continuous Deployment? • Spikes

    ◦ After the spike, start from scratch in the main branch • Most OSS projects (super-async, no pairing, distributed timezones…) ◦ Lack of trust (but inside a company, lack of trust is unacceptable) • When someone worked on their own on something complex/critical/new • Complex infrastructure changes not possible to do in “baby steps” • When the team doesn’t feel “comfortable enough” ◦ Really hard to achieve if nobody in the team has previous experience. 48
  33. When you have other “basic problems” to be solved, e.g.:

    • Lack of team empowerment and psychological safety • Safety net not robust enough (e.g. automated testing: acceptance, e2e, integration) • No simple/easy rollback mechanism • High incidents/bugs/”change failure” rate in Production (first stabilize it…) When NOT to jump into Continuous Deployment 49
  34. What about highly regulated environments? Case Studies: Modern Development Practices

    In Highly Regulated Environments (by Charity Majors) 50
  35. • Get support (hire/contract people with experience in these principles

    and practices) • Understand why we work the way we work: observe, ask, do not judge! • Get a clear high level vision about the value streams (the e2e processes) • Explain and share the goals, the current pains, the improvements expected. • Start working in pairs (even better: try ensemble/mob!!) ◦ Collaborate more and more, build trust, break silos… • Write automated tests (of “all needed kinds”) and include them in the pipeline • Create deliberated learning spaces: e.g. katas, book club, etc. • Set up a continuous integration workflow/process that can monitor your branches and run the tests automatically for every new commit merged. How to start? (I) 52
  36. How to start? (II) •Simplify your branching strategy: ◦ Ship,

    show, ask ◦ e.g. start with GitHub Flow • Limit the work in progress a.k.a. WIP (e.g. half the number of developers) •Automate your deployment to Dev/Staging/Production (even if it has several steps) ◦ Configure a release/deployment pipeline which breaks if needed (Andon a.k.a. “Stop the line”). A pipeline break should be a top priority (only below a production incident) • [only if needed] Get a “pre-production server” where you improve the safety net ◦ So similar to production as possible (Infrastructure as Code for the win!) ◦ Run there tests like Acceptance, End to end, Load tests?, etc. 53
  37. How to start? (III) • Make your branches ages visible

    and limit it to 24 hours • Start using feature toggles/flags (e.g. for changes which will take more than 2-3 days) • Improve your baby steps and baby commits skills ◦ Learn about parallel changes techniques ◦ Learn to take many much smaller pieces to be deployed • Improve your vertical slicing and business simplification skills • Reduce the branches lifetime to 2 hours. • Automate your rollbacks (or at least have a “one-click” operation) 54
  38. How to start? (IV) • Try Test-Driven Development (again: get

    support if nobody has deep experience) • Speed up your tests and pipeline if needed (ideally whole process under 15 min.) • Avoid PRs/MRs by default, work with one single main remote branch by default. • Pick up a non-critical service (e.g. a simple API) and start trying Continuous Delivery (a push deploys your code to the staging environment). • Remove the manual step: deploy to Production after Dev/Staging if everything works! In the “worst case scenario”, if you don’t finish with a Continuous Deployment pipeline, you will have learnt a lot and improved in many other things!!! 🥳 55
  39. • Continuous Deployment is not about “creating and configuring a

    pipeline”, but about: ◦ Risk management (both business and technical risk) ◦ Reduced lead time ▪ Continuous and sustainable delivery of value to our USERS/CUSTOMERS/BUSINESS ▪ Fast feedback loops (users and systems) for learning • Continuous Deployment is a great excuse to improve and detect many inefficiencies: ◦ What is stopping us to be able to start working like that tomorrow? ◦ What should be “changed” to achieve it? ◦ Everything is “connected”!! (systems thinking) Final Recap 57
  40. The challenge here is not technological but sociotechnical Keep calm

    and start in baby steps… but start! (experimentation mindset for the win!) Final Recap 58
  41. People to follow • Eduardo Ferro ◦ https://twitter.com/eferro ◦ https://www.eferro.net/

    • Jessica Kerr ◦ https://twitter.com/jessitron ◦ https://jessitron.com/ • Dave Farley ◦ https://twitter.com/davefarley77 ◦ https://www.youtube.com/c/ContinuousDelivery • Charity Majors ◦ https://twitter.com/mipsytipsy ◦ https://charity.wtf/ • Jez Humble ◦ https://twitter.com/jezhumble ◦ https://continuousdelivery.com/ • Geepaw Hill ◦ https://mastodon.social/@GeePawHill ◦ https://www.geepawhill.org/ • Clare Sudbery ◦ https://twitter.com/ClareSudbery ◦ https://insimpleterms.blog/ • Kent Beck ◦ https://twitter.com/KentBeck ◦ https://www.kentbeck.com/ • Emily Bache ◦ https://twitter.com/emilybache ◦ https://www.youtube.com/@EmilyBache-tech-coach • Martin Fowler ◦ https://twitter.com/martinfowler ◦ https://martinfowler.com/ 63
  42. Resources (I) • Continuous Integration, Delivery and Deployment ◦ “A

    major revision of Continuous Integration” (by Martin Fowler) ◦ Continuous Integration: That’s Not What They Meant • Clare Sudbery • YOW! 2023 (video) ◦ Minimum Viable Continuous Delivery (minimumcd.org) ◦ CI/CD: the what, why and how ◦ Continuous Delivery vs Continuous Deployment (video by Dave Farley) ◦ https://martinfowler.com/articles/continuousIntegration.html ◦ https://www.infoq.com/articles/continuous-delivery-teamplay/ ◦ https://martinfowler.com/bliki/ContinuousDelivery.html ◦ Card game to design Continuous Delivery pipelines (by Emily Bache) ◦ Continuous integration vs. delivery vs. deployment ◦ https://charity.wtf/2021/08/27/software-deploys-and-cognitive-biases/ ◦ https://martinfowler.com/bliki/FrequencyReducesDifficulty.html 64
  43. Resources (II) • Lean and Flow ◦ https://abrahamvallez.medium.com/one-piece-flow-7d5ade46d203 ◦ https://jessitron.com/2021/01/18/when-costs-are-nonlinear-keep-it-small/

    ◦ https://www.geepawhill.org/2021/05/18/big-batch-releases ◦ https://www.eferro.net/2021/01/small-batches-for-win-continuous.html ◦ https://refactoring.fm/p/how-shipping-fast-changes-your-life ◦ https://innolution.com/blog/limiting-work-in-process-wip-and-having-small-batches-is-antifragile ◦ Henrik Kniberg : Multiple WIP vs One Piece Flow Example [Video] 65
  44. Resources (III) • Parallel changes ◦ https://martinfowler.com/bliki/ParallelChange.html ◦ Feature toggles

    (aka “feature flags”) ◦ Branch by abstraction ◦ Strangler Fig pattern ▪ Strangler Fig Application ▪ Refactoring Legacy Code with the Strangler Fig Pattern ◦ Anti-Corruption Layer ◦ The Limited Red Society 66
  45. Resources (IV) • Workshops about parallel changes: ◦ Small Safe

    Steps 3s workshop - Agile (slides), by Eduardo Ferro ◦ https://paucls.wordpress.com/2020/02/26/ci-cd-non-breaking-changes-exercise/ • Canary releases and Blue-green deployment ◦ https://martinfowler.com/bliki/CanaryRelease.html ◦ https://martinfowler.com/bliki/BlueGreenDeployment.html ◦ https://www.redhat.com/en/topics/devops/what-is-blue-green-deployment ◦ https://learn.hashicorp.com/tutorials/terraform/blue-green-canary-tests-deployments ◦ https://aws.amazon.com/es/blogs/compute/bluegreen-deployments-with-amazon-ecs 67
  46. Resources (V) • Trunk-Based Development ◦ Lightning talk: “Pusheando a

    master, que es gerundio” (in Spanish) ◦ Why I prefer trunk-based development (Trisha Gee) ◦ Perceived Barriers to Trunk-Based Development (Dave Farley) ◦ Neil Crawdford - Trunk based development, continuous deployment and why you should adopt them (video) ◦ Of Death and Dying: A Journey to Trunk Based Development (video) ◦ What is Trunk-Based Development? (Clare Sudbery) ◦ https://nelis.boucke.be/post/trunk-based-development/ 68
  47. Resources (VI) • Branching, Pull requests and async code reviews

    ◦ How to stop branching ◦ https://thinkinglabs.io/articles/2022/05/30/on-the-evilness-of-feature-branching-the-problems.html ◦ Async Code Reviews Are Killing Your Company's Throughput [Video] ◦ https://martinfowler.com/articles/branching-patterns.html ◦ Other experiences: ▪ https://twitter.com/jlangr/status/1487135760012627973 ▪ https://twitter.com/aahoogendoorn/status/1531709454701453312 69
  48. Resources (VII) • General ◦ https://wiki.c2.com/?SoftwareAsLiability ◦ What is Value?

    by Jez Humble (video) ◦ Estimar es timar: Example Mapping ◦ Evolutionary Database Design ◦ https://twitter.com/tottinge/status/1748277312200540523 • (Vertical) Slicing: ◦ Agile Delivery - Lean Coffee - Slicing (video) ◦ https://gojko.net/2012/01/23/splitting-user-stories-the-hamburger-method/ ◦ https://xp123.com/articles/slicing-functionality-alternate-paths/ ◦ How to split a user story? (PDF flowchart) ◦ Elephant Carpaccio facilitation guide 70
  49. 71

  50. 73