Slide 1

Slide 1 text

27 de enero de 2024 Valor por encima de código: el poder del Despliegue Continuo Isidro López

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Ego slide Hemos venido a jugar... Product developer/engineer (and Product Engineering Coach and…) @islomar [email protected] Teleco, Psicología, Antropología Social y Cultural, Executive MBA… 3

Slide 4

Slide 4 text

4

Slide 5

Slide 5 text

5

Slide 6

Slide 6 text

6

Slide 7

Slide 7 text

Photo: https://www.indiatimes.com/health/tips-tricks/this-harvard-psychologist-reveals-the-first-couple-of-criteria-people-judge-you-on-351236 .html NO JUDGEMENT !! 7

Slide 8

Slide 8 text

8

Slide 9

Slide 9 text

#NoTools #NoCode 9

Slide 10

Slide 10 text

Agenda ● Continuous X ● “Why” and “what for”: some principles of software development ● Which are some “best” practices for Continuous Deployment? ● Real life™ examples: Continuous Deployment pipelines ● When NOT to do Continuous Deployment ● How to start (a humble proposal) ● Final recap ● Recommended people, books and resources ● Q&A and discussions 10

Slide 11

Slide 11 text

Continuous X 11

Slide 12

Slide 12 text

Continuous X ● Continuous Integration ● Continuous Delivery ● Continuous Deployment 12

Slide 13

Slide 13 text

What is Continuous Integration? 13

Slide 14

Slide 14 text

Continuous Delivery Every commit to the main branch leaves your code continuously ready to be safely and quickly deployed to Production (with one single and simple manual step). Our software is always in a deployable state. ● Continuous Integration is a prerequisite ● Probably you will need feature flags (among other techniques) ● Deployments need to be automated 14

Slide 15

Slide 15 text

Continuous Deployment Every commit done to the main branch goes through the pipeline and automatically gets put into Production. ● Continuous Delivery is a prerequisite ● No human intervention You need many things, for example: ● Great safety net (different kind of automated tests, statics analysis, monitoring, etc.) ● You might certainly need feature flags (or/plus other parallel changes techniques) ● Easy way to rollback 15

Slide 16

Slide 16 text

https://faun.pub/ci-cd-in-basic-terms-8cf739914d20 It is NOT Continuous Integration if you build and test in a branch different to the main branch 16

Slide 17

Slide 17 text

Deployment != Release ● Decouple “Deployment” and “Release” when needed (e.g. with feature flags) ○ Deployed: the code is “put” in the Production systems, maybe not visible. ■ With a feature toggle/flag, or an unknown URL or a hidden HTML element, etc. ○ Released: visible to the users ● Deploying is a technical decision ● Releasing is a business decision 17

Slide 18

Slide 18 text

Continuous Deployment/Release ● Deploying to Production (without releasing) has value in itself: ○ The whole pipeline works fine (e.g. tests, infrastructure, etc.) ○ We check that nothing breaks (even not being active). ○ We might want to partially activate it (e.g. with feature toggles/flags). E.g. canary releases. ○ We learn about the HOW. ● Release: we get continuous feedback from our users/customers/business (are we building the right thing?). We learn about the WHAT. 18

Slide 19

Slide 19 text

“Why” and “what for”: some principles of software development 19

Slide 20

Slide 20 text

We want a feedback cycle “as short as possible” Longer delay between integrations/ deployments https://www.goodreads.com/book/show/34695798-release-it Higher risk of bugs and downtime More changes in each integration/ deployment Review processes A great vicious cycle! We aim for: ● ↓ Lead time ● ↓ Product delivery lead time 20

Slide 21

Slide 21 text

MMMSS: Many More Much Smaller Steps The smaller the chunks/pieces to integrate/deploy/release… ● Less probability of individual errors (an error in one piece) ● Less probability of combined errors (errors because of the interaction of the pieces) ● Easier to know what failed and why ● Easier to fix it ● Less traumatic to revert/undo the last failing chunk/piece/step (if needed) ● Less human stress, less burnout. Less drama. ● Faster learning, much faster feedback (from business/users and from systems) ● More focus on the important things: thinking, improving and solving business problems ● Everything gets easier: bigger motivation and reinforcement to take even smaller steps https://www.geepawhill.org/series/many-more-much-smaller-steps/ 21

Slide 22

Slide 22 text

But wait… let’s take several steps back... 22

Slide 23

Slide 23 text

Why and what for do we develop software professionally? To solve problems which can not be solved in a more optimal/efficient way without software 23

Slide 24

Slide 24 text

● The best line of code is the one not written. ○ “Fewest elements”: https://martinfowler.com/bliki/BeckDesignRules.html ● Code needs to be maintained. ● Code needs to be tested. ● Code needs to be understood. ● A line of code is one more possible point of failure. ● It is inventory with unknown bugs until it is deployed and released. ● Software has a COST (and it delivers no value until it is in hands of the users). https://www.eferro.net/2021/02/basal-cost-of-software.html https://wiki.c2.com/?SoftwareAsLiability 24 Code is not an asset, it is liability!

Slide 25

Slide 25 text

Which are some “best” practices for Continuous Deployment? 25

Slide 26

Slide 26 text

Branching strategies (over-simplified) ● Feature branching ○ One branch per feature (sometimes it means per developer) ● GitFlow ● GitHub Flow ● Other combined strategies ○ Feature + Main + Release branches ○ Release trains ● Ship, show, ask https://martinfowler.com/articles/branching-patterns.html https://nvie.com/posts/a-successful-git-branching-model/ 26

Slide 27

Slide 27 text

● For feeling safer ○ Peer review: Someone else can take a look to it and detect problems ○ Manual testing ● For rising the code quality: code reviews ● For sharing/spreading knowledge ● For getting the control (lack of trust): ○ "Safe" way to accept contributions from other people/teams ○ Final review centralized in a few people (e.g. QA, PM, committee, etc.) Why do we work with branches, code reviews and Pull Requests? 27

Slide 28

Slide 28 text

● It completely breaks the flow ○ Increased lead time: no Continuous Delivery/Deployment ■ Blocks, dependencies, waiting times… ○ The cost of context switching ● Increased risk ○ It’s too late: someone already worked several hours (days?) on it…. Waste. ○ Big batch: more probability of things breaking or not fitting… or not being needed anymore. ○ Sometimes false feeling of “safety” because of superficial code reviews (e.g. tenths/hundreds of lines changed: are you kidding? LGTM 👌). 28 Problems with branches and Pull Requests

Slide 29

Slide 29 text

Problems with branches and Pull Requests ● Async blocking code reviews: ○ you don’t have all the context of the person for every decision made. (USUALLY) A SOURCE OF INEFFICIENCIES AND HUMAN CONFLICTS 29 https://twitter.com/d_stepanovic/status/1379451260638785536

Slide 30

Slide 30 text

(IMO) When do branches or pull/merge requests might make sense? ● OSS: very async, worldwide distributed unknown people ● Contributions between teams inside a company (“InnerSource”) ○ Though, even better: pair with someone from the other team ● For some “exceptional” reason, someone working on their own in something “complex” or with lots of new decisions ○ “Ship / Show / Ask” branching strategy 30

Slide 31

Slide 31 text

This is what it REALLY happens when branching... https://martinfowler.com/articles/branching-patterns.html#continuous-integration https://twitter.com/jahnnie/status/937917022247120898 31

Slide 32

Slide 32 text

Which branching strategy would “better” fit with reducing the risk and optimizing the continuous learning and the continuous and sustainable delivery of value? (and having less people suffering!!) 32

Slide 33

Slide 33 text

33

Slide 34

Slide 34 text

Trunk-Based Development for the win!! https://unsplash.com/photos/abkEAOjnY0s 34

Slide 35

Slide 35 text

35 Trunk-Based Development == Continuous Integration https://martinfowler.com/bliki/SemanticDiffusion.html

Slide 36

Slide 36 text

Keeping one single branch “by default” and pushing your code there every few minutes (… living together with many other practices, e.g. pairing + TDD + micro-commits + strong/safe/fast CD pipeline +...) How to get the most out of Trunk-Based Development? 36

Slide 37

Slide 37 text

Some practices towards Continuous Deployment Trunk-Based Development Parallel changes techniques (e.g. feature toggles, branch by abstraction) Baby commits (Many More Much Smaller Steps) A “Lean based” product management vision Fast, automated and easy rollbacks Continuous Deployment pipeline (fast, safe and automated) “Loosely-coupled” and autonomous teams/services An engineering culture that supports it pair/mob/ensemble programming Technical “excellence”: code easy to understand/change/test Monitoring + Logging + Traces + o11y Test-Driven Development (TDD / BDD) 37

Slide 38

Slide 38 text

A “Lean based” product management vision ● Vertical slicing: ○ Simplified and small end-to-end chunks (“maximize the amount of work not done”) ○ Business (user stories): GREAT (release) > we learn the WHAT ○ Technical: good (deployment) > we learn the HOW ● Continuous prioritization ● Focus on value/impact (aligned with “higher company goals”) ○ Flow efficiency, not resource efficiency ○ #NoEstimates: small chunks continuously prioritized and delivered to our users ● Pull system (keep it flowing!!) ○ Limit WIP (Work In Progress): stop starting, start finishing! ○ Guided by the system/team capacity + JIT (last responsible moment - “Defer commitment”) 38

Slide 39

Slide 39 text

• https://martinfowler.com/bliki/ParallelChange.html • Postel’s law: “Be conservative in what you do, be liberal in what you accept from others” • Feature toggles (aka “feature flags”) ○ Remember to clean them up!! • Branch by abstraction • Strangler Fig pattern ○ Strangler Fig Application ○ Refactoring Legacy Code with the Strangler Fig Pattern • Anti-Corruption Layer (ACL, one of the DDD patterns) Parallel changes 101 39

Slide 40

Slide 40 text

Deployment techniques ● Canary releases ○ Technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure and making it available to everybody. ● Blue/green deployment ○ Blue green deployment is an application release model that gradually transfers user traffic from a previous version of an app or microservice to a nearly identical new release—both of which are running in production. ● Dark Launching ○ Taking a new or changed back-end behavior and calling it from existing users without the users being able to tell it's being called. 40

Slide 41

Slide 41 text

What if I can not do ALL of that? ● Keep calm and do not jump into deploying immediately to Production (yet) ● It is sometimes a sign of deeper inefficiencies and improvement opportunities, it just makes them “explicit” (if we have enough awareness, self-criticism and humility) Congratulations! We have a great opportunity of improvement ahead! 🥳 41

Slide 42

Slide 42 text

• Risk management: ○ smaller batches/chunks more frequently integrated/deployed ○ One-piece flow (“flow efficiency” vs “resource efficiency”) • Learn “fast” (not “fail fast”): about the “WHAT” and the “HOW” ○ That’s where the BOTTLENECK resides in SW development • “Fast” feedback: from the users/business and from the system • REALLY continuous and sustainable delivery of value What is Continuous Deployment “really” about? 42

Slide 43

Slide 43 text

Real life™ examples: Continuous Deployment pipelines 43

Slide 44

Slide 44 text

Example 1: Continuous Deployment pipeline https://martinfowler.com/bliki/StranglerFigApplication.html 44

Slide 45

Slide 45 text

Example 2: Continuous Deployment pipeline (DevEx team) 45 Drama-free experience: you even forget you’re continuously deploying to Production!! 🥳

Slide 46

Slide 46 text

• Having a lot of DISCIPLINE ○ MMMSS: Many More Much Smaller Steps ○ E.g. merciless/opportunistic refactoring • Simplicity and FOCUS (stop starting, start finishing!) • Collective continuous improvement mindset, EMPATHY and HUMILITY • Slack time: not being fully occupied. Optimize “flow efficiency”. • Support from the C-level • Systemic vision and leveraging effort/impact What else matters? 46

Slide 47

Slide 47 text

When NOT to do Continuous Deployment 47

Slide 48

Slide 48 text

When I might NOT to do Continuous Deployment? ● Spikes ○ After the spike, start from scratch in the main branch ● Most OSS projects (super-async, no pairing, distributed timezones…) ○ Lack of trust (but inside a company, lack of trust is unacceptable) ● When someone worked on their own on something complex/critical/new ● Complex infrastructure changes not possible to do in “baby steps” ● When the team doesn’t feel “comfortable enough” ○ Really hard to achieve if nobody in the team has previous experience. 48

Slide 49

Slide 49 text

When you have other “basic problems” to be solved, e.g.: • Lack of team empowerment and psychological safety • Safety net not robust enough (e.g. automated testing: acceptance, e2e, integration) • No simple/easy rollback mechanism • High incidents/bugs/”change failure” rate in Production (first stabilize it…) When NOT to jump into Continuous Deployment 49

Slide 50

Slide 50 text

What about highly regulated environments? Case Studies: Modern Development Practices In Highly Regulated Environments (by Charity Majors) 50

Slide 51

Slide 51 text

How to start? (a humble proposal with a few ideas) 51

Slide 52

Slide 52 text

• Get support (hire/contract people with experience in these principles and practices) • Understand why we work the way we work: observe, ask, do not judge! • Get a clear high level vision about the value streams (the e2e processes) • Explain and share the goals, the current pains, the improvements expected. • Start working in pairs (even better: try ensemble/mob!!) ○ Collaborate more and more, build trust, break silos… • Write automated tests (of “all needed kinds”) and include them in the pipeline • Create deliberated learning spaces: e.g. katas, book club, etc. • Set up a continuous integration workflow/process that can monitor your branches and run the tests automatically for every new commit merged. How to start? (I) 52

Slide 53

Slide 53 text

How to start? (II) •Simplify your branching strategy: ○ Ship, show, ask ○ e.g. start with GitHub Flow • Limit the work in progress a.k.a. WIP (e.g. half the number of developers) •Automate your deployment to Dev/Staging/Production (even if it has several steps) ○ Configure a release/deployment pipeline which breaks if needed (Andon a.k.a. “Stop the line”). A pipeline break should be a top priority (only below a production incident) • [only if needed] Get a “pre-production server” where you improve the safety net ○ So similar to production as possible (Infrastructure as Code for the win!) ○ Run there tests like Acceptance, End to end, Load tests?, etc. 53

Slide 54

Slide 54 text

How to start? (III) • Make your branches ages visible and limit it to 24 hours • Start using feature toggles/flags (e.g. for changes which will take more than 2-3 days) • Improve your baby steps and baby commits skills ○ Learn about parallel changes techniques ○ Learn to take many much smaller pieces to be deployed • Improve your vertical slicing and business simplification skills • Reduce the branches lifetime to 2 hours. • Automate your rollbacks (or at least have a “one-click” operation) 54

Slide 55

Slide 55 text

How to start? (IV) • Try Test-Driven Development (again: get support if nobody has deep experience) • Speed up your tests and pipeline if needed (ideally whole process under 15 min.) • Avoid PRs/MRs by default, work with one single main remote branch by default. • Pick up a non-critical service (e.g. a simple API) and start trying Continuous Delivery (a push deploys your code to the staging environment). • Remove the manual step: deploy to Production after Dev/Staging if everything works! In the “worst case scenario”, if you don’t finish with a Continuous Deployment pipeline, you will have learnt a lot and improved in many other things!!! 🥳 55

Slide 56

Slide 56 text

Final Recap 56

Slide 57

Slide 57 text

• Continuous Deployment is not about “creating and configuring a pipeline”, but about: ○ Risk management (both business and technical risk) ○ Reduced lead time ■ Continuous and sustainable delivery of value to our USERS/CUSTOMERS/BUSINESS ■ Fast feedback loops (users and systems) for learning • Continuous Deployment is a great excuse to improve and detect many inefficiencies: ○ What is stopping us to be able to start working like that tomorrow? ○ What should be “changed” to achieve it? ○ Everything is “connected”!! (systems thinking) Final Recap 57

Slide 58

Slide 58 text

The challenge here is not technological but sociotechnical Keep calm and start in baby steps… but start! (experimentation mindset for the win!) Final Recap 58

Slide 59

Slide 59 text

Recommended people, books and resources 59

Slide 60

Slide 60 text

Recommended books 60

Slide 61

Slide 61 text

Recommended books 61

Slide 62

Slide 62 text

Recommended books 62

Slide 63

Slide 63 text

People to follow ● Eduardo Ferro ○ https://twitter.com/eferro ○ https://www.eferro.net/ ● Jessica Kerr ○ https://twitter.com/jessitron ○ https://jessitron.com/ ● Dave Farley ○ https://twitter.com/davefarley77 ○ https://www.youtube.com/c/ContinuousDelivery ● Charity Majors ○ https://twitter.com/mipsytipsy ○ https://charity.wtf/ ● Jez Humble ○ https://twitter.com/jezhumble ○ https://continuousdelivery.com/ ● Geepaw Hill ○ https://mastodon.social/@GeePawHill ○ https://www.geepawhill.org/ ● Clare Sudbery ○ https://twitter.com/ClareSudbery ○ https://insimpleterms.blog/ ● Kent Beck ○ https://twitter.com/KentBeck ○ https://www.kentbeck.com/ ● Emily Bache ○ https://twitter.com/emilybache ○ https://www.youtube.com/@EmilyBache-tech-coach ● Martin Fowler ○ https://twitter.com/martinfowler ○ https://martinfowler.com/ 63

Slide 64

Slide 64 text

Resources (I) ● Continuous Integration, Delivery and Deployment ○ “A major revision of Continuous Integration” (by Martin Fowler) ○ Continuous Integration: That’s Not What They Meant • Clare Sudbery • YOW! 2023 (video) ○ Minimum Viable Continuous Delivery (minimumcd.org) ○ CI/CD: the what, why and how ○ Continuous Delivery vs Continuous Deployment (video by Dave Farley) ○ https://martinfowler.com/articles/continuousIntegration.html ○ https://www.infoq.com/articles/continuous-delivery-teamplay/ ○ https://martinfowler.com/bliki/ContinuousDelivery.html ○ Card game to design Continuous Delivery pipelines (by Emily Bache) ○ Continuous integration vs. delivery vs. deployment ○ https://charity.wtf/2021/08/27/software-deploys-and-cognitive-biases/ ○ https://martinfowler.com/bliki/FrequencyReducesDifficulty.html 64

Slide 65

Slide 65 text

Resources (II) ● Lean and Flow ○ https://abrahamvallez.medium.com/one-piece-flow-7d5ade46d203 ○ https://jessitron.com/2021/01/18/when-costs-are-nonlinear-keep-it-small/ ○ https://www.geepawhill.org/2021/05/18/big-batch-releases ○ https://www.eferro.net/2021/01/small-batches-for-win-continuous.html ○ https://refactoring.fm/p/how-shipping-fast-changes-your-life ○ https://innolution.com/blog/limiting-work-in-process-wip-and-having-small-batches-is-antifragile ○ Henrik Kniberg : Multiple WIP vs One Piece Flow Example [Video] 65

Slide 66

Slide 66 text

Resources (III) ● Parallel changes ○ https://martinfowler.com/bliki/ParallelChange.html ○ Feature toggles (aka “feature flags”) ○ Branch by abstraction ○ Strangler Fig pattern ■ Strangler Fig Application ■ Refactoring Legacy Code with the Strangler Fig Pattern ○ Anti-Corruption Layer ○ The Limited Red Society 66

Slide 67

Slide 67 text

Resources (IV) ● Workshops about parallel changes: ○ Small Safe Steps 3s workshop - Agile (slides), by Eduardo Ferro ○ https://paucls.wordpress.com/2020/02/26/ci-cd-non-breaking-changes-exercise/ ● Canary releases and Blue-green deployment ○ https://martinfowler.com/bliki/CanaryRelease.html ○ https://martinfowler.com/bliki/BlueGreenDeployment.html ○ https://www.redhat.com/en/topics/devops/what-is-blue-green-deployment ○ https://learn.hashicorp.com/tutorials/terraform/blue-green-canary-tests-deployments ○ https://aws.amazon.com/es/blogs/compute/bluegreen-deployments-with-amazon-ecs 67

Slide 68

Slide 68 text

Resources (V) ● Trunk-Based Development ○ Lightning talk: “Pusheando a master, que es gerundio” (in Spanish) ○ Why I prefer trunk-based development (Trisha Gee) ○ Perceived Barriers to Trunk-Based Development (Dave Farley) ○ Neil Crawdford - Trunk based development, continuous deployment and why you should adopt them (video) ○ Of Death and Dying: A Journey to Trunk Based Development (video) ○ What is Trunk-Based Development? (Clare Sudbery) ○ https://nelis.boucke.be/post/trunk-based-development/ 68

Slide 69

Slide 69 text

Resources (VI) ● Branching, Pull requests and async code reviews ○ How to stop branching ○ https://thinkinglabs.io/articles/2022/05/30/on-the-evilness-of-feature-branching-the-problems.html ○ Async Code Reviews Are Killing Your Company's Throughput [Video] ○ https://martinfowler.com/articles/branching-patterns.html ○ Other experiences: ■ https://twitter.com/jlangr/status/1487135760012627973 ■ https://twitter.com/aahoogendoorn/status/1531709454701453312 69

Slide 70

Slide 70 text

Resources (VII) ● General ○ https://wiki.c2.com/?SoftwareAsLiability ○ What is Value? by Jez Humble (video) ○ Estimar es timar: Example Mapping ○ Evolutionary Database Design ○ https://twitter.com/tottinge/status/1748277312200540523 ● (Vertical) Slicing: ○ Agile Delivery - Lean Coffee - Slicing (video) ○ https://gojko.net/2012/01/23/splitting-user-stories-the-hamburger-method/ ○ https://xp123.com/articles/slicing-functionality-alternate-paths/ ○ How to split a user story? (PDF flowchart) ○ Elephant Carpaccio facilitation guide 70

Slide 71

Slide 71 text

71

Slide 72

Slide 72 text

¿Me ayudas a mejorar? 🙏 https://forms.gle/4omyq7PWoVbkWEJr5 72 @islomar

Slide 73

Slide 73 text

73