Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Yet another Continuous Delivery talk - Incontro DevOps 2021

Yet another Continuous Delivery talk - Incontro DevOps 2021

Continuous Delivery practices applied to a multi-cloud product born in 2013, through constant evolution of the pipelines to deliver software reliably and manage operations without fear.

Giovanni Toraldo

September 28, 2021

More Decks by Giovanni Toraldo

Other Decks in Programming


  1. © Engineering Yet another Continuous Delivery talk Giovanni Toraldo /

    Solution Developer Engineering D.HUB Incontro Devops 2021 #idi2021 CONTINUOUS DELIVERY
  2. © Engineering About me Open source Remote-first software developer /

    devops Solution developer in Engineering D.HUB Often teleported into the 14th century 🛡⚔🎯 CONTINUOUS DELIVERY / BIO Giovanni Toraldo
  3. © Engineering About Cloudesire Whitelabel marketplace SaaS billing Channel management

    IaaS, PaaS, SaaS catalog 2012 - R&D project 2013 - Cloudesire.com 2016 - Tim Venture seed 600K 2020 - Engineering acquisition CONTINUOUS DELIVERY / BIO
  4. © Engineering Antipatterns are everywhere – Manual deployments are a

    waste of time – Infrequent deployments cause late feedback – Manual configuration of environments leads to configuration drift – Manual testing leads to regressions – Can we do better? CONTINUOUS DELIVERY / THEORY
  5. © Engineering What Continuous Delivery can do – Short cycles

    – Reliability – Early feedback – Market pressure – Customer satisfaction CONTINUOUS DELIVERY / THEORY
  6. © Engineering Backlog ‒ One place for high-level issues (epic,

    stories) ‒ Focus on describing the problem you want to solve, not solutions • Developers knows what is good for you ‒ Prioritization as a contract with stakeholders ‒ Commit for a plan and try to stick with it (#noestimates) ‒ Current status should be transparent • Feedback to non-developers when releases happens • Meetings are ok but everyone happy and no meetings is better CONTINUOUS DELIVERY / PRACTICES
  7. © Engineering Automated testing / TDD – Probably the most

    underrated practice for achieving velocity ‒ Writing tests is HARD for newcomers ‒ Maintaining test suites is HARD for everyone (e.g.: flappers) ‒ Still, it’s worth the time ‒ Optimize build times to avoid slow feedback ‒ Tests give hints on how a function should work CONTINUOUS DELIVERY / PRACTICES
  8. © Engineering Continuous integration ‒ Treat it like your best

    friend, you can rely on it ‒ Ensure that you are not going to break everything in a single move ‒ Repetitive tasks belongs to here • Compile code • Execute regression tests • Generate documentation • Publish new artifacts CONTINUOUS DELIVERY / PRACTICES
  9. © Engineering Code reviews ‒ Spot bugs before reaching users

    ‒ Share ownership and abandon blame culture ‒ Meet specifications ‒ Keep the bar high ‒ Written trail of design decisions ‒ Asynchronous feedback ‒ Mentor juniors without doing anything special CONTINUOUS DELIVERY / PRACTICES
  10. © Engineering Packaging / Distribution ‒ A strict requirement for

    automated releases ‒ Every commit is a release candidate ‒ Standard packaging (e.g. Docker) ‒ Central artifacts repository (Nexus, Artifactory) • Accessible from all the execution environments CONTINUOUS DELIVERY / PRACTICES
  11. © Engineering Automated releases ‒ Every environment should be a

    prod-like environment • Have at least a test environment for every prod environment ‒ Repeatable procedures reusable in every environment ‒ Rollback should be easy as a standard release ‒ Releases should be tracked somewhere (GIT, CI) ‒ Releases must not affects customers CONTINUOUS DELIVERY / PRACTICES
  12. © Engineering Feedback loop ‒ Automatic deployment to one prod-like

    environment ‒ Release daily on staging environments ‒ Ask for explicit feedback after changes from tester/QA/stakeholders ‒ Meaningful and searchable application logs ‒ Proactively monitor exceptions and errors CONTINUOUS DELIVERY / PRACTICES
  13. © Engineering Infrastructure as code ‒ Forgot SSH commands or

    clicking buttons ‒ Repeatable procedures ‒ Standardization ‒ Objects Inventory ‒ Document without actually writing documentation CONTINUOUS DELIVERY / PRACTICES
  14. © Engineering Microservices architectures ‒ Maintaining a unique codebase has

    exponential difficulty • Divide et impera ‒ Requires a trained release pipeline ‒ Avoid repeating yourself via: • Boilerplate projects • Shared libraries • Automated updates • DOCUMENTATION CONTINUOUS DELIVERY / PRACTICES
  15. © Engineering Github ‒ GIT repository hosting ‒ Lightweight issue

    management • There is a label for everything • Type: Bug, new feature, change feature, operations, devops • Priority: P1, P2, P3, P4 • Effort: light, medium, high, epic • Workflow: in-progress, ready-for-staging, test, ready, awaiting-feedback, blocked • Topic: UI/UX, billing, ERP... • Customer • Stalebot that warns and close inactive issue for more than 3 months ‒ Documentation as markdown files ‒ Planning via weekly milestones • Internal dashboard to track milestones progress CONTINUOUS DELIVERY / TOOLING
  16. © Engineering Dependabot ‒ Automatic dependencies updates for your repositories

    on Github ‒ PR get opened automatically with changelog between versions ‒ Triggered weekly by a schedule on early monday • PR are already built and ready to be merged ‒ Security updates with links to CVEs ‒ Supports most dependency management tools and Dockerfile ‒ Enabled on all repositories via Terraform CONTINUOUS DELIVERY / TOOLING
  17. © Engineering .github/dependabot.yml version: 2 registries: maven-internal-releases: type: maven-repository url:

    https://repo.cloudesire.com/repository/internal username: ${{secrets.MAVEN_USER}} password: ${{secrets.MAVEN_PASS}} updates: - package-ecosystem: maven directory: /cloudesire-root schedule: interval: weekly day: sunday registries: - maven-internal-releases assignees: - gionn - malteo - package-ecosystem: docker directory: /cloudesire-root/cmw schedule: interval: weekly day: sunday assignees: - gionn - malteo CONTINUOUS DELIVERY / TOOLING
  18. © Engineering CircleCI ‒ Cloud CI services better than in-house:

    • Scalability • Predictable pricing ‒ Batteries included compared to Jenkins ‒ A bit of downsides: • Not so configurable (e.g. tests parallelization) • Can’t add new projects via API • Docker integration slightly cumbersome since beginning ‒ Reuse pipeline configuration via ORB CONTINUOUS DELIVERY / TOOLING
  19. © Engineering Sonar (static analysis) ‒ Catch bugs automatically ‒

    Discovery and learn (why is this an issue?) ‒ Enforce standards (quality gates) ‒ Applied systematically for every project ‒ Reports integrated in PR (developer edition) ‒ Security audits (Hotspots) CONTINUOUS DELIVERY / TOOLING
  20. © Engineering Terraform (Infrastructure as code) ‒ Supports many providers:

    • Iaas: AWS, Azure, GCP, ... • Saas: Cloudflare, Github, ... • API: Vault, ... ‒ Follow common development workflow • PR • Code review • CI pipelines CONTINUOUS DELIVERY / TOOLING
  21. © Engineering Chef (configuration management) ‒ Still the configuration management

    tool after 8+ yrs ‒ Provision new production environment in 30 minutes • script to generate new chef environments ‒ Nightly integration tests on cloud infrastructure ‒ Provision can run from CI or developers box CONTINUOUS DELIVERY / TOOLING
  22. © Engineering Graylog ‒ Centralized logs collection • Applications via

    GELF • Syslog • Docker logs ‒ Elasticsearch monthly indexes • configurable retention ‒ Full-text search on every field, cross-environment CONTINUOUS DELIVERY / TOOLING
  23. © Engineering Sentry ‒ Push applications exceptions via API ‒

    Aggregate similar exceptions ‒ Mark as Resolved • Flagged as regression if found again ‒ Mark as Ignored for n time ‒ Searchable ‒ Slack integration • Each component in a different channel • Only for new errors, regressions or repeated errors in a timeframe CONTINUOUS DELIVERY / TOOLING
  24. © Engineering Step 1: open a release PR on ops

    repo via Merger script ‒ Select which chef environment you want to update ‒ Iterate on all the platform components installed on the environment to: • Retrieve the latest successful build of master branch on CircleCI • Compare build number with currently installed docker image version • CI deploy jobs tag docker images with build number • If a new version is available, compare current and previous commit-id on Github • Parse commits to search for PR and referenced github issues ‒ Open a pull request with version changes on chef environment • Branch name is always `release-envName` • PR body contains the changelog for every components and links to issue • Issues are linked via a redirector domain to avoid references spam CONTINUOUS DELIVERY / RELEASE
  25. © Engineering Step 2: PR review, approve, merge, provision ‒

    Follows the standard development workflow (review, approve, merge) ‒ When building master, CI pipeline detect if the built commit is: • A release PR generated by Merger • A generic PR with a label `deploy/envName` ‒ Run Chef sequentially on all the nodes of the environment CONTINUOUS DELIVERY / RELEASE
  26. © Engineering Step 3: profit ‒ If the release fails,

    the CI send an email to release owners and exceptions are pushed to sentry/slack ‒ If the release is successful: • The release PR get labeled as `deployed` and a comment is left with the link of the build • Every referenced issue in the release PR body receive a comment stating that a release is just happened • If the referenced issue had a label: • `workflow/ready-for-staging` is replaced by `workflow/test` • `workflow/ready` the issue is closed CONTINUOUS DELIVERY / RELEASE
  27. © Engineering Release schedule via CircleCI ‒ Release PR are

    automatically opened following a weekly schedule: • Main staging environment every weekday until friday • Main demo environment on friday, copying main staging versions • Main production environment on monday, copying main demo versions • Secondary (clients) staging environment on different day of the week, copying main production versions • Secondary (clients) production environment on different days of the week, copying from their staging environment CONTINUOUS DELIVERY / RELEASE
  28. © Engineering Wrap-up ‒ Continuous delivery is a game changer

    when developing products ‒ Try to continuously improve, one step at a time ‒ Automate all the things CONTINUOUS DELIVERY / SUMMARY