© Engineering Yet another Continuous Delivery talk Giovanni Toraldo / Solution Developer Engineering D.HUB Incontro Devops 2021 #idi2021 CONTINUOUS DELIVERY 

© Engineering About me Open source Remote-first software developer / devops Solution developer in Engineering D.HUB Often teleported into the 14th century 🛡⚔🎯 CONTINUOUS DELIVERY / BIO Giovanni Toraldo 

© Engineering About Cloudesire Whitelabel marketplace SaaS billing Channel management IaaS, PaaS, SaaS catalog 2012 - R&D project 2013 - Cloudesire.com 2016 - Tim Venture seed 600K 2020 - Engineering acquisition CONTINUOUS DELIVERY / BIO 

© Engineering Antipatterns are everywhere – Manual deployments are a waste of time – Infrequent deployments cause late feedback – Manual configuration of environments leads to configuration drift – Manual testing leads to regressions – Can we do better? CONTINUOUS DELIVERY / THEORY 

© Engineering What Continuous Delivery can do – Short cycles – Reliability – Early feedback – Market pressure – Customer satisfaction CONTINUOUS DELIVERY / THEORY 

© Engineering Continuous Delivery phases CONTINUOUS DELIVERY / THEORY 

© Engineering How we do achieve our goal? CONTINUOUS DELIVERY / PRACTICES 

© Engineering Backlog ‒ One place for high-level issues (epic, stories) ‒ Focus on describing the problem you want to solve, not solutions ● Developers knows what is good for you ‒ Prioritization as a contract with stakeholders ‒ Commit for a plan and try to stick with it (#noestimates) ‒ Current status should be transparent ● Feedback to non-developers when releases happens ● Meetings are ok but everyone happy and no meetings is better CONTINUOUS DELIVERY / PRACTICES 

© Engineering Automated testing / TDD – Probably the most underrated practice for achieving velocity ‒ Writing tests is HARD for newcomers ‒ Maintaining test suites is HARD for everyone (e.g.: flappers) ‒ Still, it’s worth the time ‒ Optimize build times to avoid slow feedback ‒ Tests give hints on how a function should work CONTINUOUS DELIVERY / PRACTICES 

© Engineering Continuous integration ‒ Treat it like your best friend, you can rely on it ‒ Ensure that you are not going to break everything in a single move ‒ Repetitive tasks belongs to here ● Compile code ● Execute regression tests ● Generate documentation ● Publish new artifacts CONTINUOUS DELIVERY / PRACTICES 

© Engineering Code reviews ‒ Spot bugs before reaching users ‒ Share ownership and abandon blame culture ‒ Meet specifications ‒ Keep the bar high ‒ Written trail of design decisions ‒ Asynchronous feedback ‒ Mentor juniors without doing anything special CONTINUOUS DELIVERY / PRACTICES 

© Engineering Packaging / Distribution ‒ A strict requirement for automated releases ‒ Every commit is a release candidate ‒ Standard packaging (e.g. Docker) ‒ Central artifacts repository (Nexus, Artifactory) ● Accessible from all the execution environments CONTINUOUS DELIVERY / PRACTICES 

© Engineering Automated releases ‒ Every environment should be a prod-like environment ● Have at least a test environment for every prod environment ‒ Repeatable procedures reusable in every environment ‒ Rollback should be easy as a standard release ‒ Releases should be tracked somewhere (GIT, CI) ‒ Releases must not affects customers CONTINUOUS DELIVERY / PRACTICES 

© Engineering Feedback loop ‒ Automatic deployment to one prod-like environment ‒ Release daily on staging environments ‒ Ask for explicit feedback after changes from tester/QA/stakeholders ‒ Meaningful and searchable application logs ‒ Proactively monitor exceptions and errors CONTINUOUS DELIVERY / PRACTICES 

© Engineering Infrastructure as code ‒ Forgot SSH commands or clicking buttons ‒ Repeatable procedures ‒ Standardization ‒ Objects Inventory ‒ Document without actually writing documentation CONTINUOUS DELIVERY / PRACTICES 

© Engineering Microservices architectures ‒ Maintaining a unique codebase has exponential difficulty ● Divide et impera ‒ Requires a trained release pipeline ‒ Avoid repeating yourself via: ● Boilerplate projects ● Shared libraries ● Automated updates ● DOCUMENTATION CONTINUOUS DELIVERY / PRACTICES 

© Engineering What’s in our toolbox CONTINUOUS DELIVERY / TOOLING 

© Engineering Github ‒ GIT repository hosting ‒ Lightweight issue management ● There is a label for everything ● Type: Bug, new feature, change feature, operations, devops ● Priority: P1, P2, P3, P4 ● Effort: light, medium, high, epic ● Workflow: in-progress, ready-for-staging, test, ready, awaiting-feedback, blocked ● Topic: UI/UX, billing, ERP... ● Customer ● Stalebot that warns and close inactive issue for more than 3 months ‒ Documentation as markdown files ‒ Planning via weekly milestones ● Internal dashboard to track milestones progress CONTINUOUS DELIVERY / TOOLING 

© Engineering Internal dashboard for milestones tracking CONTINUOUS DELIVERY / TOOLING 

© Engineering Dependabot ‒ Automatic dependencies updates for your repositories on Github ‒ PR get opened automatically with changelog between versions ‒ Triggered weekly by a schedule on early monday ● PR are already built and ready to be merged ‒ Security updates with links to CVEs ‒ Supports most dependency management tools and Dockerfile ‒ Enabled on all repositories via Terraform CONTINUOUS DELIVERY / TOOLING 

© Engineering .github/dependabot.yml version: 2 registries: maven-internal-releases: type: maven-repository url: https://repo.cloudesire.com/repository/internal username: ${{secrets.MAVEN_USER}} password: ${{secrets.MAVEN_PASS}} updates: - package-ecosystem: maven directory: /cloudesire-root schedule: interval: weekly day: sunday registries: - maven-internal-releases assignees: - gionn - malteo - package-ecosystem: docker directory: /cloudesire-root/cmw schedule: interval: weekly day: sunday assignees: - gionn - malteo CONTINUOUS DELIVERY / TOOLING 

© Engineering Dependabot automated pull request CONTINUOUS DELIVERY / TOOLING 

© Engineering CircleCI ‒ Cloud CI services better than in-house: ● Scalability ● Predictable pricing ‒ Batteries included compared to Jenkins ‒ A bit of downsides: ● Not so configurable (e.g. tests parallelization) ● Can’t add new projects via API ● Docker integration slightly cumbersome since beginning ‒ Reuse pipeline configuration via ORB CONTINUOUS DELIVERY / TOOLING 

© Engineering CircleCI ORB example CONTINUOUS DELIVERY / TOOLING 

© Engineering CircleCI workflow diagram CONTINUOUS DELIVERY / TOOLING 

© Engineering CircleCI job detail CONTINUOUS DELIVERY / TOOLING 

© Engineering Sonar (static analysis) ‒ Catch bugs automatically ‒ Discovery and learn (why is this an issue?) ‒ Enforce standards (quality gates) ‒ Applied systematically for every project ‒ Reports integrated in PR (developer edition) ‒ Security audits (Hotspots) CONTINUOUS DELIVERY / TOOLING 

© Engineering Terraform (Infrastructure as code) ‒ Supports many providers: ● Iaas: AWS, Azure, GCP, ... ● Saas: Cloudflare, Github, ... ● API: Vault, ... ‒ Follow common development workflow ● PR ● Code review ● CI pipelines CONTINUOUS DELIVERY / TOOLING 

© Engineering Automated terraform plan on pull requests CONTINUOUS DELIVERY / TOOLING 

© Engineering CircleCI workflow for terraform repository CONTINUOUS DELIVERY / TOOLING 

© Engineering Chef (configuration management) ‒ Still the configuration management tool after 8+ yrs ‒ Provision new production environment in 30 minutes ● script to generate new chef environments ‒ Nightly integration tests on cloud infrastructure ‒ Provision can run from CI or developers box CONTINUOUS DELIVERY / TOOLING 

© Engineering Graylog ‒ Centralized logs collection ● Applications via GELF ● Syslog ● Docker logs ‒ Elasticsearch monthly indexes ● configurable retention ‒ Full-text search on every field, cross-environment CONTINUOUS DELIVERY / TOOLING 

© Engineering Sentry ‒ Push applications exceptions via API ‒ Aggregate similar exceptions ‒ Mark as Resolved ● Flagged as regression if found again ‒ Mark as Ignored for n time ‒ Searchable ‒ Slack integration ● Each component in a different channel ● Only for new errors, regressions or repeated errors in a timeframe CONTINUOUS DELIVERY / TOOLING 

© Engineering Sentry issue example CONTINUOUS DELIVERY / TOOLING 

© Engineering How to release CONTINUOUS DELIVERY / RELEASE 

© Engineering Step 1: open a release PR on ops repo via Merger script ‒ Select which chef environment you want to update ‒ Iterate on all the platform components installed on the environment to: ● Retrieve the latest successful build of master branch on CircleCI ● Compare build number with currently installed docker image version ● CI deploy jobs tag docker images with build number ● If a new version is available, compare current and previous commit-id on Github ● Parse commits to search for PR and referenced github issues ‒ Open a pull request with version changes on chef environment ● Branch name is always `release-envName` ● PR body contains the changelog for every components and links to issue ● Issues are linked via a redirector domain to avoid references spam CONTINUOUS DELIVERY / RELEASE 

© Engineering Example Release PR CONTINUOUS DELIVERY / TOOLING 

© Engineering Step 2: PR review, approve, merge, provision ‒ Follows the standard development workflow (review, approve, merge) ‒ When building master, CI pipeline detect if the built commit is: ● A release PR generated by Merger ● A generic PR with a label `deploy/envName` ‒ Run Chef sequentially on all the nodes of the environment CONTINUOUS DELIVERY / RELEASE 

© Engineering Step 3: profit ‒ If the release fails, the CI send an email to release owners and exceptions are pushed to sentry/slack ‒ If the release is successful: ● The release PR get labeled as `deployed` and a comment is left with the link of the build ● Every referenced issue in the release PR body receive a comment stating that a release is just happened ● If the referenced issue had a label: ● `workflow/ready-for-staging` is replaced by `workflow/test` ● `workflow/ready` the issue is closed CONTINUOUS DELIVERY / RELEASE 

© Engineering Release schedule via CircleCI ‒ Release PR are automatically opened following a weekly schedule: ● Main staging environment every weekday until friday ● Main demo environment on friday, copying main staging versions ● Main production environment on monday, copying main demo versions ● Secondary (clients) staging environment on different day of the week, copying main production versions ● Secondary (clients) production environment on different days of the week, copying from their staging environment CONTINUOUS DELIVERY / RELEASE 

© Engineering Wrap-up ‒ Continuous delivery is a game changer when developing products ‒ Try to continuously improve, one step at a time ‒ Automate all the things CONTINUOUS DELIVERY / SUMMARY 

© Engineering www.eng.it Engineering Ingegneria Informatica SpA @EngineeringSpa gruppo.engineering LifeAtEngineering Thanks! Giovanni Toraldo @gionn