Yet another Continuous Delivery talk - Incontro DevOps 2021

Continuous Delivery practices applied to a multi-cloud product born in 2013, through constant evolution of the pipelines to deliver software reliably and manage operations without fear.

Transcript

1. © Engineering
   Yet another
   Continuous Delivery talk
   Giovanni Toraldo / Solution Developer
   Engineering D.HUB
   Incontro Devops 2021 #idi2021
   CONTINUOUS DELIVERY
   

   View Slide

2. © Engineering
   About me
   Open source
   Remote-first
   software developer / devops
   Solution developer in Engineering D.HUB
   Often teleported into the 14th century 🛡⚔🎯
   CONTINUOUS DELIVERY / BIO
   Giovanni Toraldo
   

   View Slide

3. © Engineering
   About Cloudesire
   Whitelabel marketplace
   SaaS billing
   Channel management
   IaaS, PaaS, SaaS catalog
   2012 - R&D project
   2013 - Cloudesire.com
   2016 - Tim Venture seed 600K
   2020 - Engineering acquisition
   CONTINUOUS DELIVERY / BIO
   

   View Slide

4. © Engineering
   Antipatterns are everywhere
   – Manual deployments are a waste of time
   – Infrequent deployments cause late feedback
   – Manual configuration of environments leads to configuration drift
   – Manual testing leads to regressions
   – Can we do better?
   CONTINUOUS DELIVERY / THEORY
   

   View Slide

5. © Engineering
   What Continuous Delivery can do
   – Short cycles
   – Reliability
   – Early feedback
   – Market pressure
   – Customer satisfaction
   CONTINUOUS DELIVERY / THEORY
   

   View Slide

6. © Engineering
   Continuous Delivery phases
   CONTINUOUS DELIVERY / THEORY
   

   View Slide

7. © Engineering
   How we do achieve our goal?
   CONTINUOUS DELIVERY / PRACTICES
   

   View Slide

8. © Engineering
   Backlog
   ‒ One place for high-level issues (epic, stories)
   ‒ Focus on describing the problem you want to solve, not solutions
   ● Developers knows what is good for you
   ‒ Prioritization as a contract with stakeholders
   ‒ Commit for a plan and try to stick with it (#noestimates)
   ‒ Current status should be transparent
   ● Feedback to non-developers when releases happens
   ● Meetings are ok but everyone happy and no meetings is better
   CONTINUOUS DELIVERY / PRACTICES
   

   View Slide

9. © Engineering
   Automated testing / TDD
   – Probably the most underrated practice for achieving velocity
   ‒ Writing tests is HARD for newcomers
   ‒ Maintaining test suites is HARD for everyone (e.g.: flappers)
   ‒ Still, it’s worth the time
   ‒ Optimize build times to avoid slow feedback
   ‒ Tests give hints on how a function should work
   CONTINUOUS DELIVERY / PRACTICES
   

   View Slide

10. © Engineering
    Continuous integration
    ‒ Treat it like your best friend, you can rely on it
    ‒ Ensure that you are not going to break everything in a single move
    ‒ Repetitive tasks belongs to here
    ● Compile code
    ● Execute regression tests
    ● Generate documentation
    ● Publish new artifacts
    CONTINUOUS DELIVERY / PRACTICES
    

    View Slide

11. © Engineering
    Code reviews
    ‒ Spot bugs before reaching users
    ‒ Share ownership and abandon blame culture
    ‒ Meet specifications
    ‒ Keep the bar high
    ‒ Written trail of design decisions
    ‒ Asynchronous feedback
    ‒ Mentor juniors without doing anything special
    CONTINUOUS DELIVERY / PRACTICES
    

    View Slide

12. © Engineering
    Packaging / Distribution
    ‒ A strict requirement for automated releases
    ‒ Every commit is a release candidate
    ‒ Standard packaging (e.g. Docker)
    ‒ Central artifacts repository (Nexus, Artifactory)
    ● Accessible from all the execution environments
    CONTINUOUS DELIVERY / PRACTICES
    

    View Slide

13. © Engineering
    Automated releases
    ‒ Every environment should be a prod-like environment
    ● Have at least a test environment for every prod environment
    ‒ Repeatable procedures reusable in every environment
    ‒ Rollback should be easy as a standard release
    ‒ Releases should be tracked somewhere (GIT, CI)
    ‒ Releases must not affects customers
    CONTINUOUS DELIVERY / PRACTICES
    

    View Slide

14. © Engineering
    Feedback loop
    ‒ Automatic deployment to one prod-like environment
    ‒ Release daily on staging environments
    ‒ Ask for explicit feedback after changes from
    tester/QA/stakeholders
    ‒ Meaningful and searchable application logs
    ‒ Proactively monitor exceptions and errors
    CONTINUOUS DELIVERY / PRACTICES
    

    View Slide

15. © Engineering
    Infrastructure as code
    ‒ Forgot SSH commands or clicking buttons
    ‒ Repeatable procedures
    ‒ Standardization
    ‒ Objects Inventory
    ‒ Document without actually writing documentation
    CONTINUOUS DELIVERY / PRACTICES
    

    View Slide

16. © Engineering
    Microservices architectures
    ‒ Maintaining a unique codebase has exponential difficulty
    ● Divide et impera
    ‒ Requires a trained release pipeline
    ‒ Avoid repeating yourself via:
    ● Boilerplate projects
    ● Shared libraries
    ● Automated updates
    ● DOCUMENTATION
    CONTINUOUS DELIVERY / PRACTICES
    

    View Slide

17. © Engineering
    What’s in our toolbox
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

18. © Engineering
    Github
    ‒ GIT repository hosting
    ‒ Lightweight issue management
    ● There is a label for everything
    ● Type: Bug, new feature, change feature, operations, devops
    ● Priority: P1, P2, P3, P4
    ● Effort: light, medium, high, epic
    ● Workflow: in-progress, ready-for-staging, test, ready, awaiting-feedback, blocked
    ● Topic: UI/UX, billing, ERP...
    ● Customer
    ● Stalebot that warns and close inactive issue for more than 3 months
    ‒ Documentation as markdown files
    ‒ Planning via weekly milestones
    ● Internal dashboard to track milestones progress
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

19. © Engineering
    Internal dashboard for milestones tracking
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

20. © Engineering
    Dependabot
    ‒ Automatic dependencies updates for your repositories on Github
    ‒ PR get opened automatically with changelog between versions
    ‒ Triggered weekly by a schedule on early monday
    ● PR are already built and ready to be merged
    ‒ Security updates with links to CVEs
    ‒ Supports most dependency management tools and Dockerfile
    ‒ Enabled on all repositories via Terraform
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

21. © Engineering
    .github/dependabot.yml
    version: 2
    registries:
    maven-internal-releases:
    type: maven-repository
    url: https://repo.cloudesire.com/repository/internal
    username: ${{secrets.MAVEN_USER}}
    password: ${{secrets.MAVEN_PASS}}
    updates:
    - package-ecosystem: maven
    directory: /cloudesire-root
    schedule:
    interval: weekly
    day: sunday
    registries:
    - maven-internal-releases
    assignees:
    - gionn
    - malteo
    - package-ecosystem: docker
    directory: /cloudesire-root/cmw
    schedule:
    interval: weekly
    day: sunday
    assignees:
    - gionn
    - malteo
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

22. © Engineering
    Dependabot automated pull request
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

23. © Engineering
    CircleCI
    ‒ Cloud CI services better than in-house:
    ● Scalability
    ● Predictable pricing
    ‒ Batteries included compared to Jenkins
    ‒ A bit of downsides:
    ● Not so configurable (e.g. tests parallelization)
    ● Can’t add new projects via API
    ● Docker integration slightly cumbersome since beginning
    ‒ Reuse pipeline configuration via ORB
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

24. © Engineering
    CircleCI ORB example
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

25. © Engineering
    CircleCI workflow diagram
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

26. © Engineering
    CircleCI job detail
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

27. © Engineering
    Sonar (static analysis)
    ‒ Catch bugs automatically
    ‒ Discovery and learn (why is this an issue?)
    ‒ Enforce standards (quality gates)
    ‒ Applied systematically for every project
    ‒ Reports integrated in PR (developer edition)
    ‒ Security audits (Hotspots)
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

28. © Engineering
    Terraform (Infrastructure as code)
    ‒ Supports many providers:
    ● Iaas: AWS, Azure, GCP, ...
    ● Saas: Cloudflare, Github, ...
    ● API: Vault, ...
    ‒ Follow common development workflow
    ● PR
    ● Code review
    ● CI pipelines
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

29. © Engineering
    Automated terraform plan on pull requests
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

30. © Engineering
    CircleCI workflow for terraform repository
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

31. © Engineering
    Chef (configuration management)
    ‒ Still the configuration management tool after 8+ yrs
    ‒ Provision new production environment in 30 minutes
    ● script to generate new chef environments
    ‒ Nightly integration tests on cloud infrastructure
    ‒ Provision can run from CI or developers box
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

32. © Engineering
    Graylog
    ‒ Centralized logs collection
    ● Applications via GELF
    ● Syslog
    ● Docker logs
    ‒ Elasticsearch monthly indexes
    ● configurable retention
    ‒ Full-text search on every field, cross-environment
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

33. © Engineering
    Sentry
    ‒ Push applications exceptions via API
    ‒ Aggregate similar exceptions
    ‒ Mark as Resolved
    ● Flagged as regression if found again
    ‒ Mark as Ignored for n time
    ‒ Searchable
    ‒ Slack integration
    ● Each component in a different channel
    ● Only for new errors, regressions or repeated errors in a timeframe
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

34. © Engineering
    Sentry issue example
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

35. © Engineering
    How to release
    CONTINUOUS DELIVERY / RELEASE
    

    View Slide

36. © Engineering
    Step 1: open a release PR on ops repo via Merger script
    ‒ Select which chef environment you want to update
    ‒ Iterate on all the platform components installed on the environment to:
    ● Retrieve the latest successful build of master branch on CircleCI
    ● Compare build number with currently installed docker image version
    ● CI deploy jobs tag docker images with build number
    ● If a new version is available, compare current and previous commit-id on Github
    ● Parse commits to search for PR and referenced github issues
    ‒ Open a pull request with version changes on chef environment
    ● Branch name is always `release-envName`
    ● PR body contains the changelog for every components and links to issue
    ● Issues are linked via a redirector domain to avoid references spam
    CONTINUOUS DELIVERY / RELEASE
    

    View Slide

37. © Engineering
    Example Release PR
    CONTINUOUS DELIVERY / TOOLING
    

    View Slide

38. © Engineering
    Step 2: PR review, approve, merge, provision
    ‒ Follows the standard development workflow (review, approve, merge)
    ‒ When building master, CI pipeline detect if the built commit is:
    ● A release PR generated by Merger
    ● A generic PR with a label `deploy/envName`
    ‒ Run Chef sequentially on all the nodes of the environment
    CONTINUOUS DELIVERY / RELEASE
    

    View Slide

39. © Engineering
    Step 3: profit
    ‒ If the release fails, the CI send an email to release owners and exceptions are
    pushed to sentry/slack
    ‒ If the release is successful:
    ● The release PR get labeled as `deployed` and a comment is left with the link of
    the build
    ● Every referenced issue in the release PR body receive a comment stating that a
    release is just happened
    ● If the referenced issue had a label:
    ● `workflow/ready-for-staging` is replaced by `workflow/test`
    ● `workflow/ready` the issue is closed
    CONTINUOUS DELIVERY / RELEASE
    

    View Slide

40. © Engineering
    Release schedule via CircleCI
    ‒ Release PR are automatically opened following a weekly schedule:
    ● Main staging environment every weekday until friday
    ● Main demo environment on friday, copying main staging versions
    ● Main production environment on monday, copying main demo
    versions
    ● Secondary (clients) staging environment on different day of the
    week, copying main production versions
    ● Secondary (clients) production environment on different days of the
    week, copying from their staging environment
    CONTINUOUS DELIVERY / RELEASE
    

    View Slide

41. © Engineering
    Wrap-up
    ‒ Continuous delivery is a game changer when developing
    products
    ‒ Try to continuously improve, one step at a time
    ‒ Automate all the things
    CONTINUOUS DELIVERY / SUMMARY
    

    View Slide

42. © Engineering
    www.eng.it
    Engineering Ingegneria Informatica SpA
    @EngineeringSpa
    gruppo.engineering
    LifeAtEngineering
    Thanks!
    Giovanni Toraldo
    @gionn
    

    View Slide