ADDO_2021: Why and how to include database changes in the deployment pipeline

Transcript

1. TRACK: CI/CD CONTINUOUS EVERYTHING
   NOVEMBER 12, 2020
   Eduardo Piairo
   Why and how to
   include database
   changes in the
   deployment
   pipeline
   

   View Slide

2. TRACK: CI/CD CONTINUOUS EVERYTHING
   ● Engineering Operations Director @ Deeper Insights
   ● Co-Founder and Leader @ DevOps Porto
   About Me
   @EdPiairo
   https://pt.linkedin.com/in/eduardopiairo
   eduardopiairo@gmail.com
   https://www.eduardopiairo.com
   

   View Slide

3. TRACK: CI/CD CONTINUOUS EVERYTHING
   ● Deployment pipeline
   ● Database challenges
   ● Deployment pipeline stages
   ● Deployment pipeline scenarios
   ● Final thoughts
   Agenda
   

   View Slide

4. TRACK: CI/CD CONTINUOUS EVERYTHING
   A deployment pipeline should enable collaboration between the various
   groups involved in delivering software and provide everyone visibility
   about the flow of changes in the system, together with a thorough audit
   trail.
   by Martin Fowler
   @ https://martinfowler.com/bliki/DeploymentPipeline.html
   Definition
   

   View Slide

5. TRACK: CI/CD CONTINUOUS EVERYTHING
   ● Stage 1 – Change description (Source Control)
   ● Stage 2 – Change validation (Continuous Integration)
   ● Stage 3 – Change implementation (Continuous Delivery)
   Stages
   Source
   Control
   Continuous
   Integration
   Continuous
   Delivery
   

   View Slide

6. TRACK: CI/CD CONTINUOUS EVERYTHING
   Database & Deployment Pipeline
   Source
   Control
   Continuous
   Integration
   Continuous
   Delivery
   Database
   +
   Application
   

   View Slide

7. TRACK: CI/CD CONTINUOUS EVERYTHING
   What’s so special about
   databases?
   

   View Slide

8. TRACK: CI/CD CONTINUOUS EVERYTHING
   Database Challenges
   • Database and application changes (most of the time) are
   handled differently
   • Different deployment pipelines - synchronization needed
   • Database is not included (manual work)
   

   View Slide

9. TRACK: CI/CD CONTINUOUS EVERYTHING
   Database Challenges
   • Manual work …
   • Lack of traceability of database changes (changes history)
   • It’s the most expensive and risky work type
   • Prevent CI and CD utilization in their full extent
   • Promote the fear of changes
   

   View Slide

10. TRACK: CI/CD CONTINUOUS EVERYTHING
    Databases become a bottleneck
    in the delivery process
    

    View Slide

11. TRACK: CI/CD CONTINUOUS EVERYTHING
    

    View Slide

12. TRACK: CI/CD CONTINUOUS EVERYTHING
    The Value Of Automation
    • Enable control over database development
    • Increase speed of response to change
    • Greater reliability, repeatability and consistency
    • Enables failing in a controlled and known way
    • Remove/reduce human intervention in the release process
    

    View Slide

13. TRACK: CI/CD CONTINUOUS EVERYTHING
    Fearless database changes
    

    View Slide

14. TRACK: CI/CD CONTINUOUS EVERYTHING
    Source Control
    • Every database change can be described with an SQL script
    • Traceability through change history
    • SQL as documentation
    • Shared code-base and shared process
    • Enforce standards to reduce conflicts
    

    View Slide

15. TRACK: CI/CD CONTINUOUS EVERYTHING
    Migrations vs State
    • State based solutions
    • How the database should be
    • Migrations based solutions
    • How the database should change
    Migration Migration
    State
    Delta
    

    View Slide

16. TRACK: CI/CD CONTINUOUS EVERYTHING
    Scripting Guideline
    • One script, one operation type, one object (small batches)
    • Merge conflicts management
    • Patterns identification
    • File system scripts history search
    

    View Slide

17. TRACK: CI/CD CONTINUOUS EVERYTHING
    Continuous Integration
    • Integrate and validate changes
    • Unit and integration tests
    • Small batches, less risk
    • Considerations
    • What should be tested? And when?
    • How can the application contribute to this stage?
    

    View Slide

18. TRACK: CI/CD CONTINUOUS EVERYTHING
    Continuous Delivery
    • Delivering change(s) in the target environment
    • Considerations
    • Downtime
    • Time to recover
    • Affected applications
    

    View Slide

19. TRACK: CI/CD CONTINUOUS EVERYTHING
    Failed Deployment
    • Scripts (roll-back, roll-front)
    • Very fast
    • (Please) Do not make mistakes
    • Working backwards
    

    View Slide

20. TRACK: CI/CD CONTINUOUS EVERYTHING
    Failed Deployment
    • Backups (before deploy)
    • Safer but slower
    • Keep databases small
    • Avoid shared databases
    

    View Slide

21. TRACK: CI/CD CONTINUOUS EVERYTHING
    Scenario #1
    Source Control Continuous Integration Continuous Delivery
    APP
    DB
    • Independent deployment pipelines
    • High need of synchronization between DBA and development team
    

    View Slide

22. TRACK: CI/CD CONTINUOUS EVERYTHING
    Scenario #2
    Source Control Continuous Integration Continuous Delivery
    APP
    DB
    • Different code repositories
    • Independent CI process
    • The contact point will be the deployment moment
    

    View Slide

23. TRACK: CI/CD CONTINUOUS EVERYTHING
    Scenario #3
    Source Control Continuous Integration Continuous Delivery
    APP
    DB
    • Different code repositories
    • Connected/dependent CI process
    • Short learning step towards scenario #4
    

    View Slide

24. TRACK: CI/CD CONTINUOUS EVERYTHING
    Scenario #4
    Source Control Continuous Integration Continuous Delivery
    APP
    DB
    • Share the some code repository, CI process and CD process
    • No need of synchronization
    • Promotes learning within teams (database code reviews)
    

    View Slide

25. TRACK: CI/CD CONTINUOUS EVERYTHING
    My Definition
    Pipeline: Cultural and technical tool for managing changes in the
    software development process (should contemplate databases, applications
    and infrastructure).
    

    View Slide

26. TRACK: CI/CD CONTINUOUS EVERYTHING
    DevOps Way
    • Increase flow visibility
    • Increase feedback
    • Increase knowledge base and exploration
    

    View Slide

27. TRACK: CI/CD CONTINUOUS EVERYTHING
    Q&A
    @EdPiairo
    https://pt.linkedin.com/in/eduardopiairo
    eduardopiairo@gmail.com
    http://www.eduardopiairo.com/
    #2021-ci-cd
    

    View Slide