Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevOpsPorto Meetup12: Operations for databases - the agile/devops journey by Eduardo Piairo

DevOpsPorto Meetup12: Operations for databases - the agile/devops journey by Eduardo Piairo

Talk delivered by Eduardo Piairo

DevOpsPorto

January 19, 2018
Tweet

More Decks by DevOpsPorto

Other Decks in Technology

Transcript

  1. OPERATIONS FOR DATABASES Eduardo Piairo
    @EdPiairo
    #DevOpsPorto
    The Agile/DevOps Journey

    View Slide

  2. ABOUT ME
    Operations for databases
    @EdPiairo, #DevOpsPorto
    @EdPiairo
    https://pt.linkedin.com/in/jesuspiairo
    [email protected]
    http://www.eduardopiairo.com/
    Eduardo Piairo
    Operations Engineer
    DevOps Porto Founder

    View Slide

  3. THIS PRESENTATION
    Agile journey
    Scrum
    Kanban
    DevOps
    Operations for databases
    Database operations
    Automation
    Source Control
    Continuous Integration
    Continuous Delivery
    The definition and the interpretation of the covered concepts
    are attached to the cultural context of a specific organization
    on a certain period!
    @EdPiairo, #DevOpsPorto

    View Slide

  4. AGILE JOURNEY
    1.Before Scrum
    2.Scrum
    3.Kanban
    4.Scrum + Kanban
    Operations for databases
    @EdPiairo, #DevOpsPorto

    View Slide

  5. AGILE JOURNEY
    1.Before Scrum
    2.Scrum
    3.Kanban
    4.Scrum + Kanban
    Operations for databases
    @EdPiairo, #DevOpsPorto

    View Slide

  6. BEFORE SCRUM
    Operations for databases
    • Before Scrum - The “real agile” method
    • Centralized decision mechanism
    • Growing phase
    • Size (5 to 15)
    • Complexity (more and more components)
    @EdPiairo, #DevOpsPorto

    View Slide

  7. BEFORE SCRUM
    Operations for databases
    Application #1
    Application #2
    Application #3
    Shared database(s)
    “Team” #1
    “Team” #2
    “Team” #3
    “Team” #T-SQL
    @EdPiairo, #DevOpsPorto

    View Slide

  8. AGILE JOURNEY
    1.Before Scrum
    2.Scrum
    3.Kanban
    4.Scrum + Kanban
    Operations for databases
    @EdPiairo, #DevOpsPorto

    View Slide

  9. SCRUM
    Operations for databases
    • Scrum - The “magic” scrum
    • Scrum for the masses (>20)
    • Pure Scrum approach – operations as development team
    • 2 week sprint, sprint goal
    • The team was “too reactive”
    @EdPiairo, #DevOpsPorto

    View Slide

  10. DEVOPS FOR DATABASES – THE BEGINNING
    Operations for databases
    Application #1
    Application #2
    Application #3
    Shared database(s)
    Team #1
    Team #2
    Team #3
    Team #T-SQL
    @EdPiairo, #DevOpsPorto

    View Slide

  11. DEVOPS FOR DATABASES – BUILDING A DEPLOYMENT PIPELINE
    Operations for databases
    Source
    Control
    Continuous
    Integration
    Continuous
    Delivery
    Database
    +
    Application
    @EdPiairo, #DevOpsPorto

    View Slide

  12. DEVOPS FOR DATABASES – THE SPECIAL CASE OF THE DATABASES
    What’s so special about databases?
    Operations for databases
    @EdPiairo, #DevOpsPorto

    View Slide

  13. DEVOPS FOR DATABASES – PROBLEMS TO FIX
    Operations for databases
    • Databases are out of pace with application development
    • Synchronization between development and DBA teams
    • No traceability of database changes (changes history)
    • What changed? Who? When? Why?
    • Manual databases processes prevent the CI and CD utilization in their full extent
    • Time consuming and error prone
    • Releases are less frequent and risky
    @EdPiairo, #DevOpsPorto

    View Slide

  14. DEVOPS FOR DATABASES – PROBLEMS TO FIX
    Operations for databases
    • Bugs in production environment
    • Database related bugs are only discovered after deployment to production
    • Manual or inexistent tests
    • Fixes and hotfixes have time cost, what can lead to delay a release
    • High dependency in a person (database specialist)
    • Fear of making changes
    @EdPiairo, #DevOpsPorto

    View Slide

  15. DEVOPS FOR DATABASES – PROBLEMS TO FIX
    Operations for databases
    Databases became a bottleneck in an
    agile delivery process
    @EdPiairo, #DevOpsPorto

    View Slide

  16. DEVOPS FOR DATABASES – THE SOLUTION
    Operations for databases
    Source
    Control
    Continuous
    Integration
    Continuous
    Delivery
    Automation
    +
    Change control
    @EdPiairo, #DevOpsPorto

    View Slide

  17. DEVOPS FOR DATABASES – THE SOLUTION
    Decision #1
    Let’s automate
    Let’s do source control
    Operations for databases
    @EdPiairo, #DevOpsPorto

    View Slide

  18. DEVOPS FOR DATABASES – THE VALUE OF AUTOMATION
    Operations for databases
    • Enable control over database development
    • Increase speed of response to change
    • Keep a versioned “history” of database states
    • Greater reliability of the release process
    • Increase release frequency through repeatability of processes
    • Reduce time spent fixing bugs
    @EdPiairo, #DevOpsPorto

    View Slide

  19. DEVOPS FOR DATABASES – THE VALUE OF AUTOMATION
    Operations for databases
    • Remove/reduce human intervention in the release process
    • The build step is automatic triggered by a “push” into source control repository
    • The deploy step is automatic triggered by a successfully build process
    @EdPiairo, #DevOpsPorto

    View Slide

  20. DEVOPS FOR DATABASES – THE VALUE OF AUTOMATION
    Operations for databases
    Fearless database changes
    @EdPiairo, #NetPonto

    View Slide

  21. DEVOPS FOR DATABASES – THE 1ST STEP: SOURCE CONTROL
    Operations for databases
    • First step in your database deployment pipeline
    • Traceability through change history
    • SQL as documentation
    • Shared code-base and shared process
    • Enforceable standards to reduce conflicts
    • Fundamental resource: SQL script
    @EdPiairo, #DevOpsPorto

    View Slide

  22. DEVOPS FOR DATABASES – THE 1ST STEP: SOURCE CONTROL
    Decision #2
    Database source control:
    Migrations vs State
    Operations for databases
    @EdPiairo, #DevOpsPorto

    View Slide

  23. DEVOPS FOR DATABASES – MIGRATIONS VS STATE
    Operations for databases
    • State based solutions
    • Script represents the current database state
    • Your source of truth is how the database should be
    • Migrations based solutions
    • Script represents a migration
    • Your source of truth is how the database should change
    Flyway: open source database migration tool
    • Simplicity: easy to setup, no need to install
    • Zero dependencies (java + jdbc)
    • Scripts are written in SQL
    @EdPiairo, #DevOpsPorto

    View Slide

  24. DEVOPS FOR DATABASES – COLLABORATION THROUGH A CONTRACT
    Operations for databases
    • Set of rules and expectations
    • Define responsibility frontiers
    • Sets a common language
    Application #1
    Application #2
    Application #3
    Shared database(s)
    Team #1
    T-SQL Script
    Team #2
    Team #3
    Team #T-SQL
    @EdPiairo, #DevOpsPorto

    View Slide

  25. DEVOPS FOR DATABASES - RULES
    Operations for databases
    • Scripting rules
    • Rule 1: Script version (timestamp)
    • Rule 2: Operation type
    • Rule 3: Object type
    • Rule 4: Object name
    Example:
    V20160220.1100__Create_TB_MyTable.sql
    • One script, one operation type, one object
    @EdPiairo, #DevOpsPorto
    SMALL BATCHES

    View Slide

  26. DEVOPS FOR DATABASES - THE PIPELINE
    Operations for databases
    • Contract – communication, collaboration mechanism for change management
    • Should be reflected in your development pipeline
    • The better/clearer your pipeline, the less you need to document (your code is your documentation)
    • Everything is negotiable in the contract, except its application
    @EdPiairo, #DevOpsPorto

    View Slide

  27. AGILE JOURNEY
    1.Before Scrum
    2.Scrum
    3.Kanban
    4.Scrum + Kanban
    Operations for databases
    @EdPiairo, #DevOpsPorto

    View Slide

  28. KANBAN
    Operations for databases
    • Kanban 101
    • Focus on development teams necessities
    • 4 week iterations
    • A bad choice
    • Different “language” from other teams
    • Desynchronization between operations and development teams
    @EdPiairo, #DevOpsPorto

    View Slide

  29. DEVOPS FOR DATABASES – THE PIPELINE
    Operations for databases
    • Change description (Source Control)
    • Change validation (Continuous Integration)
    • Change implementation (Continuous Delivery)
    @EdPiairo, #DevOpsPorto

    View Slide

  30. AGILE JOURNEY
    1.Before Scrum
    2.Scrum
    3.Kanban
    4.Scrum + Kanban
    Operations for databases
    @EdPiairo, #DevOpsPorto

    View Slide

  31. SCRUM + KANBAN
    Operations for databases
    • Scrum + Kanban – The best of two worlds
    • 2 week sprint, sprint goal
    • Task definition was based on teams necessities and our necessities
    • Team capacity < 70% (enough bandwidth to react)
    • Strong and disciplined team
    • Integration in solutions design
    @EdPiairo, #DevOpsPorto

    View Slide

  32. DEVOPS EVERYWEHERE
    Operations for databases
    • Applications
    • Databases
    • Infrastructure
    @EdPiairo, #DevOpsPorto

    View Slide

  33. WHY DEVOPS?
    Operations for databases
    • Developing software is not enough, you have to deliver it
    • Communication and collaboration framework for manage changes
    • You can not stop change, but you can control it
    • CAMS
    • We change the culture (with lot of negotiation and evangelization)
    • We invested in automation (started with 0 automation in a never ending automation journey)
    • We conquer work flow visibility, so we were able to start measure
    • We spread the contract across the organization
    @EdPiairo, #DevOpsPorto

    View Slide

  34. SOME METRICS/RESULTS
    Operations for databases
    • We went from 100% manual databases changes to about 98% automatic changes (100% controlled changes)
    • On average 200 scripts were created per day
    • In one year we achieved 10k scripts
    • We went from supporting only 1costumer to 4 costumers at the same time
    • Each week, costumer requests were delivered
    • Fixes were applied whenever necessary
    @EdPiairo, #DevOpsPorto

    View Slide

  35. THE DEVOPS WAY
    Operations for databases
    • DevOps: contract for collaboration and communication (change management “framework”)
    • Change description (Source Control)
    • Change validation (Continuous Integration)
    • Change implementation (Continuous Delivery)
    • Applications, databases, infrastructure
    @EdPiairo, #DevOpsPorto

    View Slide

  36. OPERATIONS WORK MANAGEMENT
    Operations for databases
    • Scrum or Kanban?
    • Depends on
    • DevOps team structure (http://web.devopstopologies.com/)
    • How many different topologies
    • Your clients
    • Good ideas
    • Have cadence/rhythm
    • Alignment with development team
    • Visibility, transparency and sharing (from doers to operations enablers)
    @EdPiairo, #DevOpsPorto

    View Slide

  37. READING MATERIAL
    Operations for databases
    @EdPiairo, #DevOpsPorto

    View Slide

  38. Q&A
    Operations for databases
    @EdPiairo
    https://pt.linkedin.com/in/jesuspiairo
    [email protected]
    http://www.eduardopiairo.com/
    @EdPiairo, #DevOpsPorto

    View Slide