Operations for databases – The DevOps journey

Talk presented @ CommitPorto (30-06-2018)

Transcript

1. OPERATIONS FOR DATABASES Eduardo Piairo @EdPiairo #commitporto The DevOps Journey
2. ABOUT ME Operations for databases – The DevOps journey @EdPiairo, #commitporto @EdPiairo https://pt.linkedin.com/in/jesuspiairo eduardopiairo@gmail.com http://www.eduardopiairo.com/ Eduardo Piairo DevOps Coach @ DevOps Porto Founder
3. THIS PRESENTATION Agile journey Scrum Kanban DevOps Operations for databases – The DevOps journey Database operations Automation Source Control Continuous Integration Continuous Delivery The definition and the interpretation of the covered concepts are attached to the cultural context of a specific organization on a certain period! “Context is everything” @EdPiairo, #commitporto
4. AGILE JOURNEY 1.Before Scrum 2.Scrum 3.Kanban 4.Scrum + Kanban Operations for databases – The DevOps journey @EdPiairo, #commitporto
5. AGILE JOURNEY 1.Before Scrum 2.Scrum 3.Kanban 4.Scrum + Kanban Operations for databases – The DevOps journey @EdPiairo, #commitporto
6. BEFORE SCRUM Operations for databases – The DevOps journey • Before Scrum - The “real agile” method • Centralized decision mechanism • Growing phase • Size (5 to 15) • Complexity (more and more components) @EdPiairo, #commitporto
7. BEFORE SCRUM Operations for databases – The DevOps journey Application #1 Application #2 Application #3 Shared database(s) “Team” #1 “Team” #2 “Team” #3 “Team” #T-SQL @EdPiairo, #commitporto
8. AGILE JOURNEY 1.Before Scrum 2.Scrum 3.Kanban 4.Scrum + Kanban Operations for databases – The DevOps journey @EdPiairo, #commitporto
9. SCRUM Operations for databases – The DevOps journey • Scrum - The “magic” scrum • Scrum for the masses (>20) • Pure Scrum approach – operations as development team • 2 week sprint, sprint goal • The team was “too reactive” @EdPiairo, #commitporto
10. DEVOPS FOR DATABASES – THE BEGINNING Operations for databases – The DevOps journey Application #1 Application #2 Application #3 Shared database(s) Team #1 Team #2 Team #3 Team #T-SQL @EdPiairo, #commitporto
11. DEVOPS FOR DATABASES – BUILDING A DEPLOYMENT PIPELINE Operations for databases – The DevOps journey Source Control Continuous Integration Continuous Delivery Database + Application @EdPiairo, #commitporto
12. DEVOPS FOR DATABASES – THE SPECIAL CASE OF THE DATABASES What’s so special about databases? Operations for databases – The DevOps journey @EdPiairo, #commitporto
13. DEVOPS FOR DATABASES – PROBLEMS TO FIX Operations for databases – The DevOps journey • Databases are out of pace with application development • Synchronization between development and DBA teams • No traceability of database changes (changes history) • What changed? Who? When? Why? • Manual databases processes prevent the CI and CD utilization in their full extent • Time consuming and error prone • Releases are less frequent and risky @EdPiairo, #commitporto
14. DEVOPS FOR DATABASES – PROBLEMS TO FIX Operations for databases – The DevOps journey • Bugs in production environment • Database related bugs are only discovered after deployment to production • Manual or inexistent tests • Fixes and hotfixes have time cost, what can lead to delay a release • High dependency in a person (database specialist) • Fear of making changes @EdPiairo, #commitporto
15. DEVOPS FOR DATABASES – PROBLEMS TO FIX Operations for databases – The DevOps journey Databases became a bottleneck in an agile delivery process @EdPiairo, #commitporto
16. DEVOPS FOR DATABASES – THE SOLUTION Operations for databases – The DevOps journey Source Control Continuous Integration Continuous Delivery Automation + Change control @EdPiairo, #commitporto
17. DEVOPS FOR DATABASES – THE SOLUTION Decision #1 Let’s automate Let’s do source control Operations for databases – The DevOps journey @EdPiairo, #commitporto
18. DEVOPS FOR DATABASES – THE VALUE OF AUTOMATION Operations for databases – The DevOps journey • Enable control over database development • Increase speed of response to change • Keep a versioned “history” of database states • Greater reliability of the release process • Remove/reduce human intervention @EdPiairo, #commitporto From The art of deployment to The engineering of deployment
19. DEVOPS FOR DATABASES – THE VALUE OF AUTOMATION Operations for databases – The DevOps journey Fearless database changes @EdPiairo, #commitporto
20. DEVOPS FOR DATABASES – THE 1ST STEP: SOURCE CONTROL Operations for databases – The DevOps journey • First step in your database deployment pipeline • Traceability through change history • SQL as documentation • Shared code-base and shared process • Enforceable standards to reduce conflicts @EdPiairo, #commitporto SQL Script
21. DEVOPS FOR DATABASES – THE 1ST STEP: SOURCE CONTROL Decision #2 Database source control: Migrations vs State Operations for databases – The DevOps journey @EdPiairo, #commitporto
22. DEVOPS FOR DATABASES – MIGRATIONS VS STATE Operations for databases – The DevOps journey • State based solutions • Script represents the current database state • Your source of truth is how the database should be • Migrations based solutions • Script represents a migration • Your source of truth is how the database should change @EdPiairo, #commitporto Migration Migration State Delta
23. DEVOPS FOR DATABASES – COLLABORATION THROUGH A CONTRACT Operations for databases – The DevOps journey • Set of rules and expectations • Define responsibility frontiers • Sets a common language Application #1 Application #2 Application #3 Shared database(s) Team #1 T-SQL Script Team #2 Team #3 Team #T-SQL @EdPiairo, #commitporto
24. DEVOPS FOR DATABASES – RULES / GUIDELINES Operations for databases – The DevOps journey • Scripting rules • Rule 1: Script version (timestamp) • Rule 2: Operation type • Rule 3: Object type • Rule 4: Object name Example: V20160220.1100__Create_TB_MyTable.sql @EdPiairo, #commitporto • Small batches (One script, one operation type, one object) • Merge conflicts management • Patterns identification • File system scripts history search
25. DEVOPS FOR DATABASES - THE PIPELINE Operations for databases – The DevOps journey • Contract – communication, collaboration mechanism for change management • Don’t lie to the pipeline • Everything is negotiable in the contract, except its application @EdPiairo, #commitporto
26. AGILE JOURNEY 1.Before Scrum 2.Scrum 3.Kanban 4.Scrum + Kanban Operations for databases – The DevOps journey @EdPiairo, #commitporto
27. KANBAN Operations for databases – The DevOps journey • Kanban 101 • Focus on development teams necessities • 4 week iterations • A bad choice • Different “language” from other teams • Desynchronization between operations and development teams @EdPiairo, #commitporto
28. DEVOPS FOR DATABASES – THE PIPELINE Operations for databases – The DevOps journey • Change description (Source Control) • Change validation (Continuous Integration) • Change implementation (Continuous Delivery) @EdPiairo, #commitporto
29. AGILE JOURNEY 1.Before Scrum 2.Scrum 3.Kanban 4.Scrum + Kanban Operations for databases – The DevOps journey @EdPiairo, #commitporto
30. SCRUM + KANBAN Operations for databases – The DevOps journey • Scrum + Kanban – The best of two worlds • 2 week sprint, sprint goal • Task definition was based on teams needs and our needs • Team capacity < 70% (enough bandwidth to react) • Strong and disciplined team • Integration in solutions design @EdPiairo, #commitporto
31. DEVOPS EVERYWHERE Operations for databases – The DevOps journey • Applications • Databases • Infrastructure @EdPiairo, #commitporto
32. WHY DEVOPS? Operations for databases – The DevOps journey • Developing software is not enough, you have to deliver it • You can not stop change, but you can control it • Different intervenients need to communication and collaborate • CAMS • We change the culture (with lot of negotiation and evangelization) • We invested in automation (started with 0 automation in a never ending automation journey) • We conquer work flow visibility, so we were able to start measure • We spread the contract across the organization @EdPiairo, #commitporto
33. THE DEVOPS WAY Operations for databases – The DevOps journey @EdPiairo, #commitporto • DevOps way • Increase flow visibility • Increase feedback • Increase knowledge base
34. SOME METRICS/RESULTS Operations for databases – The DevOps journey • We went from 100% manual databases changes to about 98% automatic changes (100% controlled changes) • On average 200 scripts were created per day • In one year we achieved 10k scripts • We went from supporting only 1costumer to 4 costumers at the same time • Each week, costumer requests were delivered • Fixes were applied whenever necessary @EdPiairo, #commitporto
35. Q&A Operations for databases – The DevOps journey @EdPiairo https://pt.linkedin.com/in/jesuspiairo eduardopiairo@gmail.com http://www.eduardopiairo.com/ @EdPiairo, #commitporto