Tweet
Tweet

Slide 1

Slide 1 text

OPERATIONS FOR DATABASES Eduardo Piairo @EdPiairo #DevOpsPorto The Agile/DevOps Journey

Slide 2

Slide 2 text

ABOUT ME Operations for databases @EdPiairo, #DevOpsPorto @EdPiairo https://pt.linkedin.com/in/jesuspiairo [email protected] http://www.eduardopiairo.com/ Eduardo Piairo Operations Engineer DevOps Porto Founder

Slide 3

Slide 3 text

THIS PRESENTATION Agile journey Scrum Kanban DevOps Operations for databases Database operations Automation Source Control Continuous Integration Continuous Delivery The definition and the interpretation of the covered concepts are attached to the cultural context of a specific organization on a certain period! @EdPiairo, #DevOpsPorto

Slide 4

Slide 4 text

AGILE JOURNEY 1.Before Scrum 2.Scrum 3.Kanban 4.Scrum + Kanban Operations for databases @EdPiairo, #DevOpsPorto

Slide 5

Slide 5 text

AGILE JOURNEY 1.Before Scrum 2.Scrum 3.Kanban 4.Scrum + Kanban Operations for databases @EdPiairo, #DevOpsPorto

Slide 6

Slide 6 text

BEFORE SCRUM Operations for databases • Before Scrum - The “real agile” method • Centralized decision mechanism • Growing phase • Size (5 to 15) • Complexity (more and more components) @EdPiairo, #DevOpsPorto

Slide 7

Slide 7 text

BEFORE SCRUM Operations for databases Application #1 Application #2 Application #3 Shared database(s) “Team” #1 “Team” #2 “Team” #3 “Team” #T-SQL @EdPiairo, #DevOpsPorto

Slide 8

Slide 8 text

AGILE JOURNEY 1.Before Scrum 2.Scrum 3.Kanban 4.Scrum + Kanban Operations for databases @EdPiairo, #DevOpsPorto

Slide 9

Slide 9 text

SCRUM Operations for databases • Scrum - The “magic” scrum • Scrum for the masses (>20) • Pure Scrum approach – operations as development team • 2 week sprint, sprint goal • The team was “too reactive” @EdPiairo, #DevOpsPorto

Slide 10

Slide 10 text

DEVOPS FOR DATABASES – THE BEGINNING Operations for databases Application #1 Application #2 Application #3 Shared database(s) Team #1 Team #2 Team #3 Team #T-SQL @EdPiairo, #DevOpsPorto

Slide 11

Slide 11 text

DEVOPS FOR DATABASES – BUILDING A DEPLOYMENT PIPELINE Operations for databases Source Control Continuous Integration Continuous Delivery Database + Application @EdPiairo, #DevOpsPorto

Slide 12

Slide 12 text

DEVOPS FOR DATABASES – THE SPECIAL CASE OF THE DATABASES What’s so special about databases? Operations for databases @EdPiairo, #DevOpsPorto

Slide 13

Slide 13 text

DEVOPS FOR DATABASES – PROBLEMS TO FIX Operations for databases • Databases are out of pace with application development • Synchronization between development and DBA teams • No traceability of database changes (changes history) • What changed? Who? When? Why? • Manual databases processes prevent the CI and CD utilization in their full extent • Time consuming and error prone • Releases are less frequent and risky @EdPiairo, #DevOpsPorto

Slide 14

Slide 14 text

DEVOPS FOR DATABASES – PROBLEMS TO FIX Operations for databases • Bugs in production environment • Database related bugs are only discovered after deployment to production • Manual or inexistent tests • Fixes and hotfixes have time cost, what can lead to delay a release • High dependency in a person (database specialist) • Fear of making changes @EdPiairo, #DevOpsPorto

Slide 15

Slide 15 text

DEVOPS FOR DATABASES – PROBLEMS TO FIX Operations for databases Databases became a bottleneck in an agile delivery process @EdPiairo, #DevOpsPorto

Slide 16

Slide 16 text

DEVOPS FOR DATABASES – THE SOLUTION Operations for databases Source Control Continuous Integration Continuous Delivery Automation + Change control @EdPiairo, #DevOpsPorto

Slide 17

Slide 17 text

DEVOPS FOR DATABASES – THE SOLUTION Decision #1 Let’s automate Let’s do source control Operations for databases @EdPiairo, #DevOpsPorto

Slide 18

Slide 18 text

DEVOPS FOR DATABASES – THE VALUE OF AUTOMATION Operations for databases • Enable control over database development • Increase speed of response to change • Keep a versioned “history” of database states • Greater reliability of the release process • Increase release frequency through repeatability of processes • Reduce time spent fixing bugs @EdPiairo, #DevOpsPorto

Slide 19

Slide 19 text

DEVOPS FOR DATABASES – THE VALUE OF AUTOMATION Operations for databases • Remove/reduce human intervention in the release process • The build step is automatic triggered by a “push” into source control repository • The deploy step is automatic triggered by a successfully build process @EdPiairo, #DevOpsPorto

Slide 20

Slide 20 text

DEVOPS FOR DATABASES – THE VALUE OF AUTOMATION Operations for databases Fearless database changes @EdPiairo, #NetPonto

Slide 21

Slide 21 text

DEVOPS FOR DATABASES – THE 1ST STEP: SOURCE CONTROL Operations for databases • First step in your database deployment pipeline • Traceability through change history • SQL as documentation • Shared code-base and shared process • Enforceable standards to reduce conflicts • Fundamental resource: SQL script @EdPiairo, #DevOpsPorto

Slide 22

Slide 22 text

DEVOPS FOR DATABASES – THE 1ST STEP: SOURCE CONTROL Decision #2 Database source control: Migrations vs State Operations for databases @EdPiairo, #DevOpsPorto

Slide 23

Slide 23 text

DEVOPS FOR DATABASES – MIGRATIONS VS STATE Operations for databases • State based solutions • Script represents the current database state • Your source of truth is how the database should be • Migrations based solutions • Script represents a migration • Your source of truth is how the database should change Flyway: open source database migration tool • Simplicity: easy to setup, no need to install • Zero dependencies (java + jdbc) • Scripts are written in SQL @EdPiairo, #DevOpsPorto

Slide 24

Slide 24 text

DEVOPS FOR DATABASES – COLLABORATION THROUGH A CONTRACT Operations for databases • Set of rules and expectations • Define responsibility frontiers • Sets a common language Application #1 Application #2 Application #3 Shared database(s) Team #1 T-SQL Script Team #2 Team #3 Team #T-SQL @EdPiairo, #DevOpsPorto

Slide 25

Slide 25 text

DEVOPS FOR DATABASES - RULES Operations for databases • Scripting rules • Rule 1: Script version (timestamp) • Rule 2: Operation type • Rule 3: Object type • Rule 4: Object name Example: V20160220.1100__Create_TB_MyTable.sql • One script, one operation type, one object @EdPiairo, #DevOpsPorto SMALL BATCHES

Slide 26

Slide 26 text

DEVOPS FOR DATABASES - THE PIPELINE Operations for databases • Contract – communication, collaboration mechanism for change management • Should be reflected in your development pipeline • The better/clearer your pipeline, the less you need to document (your code is your documentation) • Everything is negotiable in the contract, except its application @EdPiairo, #DevOpsPorto

Slide 27

Slide 27 text

AGILE JOURNEY 1.Before Scrum 2.Scrum 3.Kanban 4.Scrum + Kanban Operations for databases @EdPiairo, #DevOpsPorto

Slide 28

Slide 28 text

KANBAN Operations for databases • Kanban 101 • Focus on development teams necessities • 4 week iterations • A bad choice • Different “language” from other teams • Desynchronization between operations and development teams @EdPiairo, #DevOpsPorto

Slide 29

Slide 29 text

DEVOPS FOR DATABASES – THE PIPELINE Operations for databases • Change description (Source Control) • Change validation (Continuous Integration) • Change implementation (Continuous Delivery) @EdPiairo, #DevOpsPorto

Slide 30

Slide 30 text

AGILE JOURNEY 1.Before Scrum 2.Scrum 3.Kanban 4.Scrum + Kanban Operations for databases @EdPiairo, #DevOpsPorto

Slide 31

Slide 31 text

SCRUM + KANBAN Operations for databases • Scrum + Kanban – The best of two worlds • 2 week sprint, sprint goal • Task definition was based on teams necessities and our necessities • Team capacity < 70% (enough bandwidth to react) • Strong and disciplined team • Integration in solutions design @EdPiairo, #DevOpsPorto

Slide 32

Slide 32 text

DEVOPS EVERYWEHERE Operations for databases • Applications • Databases • Infrastructure @EdPiairo, #DevOpsPorto

Slide 33

Slide 33 text

WHY DEVOPS? Operations for databases • Developing software is not enough, you have to deliver it • Communication and collaboration framework for manage changes • You can not stop change, but you can control it • CAMS • We change the culture (with lot of negotiation and evangelization) • We invested in automation (started with 0 automation in a never ending automation journey) • We conquer work flow visibility, so we were able to start measure • We spread the contract across the organization @EdPiairo, #DevOpsPorto

Slide 34

Slide 34 text

SOME METRICS/RESULTS Operations for databases • We went from 100% manual databases changes to about 98% automatic changes (100% controlled changes) • On average 200 scripts were created per day • In one year we achieved 10k scripts • We went from supporting only 1costumer to 4 costumers at the same time • Each week, costumer requests were delivered • Fixes were applied whenever necessary @EdPiairo, #DevOpsPorto

Slide 35

Slide 35 text

THE DEVOPS WAY Operations for databases • DevOps: contract for collaboration and communication (change management “framework”) • Change description (Source Control) • Change validation (Continuous Integration) • Change implementation (Continuous Delivery) • Applications, databases, infrastructure @EdPiairo, #DevOpsPorto

Slide 36

Slide 36 text

OPERATIONS WORK MANAGEMENT Operations for databases • Scrum or Kanban? • Depends on • DevOps team structure (http://web.devopstopologies.com/) • How many different topologies • Your clients • Good ideas • Have cadence/rhythm • Alignment with development team • Visibility, transparency and sharing (from doers to operations enablers) @EdPiairo, #DevOpsPorto

Slide 37

Slide 37 text

READING MATERIAL Operations for databases @EdPiairo, #DevOpsPorto

Slide 38

Slide 38 text

Q&A Operations for databases @EdPiairo https://pt.linkedin.com/in/jesuspiairo [email protected] http://www.eduardopiairo.com/ @EdPiairo, #DevOpsPorto