$30 off During Our Annual Pro Sale. View Details »

Aren't we forgetting someone?

Aren't we forgetting someone?

How do DBAs fit into the world of DevOps? We usually think of DevOps as a collaboration between developers and operations or support engineers so it’s easy to forget the database admins who occupy both the development and operations space.

Organisationally they are often separate, sometimes geographically as well as organisationally. How does the database fit into the DevOps or Continuous Delivery story, both technologically and culturally?

Dan North has been helping a number of large organisations at various stages on their journey to continuous delivery and in every case their legacy database dependencies are a major stumbling block. Where do you start unpicking the database? How can you tell if it is working? Is the Matrix just a very big Oracle instance?

Daniel Terhorst-North
PRO

February 04, 2016
Tweet

More Decks by Daniel Terhorst-North

Other Decks in Technology

Transcript

  1. Aren’t we forgetting
    someone?
    Dan North
    @tastapod

    View Slide

  2. @tastapod
    The Challenge

    View Slide

  3. @tastapod
    Can we treat the
    “database as code”?

    View Slide

  4. @tastapod
    Can we treat the
    “database as code”?
    Servers as code

    View Slide

  5. @tastapod
    Can we treat the
    “database as code”?
    Servers as code
    Network and DNS as code

    View Slide

  6. @tastapod
    Can we treat the
    “database as code”?
    Servers as code
    Network and DNS as code
    Release management as code

    View Slide

  7. @tastapod
    Can we treat the
    “database as code”?
    Servers as code
    Network and DNS as code
    Configuration as code
    Release management as code

    View Slide

  8. @tastapod
    Can we treat the
    “database as code”?
    Servers as code
    Network and DNS as code
    Configuration as code
    Release management as code
    Database as code

    View Slide

  9. @tastapod
    Database as code

    View Slide

  10. @tastapod
    Database as code
    All changes scripted and automated

    View Slide

  11. @tastapod
    Database as code
    All changes scripted and automated
    All changes under version control

    View Slide

  12. @tastapod
    Database as code
    All changes scripted and automated
    All changes under version control
    Ability to release on demand

    View Slide

  13. @tastapod
    Database as code
    All changes scripted and automated
    All changes under version control
    Ability to release on demand
    DBAs integrated with Dev and Ops

    View Slide

  14. @tastapod
    The Context

    View Slide

  15. @tastapod
    This is your architecture

    View Slide

  16. @tastapod
    This is your architecture

    View Slide

  17. View Slide

  18. id name language
    1 chef ruby
    2 puppet ruby
    3 ansible python
    4 salt python
    products

    View Slide

  19. id name language
    1 chef ruby
    2 puppet ruby
    3 ansible python
    4 salt python
    products
    relation (table)

    View Slide

  20. id name language
    1 chef ruby
    2 puppet ruby
    3 ansible python
    4 salt python
    products field (column)
    relation (table)

    View Slide

  21. id name language
    1 chef ruby
    2 puppet ruby
    3 ansible python
    4 salt python
    products field (column)
    record (row)
    relation (table)

    View Slide

  22. id name language
    1 chef ruby
    2 puppet ruby
    3 ansible python
    4 salt python
    products field (column)
    record (row)
    relation (table)
    id name product_id
    10 Puppet Labs 2
    11 Chef inc. 1
    12 SaltStack 4
    13 Red Hat 3
    vendors

    View Slide

  23. id name language
    1 chef ruby
    2 puppet ruby
    3 ansible python
    4 salt python
    products field (column)
    record (row)
    relation (table)
    id name product_id
    10 Puppet Labs 2
    11 Chef inc. 1
    12 SaltStack 4
    13 Red Hat 3
    vendors
    foreign key

    View Slide

  24. id name language
    1 chef ruby
    2 puppet ruby
    3 ansible python
    4 salt python
    products field (column)
    record (row)
    relation (table)
    id name product_id
    10 Puppet Labs 2
    11 Chef inc. 1
    12 SaltStack 4
    13 Red Hat 3
    vendors
    foreign key
    index

    View Slide

  25. id name language
    1 chef ruby
    2 puppet ruby
    3 ansible python
    4 salt python
    products field (column)
    relation (table)
    id name product_id
    10 Puppet Labs 2
    11 Chef inc. 1
    12 SaltStack 4
    13 Red Hat 3
    vendors
    foreign key
    UPDATE products
    SET name = …
    stored proc
    index
    record (row)

    View Slide

  26. id name language
    1 chef ruby
    2 puppet ruby
    3 ansible python
    4 salt python
    products field (column)
    relation (table)
    id name product_id
    10 Puppet Labs 2
    11 Chef inc. 1
    12 SaltStack 4
    13 Red Hat 3
    vendors
    foreign key
    UPDATE products
    SET name = …
    stored proc
    constraint
    index
    record (row)

    View Slide

  27. id name language
    1 chef ruby
    2 puppet ruby
    3 ansible python
    4 salt python
    products field (column)
    relation (table)
    id name product_id
    10 Puppet Labs 2
    11 Chef inc. 1
    12 SaltStack 4
    13 Red Hat 3
    vendors
    foreign key
    UPDATE products
    SET name = …
    stored proc
    constraint
    trigger
    index
    record (row)

    View Slide

  28. id name language
    1 chef ruby
    2 puppet ruby
    3 ansible python
    4 salt python
    products field (column)
    relation (table)
    id name product_id
    10 Puppet Labs 2
    11 Chef inc. 1
    12 SaltStack 4
    13 Red Hat 3
    vendors
    foreign key
    UPDATE products
    SET name = …
    stored proc
    constraint
    trigger
    index
    table_size = …
    heap_size = …
    free_space = …
    last_access = …
    heuristics
    record (row)

    View Slide

  29. structural data
    application data
    reference data
    procs, triggers
    Databases contain data

    View Slide

  30. structural
    data
    application data
    reference
    data
    procs,
    triggers
    Lots and lots of data!

    View Slide

  31. @tastapod
    The Constraints

    View Slide

  32. @tastapod
    Database changes come
    from multiple directions

    View Slide

  33. @tastapod
    Database changes come
    from multiple directions
    Application changes

    View Slide

  34. @tastapod
    Database changes come
    from multiple directions
    Application changes Data growth

    View Slide

  35. @tastapod
    Database changes come
    from multiple directions
    Application changes Data growth
    Policy changes

    View Slide

  36. @tastapod
    Database changes come
    from multiple directions
    Application changes
    Performance needs
    Data growth
    Policy changes

    View Slide

  37. @tastapod
    Database changes come
    from multiple directions
    Application changes
    Performance needs
    Data growth
    Upgrades and patches
    Policy changes

    View Slide

  38. @tastapod
    Database changes come
    from multiple directions
    Application changes
    Performance needs
    Data growth
    Upgrades and patches
    Policy changes
    Dev

    View Slide

  39. @tastapod
    Database changes come
    from multiple directions
    Application changes
    Performance needs
    Policy changes
    Dev
    Business
    Data growth
    Upgrades and patches

    View Slide

  40. @tastapod
    Database changes come
    from multiple directions
    Application changes
    Performance needs
    Data growth
    Upgrades and patches
    Dev Ops
    Policy changes
    Business

    View Slide

  41. @tastapod
    DBAs have been bridging
    Dev and Ops for years!
    Dev Ops

    View Slide

  42. @tastapod
    DBAs have been bridging
    Dev and Ops for years!
    Dev Ops
    but not how we would define it…

    View Slide

  43. reference data
    structural data
    application data
    procs, triggers
    Multiple data lifecycles

    View Slide

  44. reference data
    structural data
    application data
    procs, triggers
    Multiple data lifecycles
    changes with app changes with app

    View Slide

  45. reference data
    changes with business
    structural data
    application data
    procs, triggers
    Multiple data lifecycles
    changes with app changes with app

    View Slide

  46. reference data
    changes with business
    structural data
    application data
    procs, triggers
    Multiple data lifecycles
    changes with app changes with app
    changes with usage

    View Slide

  47. @tastapod
    Application data is
    long-lived

    View Slide

  48. @tastapod
    Application data is
    long-lived
    Outlives releases

    View Slide

  49. @tastapod
    Application data is
    long-lived
    Outlives releases
    Outlives applications

    View Slide

  50. @tastapod
    Application data is
    long-lived
    Outlives releases
    Outlives applications
    Outlives people!

    View Slide

  51. @tastapod
    Database is often an
    integration point

    View Slide

  52. @tastapod
    Database is often an
    integration point

    View Slide

  53. @tastapod
    Database is often an
    integration point

    View Slide

  54. @tastapod
    Database is often an
    integration point

    View Slide

  55. @tastapod
    Database mistakes can
    have a high impact

    View Slide

  56. @tastapod
    Database mistakes can
    have a high impact
    Performance impact

    View Slide

  57. @tastapod
    Database mistakes can
    have a high impact
    Cascading failures
    Performance impact

    View Slide

  58. @tastapod
    Database mistakes can
    have a high impact
    Cascading failures
    Performance impact
    Coincident failures

    View Slide

  59. @tastapod
    Database mistakes can
    have a high impact
    Cascading failures
    Performance impact
    Data loss
    Coincident failures

    View Slide

  60. @tastapod
    Database mistakes can
    have a high impact
    Cascading failures
    Performance impact
    Data loss
    Data corruption
    Coincident failures

    View Slide

  61. @tastapod
    Database mistakes can
    have a high impact
    Cascading failures
    Performance impact
    Data loss
    Data corruption
    Detection can be distant in space and time, and cost
    Coincident failures

    View Slide

  62. @tastapod
    DevOps relies on
    minimising impact
    impact
    likelihood

    View Slide

  63. @tastapod
    DevOps relies on
    minimising impact
    impact
    likelihood
    MTBF

    View Slide

  64. @tastapod
    DevOps relies on
    minimising impact
    impact
    likelihood
    MTBF
    MTTR

    View Slide

  65. @tastapod
    DevOps relies on
    minimising impact
    impact
    likelihood
    We used to obsess
    about this
    MTBF
    MTTR

    View Slide

  66. @tastapod
    DevOps relies on
    minimising impact
    impact
    likelihood
    We used to obsess
    about this
    DevOps cares
    about this
    MTBF
    MTTR

    View Slide

  67. @tastapod

    View Slide

  68. DBAs are conservative and change-cautious
    @tastapod

    View Slide

  69. DevOps is relentless and change-eager
    DBAs are conservative and change-cautious
    @tastapod

    View Slide

  70. DevOps is relentless and change-eager
    This is the culture clash!
    DBAs are conservative and change-cautious
    @tastapod

    View Slide

  71. @tastapod
    The Approach

    View Slide

  72. @tastapod
    The Status Quo

    View Slide

  73. @tastapod
    The Status Quo
    Most changes are manual

    View Slide

  74. @tastapod
    The Status Quo
    Most changes are manual
    Driven by tickets or change requests

    View Slide

  75. @tastapod
    The Status Quo
    Most changes are manual
    Driven by tickets or change requests
    Carefully planned and scrutinised

    View Slide

  76. @tastapod
    The Status Quo
    Most changes are manual
    Driven by tickets or change requests
    Carefully planned and scrutinised
    Fragmented audit trail

    View Slide

  77. @tastapod
    The Status Quo
    Most changes are manual
    Driven by tickets or change requests
    Carefully planned and scrutinised
    Fragmented audit trail
    Expensive and time-consuming to repeat

    View Slide

  78. @tastapod
    Database-as-code #1:
    Migrations

    View Slide

  79. @tastapod
    Database-as-code #1:
    Migrations
    current

    View Slide

  80. @tastapod
    Database-as-code #1:
    Migrations
    ALTER TABLE …
    ADD COLUMN …

    INSERT INTO
    ref_data…

    current

    View Slide

  81. @tastapod
    Database-as-code #1:
    Migrations
    ALTER TABLE …
    ADD COLUMN …

    INSERT INTO
    ref_data…

    current new

    View Slide

  82. @tastapod
    Database-as-code #1:
    Migrations
    ALTER TABLE …
    ADD COLUMN …

    INSERT INTO
    ref_data…

    current new
    VCS

    View Slide

  83. @tastapod
    Database-as-code #1:
    Migrations
    ALTER TABLE …
    ADD COLUMN …

    INSERT INTO
    ref_data…

    current new
    dbdeploy (RIP)
    VCS

    View Slide

  84. @tastapod
    Database-as-code #1:
    Migrations
    ALTER TABLE …
    ADD COLUMN …

    INSERT INTO
    ref_data…

    current new
    dbdeploy (RIP)
    rails
    VCS

    View Slide

  85. @tastapod
    Database-as-code #1:
    Migrations
    ALTER TABLE …
    ADD COLUMN …

    INSERT INTO
    ref_data…

    current new
    dbdeploy (RIP)
    rails
    flyway
    VCS

    View Slide

  86. @tastapod
    Database-as-code #1:
    Migrations
    ALTER TABLE …
    ADD COLUMN …

    INSERT INTO
    ref_data…

    current new
    dbdeploy (RIP)
    rails
    flyway
    liquibase
    VCS

    View Slide

  87. @tastapod
    Code-as-code:
    A comparison
    ALTER TABLE …
    ADD COLUMN …

    INSERT INTO
    ref_data…

    current new
    VCS

    View Slide

  88. @tastapod
    VCS
    Code-as-code:
    A comparison
    - old_code…

    + new_code…

    current new

    View Slide

  89. @tastapod
    VCS
    Code-as-code:
    A comparison
    - old_code…

    + new_code…

    current new
    Developer edits the code

    View Slide

  90. @tastapod
    VCS
    Code-as-code:
    A comparison
    - old_code…

    + new_code…

    current new
    Developer edits the code
    VCS figures out the script

    View Slide

  91. @tastapod
    VCS
    Code-as-code:
    A comparison
    - old_code…

    + new_code…

    current new
    Developer edits the code
    VCS figures out the script
    VCS can create new state

    View Slide

  92. @tastapod
    VCS
    Database-as-code #2:
    DBVCS?
    ALTER TABLE …
    ADD COLUMN …

    INSERT INTO
    ref_data…

    current new
    Developer edits the database
    VCS figures out the script
    VCS can create new state

    View Slide

  93. @tastapod
    VCS
    Database-as-code #2:
    DBVCS?
    ALTER TABLE …
    ADD COLUMN …

    INSERT INTO
    ref_data…

    current new
    Red Gate is close but
    commercial. No OSS
    equivalent
    Developer edits the database
    VCS figures out the script
    VCS can create new state

    View Slide

  94. @tastapod
    The Transition

    View Slide

  95. @tastapod
    Engage with your DBA

    View Slide

  96. @tastapod
    Engage with your DBA
    Their context is broader than yours

    View Slide

  97. @tastapod
    Engage with your DBA
    Their context is broader than yours
    They already have a foot in both camps

    View Slide

  98. @tastapod
    Engage with your DBA
    Their context is broader than yours
    They already have a foot in both camps
    Elevate the constraint

    View Slide

  99. @tastapod
    Engage with your DBA
    Their context is broader than yours
    They already have a foot in both camps
    Elevate the constraint
    Don’t “DevOps” the DBAs!

    View Slide

  100. @tastapod
    Engage with your DBA
    Their context is broader than yours
    They already have a foot in both camps
    Elevate the constraint
    Don’t “DevOps” the DBAs!
    Now, who else are we forgetting?

    View Slide

  101. @tastapod
    Thanks for listening!
    Dan North
    @tastapod
    [email protected]
    http:/
    /dannorth.net

    View Slide