Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Workshop: Delivering chnages for applications and databases

Workshop: Delivering chnages for applications and databases

Workshop presented @ Scrum Gathering Portugal (02-12-2016) by Eduardo Piairo and Miguel Alho


Eduardo Piairo

December 02, 2016


  1. DevOps delivering changes for applications and databases

  2. Quem somos Eduardo Piairo, DevOps Engineer @ Basecone @EdPiairo https://pt.linkedin.com/in/jesuspiairo

    eduardopiairo@outlook.com http://www.eduardopiairo.com/
  3. Quem Somos Miguel Alho, Software Engineer @ Faro @MytyMyky https://pt.linkedin.com/in/miguelalho-982331

    alho@miguelalho.com http://www.miguelalho.pt/
  4. Chapter 1: Introduction 1. The “DevOps” context 2. The problem

    that we want to fix 3. The database challenge 4. Infrastructure 5. Collaboration & Culture 6. Continuous Integration 7. Continuous Delivery
  5. Chapter 2: Application 1. Microservice repository creation 2. Creating the

    first functionality with TDD 3. Implement a build job / CI pipeline
  6. Chapter 3: Infrastructure 1. Physical machines 2. Virtual machines 3.

    Platform as a Service 4. Serverless 5. Infrastructure as Code 6. Provision and deployment for our microservice
  7. Chapter 4: Database 1. The special case of the database

    2. Database automation 3. Migrations vs State 4. Flyway 5. Database integration into our microservice
  8. Chapter 5: Managing Changes 1. Business requirements changes 2. How

    interaction is important to determine how to move forward 3. Versioning strategies 4. Deployment strategies 5. Implementing changes in our Microservice
  9. Chapter 6: DevOps engineering 1. Deployment pipeline 2. Why DevOps

    3. DevOps contract
  10. Chapter 1 Because everything starts with a problem...

  11. The DevOps context Chapter 1

  12. Stakeholders Sponsors / Customers Product Owner / Project Manager Coaches

    and Scrum Masters Business Stakeholders Architects, Leads and Developers Quality Assurance Engineers Release Engineers DBAs IT Operations Development Stakeholders User
  13. Business stakeholders want... Working software Software that fix the customer's

    problems Software that add value New features and fast Incremental features Frequent changes (Continuous Change) Business development and continuity Sponsors / Costumers Product Owner / Project Manager Coaches and Scrum Masters Business Stakeholders Sponsors / Customers Product Owner / Project Manager Coaches and Scrum Masters Business Stakeholders
  14. Development stakeholders want... Development Stakeholders Develop quality solutions Secure and

    stable solutions Avoid bugs and midnight calls Few obstacles Architects, Leads and Developers Quality Assurance Engineers Release Engineers DBAs IT Operations
  15. There is a problem that we need to fix Chapter

  16. Dichotomy of interests Traditionally there is collision of interests. Devops

    intends to break this collision with communication, empathy and collaboration improvement, along with useful technical practices, principally automation New features Corrections and improvements Optimizations Stability Security Troubleshooting Operations Devs
  17. Delivery of new features quickly Devs must be able to

    make changes to software quickly to integrate new features Software should be developed in order to integrate changes, instead of simply passing the tests Requires: • Use of adequate design patterns • Automation of the build process • Testing in different environments/levels • Fast functionality validation • Regression prevention • Quality measurement
  18. Software delivery consistently Putting software in production should not be

    a headache Requires: • Continuous Integration • Continuous Delivery • Process automation – consistency guarantee • Visibility – quality and metrics
  19. Scalability Rapidly answer to new needs for delivery as well

    as users load requirements Requires: • Provisioning automation • Deployment automation • Instance generation simplification • Monitoring
  20. The database challenge Chapter 1

  21. Database as bottleneck • Databases traditionally are out of pace

    with application development • Easy target to blame • Lack of traceability of databases changes (changes history) • Manual database processes prevent CI and CD
  22. Database as bottleneck Requires: • Automation • Source Control database

    changes (1st step for db deployment pipeline) • Synchronization between application and database
  23. Infrastructure Chapter 1

  24. Infrastructure How can you have the required consistency without including

    infrastructure? If you practice (development) and play (production) in different fields, why do you expect the same behavior/performance?
  25. Collaboration Not everything is possible to solve with technical solutions

    Chapter 1
  26. DevOps change

  27. Collaborative culture Respect Continuous learning Consensus Non-violent communication Disposal of

    “Command and control” attitudes
  28. It clearly can not be this:

  29. TEAM creation/building

  30. Investigation

  31. Culture is key… … but technical aspects do matter Chapter

  32. None
  33. ( ) http://www.slideshare.net/DougSeven/continuous-delivery-28357633

  34. Knight(Mare) • Knight Capital Group • 3.3 billion trades daily

    • $21.5 billion traded daily • $365M cash & equivalents ----- • NYSE launches Retail Liquidity Program on August 1, 2012, an automated routing system for equity orders • Group updates software to comply and repurpose 8-year old feature flag
  35. Knight(mare) “During the deployment of the new code, however, one

    of Knight’s technicians did not copy the new code to one of the eight SMARs computer servers.” SEC Filing | Release No. 70694 | October 16, 2013 ( http://tinyurl.com/Devops-Knightmare )
  36. Knight(mare) • 9:30 AM - Market Opens • 212 small

    retail “parent-orders” • 7 servers processing “child-orders” correctly. • 8th server using the old feature flag code failed to recognize parent-orders were fulfilled. • 8th server sent cumulative child.-orders in rapid succession.
  37. Knight(mare) • No automated fail-safe • No procedures for how

    to react • Knight uninstalled the correct SMARS code from the seven servers where it had been deployed correctly • 9:30 AM to 10:15 • 4 million executions in 154 stocks • 357 million shares
  38. DevOps changes in Technical Practices • Continuous Integration • Continuous

    Delivery • Provisioning (Infrastructure as Code) • Containerization • Microservices • Database migrations
  39. Continuous Integration Chapter 1

  40. Continuous Integration Increases your opportunities feedback (fast) feedback Build software

    at every change Features: A connection to a version control repository A build script A feedback mechanism A process for integrating the source code changes
  41. Continuous Integration To implement it, we need: • Source Control

    - Git, Hg, Svn, TFS, ... • Build System ◦ Scripting - Cake, Psake, FAKE, Gulp … ◦ Compiler and tools - MsBuild, Node/npm, GitVersion, … ◦ Build Server - VSTS, TeamCity, AppVeyor, CircleCi • Testing Framework - nUnit, xUnit, Fixie, tSQLt, ... • Packaging mechanism - Nuget, Docker container, zip file…
  42. Continuous Delivery Chapter 1

  43. Continuous Delivery Deployment Pipeline Automated implementation of the application’s build,

    deploy, test, and release processes
  44. Chapter 2 Application, the logical world

  45. Here’s what we are going to do: • Create a

    project in VSTS • Create a repository for a Microservice, with in memory data • Implement the microservice using TDD • Create a basic build job for our microservice in VSTS • Se this whole setup run
  46. Chapter 3 Infrastructure (as code)

  47. Physical machines High hardware cost High human cost Very low

  48. Virtual Machines In the cloud Upfront cost First step to

    IaC Requires configuration management
  49. Platform as a Service (PaaS) Very flexible approach Costs grow

    with the application Fast and simple development Bring infrastructure/operations closer to the development
  50. Serverless (FaaS) Azure Functions AWS Lambda Pay what you use

  51. Infrastructure as Code (IaC) Thank you Virtualization!! Managing and provisioning

    infrastructure Is not simply writing scripts Allows to apply proven software development practices Version control, testing, small deployments
  52. Chapter 4 Database development

  53. What’s so special about the database?

  54. What’s so special about the database?

  55. Database Lifecycle Management (DLM) Data management and data migration Data

    monitoring Data recovery
  56. Database challenges Databases are out of pace with application development

    Lack of traceability of database changes (changes history) Manual databases processes prevent the CI and CD utilization Lack of testability Database setup time for new environments Bugs in production
  57. Database challenges Databases becoming a bottleneck in an agile delivery

  58. None
  59. Why database automation? Enable control over database development Increase speed

    of response to change Keep a versioned “history” of database states and changes Reliability of the release process Increase release frequency though repeatability of processes Reduce time spent fixing bugs - automated tests Remove/reduce human intervention in the release process
  60. Why database automation? Fearless database changes

  61. 1st Step: Source Control First step in your database deployment

    pipeline Traceability through change history SQL as documentation Shared code-base and shared process Enforceable standards to reduce conflicts
  62. 1st Step: Source Control Fundamental resource: SQL Script

  63. Migrations vs State State based solutions Script represents the current

    database state Your source of truth is how the database should be Migrations based solutions Script represents a migration Migration represents how to transition to the next database version Your source of truth is how the database should change
  64. Flyway Flyway – “Database Migrations made Easy” http://flywaydb.org/ Open source

    database migration tool Flyway command-line enabled Simplicity: easy to setup, no need to install Zero dependencies (java + jdbc) Scripts are written in SQL
  65. Flyway: How it works Flyway commands: migrate, clean, info, validate,

    baseline, repair Metadata table: Used to track the state of the database If the database is empty, flyway won’t find it and will create it Flyway scans the migrations directory and check migrations against the metadata table Migrations are sorted based on their version number and applied in order
  66. Version Control System Two components Database scripts repo (git) Flyway

    - database migrations Rule 1: Script version (timestamp) Rule 2: Operation type Rule 3: Object type Rule 4: Object name Example: V20160728.1100__Create_TB_MyTable.sql
  67. Time to add a database to our service • Provision

    development Database • Add test project in Code • Implement database features in our microservice • Test • Add steps in our build system to support this • Deploy changes
  68. Chapter 5 Managing changes...

  69. Change is unavoidable Businesses change naturally • competitive advantage requires

    enhancements • products need to move towards customer’s real needs • social and economic context affects how business is done Software NEEDS to change to enable this!
  70. Change is not easy in software Ideally, changes contained in

    one component should NOT break all the others! Unfortunately, this is not always possible • more than one app consumes a service • more than one app consumes a database • binary data contracts (dlls) need to be updated everywhere • environment variables and changes affect program’s expected behaviour • lack of tests brings uncertainty to change outcome Many of these are anti-patterns!
  71. Collaboration is key • Need for change needs to be

    communicated • Risk of change needs to be analysed • Effects need to be listed to prepare for possible problems and avoid fire- fighting • Teams need to announce how changes will impact existing processes and work. • Teams need to determine if the change is done in a single step or in multiple iterations.
  72. Versioning strategies • Versioning allows team to tag artifacts to

    a specific state • Collaborating components can assert their validity based on versions ◦ component A requires B version X or higher • Semantic versioning gives meaning to number ◦ prefer SemVer • Automate versioning as much as possible ◦ tooling and commit messages can be enough to indicate the type of change • Important part of a CI pipeline - end result of a build is a specific version of an
  73. SemVer 1.3.5-beta0003 major minor patch prerelease increment on a breaking

    change increment on a feature addition increment on a bug fix increment while unreleased
  74. GitFlow

  75. 0.1.1-beta0001 0.1.1 0.2.0

  76. Versioning is not easy • But it is extremely important

    • Semantics matter ◦ minors and patches mean clients shouldn’t have to change anything, only update ◦ Majors are breaking so clients need to take some maintenance and realignment action • It’s also automatic documentation
  77. Deployment strategies You choose based on what your context is

    (traffic volume, working hours, SLAs) • Single Server or service ◦ Stop service, upgrade, restart • Multiple Servers ◦ Stop service, upgrade, restart ◦ Blue-Green deployment ◦ Canary Deployment Whatever the scenario, automation is key
  78. Deployments with DBs make things harder • Database’s data is

    mutable. It’s super hard to rollback! ◦ How do you rollback changes that delete a column of data? • Application is typically immutable. It’s way easier to rollback ◦ Just need the database schema to be compatible This means that we typically need to make changes in multiple steps to avoid breaking the service. Schema should valid for backward compatible to at least 1 version of the application. Marking things as deprecated can / should make a difference.
  79. Chapter 6 DevOps as engineering practice...

  80. What we just saw

  81. Why DevOps Developing software is not enough, you have to

    deliver it Communication framework for change management You can not stop change, but you can control it Perspectives Need for speed (time-to- market) (management people) Need for control (error control) (operations people)
  82. DevOps: contract for collaboration and communication Contract - change communication

    management tool Change description (Source Control) Change validation (Continuous Integration) Change implementation (Continuous Delivery) Applications, databases, infrastructure
  83. DevOps advices Operations requirements in Backlog Operations engineers in Product

    Team Shared goals Shared responsibilities Devs monitor apps Devs deploy apps
  84. Reading material

  85. None
  86. Links

  87. Demo application “Purchases-DbMigration-sample” https://github.com/MiguelAlho/Purchases-DbMigration-sample Flyway https://flywaydb.org/ DbUp http://dbup.github.io/ Octopus Deploy

    https://octopus.com/ http://docs.octopusdeploy.com/display/OD/Deploying+ASP.NET+Core+Web+Applications Visual Studio https://www.visualstudio.com/
  88. Final Q&A