Workshop: Delivering chnages for applications and databases

Workshop presented @ Scrum Gathering Portugal (02-12-2016) by Eduardo Piairo and Miguel Alho

1. DevOps delivering changes for applications and databases
2. Quem somos Eduardo Piairo, DevOps Engineer @ Basecone @EdPiairo https://pt.linkedin.com/in/jesuspiairo eduardopiairo@outlook.com http://www.eduardopiairo.com/
3. Quem Somos Miguel Alho, Software Engineer @ Faro @MytyMyky https://pt.linkedin.com/in/miguelalho-982331 alho@miguelalho.com http://www.miguelalho.pt/
4. Chapter 1: Introduction 1. The “DevOps” context 2. The problem that we want to fix 3. The database challenge 4. Infrastructure 5. Collaboration & Culture 6. Continuous Integration 7. Continuous Delivery
5. Chapter 2: Application 1. Microservice repository creation 2. Creating the first functionality with TDD 3. Implement a build job / CI pipeline
6. Chapter 3: Infrastructure 1. Physical machines 2. Virtual machines 3. Platform as a Service 4. Serverless 5. Infrastructure as Code 6. Provision and deployment for our microservice
7. Chapter 4: Database 1. The special case of the database 2. Database automation 3. Migrations vs State 4. Flyway 5. Database integration into our microservice
8. Chapter 5: Managing Changes 1. Business requirements changes 2. How interaction is important to determine how to move forward 3. Versioning strategies 4. Deployment strategies 5. Implementing changes in our Microservice
9. Chapter 6: DevOps engineering 1. Deployment pipeline 2. Why DevOps 3. DevOps contract
10. Chapter 1 Because everything starts with a problem...
11. The DevOps context Chapter 1
12. Stakeholders Sponsors / Customers Product Owner / Project Manager Coaches and Scrum Masters Business Stakeholders Architects, Leads and Developers Quality Assurance Engineers Release Engineers DBAs IT Operations Development Stakeholders User
13. Business stakeholders want... Working software Software that fix the customer's problems Software that add value New features and fast Incremental features Frequent changes (Continuous Change) Business development and continuity Sponsors / Costumers Product Owner / Project Manager Coaches and Scrum Masters Business Stakeholders Sponsors / Customers Product Owner / Project Manager Coaches and Scrum Masters Business Stakeholders
14. Development stakeholders want... Development Stakeholders Develop quality solutions Secure and stable solutions Avoid bugs and midnight calls Few obstacles Architects, Leads and Developers Quality Assurance Engineers Release Engineers DBAs IT Operations
15. There is a problem that we need to fix Chapter 1
16. Dichotomy of interests Traditionally there is collision of interests. Devops intends to break this collision with communication, empathy and collaboration improvement, along with useful technical practices, principally automation New features Corrections and improvements Optimizations Stability Security Troubleshooting Operations Devs
17. Delivery of new features quickly Devs must be able to make changes to software quickly to integrate new features Software should be developed in order to integrate changes, instead of simply passing the tests Requires: • Use of adequate design patterns • Automation of the build process • Testing in different environments/levels • Fast functionality validation • Regression prevention • Quality measurement
18. Software delivery consistently Putting software in production should not be a headache Requires: • Continuous Integration • Continuous Delivery • Process automation – consistency guarantee • Visibility – quality and metrics
19. Scalability Rapidly answer to new needs for delivery as well as users load requirements Requires: • Provisioning automation • Deployment automation • Instance generation simplification • Monitoring
20. The database challenge Chapter 1
21. Database as bottleneck • Databases traditionally are out of pace with application development • Easy target to blame • Lack of traceability of databases changes (changes history) • Manual database processes prevent CI and CD
22. Database as bottleneck Requires: • Automation • Source Control database changes (1st step for db deployment pipeline) • Synchronization between application and database
23. Infrastructure Chapter 1
24. Infrastructure How can you have the required consistency without including infrastructure? If you practice (development) and play (production) in different fields, why do you expect the same behavior/performance?
25. Collaboration Not everything is possible to solve with technical solutions Chapter 1
26. DevOps change
27. Collaborative culture Respect Continuous learning Consensus Non-violent communication Disposal of “Command and control” attitudes
28. It clearly can not be this:
29. TEAM creation/building
30. Investigation
31. Culture is key… … but technical aspects do matter Chapter 1
33. ( ) http://www.slideshare.net/DougSeven/continuous-delivery-28357633
34. Knight(Mare) • Knight Capital Group • 3.3 billion trades daily • $21.5 billion traded daily • $365M cash & equivalents ----- • NYSE launches Retail Liquidity Program on August 1, 2012, an automated routing system for equity orders • Group updates software to comply and repurpose 8-year old feature flag
35. Knight(mare) “During the deployment of the new code, however, one of Knight’s technicians did not copy the new code to one of the eight SMARs computer servers.” SEC Filing | Release No. 70694 | October 16, 2013 ( http://tinyurl.com/Devops-Knightmare )
36. Knight(mare) • 9:30 AM - Market Opens • 212 small retail “parent-orders” • 7 servers processing “child-orders” correctly. • 8th server using the old feature flag code failed to recognize parent-orders were fulfilled. • 8th server sent cumulative child.-orders in rapid succession.
37. Knight(mare) • No automated fail-safe • No procedures for how to react • Knight uninstalled the correct SMARS code from the seven servers where it had been deployed correctly • 9:30 AM to 10:15 • 4 million executions in 154 stocks • 357 million shares
38. DevOps changes in Technical Practices • Continuous Integration • Continuous Delivery • Provisioning (Infrastructure as Code) • Containerization • Microservices • Database migrations
39. Continuous Integration Chapter 1
40. Continuous Integration Increases your opportunities feedback (fast) feedback Build software at every change Features: A connection to a version control repository A build script A feedback mechanism A process for integrating the source code changes
41. Continuous Integration To implement it, we need: • Source Control - Git, Hg, Svn, TFS, ... • Build System ◦ Scripting - Cake, Psake, FAKE, Gulp … ◦ Compiler and tools - MsBuild, Node/npm, GitVersion, … ◦ Build Server - VSTS, TeamCity, AppVeyor, CircleCi • Testing Framework - nUnit, xUnit, Fixie, tSQLt, ... • Packaging mechanism - Nuget, Docker container, zip file…
42. Continuous Delivery Chapter 1
43. Continuous Delivery Deployment Pipeline Automated implementation of the application’s build, deploy, test, and release processes
44. Chapter 2 Application, the logical world
45. Here’s what we are going to do: • Create a project in VSTS • Create a repository for a Microservice, with in memory data • Implement the microservice using TDD • Create a basic build job for our microservice in VSTS • Se this whole setup run
46. Chapter 3 Infrastructure (as code)
47. Physical machines High hardware cost High human cost Very low flexibility
48. Virtual Machines In the cloud Upfront cost First step to IaC Requires configuration management
49. Platform as a Service (PaaS) Very flexible approach Costs grow with the application Fast and simple development Bring infrastructure/operations closer to the development
50. Serverless (FaaS) Azure Functions AWS Lambda Pay what you use
51. Infrastructure as Code (IaC) Thank you Virtualization!! Managing and provisioning infrastructure Is not simply writing scripts Allows to apply proven software development practices Version control, testing, small deployments
52. Chapter 4 Database development
53. What’s so special about the database?
54. What’s so special about the database?
55. Database Lifecycle Management (DLM) Data management and data migration Data monitoring Data recovery
56. Database challenges Databases are out of pace with application development Lack of traceability of database changes (changes history) Manual databases processes prevent the CI and CD utilization Lack of testability Database setup time for new environments Bugs in production
57. Database challenges Databases becoming a bottleneck in an agile delivery process
59. Why database automation? Enable control over database development Increase speed of response to change Keep a versioned “history” of database states and changes Reliability of the release process Increase release frequency though repeatability of processes Reduce time spent fixing bugs - automated tests Remove/reduce human intervention in the release process
60. Why database automation? Fearless database changes
61. 1st Step: Source Control First step in your database deployment pipeline Traceability through change history SQL as documentation Shared code-base and shared process Enforceable standards to reduce conflicts
62. 1st Step: Source Control Fundamental resource: SQL Script
63. Migrations vs State State based solutions Script represents the current database state Your source of truth is how the database should be Migrations based solutions Script represents a migration Migration represents how to transition to the next database version Your source of truth is how the database should change
64. Flyway Flyway – “Database Migrations made Easy” http://flywaydb.org/ Open source database migration tool Flyway command-line enabled Simplicity: easy to setup, no need to install Zero dependencies (java + jdbc) Scripts are written in SQL
65. Flyway: How it works Flyway commands: migrate, clean, info, validate, baseline, repair Metadata table: Used to track the state of the database If the database is empty, flyway won’t find it and will create it Flyway scans the migrations directory and check migrations against the metadata table Migrations are sorted based on their version number and applied in order
66. Version Control System Two components Database scripts repo (git) Flyway - database migrations Rule 1: Script version (timestamp) Rule 2: Operation type Rule 3: Object type Rule 4: Object name Example: V20160728.1100__Create_TB_MyTable.sql
67. Time to add a database to our service • Provision development Database • Add test project in Code • Implement database features in our microservice • Test • Add steps in our build system to support this • Deploy changes
68. Chapter 5 Managing changes...
69. Change is unavoidable Businesses change naturally • competitive advantage requires enhancements • products need to move towards customer’s real needs • social and economic context affects how business is done Software NEEDS to change to enable this!
70. Change is not easy in software Ideally, changes contained in one component should NOT break all the others! Unfortunately, this is not always possible • more than one app consumes a service • more than one app consumes a database • binary data contracts (dlls) need to be updated everywhere • environment variables and changes affect program’s expected behaviour • lack of tests brings uncertainty to change outcome Many of these are anti-patterns!
71. Collaboration is key • Need for change needs to be communicated • Risk of change needs to be analysed • Effects need to be listed to prepare for possible problems and avoid fire- fighting • Teams need to announce how changes will impact existing processes and work. • Teams need to determine if the change is done in a single step or in multiple iterations.
72. Versioning strategies • Versioning allows team to tag artifacts to a specific state • Collaborating components can assert their validity based on versions ◦ component A requires B version X or higher • Semantic versioning gives meaning to number ◦ prefer SemVer • Automate versioning as much as possible ◦ tooling and commit messages can be enough to indicate the type of change • Important part of a CI pipeline - end result of a build is a specific version of an
73. SemVer 1.3.5-beta0003 major minor patch prerelease increment on a breaking change increment on a feature addition increment on a bug fix increment while unreleased
74. GitFlow
75. 0.1.1-beta0001 0.1.1 0.2.0
76. Versioning is not easy • But it is extremely important • Semantics matter ◦ minors and patches mean clients shouldn’t have to change anything, only update ◦ Majors are breaking so clients need to take some maintenance and realignment action • It’s also automatic documentation
77. Deployment strategies You choose based on what your context is (traffic volume, working hours, SLAs) • Single Server or service ◦ Stop service, upgrade, restart • Multiple Servers ◦ Stop service, upgrade, restart ◦ Blue-Green deployment ◦ Canary Deployment Whatever the scenario, automation is key
78. Deployments with DBs make things harder • Database’s data is mutable. It’s super hard to rollback! ◦ How do you rollback changes that delete a column of data? • Application is typically immutable. It’s way easier to rollback ◦ Just need the database schema to be compatible This means that we typically need to make changes in multiple steps to avoid breaking the service. Schema should valid for backward compatible to at least 1 version of the application. Marking things as deprecated can / should make a difference.
79. Chapter 6 DevOps as engineering practice...
80. What we just saw
81. Why DevOps Developing software is not enough, you have to deliver it Communication framework for change management You can not stop change, but you can control it Perspectives Need for speed (time-to- market) (management people) Need for control (error control) (operations people)
82. DevOps: contract for collaboration and communication Contract - change communication management tool Change description (Source Control) Change validation (Continuous Integration) Change implementation (Continuous Delivery) Applications, databases, infrastructure
83. DevOps advices Operations requirements in Backlog Operations engineers in Product Team Shared goals Shared responsibilities Devs monitor apps Devs deploy apps
84. Reading material
86. Links
87. Demo application “Purchases-DbMigration-sample” https://github.com/MiguelAlho/Purchases-DbMigration-sample Flyway https://flywaydb.org/ DbUp http://dbup.github.io/ Octopus Deploy https://octopus.com/ http://docs.octopusdeploy.com/display/OD/Deploying+ASP.NET+Core+Web+Applications Visual Studio https://www.visualstudio.com/
88. Final Q&A