Death to Manual Deployments

Manual deployments are terrible because they they ask people to be as repeatable as machines. This presentation looks at why manual deployments fail, what automation brings to the table, and how to build on top of automated deployments.

Transcript

1. Death to Manual Deployments Implementing repeatable, reliable releases Eric Minick DevOps Evangelist Aug, 2013 @EricMinick
2. Who’s that guy? Eric Minick DevOps Evangelist eminick@us.ibm.com § Background as a developer, tester and tools guy § Helping clients implement build, deploy & release tooling for the past 10 years
3. Confession: I hate manual deployments § They’re slow § They aren’t consistent across environments § You fail in production § You don’t know who deployed what, where or when
4. Why do people like manual deployments? § I feel more in control from the command line § It’s how I’ve always done it § I know what I did and see the output of the commands I ran § When something goes wrong, I can be the hero and fix it ?
5. Scripts to the rescue!? § I don’t have a manual deployment: I have scripts! – How many scripts? – What order? – Who has permissions to run them? – Are they the same across environments? – What else do you need to do? § Scripts are only part of a fully automated deployment.
6. I love automated deployments § They’re NOT slow § They ARE consistent across environments § You DON’T fail in production § You DO know who deployed what, where or when
7. Automated deployments provide benefits § Provide a platform you can build on – continuous deploys, smoke tests, other goodies § Empower people while increasing control – stop waiting on other people – stop being harassed by other people § Better, Faster, Cheaper (Pick any 3)
8. The Program § Today’s Manual Deployments § Elements of an Automated Solution § Best Practices with Automated Deployments
9. The Program § Today’s Manual Deployments § Elements of an Automated Solution § Best Practices with Automated Deployments
10. … you might have a manual deployment § Hand-offs are through email (or file shares) § Developers, QA, and Ops deploy differently § Tweaks communicated on an as-broken basis § Deployments steps are a huge document § The term “deployment weekend” is used § Developers have production passwords
11. Hand-offs are through email (or file shares) § They’re slow § They aren’t consistent across environments § You fail in production § You don’t know who deployed what, where or when ?
12. Developers, QA, and Ops deploy differently § They’re slow § They aren’t consistent across environments § You fail in production § You don’t know who deployed what, where or when
13. Tweaks communicated as-broken § They’re slow § They aren’t consistent across environments § You fail in production § You don’t know who deployed what, where or when The main page won’t load? Oh… Right, you needed to add another data source.
14. Deployments steps are a huge document § They’re slow § They aren’t consistent across environments § You fail in production § You don’t know who deployed what, where or when
15. The term “deployment weekend” is used § They’re slow § They aren’t consistent across environments § You fail in production § You don’t know who deployed what, where or when That’s ok, I don’t really like my kids anyway
16. Developers have production passwords § They’re slow § They aren’t consistent across environments § You fail in production § You don’t know who deployed what, where or when 㽈
17. If only people were more disciplined… § Deployment docs – could capture everything – always be updated – could be followed them, perfectly, every time. § Audit spreadsheets – could capture everything – could update them Image from: http://www.ultimatebasictraining.com/blog/2010/02/what-area-drill-sergeants-like/
18. People aren’t machines § People: good at using creative judgment § Machines: good at consistency Automated deployments leverage the strengths of people and machines. Alistar Cockburn, “Characterizing People as Non-Linear, First-Order Components in Software Development”
19. DevOps isn’t natural Conway’s Law: organizations are constrained to produce solutions that copy the structure of their organization
20. DevOps isn’t natural Conway’s Law: organizations are constrained to produce solutions that copy the structure of their organization § Ops wants to maintain control – doesn’t care about DEV, can’t keep up with the frequent changes § Development doesn’t want responsibility for production deployments § QA lacks influence in DEV & PROD
21. The Program § Today’s Manual Deployments § Elements of an Automated Solution § Best Practices with Automated Deployments
22. Elements of an automated solution § Automated process § Coordinating tiers § Environment definitions § Artifact repository § Access control § Audit trail
23. Automated process Status Quo § Don’t exist § Incomplete § Multiple order-dependent scripts § Multiple versions per environment (or machine) § Uncontrolled What we need § Single script reused across environments § Controlled access to deployment steps § Auto record what happened Fast, consistent and repeatable
24. Coordinating Tiers / Components Status Quo § Based on oral history § In a Release Manager’s spreadsheet § Total plan based on documented changes for each release – Redesign every time What we need § A model for whole systems § Standard deployment based on changes to all pieces and parts § Actual deployment runs the needed subset. Move all tiers, services, config etc together
25. Environment definitions Status Quo § Configured manually § Details embedded in binaries, scripts, property files, ini files § Environment information either uncontrolled or in source control What we need § Environments a first-level entity § Properties controlled at the environment level § Secure storage of sensitive values (passwords) Per-environment configuration and passwords
26. Artifact repository Status Quo § Is email a repository § File share on the network § Source control for binary control § Uncontrolled § Crude control over what to keep What we need § Definitive Software Library § Consistent set of artifacts across environments § Tamper-proof artifacts § Automated retention policies based on deployments Know what you’re getting
27. Access control Status Quo § Controlled by who can log onto the machine § Access tends to grow over time in response to crisis § Access control distributed across machines What we need § Role-based access control § Access control by environment § Single source of access control § LDAP / Active Directory authentication (or authorization) Who can deploy to which environment
28. Audit trail Status Quo § What audit trail? § I have the spreadsheet § Our Change Management System has what we think we did § Audit trail starts at PROD What we need § Win at Clue: – Who – What – Where – When / How (better than Clue!) § Require no additional work § End-to-end traceability Know what happened
29. Solution Benefits § We have confidence that our production deployments will succeed § QA can take new builds faster § DEV deployments fast enough to keep up with our rate of change § We have an audit trail from day one § We have separation of duties
30. The Program § Today’s Manual Deployments § Elements of an Automated Solution § Best Practices with Automated Deployments
31. Automated Deployments Best Practices § Deferred deployments § Self-service deployments § Automatic deployment § Automatic smoke-test § Automated rollbacks § Continuous production deployment
32. Deferred deployments § Story: I want to do my deployment at 11 pm on Friday after my users are offline. § Solution: Schedule deployment for desired time, go home. Wait for notification of success or failure.
33. Self-service deployments § Story: QA wants to be able to take a new build when they need one. § Solution: Use per-environment access control to allow the QA people to deploy on-demand, but only to the test environment.
34. Automatic deployment § Story: Want to keep the development test server in sync with the latest changes in source control. § Solution: After building, automatically deploy to the development server.
35. Automatic smoke-test § Story: I want to know if something breaks immediately. § Solution: After deploying, automatically trigger a suite of automated tests and report the results.
36. Automated rollback § Story: If my deployment fails I want to restore the previous build. § Solution: When a deployment step fails, trigger the restoration of the last successfully deployed build.
37. Continuous production deployments § Story: When a change is committed it should be deployed to production if it doesn’t break anything. § Solution: After deployment to test environment run automated test suites. If they all pass, trigger a deployment to production, then test that production deploy was successful. Rollback on failure.
38. Summary § Manual deployments are slow and broken because they require people to act like machines § Computers make good machines § Automated deployments provide a solid foundation for further improvement
39. References http://urbancode.com/resources § Deployment Automation Basics § Lean Build & Deployment Automation § ITIL Release Management and Automation Blogs.urbancode.com Twitter.com/UrbanCode
40. Deployment and Release Products § uDeploy – Application Release Automation § uRelease – Release management planning and release weekend execution
41. 41 § Slideshare.net/Urbancode § Urbancode.com/resources/ § Blogs.urbancode.com/ § eMinick@us.ibm.com §  @EricMinick