Change Management at Scale

Reconciling process and agility

1. CHANGE MANAGEMENT AT CHANGE MANAGEMENT AT SCALE SCALE STAYING ORGANIZED IN THE FACE OF AGILITY STAYING ORGANIZED IN THE FACE OF AGILITY 1
2. CTO at Exoscale: The European Cloud Provider Early devops implementer: Orange, Lagardere Early big-data & ML implementer: paper.li @pyr 2 . 1
3. EXOSCALE EXOSCALE 3 . 1
4. EXOSCALE EXOSCALE provider "exoscale" { api_key = "${var.exoscale_api_key}" secret_key = "${var.exoscale_secret_key}" } resource "exoscale_instance" "web" { template = "Ubuntu 17.04" disk_size = "50g" profile = "medium" ssh_key = "production" } 4 . 1
5. WHAT'S CHANGE ? WHAT'S CHANGE ? The addition, modification or removal of anything that could have an effect on IT services 5 . 1
6. WHY CHANGE MATTERS WHY CHANGE MATTERS IT used to be a support function Inventory Payroll Accounting 6 . 1
7. WHY CHANGE MATTERS WHY CHANGE MATTERS 7 . 1
8. WHAT'S IN AN APP WHAT'S IN AN APP Inventory User preferences Real-time analytics 8 . 1
9. AIM OF THE TALK AIM OF THE TALK Addressing the tension between process, security, and agility 9 . 1
10. OUTLINE OUTLINE Process in the Entreprise ITIL: the good parts Agile operations Tooling 10 . 1
11. PROCESS IN THE ENTREPRISE PROCESS IN THE ENTREPRISE 11 . 1
12. WHAT IT LOOKS LIKE WHAT IT LOOKS LIKE 12 . 1
13. CANONICAL USE-CASE: ADDING CANONICAL USE-CASE: ADDING OUTBOUND MAIL OUTBOUND MAIL Please open outbound TCP 25 Sounds like you want TCP 587 to our internal mailers <wait 2 weeks> 13 . 1
14. MEANWHILE MEANWHILE laptop> ssh root@app01 app01> iptables -A OUTPUT -p tcp --dport 25 -j ACCEPT 14 . 1
15. ABSTRACTING THE PROCESS ABSTRACTING THE PROCESS 15 . 1
16. WHAT WE'RE INTERESTED IN WHAT WE'RE INTERESTED IN Traceability Reversibility Inventory 16 . 1
17. STANDARDS STANDARDS ISO 27001 and ISO27018 CSA: A good basis for IaaS, PaaS and SaaS vendors ITIL: Best practices 17 . 1
18. ITIL: THE GOOD PARTS ITIL: THE GOOD PARTS 18 . 1
19. ITIL ITIL Information Technology Infrastructure Library Best practices for dealing with (large) IT orgs. Common lingo No tools 19 . 1
20. ITIL ITIL 20 . 1
21. ITIL ITIL Service design Service transition Service operation 21 . 1
22. SERVICE DESIGN SERVICE DESIGN Service catalog Capacity Management 22 . 1
23. SERVICE OPERATION SERVICE OPERATION Problem management Incident management 23 . 1
24. SERVICE TRANSITION SERVICE TRANSITION Configuration Management Change management 24 . 1
25. CMDB CMDB Configuration Management Database Holds configuration items and their relationship Somewhat conflates Asset Management and Configuration Management 25 . 1
26. CHANGE MANAGEMENT CHANGE MANAGEMENT Defines change lifecycle RFCs and classification (standard, emergency, normal) Change Acceptance Board (CAB, ECAB) Change records 26 . 1
27. AGILE OPERATIONS AGILE OPERATIONS 27 . 1
28. SCALING PROCESS SCALING PROCESS New objectives Fast iteration cycle Reduced interference Not just startups & small orgs How do we map valid ITIL concerns with agile orgs ? 28 . 1
29. SERVICE DESIGN SERVICE DESIGN The service catalog should be self-service This is much easier when co-opting IaaS 29 . 1
30. SERVICE OPERATION SERVICE OPERATION Problem and incident management are most likely already well addressed Work needed on reducing communication latency 30 . 1
31. SERVICE TRANSITION SERVICE TRANSITION 31 . 1
32. CMDB CMDB Configuration Management Standard configuration management frameworks apply Cattle vs Pet approach has the underlying notion of service catalog Asset Management Virtualization and Cloud provide dynamic inventory 32 . 1
33. CHANGE MANAGEMENT RECAP CHANGE MANAGEMENT RECAP RFCs Change acceptance board Change records 33 . 1
34. CHANGE MANAGEMENT: RFCS CHANGE MANAGEMENT: RFCS Useful elements Motivation & Objective Tentative timeframe Integration & Non-regression tests May be polymorphic Text document Configuration management update Command and control recipe 34 . 1
35. CHANGE MANAGEMENT: CAB CHANGE MANAGEMENT: CAB Peer review of runbooks Breaks long release cycle Reaches CAB objectives of traceability Recurring tasks can be auto-validated (standard changes) For instance adding a vhost A good prerequisite: no manual intervention 35 . 1
36. CHANGE MANAGEMENT: CHANGE CHANGE MANAGEMENT: CHANGE RECORDS RECORDS Text based log of change Accessible by all parties at time of change 36 . 1
37. TOOLING TOOLING 37 . 1
38. SERVICE DESIGN SERVICE DESIGN Cloud hosting platforms Kubernetes Internal PaaS systems (OpenShi ) 38 . 1
39. SERVICE OPERATION SERVICE OPERATION Co-opt development tools to reduce barriers and friction Status pages for critical services 39 . 1
40. SERVICE TRANSITION SERVICE TRANSITION 40 . 1
41. CMDB IS EASY CMDB IS EASY You already have configuration management It's already stored in git # nodes.yaml node "network-lb01" { include network::lb } node "portal-front01" { include portal::front } node "portal-front02" { include portal::front } node "portal-front03" { include portal::front } node "portal-db01" { include portal::db } 41 . 1
42. CHANGE MANAGEMENT IS LESS CHANGE MANAGEMENT IS LESS ADRESSED ADRESSED No common runbook tool No standard execution logging 42 . 1
43. COMMON THEME: REUSE YOUR EXISTING COMMON THEME: REUSE YOUR EXISTING INFRASTRUCTURE INFRASTRUCTURE Pull requests are great for peer review Git provides a nice way to archive text data Introduce as few new tools as possible Avoid adding bloat with process 43 . 1
44. EXAMPLE APPROACH: EXAMPLE APPROACH: RFCs Standard Changes: Config Mgmt or Command-and-Control recipe Normal Changes: Doc update for the corresponding service platform CAB: Pull requests Change records: Archived chat logs Hubot is great for this 44 . 1
45. LEVERAGE YOUR CHAT ROOM LEVERAGE YOUR CHAT ROOM 45 . 1
46. PARTING WORDS PARTING WORDS 46 . 1
47. IT'S ABOUT VISIBILITY IT'S ABOUT VISIBILITY Finding out what happened 37 days ago is a git log away 47 . 1
48. AVOID DOGMATISM AVOID DOGMATISM Don't fall in the SCRUM trap 48 . 1
49. CAREFULLY WEIGHT FRICTION TO CAREFULLY WEIGHT FRICTION TO BENEFIT RATIO BENEFIT RATIO Don't lose sight of the objectives 49 . 1
50. GO STEP BY STEP GO STEP BY STEP It's ok to still log-in if you need it Consider it's a failure and see how you can remediate 50 . 1
51. QUESTIONS? QUESTIONS? 51 . 1
52. PYR PYR 52 . 1