Change management at scale

Devops Days Paris Talk 2015

Transcript

1. CHANGE MANAGEMENT AT SCALE PIERRE-YVES RITSCHARD @PYR 0
2. @PYR CTO at Exoscale: swiss cloud hosting. Background in both big shops & startups Open source developer
3. WHAT'S CHANGE ? The addition, modification or removal of anything that could have an effect on IT services
4. AIM OF THE TALK Reflect on how to approach change in production.
5. OUTLINE Process in the Entreprise ITIL: the good parts Agile operations Tooling
6. PROCESS IN THE ENTREPRISE
7. WHAT IT LOOKS LIKE
8. CANONICAL USE-CASE: ADDING OUTBOUND MAIL Please open outbound TCP 25 Sounds like you want TCP 587 to our internal mailers < w a i t 2 w e e k s >
9. MEANWHILE l a p t o p > s s h r o o t @ a p p 0 1 a p p 0 1 > i p t a b l e s - A O U T P U T - p t c p - - d p o r t 2 5 - j A C C E P T
10. PROMISES Traceability Reversibility CYAP
11. ABSTRACTING THE PROCESS
12. WHAT WE'RE INTERESTED IN Traceability Reversibility
13. STANDARDS ISO 27001: (mostly crazyness) CSA: A good basis for IaaS, PaaS and SaaS vendors ITIL: Best practices
14. ITIL: THE GOOD PARTS
15. ITIL Information Technology Infrastructure Library Best practices for dealing with (large) IT orgs. Common lingo No tools
16. ITIL
17. ITIL Service design Service transition Service operation
18. SERVICE DESIGN Service catalog Capacity Management
19. SERVICE OPERATION Problem management Incident management
20. SERVICE TRANSITION Configuration Management (CM) Change management
21. CMDB Configuration Management Database Holds configuration items and their relationship Somewhat conflates Asset Management and Configuration Management
22. CHANGE MANAGEMENT Defines change lifecycle RFCs and classification (standard, emergency, normal) Change Acceptance Board (CAB, ECAB) Change records Technical and organizational challenges
23. AGILE OPERATIONS
24. SCALING PROCESS New objectives Fast iteration cycle Reduced interference Not just startups & small orgs How do we map valid ITIL concerns with agile orgs ?
25. CMDB Standard configuration management frameworks apply Cattle vs Pet approach has the underlying notion of service catalog
26. CHANGE MANAGEMENT RECAP RFCs Change acceptance board Change records
27. CHANGE MANAGEMENT: RFCS Hardest step Automation prepared us Useful elements Motivation & Objective Tentative timeframe Functional tests May be polymorphic Text document Configuration management update Command and control recipe
28. CHANGE MANAGEMENT: CAB Peer review of runbooks Breaks long release cycle Reaches CAB objectives of traceability Recurring tasks can be auto-validated (standard changes) For instance adding a vhost A good prerequisite: no manual intervention
29. CHANGE MANAGEMENT: CHANGE RECORDS Text based log of change Accessible by all parties at time of change
30. TOOLING
31. CMDB IS EASY You already have configuration management It's already stored in git # n o d e s . y a m l n o d e " n e t w o r k - l b 0 1 " { i n c l u d e n e t w o r k : : l b } n o d e " p o r t a l - f r o n t 0 1 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - f r o n t 0 2 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - f r o n t 0 3 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - d b 0 1 " { i n c l u d e p o r t a l : : d b }
32. CHANGE MANAGEMENT IS LESS ADRESSED No common runbook tool No standard execution logging
33. COMMON THEME: REUSE YOUR EXISTING INFRASTRUCTURE Pull requests are great for peer review Git provides a nice way to archive text data Introduce as few new tools as possible Avoid adding bloat with process
34. EXAMPLE APPROACH: RFCs Standard Changes: Config Mgmt or Command-and-Control recipe Normal Changes: Doc update for the corresponding service platform CAB: Pull requests Change records: Archived chat logs Hubot is great for this
35. LEVERAGE YOUR CHAT ROOM
36. ADDITIONAL TOOLING: WARP Command and Control. Web + Chat integration. https://github.com/pyr/warp
37. ADDITIONAL TOOLING: WARP # # R u n b o o k : u p d a t e t o l a t e s t w e b s i t e # # = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = # # # # 1 . M a k e s s u r e w e ' r e r u n n i n g a g a i n s t l a t e s t p a c k a g e s # # 2 . U p d a t e s a p p r o p r i a t e p a c k a g e # # 3 . R e l o a d s n g i n x s c r i p t _ n a m e : d e p l o y - w e b s i t e m a t c h : a n d : - { f a c t : e n v i r o n m e n t , v a l u e : s t a g i n g } - { f a c t : p l a t f o r m , v a l u e : w e b i s t e } # # D e f a u l t s t o s t a g i n g , T h i s p r o v i d e s a p r o d u c t i o n p r o f i l e : p r o f i l e s : p r o d u c t i o n : m a t c h : a n d : - { f a c t : e n v i r o n m e n t , v a l u e : p r o d u c t i o n } - { f a c t : p l a t f o r m , v a l u e : w e b s i t e } t i m e o u t : 3 0 0 0 0 # T h i s w i l l r u n w i t h i n 3 0 s
38. ADDITIONAL TOOLING: WARP
39. ADDITIONAL TOOLING: WARP
40. ADDITIONAL TOOLING: WARP
41. PARTING WORDS
42. IT'S ABOUT VISIBILITY Finding out what happened 37 days ago is a g i t l o g away
43. AVOID DOGMATISM Don't fall in the SCRUM trap
44. CAREFULLY WEIGHT FRICTION TO BENEFIT RATIO Don't lose sight of the objectives
45. GO STEP BY STEP It's ok to still log-in if you need it Consider it's a failure and see how you can remediate
46. OPEN SPACE WISHLIST Process tooling: Let's figure a cool name for this Discussion around runbooks, what do you do ? We can extend the discussion to other concerns Building the service catalog Capacity management Problem & Incident management
47. THANK YOU @pyr