Change management at scale

Change management at scale

Devops Days Paris Talk 2015

2fcc875f98607b3007909fe4be99160d?s=128

Pierre-Yves Ritschard

April 15, 2015
Tweet

Transcript

  1. CHANGE MANAGEMENT AT SCALE PIERRE-YVES RITSCHARD @PYR 0

  2. @PYR CTO at Exoscale: swiss cloud hosting. Background in both

    big shops & startups Open source developer
  3. WHAT'S CHANGE ? The addition, modification or removal of anything

    that could have an effect on IT services
  4. AIM OF THE TALK Reflect on how to approach change

    in production.
  5. OUTLINE Process in the Entreprise ITIL: the good parts Agile

    operations Tooling
  6. PROCESS IN THE ENTREPRISE

  7. WHAT IT LOOKS LIKE

  8. CANONICAL USE-CASE: ADDING OUTBOUND MAIL Please open outbound TCP 25

    Sounds like you want TCP 587 to our internal mailers < w a i t 2 w e e k s >
  9. MEANWHILE l a p t o p > s s

    h r o o t @ a p p 0 1 a p p 0 1 > i p t a b l e s - A O U T P U T - p t c p - - d p o r t 2 5 - j A C C E P T
  10. PROMISES Traceability Reversibility CYAP

  11. ABSTRACTING THE PROCESS

  12. WHAT WE'RE INTERESTED IN Traceability Reversibility

  13. STANDARDS ISO 27001: (mostly crazyness) CSA: A good basis for

    IaaS, PaaS and SaaS vendors ITIL: Best practices
  14. ITIL: THE GOOD PARTS

  15. ITIL Information Technology Infrastructure Library Best practices for dealing with

    (large) IT orgs. Common lingo No tools
  16. ITIL

  17. ITIL Service design Service transition Service operation

  18. SERVICE DESIGN Service catalog Capacity Management

  19. SERVICE OPERATION Problem management Incident management

  20. SERVICE TRANSITION Configuration Management (CM) Change management

  21. CMDB Configuration Management Database Holds configuration items and their relationship

    Somewhat conflates Asset Management and Configuration Management
  22. CHANGE MANAGEMENT Defines change lifecycle RFCs and classification (standard, emergency,

    normal) Change Acceptance Board (CAB, ECAB) Change records Technical and organizational challenges
  23. AGILE OPERATIONS

  24. SCALING PROCESS New objectives Fast iteration cycle Reduced interference Not

    just startups & small orgs How do we map valid ITIL concerns with agile orgs ?
  25. CMDB Standard configuration management frameworks apply Cattle vs Pet approach

    has the underlying notion of service catalog
  26. CHANGE MANAGEMENT RECAP RFCs Change acceptance board Change records

  27. CHANGE MANAGEMENT: RFCS Hardest step Automation prepared us Useful elements

    Motivation & Objective Tentative timeframe Functional tests May be polymorphic Text document Configuration management update Command and control recipe
  28. CHANGE MANAGEMENT: CAB Peer review of runbooks Breaks long release

    cycle Reaches CAB objectives of traceability Recurring tasks can be auto-validated (standard changes) For instance adding a vhost A good prerequisite: no manual intervention
  29. CHANGE MANAGEMENT: CHANGE RECORDS Text based log of change Accessible

    by all parties at time of change
  30. TOOLING

  31. CMDB IS EASY You already have configuration management It's already

    stored in git # n o d e s . y a m l n o d e " n e t w o r k - l b 0 1 " { i n c l u d e n e t w o r k : : l b } n o d e " p o r t a l - f r o n t 0 1 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - f r o n t 0 2 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - f r o n t 0 3 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - d b 0 1 " { i n c l u d e p o r t a l : : d b }
  32. CHANGE MANAGEMENT IS LESS ADRESSED No common runbook tool No

    standard execution logging
  33. COMMON THEME: REUSE YOUR EXISTING INFRASTRUCTURE Pull requests are great

    for peer review Git provides a nice way to archive text data Introduce as few new tools as possible Avoid adding bloat with process
  34. EXAMPLE APPROACH: RFCs Standard Changes: Config Mgmt or Command-and-Control recipe

    Normal Changes: Doc update for the corresponding service platform CAB: Pull requests Change records: Archived chat logs Hubot is great for this
  35. LEVERAGE YOUR CHAT ROOM

  36. ADDITIONAL TOOLING: WARP Command and Control. Web + Chat integration.

    https://github.com/pyr/warp
  37. ADDITIONAL TOOLING: WARP # # R u n b o

    o k : u p d a t e t o l a t e s t w e b s i t e # # = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = # # # # 1 . M a k e s s u r e w e ' r e r u n n i n g a g a i n s t l a t e s t p a c k a g e s # # 2 . U p d a t e s a p p r o p r i a t e p a c k a g e # # 3 . R e l o a d s n g i n x s c r i p t _ n a m e : d e p l o y - w e b s i t e m a t c h : a n d : - { f a c t : e n v i r o n m e n t , v a l u e : s t a g i n g } - { f a c t : p l a t f o r m , v a l u e : w e b i s t e } # # D e f a u l t s t o s t a g i n g , T h i s p r o v i d e s a p r o d u c t i o n p r o f i l e : p r o f i l e s : p r o d u c t i o n : m a t c h : a n d : - { f a c t : e n v i r o n m e n t , v a l u e : p r o d u c t i o n } - { f a c t : p l a t f o r m , v a l u e : w e b s i t e } t i m e o u t : 3 0 0 0 0 # T h i s w i l l r u n w i t h i n 3 0 s
  38. ADDITIONAL TOOLING: WARP

  39. ADDITIONAL TOOLING: WARP

  40. ADDITIONAL TOOLING: WARP

  41. PARTING WORDS

  42. IT'S ABOUT VISIBILITY Finding out what happened 37 days ago

    is a g i t l o g away
  43. AVOID DOGMATISM Don't fall in the SCRUM trap

  44. CAREFULLY WEIGHT FRICTION TO BENEFIT RATIO Don't lose sight of

    the objectives
  45. GO STEP BY STEP It's ok to still log-in if

    you need it Consider it's a failure and see how you can remediate
  46. OPEN SPACE WISHLIST Process tooling: Let's figure a cool name

    for this Discussion around runbooks, what do you do ? We can extend the discussion to other concerns Building the service catalog Capacity management Problem & Incident management
  47. THANK YOU @pyr