Change management at scale

Change management at scale

Devops Days Paris Talk 2015

2fcc875f98607b3007909fe4be99160d?s=128

Pierre-Yves Ritschard

April 15, 2015
Tweet

Transcript

  1. 2.

    @PYR CTO at Exoscale: swiss cloud hosting. Background in both

    big shops & startups Open source developer
  2. 3.
  3. 8.

    CANONICAL USE-CASE: ADDING OUTBOUND MAIL Please open outbound TCP 25

    Sounds like you want TCP 587 to our internal mailers < w a i t 2 w e e k s >
  4. 9.

    MEANWHILE l a p t o p > s s

    h r o o t @ a p p 0 1 a p p 0 1 > i p t a b l e s - A O U T P U T - p t c p - - d p o r t 2 5 - j A C C E P T
  5. 13.

    STANDARDS ISO 27001: (mostly crazyness) CSA: A good basis for

    IaaS, PaaS and SaaS vendors ITIL: Best practices
  6. 16.
  7. 21.

    CMDB Configuration Management Database Holds configuration items and their relationship

    Somewhat conflates Asset Management and Configuration Management
  8. 22.

    CHANGE MANAGEMENT Defines change lifecycle RFCs and classification (standard, emergency,

    normal) Change Acceptance Board (CAB, ECAB) Change records Technical and organizational challenges
  9. 24.

    SCALING PROCESS New objectives Fast iteration cycle Reduced interference Not

    just startups & small orgs How do we map valid ITIL concerns with agile orgs ?
  10. 27.

    CHANGE MANAGEMENT: RFCS Hardest step Automation prepared us Useful elements

    Motivation & Objective Tentative timeframe Functional tests May be polymorphic Text document Configuration management update Command and control recipe
  11. 28.

    CHANGE MANAGEMENT: CAB Peer review of runbooks Breaks long release

    cycle Reaches CAB objectives of traceability Recurring tasks can be auto-validated (standard changes) For instance adding a vhost A good prerequisite: no manual intervention
  12. 30.
  13. 31.

    CMDB IS EASY You already have configuration management It's already

    stored in git # n o d e s . y a m l n o d e " n e t w o r k - l b 0 1 " { i n c l u d e n e t w o r k : : l b } n o d e " p o r t a l - f r o n t 0 1 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - f r o n t 0 2 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - f r o n t 0 3 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - d b 0 1 " { i n c l u d e p o r t a l : : d b }
  14. 33.

    COMMON THEME: REUSE YOUR EXISTING INFRASTRUCTURE Pull requests are great

    for peer review Git provides a nice way to archive text data Introduce as few new tools as possible Avoid adding bloat with process
  15. 34.

    EXAMPLE APPROACH: RFCs Standard Changes: Config Mgmt or Command-and-Control recipe

    Normal Changes: Doc update for the corresponding service platform CAB: Pull requests Change records: Archived chat logs Hubot is great for this
  16. 37.

    ADDITIONAL TOOLING: WARP # # R u n b o

    o k : u p d a t e t o l a t e s t w e b s i t e # # = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = # # # # 1 . M a k e s s u r e w e ' r e r u n n i n g a g a i n s t l a t e s t p a c k a g e s # # 2 . U p d a t e s a p p r o p r i a t e p a c k a g e # # 3 . R e l o a d s n g i n x s c r i p t _ n a m e : d e p l o y - w e b s i t e m a t c h : a n d : - { f a c t : e n v i r o n m e n t , v a l u e : s t a g i n g } - { f a c t : p l a t f o r m , v a l u e : w e b i s t e } # # D e f a u l t s t o s t a g i n g , T h i s p r o v i d e s a p r o d u c t i o n p r o f i l e : p r o f i l e s : p r o d u c t i o n : m a t c h : a n d : - { f a c t : e n v i r o n m e n t , v a l u e : p r o d u c t i o n } - { f a c t : p l a t f o r m , v a l u e : w e b s i t e } t i m e o u t : 3 0 0 0 0 # T h i s w i l l r u n w i t h i n 3 0 s
  17. 45.

    GO STEP BY STEP It's ok to still log-in if

    you need it Consider it's a failure and see how you can remediate
  18. 46.

    OPEN SPACE WISHLIST Process tooling: Let's figure a cool name

    for this Discussion around runbooks, what do you do ? We can extend the discussion to other concerns Building the service catalog Capacity management Problem & Incident management