Change management at scale

Devops Days Paris Talk 2015

Transcript

1. CHANGE MANAGEMENT AT SCALE
   PIERRE-YVES RITSCHARD
   @PYR
   0
   

   View Slide

2. @PYR
   CTO at Exoscale: swiss cloud hosting.
   Background in both big shops & startups
   Open source developer
   

   View Slide

3. WHAT'S CHANGE ?
   The addition, modification or removal of
   anything that could have an effect on IT services
   

   View Slide

4. AIM OF THE TALK
   Reflect on how to approach change in production.
   

   View Slide

5. OUTLINE
   Process in the Entreprise
   ITIL: the good parts
   Agile operations
   Tooling
   

   View Slide

6. PROCESS IN THE ENTREPRISE
   

   View Slide

7. WHAT IT LOOKS LIKE
   

   View Slide

8. CANONICAL USE-CASE: ADDING OUTBOUND MAIL
   Please open outbound TCP 25
   Sounds like you want TCP 587 to our internal mailers
   <
   w
   a
   i
   t 2 w
   e
   e
   k
   s
   >
   

   View Slide

9. MEANWHILE
   l
   a
   p
   t
   o
   p
   > s
   s
   h r
   o
   o
   t
   @
   a
   p
   p
   0
   1
   a
   p
   p
   0
   1
   > i
   p
   t
   a
   b
   l
   e
   s -
   A O
   U
   T
   P
   U
   T -
   p t
   c
   p -
   -
   d
   p
   o
   r
   t 2
   5 -
   j A
   C
   C
   E
   P
   T
   

   View Slide

10. PROMISES
    Traceability
    Reversibility
    CYAP
    

    View Slide

11. ABSTRACTING THE PROCESS
    

    View Slide

12. WHAT WE'RE INTERESTED IN
    Traceability
    Reversibility
    

    View Slide

13. STANDARDS
    ISO 27001: (mostly crazyness)
    CSA: A good basis for IaaS, PaaS and SaaS vendors
    ITIL: Best practices
    

    View Slide

14. ITIL: THE GOOD PARTS
    

    View Slide

15. ITIL
    Information Technology Infrastructure Library
    Best practices for dealing with (large) IT orgs.
    Common lingo
    No tools
    

    View Slide

16. ITIL
    

    View Slide

17. ITIL
    Service design
    Service transition
    Service operation
    

    View Slide

18. SERVICE DESIGN
    Service catalog
    Capacity Management
    

    View Slide

19. SERVICE OPERATION
    Problem management
    Incident management
    

    View Slide

20. SERVICE TRANSITION
    Configuration Management (CM)
    Change management
    

    View Slide

21. CMDB
    Configuration Management Database
    Holds configuration items and their relationship
    Somewhat conflates Asset Management and Configuration
    Management
    

    View Slide

22. CHANGE MANAGEMENT
    Defines change lifecycle
    RFCs and classification (standard, emergency, normal)
    Change Acceptance Board (CAB, ECAB)
    Change records
    Technical and organizational challenges
    

    View Slide

23. AGILE OPERATIONS
    

    View Slide

24. SCALING PROCESS
    New objectives
    Fast iteration cycle
    Reduced interference
    Not just startups & small orgs
    How do we map valid ITIL concerns with agile orgs ?
    

    View Slide

25. CMDB
    Standard configuration management frameworks apply
    Cattle vs Pet approach has the underlying notion of service
    catalog
    

    View Slide

26. CHANGE MANAGEMENT RECAP
    RFCs
    Change acceptance board
    Change records
    

    View Slide

27. CHANGE MANAGEMENT: RFCS
    Hardest step
    Automation prepared us
    Useful elements
    Motivation & Objective
    Tentative timeframe
    Functional tests
    May be polymorphic
    Text document
    Configuration management update
    Command and control recipe
    

    View Slide

28. CHANGE MANAGEMENT: CAB
    Peer review of runbooks
    Breaks long release cycle
    Reaches CAB objectives of traceability
    Recurring tasks can be auto-validated (standard changes)
    For instance adding a vhost
    A good prerequisite: no manual intervention
    

    View Slide

29. CHANGE MANAGEMENT: CHANGE RECORDS
    Text based log of change
    Accessible by all parties at time of change
    

    View Slide

30. TOOLING
    

    View Slide

31. CMDB IS EASY
    You already have configuration management
    It's already stored in git
    # n
    o
    d
    e
    s
    .
    y
    a
    m
    l
    n
    o
    d
    e "
    n
    e
    t
    w
    o
    r
    k
    -
    l
    b
    0
    1
    " { i
    n
    c
    l
    u
    d
    e n
    e
    t
    w
    o
    r
    k
    :
    :
    l
    b }
    n
    o
    d
    e "
    p
    o
    r
    t
    a
    l
    -
    f
    r
    o
    n
    t
    0
    1
    " { i
    n
    c
    l
    u
    d
    e p
    o
    r
    t
    a
    l
    :
    :
    f
    r
    o
    n
    t }
    n
    o
    d
    e "
    p
    o
    r
    t
    a
    l
    -
    f
    r
    o
    n
    t
    0
    2
    " { i
    n
    c
    l
    u
    d
    e p
    o
    r
    t
    a
    l
    :
    :
    f
    r
    o
    n
    t }
    n
    o
    d
    e "
    p
    o
    r
    t
    a
    l
    -
    f
    r
    o
    n
    t
    0
    3
    " { i
    n
    c
    l
    u
    d
    e p
    o
    r
    t
    a
    l
    :
    :
    f
    r
    o
    n
    t }
    n
    o
    d
    e "
    p
    o
    r
    t
    a
    l
    -
    d
    b
    0
    1
    " { i
    n
    c
    l
    u
    d
    e p
    o
    r
    t
    a
    l
    :
    :
    d
    b }
    

    View Slide

32. CHANGE MANAGEMENT IS LESS ADRESSED
    No common runbook tool
    No standard execution logging
    

    View Slide

33. COMMON THEME: REUSE YOUR EXISTING INFRASTRUCTURE
    Pull requests are great for peer review
    Git provides a nice way to archive text data
    Introduce as few new tools as possible
    Avoid adding bloat with process
    

    View Slide

34. EXAMPLE APPROACH:
    RFCs
    Standard Changes: Config Mgmt or Command-and-Control
    recipe
    Normal Changes: Doc update for the corresponding service
    platform
    CAB: Pull requests
    Change records: Archived chat logs
    Hubot is great for this
    

    View Slide

35. LEVERAGE YOUR CHAT ROOM
    

    View Slide

36. ADDITIONAL TOOLING: WARP
    Command and Control.
    Web + Chat integration.
    https://github.com/pyr/warp
    

    View Slide

37. ADDITIONAL TOOLING: WARP
    #
    # R
    u
    n
    b
    o
    o
    k
    : u
    p
    d
    a
    t
    e t
    o l
    a
    t
    e
    s
    t w
    e
    b
    s
    i
    t
    e
    #
    # =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    #
    #
    #
    # 1
    . M
    a
    k
    e
    s s
    u
    r
    e w
    e
    '
    r
    e r
    u
    n
    n
    i
    n
    g a
    g
    a
    i
    n
    s
    t l
    a
    t
    e
    s
    t p
    a
    c
    k
    a
    g
    e
    s
    #
    # 2
    . U
    p
    d
    a
    t
    e
    s a
    p
    p
    r
    o
    p
    r
    i
    a
    t
    e p
    a
    c
    k
    a
    g
    e
    #
    # 3
    . R
    e
    l
    o
    a
    d
    s n
    g
    i
    n
    x
    s
    c
    r
    i
    p
    t
    _
    n
    a
    m
    e
    : d
    e
    p
    l
    o
    y
    -
    w
    e
    b
    s
    i
    t
    e
    m
    a
    t
    c
    h
    :
    a
    n
    d
    :
    - {
    f
    a
    c
    t
    : e
    n
    v
    i
    r
    o
    n
    m
    e
    n
    t
    , v
    a
    l
    u
    e
    : s
    t
    a
    g
    i
    n
    g
    }
    - {
    f
    a
    c
    t
    : p
    l
    a
    t
    f
    o
    r
    m
    , v
    a
    l
    u
    e
    : w
    e
    b
    i
    s
    t
    e
    }
    #
    # D
    e
    f
    a
    u
    l
    t
    s t
    o s
    t
    a
    g
    i
    n
    g
    , T
    h
    i
    s p
    r
    o
    v
    i
    d
    e
    s a p
    r
    o
    d
    u
    c
    t
    i
    o
    n p
    r
    o
    f
    i
    l
    e
    :
    p
    r
    o
    f
    i
    l
    e
    s
    :
    p
    r
    o
    d
    u
    c
    t
    i
    o
    n
    :
    m
    a
    t
    c
    h
    :
    a
    n
    d
    :
    - {
    f
    a
    c
    t
    : e
    n
    v
    i
    r
    o
    n
    m
    e
    n
    t
    , v
    a
    l
    u
    e
    : p
    r
    o
    d
    u
    c
    t
    i
    o
    n
    }
    - {
    f
    a
    c
    t
    : p
    l
    a
    t
    f
    o
    r
    m
    , v
    a
    l
    u
    e
    : w
    e
    b
    s
    i
    t
    e
    }
    t
    i
    m
    e
    o
    u
    t
    : 3
    0
    0
    0
    0 # T
    h
    i
    s w
    i
    l
    l r
    u
    n w
    i
    t
    h
    i
    n 3
    0
    s
    

    View Slide

38. ADDITIONAL TOOLING: WARP
    

    View Slide

39. ADDITIONAL TOOLING: WARP
    

    View Slide

40. ADDITIONAL TOOLING: WARP
    

    View Slide

41. PARTING WORDS
    

    View Slide

42. IT'S ABOUT VISIBILITY
    Finding out what happened 37 days ago is a g
    i
    t l
    o
    g
    away
    

    View Slide

43. AVOID DOGMATISM
    Don't fall in the SCRUM trap
    

    View Slide

44. CAREFULLY WEIGHT FRICTION TO BENEFIT RATIO
    Don't lose sight of the objectives
    

    View Slide

45. GO STEP BY STEP
    It's ok to still log-in if you need it
    Consider it's a failure and see how you can remediate
    

    View Slide

46. OPEN SPACE WISHLIST
    Process tooling: Let's figure a cool name for this
    Discussion around runbooks, what do you do ?
    We can extend the discussion to other concerns
    Building the service catalog
    Capacity management
    Problem & Incident management
    

    View Slide

47. THANK YOU
    @pyr
    

    View Slide