Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Change management at scale

Change management at scale

Devops Days Paris Talk 2015

Pierre-Yves Ritschard

April 15, 2015
Tweet

More Decks by Pierre-Yves Ritschard

Other Decks in Technology

Transcript

  1. CHANGE MANAGEMENT AT SCALE
    PIERRE-YVES RITSCHARD
    @PYR
    0

    View Slide

  2. @PYR
    CTO at Exoscale: swiss cloud hosting.
    Background in both big shops & startups
    Open source developer

    View Slide

  3. WHAT'S CHANGE ?
    The addition, modification or removal of
    anything that could have an effect on IT services

    View Slide

  4. AIM OF THE TALK
    Reflect on how to approach change in production.

    View Slide

  5. OUTLINE
    Process in the Entreprise
    ITIL: the good parts
    Agile operations
    Tooling

    View Slide

  6. PROCESS IN THE ENTREPRISE

    View Slide

  7. WHAT IT LOOKS LIKE

    View Slide

  8. CANONICAL USE-CASE: ADDING OUTBOUND MAIL
    Please open outbound TCP 25
    Sounds like you want TCP 587 to our internal mailers
    <
    w
    a
    i
    t 2 w
    e
    e
    k
    s
    >

    View Slide

  9. MEANWHILE
    l
    a
    p
    t
    o
    p
    > s
    s
    h r
    o
    o
    t
    @
    a
    p
    p
    0
    1
    a
    p
    p
    0
    1
    > i
    p
    t
    a
    b
    l
    e
    s -
    A O
    U
    T
    P
    U
    T -
    p t
    c
    p -
    -
    d
    p
    o
    r
    t 2
    5 -
    j A
    C
    C
    E
    P
    T

    View Slide

  10. PROMISES
    Traceability
    Reversibility
    CYAP

    View Slide

  11. ABSTRACTING THE PROCESS

    View Slide

  12. WHAT WE'RE INTERESTED IN
    Traceability
    Reversibility

    View Slide

  13. STANDARDS
    ISO 27001: (mostly crazyness)
    CSA: A good basis for IaaS, PaaS and SaaS vendors
    ITIL: Best practices

    View Slide

  14. ITIL: THE GOOD PARTS

    View Slide

  15. ITIL
    Information Technology Infrastructure Library
    Best practices for dealing with (large) IT orgs.
    Common lingo
    No tools

    View Slide

  16. ITIL

    View Slide

  17. ITIL
    Service design
    Service transition
    Service operation

    View Slide

  18. SERVICE DESIGN
    Service catalog
    Capacity Management

    View Slide

  19. SERVICE OPERATION
    Problem management
    Incident management

    View Slide

  20. SERVICE TRANSITION
    Configuration Management (CM)
    Change management

    View Slide

  21. CMDB
    Configuration Management Database
    Holds configuration items and their relationship
    Somewhat conflates Asset Management and Configuration
    Management

    View Slide

  22. CHANGE MANAGEMENT
    Defines change lifecycle
    RFCs and classification (standard, emergency, normal)
    Change Acceptance Board (CAB, ECAB)
    Change records
    Technical and organizational challenges

    View Slide

  23. AGILE OPERATIONS

    View Slide

  24. SCALING PROCESS
    New objectives
    Fast iteration cycle
    Reduced interference
    Not just startups & small orgs
    How do we map valid ITIL concerns with agile orgs ?

    View Slide

  25. CMDB
    Standard configuration management frameworks apply
    Cattle vs Pet approach has the underlying notion of service
    catalog

    View Slide

  26. CHANGE MANAGEMENT RECAP
    RFCs
    Change acceptance board
    Change records

    View Slide

  27. CHANGE MANAGEMENT: RFCS
    Hardest step
    Automation prepared us
    Useful elements
    Motivation & Objective
    Tentative timeframe
    Functional tests
    May be polymorphic
    Text document
    Configuration management update
    Command and control recipe

    View Slide

  28. CHANGE MANAGEMENT: CAB
    Peer review of runbooks
    Breaks long release cycle
    Reaches CAB objectives of traceability
    Recurring tasks can be auto-validated (standard changes)
    For instance adding a vhost
    A good prerequisite: no manual intervention

    View Slide

  29. CHANGE MANAGEMENT: CHANGE RECORDS
    Text based log of change
    Accessible by all parties at time of change

    View Slide

  30. TOOLING

    View Slide

  31. CMDB IS EASY
    You already have configuration management
    It's already stored in git
    # n
    o
    d
    e
    s
    .
    y
    a
    m
    l
    n
    o
    d
    e "
    n
    e
    t
    w
    o
    r
    k
    -
    l
    b
    0
    1
    " { i
    n
    c
    l
    u
    d
    e n
    e
    t
    w
    o
    r
    k
    :
    :
    l
    b }
    n
    o
    d
    e "
    p
    o
    r
    t
    a
    l
    -
    f
    r
    o
    n
    t
    0
    1
    " { i
    n
    c
    l
    u
    d
    e p
    o
    r
    t
    a
    l
    :
    :
    f
    r
    o
    n
    t }
    n
    o
    d
    e "
    p
    o
    r
    t
    a
    l
    -
    f
    r
    o
    n
    t
    0
    2
    " { i
    n
    c
    l
    u
    d
    e p
    o
    r
    t
    a
    l
    :
    :
    f
    r
    o
    n
    t }
    n
    o
    d
    e "
    p
    o
    r
    t
    a
    l
    -
    f
    r
    o
    n
    t
    0
    3
    " { i
    n
    c
    l
    u
    d
    e p
    o
    r
    t
    a
    l
    :
    :
    f
    r
    o
    n
    t }
    n
    o
    d
    e "
    p
    o
    r
    t
    a
    l
    -
    d
    b
    0
    1
    " { i
    n
    c
    l
    u
    d
    e p
    o
    r
    t
    a
    l
    :
    :
    d
    b }

    View Slide

  32. CHANGE MANAGEMENT IS LESS ADRESSED
    No common runbook tool
    No standard execution logging

    View Slide

  33. COMMON THEME: REUSE YOUR EXISTING INFRASTRUCTURE
    Pull requests are great for peer review
    Git provides a nice way to archive text data
    Introduce as few new tools as possible
    Avoid adding bloat with process

    View Slide

  34. EXAMPLE APPROACH:
    RFCs
    Standard Changes: Config Mgmt or Command-and-Control
    recipe
    Normal Changes: Doc update for the corresponding service
    platform
    CAB: Pull requests
    Change records: Archived chat logs
    Hubot is great for this

    View Slide

  35. LEVERAGE YOUR CHAT ROOM

    View Slide

  36. ADDITIONAL TOOLING: WARP
    Command and Control.
    Web + Chat integration.
    https://github.com/pyr/warp

    View Slide

  37. ADDITIONAL TOOLING: WARP
    #
    # R
    u
    n
    b
    o
    o
    k
    : u
    p
    d
    a
    t
    e t
    o l
    a
    t
    e
    s
    t w
    e
    b
    s
    i
    t
    e
    #
    # =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    =
    #
    #
    #
    # 1
    . M
    a
    k
    e
    s s
    u
    r
    e w
    e
    '
    r
    e r
    u
    n
    n
    i
    n
    g a
    g
    a
    i
    n
    s
    t l
    a
    t
    e
    s
    t p
    a
    c
    k
    a
    g
    e
    s
    #
    # 2
    . U
    p
    d
    a
    t
    e
    s a
    p
    p
    r
    o
    p
    r
    i
    a
    t
    e p
    a
    c
    k
    a
    g
    e
    #
    # 3
    . R
    e
    l
    o
    a
    d
    s n
    g
    i
    n
    x
    s
    c
    r
    i
    p
    t
    _
    n
    a
    m
    e
    : d
    e
    p
    l
    o
    y
    -
    w
    e
    b
    s
    i
    t
    e
    m
    a
    t
    c
    h
    :
    a
    n
    d
    :
    - {
    f
    a
    c
    t
    : e
    n
    v
    i
    r
    o
    n
    m
    e
    n
    t
    , v
    a
    l
    u
    e
    : s
    t
    a
    g
    i
    n
    g
    }
    - {
    f
    a
    c
    t
    : p
    l
    a
    t
    f
    o
    r
    m
    , v
    a
    l
    u
    e
    : w
    e
    b
    i
    s
    t
    e
    }
    #
    # D
    e
    f
    a
    u
    l
    t
    s t
    o s
    t
    a
    g
    i
    n
    g
    , T
    h
    i
    s p
    r
    o
    v
    i
    d
    e
    s a p
    r
    o
    d
    u
    c
    t
    i
    o
    n p
    r
    o
    f
    i
    l
    e
    :
    p
    r
    o
    f
    i
    l
    e
    s
    :
    p
    r
    o
    d
    u
    c
    t
    i
    o
    n
    :
    m
    a
    t
    c
    h
    :
    a
    n
    d
    :
    - {
    f
    a
    c
    t
    : e
    n
    v
    i
    r
    o
    n
    m
    e
    n
    t
    , v
    a
    l
    u
    e
    : p
    r
    o
    d
    u
    c
    t
    i
    o
    n
    }
    - {
    f
    a
    c
    t
    : p
    l
    a
    t
    f
    o
    r
    m
    , v
    a
    l
    u
    e
    : w
    e
    b
    s
    i
    t
    e
    }
    t
    i
    m
    e
    o
    u
    t
    : 3
    0
    0
    0
    0 # T
    h
    i
    s w
    i
    l
    l r
    u
    n w
    i
    t
    h
    i
    n 3
    0
    s

    View Slide

  38. ADDITIONAL TOOLING: WARP

    View Slide

  39. ADDITIONAL TOOLING: WARP

    View Slide

  40. ADDITIONAL TOOLING: WARP

    View Slide

  41. PARTING WORDS

    View Slide

  42. IT'S ABOUT VISIBILITY
    Finding out what happened 37 days ago is a g
    i
    t l
    o
    g
    away

    View Slide

  43. AVOID DOGMATISM
    Don't fall in the SCRUM trap

    View Slide

  44. CAREFULLY WEIGHT FRICTION TO BENEFIT RATIO
    Don't lose sight of the objectives

    View Slide

  45. GO STEP BY STEP
    It's ok to still log-in if you need it
    Consider it's a failure and see how you can remediate

    View Slide

  46. OPEN SPACE WISHLIST
    Process tooling: Let's figure a cool name for this
    Discussion around runbooks, what do you do ?
    We can extend the discussion to other concerns
    Building the service catalog
    Capacity management
    Problem & Incident management

    View Slide

  47. THANK YOU
    @pyr

    View Slide