Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Change Management at Scale

Change Management at Scale

Reconciling process and agility

Pierre-Yves Ritschard

May 02, 2018
Tweet

More Decks by Pierre-Yves Ritschard

Other Decks in Technology

Transcript

  1. CHANGE MANAGEMENT AT
    CHANGE MANAGEMENT AT
    SCALE
    SCALE
    STAYING ORGANIZED IN THE FACE OF AGILITY
    STAYING ORGANIZED IN THE FACE OF AGILITY
    1

    View full-size slide



  2. CTO at Exoscale: The European Cloud Provider
    Early devops implementer: Orange, Lagardere
    Early big-data & ML implementer: paper.li
    @pyr
    2 . 1

    View full-size slide

  3. EXOSCALE
    EXOSCALE
    3 . 1

    View full-size slide

  4. EXOSCALE
    EXOSCALE
    provider "exoscale" {
    api_key = "${var.exoscale_api_key}"
    secret_key = "${var.exoscale_secret_key}"
    }
    resource "exoscale_instance" "web" {
    template = "Ubuntu 17.04"
    disk_size = "50g"
    profile = "medium"
    ssh_key = "production"
    }
    4 . 1

    View full-size slide

  5. WHAT'S CHANGE ?
    WHAT'S CHANGE ?
    The addition, modification or removal of
    anything that could have an effect on IT services
    5 . 1

    View full-size slide

  6. WHY CHANGE MATTERS
    WHY CHANGE MATTERS
    IT used to be a support function
    Inventory
    Payroll
    Accounting
    6 . 1

    View full-size slide

  7. WHY CHANGE MATTERS
    WHY CHANGE MATTERS
    7 . 1

    View full-size slide

  8. WHAT'S IN AN APP
    WHAT'S IN AN APP
    Inventory
    User preferences
    Real-time analytics
    8 . 1

    View full-size slide

  9. AIM OF THE TALK
    AIM OF THE TALK
    Addressing the tension between process, security, and agility
    9 . 1

    View full-size slide

  10. OUTLINE
    OUTLINE
    Process in the Entreprise
    ITIL: the good parts
    Agile operations
    Tooling
    10 . 1

    View full-size slide

  11. PROCESS IN THE ENTREPRISE
    PROCESS IN THE ENTREPRISE
    11 . 1

    View full-size slide

  12. WHAT IT LOOKS LIKE
    WHAT IT LOOKS LIKE
    12 . 1

    View full-size slide

  13. CANONICAL USE-CASE: ADDING
    CANONICAL USE-CASE: ADDING
    OUTBOUND MAIL
    OUTBOUND MAIL
    Please open outbound TCP 25
    Sounds like you want TCP 587 to our internal mailers

    13 . 1

    View full-size slide

  14. MEANWHILE
    MEANWHILE
    laptop> ssh root@app01
    app01> iptables -A OUTPUT -p tcp --dport 25 -j ACCEPT
    14 . 1

    View full-size slide

  15. ABSTRACTING THE PROCESS
    ABSTRACTING THE PROCESS
    15 . 1

    View full-size slide

  16. WHAT WE'RE INTERESTED IN
    WHAT WE'RE INTERESTED IN
    Traceability
    Reversibility
    Inventory
    16 . 1

    View full-size slide

  17. STANDARDS
    STANDARDS
    ISO 27001 and ISO27018
    CSA: A good basis for IaaS, PaaS and SaaS vendors
    ITIL: Best practices
    17 . 1

    View full-size slide

  18. ITIL: THE GOOD PARTS
    ITIL: THE GOOD PARTS
    18 . 1

    View full-size slide

  19. ITIL
    ITIL
    Information Technology Infrastructure Library
    Best practices for dealing with (large) IT orgs.
    Common lingo
    No tools
    19 . 1

    View full-size slide

  20. ITIL
    ITIL
    20 . 1

    View full-size slide

  21. ITIL
    ITIL
    Service design
    Service transition
    Service operation
    21 . 1

    View full-size slide

  22. SERVICE DESIGN
    SERVICE DESIGN
    Service catalog
    Capacity Management
    22 . 1

    View full-size slide

  23. SERVICE OPERATION
    SERVICE OPERATION
    Problem management
    Incident management
    23 . 1

    View full-size slide

  24. SERVICE TRANSITION
    SERVICE TRANSITION
    Configuration Management
    Change management
    24 . 1

    View full-size slide

  25. CMDB
    CMDB
    Configuration Management Database
    Holds configuration items and their relationship
    Somewhat conflates Asset Management and Configuration
    Management
    25 . 1

    View full-size slide

  26. CHANGE MANAGEMENT
    CHANGE MANAGEMENT
    Defines change lifecycle
    RFCs and classification (standard, emergency, normal)
    Change Acceptance Board (CAB, ECAB)
    Change records
    26 . 1

    View full-size slide

  27. AGILE OPERATIONS
    AGILE OPERATIONS
    27 . 1

    View full-size slide

  28. SCALING PROCESS
    SCALING PROCESS
    New objectives
    Fast iteration cycle
    Reduced interference
    Not just startups & small orgs
    How do we map valid ITIL concerns with agile orgs ?
    28 . 1

    View full-size slide

  29. SERVICE DESIGN
    SERVICE DESIGN
    The service catalog should be self-service
    This is much easier when co-opting IaaS
    29 . 1

    View full-size slide

  30. SERVICE OPERATION
    SERVICE OPERATION
    Problem and incident management are most likely already well
    addressed
    Work needed on reducing communication latency
    30 . 1

    View full-size slide

  31. SERVICE TRANSITION
    SERVICE TRANSITION
    31 . 1

    View full-size slide

  32. CMDB
    CMDB
    Configuration Management
    Standard configuration management frameworks apply
    Cattle vs Pet approach has the underlying notion of service
    catalog
    Asset Management
    Virtualization and Cloud provide dynamic inventory
    32 . 1

    View full-size slide

  33. CHANGE MANAGEMENT RECAP
    CHANGE MANAGEMENT RECAP
    RFCs
    Change acceptance board
    Change records
    33 . 1

    View full-size slide

  34. CHANGE MANAGEMENT: RFCS
    CHANGE MANAGEMENT: RFCS
    Useful elements
    Motivation & Objective
    Tentative timeframe
    Integration & Non-regression tests
    May be polymorphic
    Text document
    Configuration management update
    Command and control recipe
    34 . 1

    View full-size slide

  35. CHANGE MANAGEMENT: CAB
    CHANGE MANAGEMENT: CAB
    Peer review of runbooks
    Breaks long release cycle
    Reaches CAB objectives of traceability
    Recurring tasks can be auto-validated (standard changes)
    For instance adding a vhost
    A good prerequisite: no manual intervention
    35 . 1

    View full-size slide

  36. CHANGE MANAGEMENT: CHANGE
    CHANGE MANAGEMENT: CHANGE
    RECORDS
    RECORDS
    Text based log of change
    Accessible by all parties at time of change
    36 . 1

    View full-size slide

  37. TOOLING
    TOOLING
    37 . 1

    View full-size slide

  38. SERVICE DESIGN
    SERVICE DESIGN
    Cloud hosting platforms
    Kubernetes
    Internal PaaS systems (OpenShi )
    38 . 1

    View full-size slide

  39. SERVICE OPERATION
    SERVICE OPERATION
    Co-opt development tools to reduce barriers and friction
    Status pages for critical services
    39 . 1

    View full-size slide

  40. SERVICE TRANSITION
    SERVICE TRANSITION
    40 . 1

    View full-size slide

  41. CMDB IS EASY
    CMDB IS EASY
    You already have configuration management
    It's already stored in git
    # nodes.yaml
    node "network-lb01" { include network::lb }
    node "portal-front01" { include portal::front }
    node "portal-front02" { include portal::front }
    node "portal-front03" { include portal::front }
    node "portal-db01" { include portal::db }
    41 . 1

    View full-size slide

  42. CHANGE MANAGEMENT IS LESS
    CHANGE MANAGEMENT IS LESS
    ADRESSED
    ADRESSED
    No common runbook tool
    No standard execution logging
    42 . 1

    View full-size slide

  43. COMMON THEME: REUSE YOUR EXISTING
    COMMON THEME: REUSE YOUR EXISTING
    INFRASTRUCTURE
    INFRASTRUCTURE
    Pull requests are great for peer review
    Git provides a nice way to archive text data
    Introduce as few new tools as possible
    Avoid adding bloat with process
    43 . 1

    View full-size slide

  44. EXAMPLE APPROACH:
    EXAMPLE APPROACH:
    RFCs
    Standard Changes: Config Mgmt or Command-and-Control
    recipe
    Normal Changes: Doc update for the corresponding service
    platform
    CAB: Pull requests
    Change records: Archived chat logs
    Hubot is great for this
    44 . 1

    View full-size slide

  45. LEVERAGE YOUR CHAT ROOM
    LEVERAGE YOUR CHAT ROOM
    45 . 1

    View full-size slide

  46. PARTING WORDS
    PARTING WORDS
    46 . 1

    View full-size slide

  47. IT'S ABOUT VISIBILITY
    IT'S ABOUT VISIBILITY
    Finding out what happened 37 days ago is a git log away
    47 . 1

    View full-size slide

  48. AVOID DOGMATISM
    AVOID DOGMATISM
    Don't fall in the SCRUM trap
    48 . 1

    View full-size slide

  49. CAREFULLY WEIGHT FRICTION TO
    CAREFULLY WEIGHT FRICTION TO
    BENEFIT RATIO
    BENEFIT RATIO
    Don't lose sight of the objectives
    49 . 1

    View full-size slide

  50. GO STEP BY STEP
    GO STEP BY STEP
    It's ok to still log-in if you need it
    Consider it's a failure and see how you can remediate
    50 . 1

    View full-size slide

  51. QUESTIONS?
    QUESTIONS?
    51 . 1

    View full-size slide

  52. PYR
    PYR
    52 . 1

    View full-size slide