Change Management at Scale

Reconciling process and agility

Transcript

1. CHANGE MANAGEMENT AT
   CHANGE MANAGEMENT AT
   SCALE
   SCALE
   STAYING ORGANIZED IN THE FACE OF AGILITY
   STAYING ORGANIZED IN THE FACE OF AGILITY
   1
   

   View Slide

2. 
   
   CTO at Exoscale: The European Cloud Provider
   Early devops implementer: Orange, Lagardere
   Early big-data & ML implementer: paper.li
   @pyr
   2 . 1
   

   View Slide

3. EXOSCALE
   EXOSCALE
   3 . 1
   

   View Slide

4. EXOSCALE
   EXOSCALE
   provider "exoscale" {
   api_key = "${var.exoscale_api_key}"
   secret_key = "${var.exoscale_secret_key}"
   }
   resource "exoscale_instance" "web" {
   template = "Ubuntu 17.04"
   disk_size = "50g"
   profile = "medium"
   ssh_key = "production"
   }
   4 . 1
   

   View Slide

5. WHAT'S CHANGE ?
   WHAT'S CHANGE ?
   The addition, modification or removal of
   anything that could have an effect on IT services
   5 . 1
   

   View Slide

6. WHY CHANGE MATTERS
   WHY CHANGE MATTERS
   IT used to be a support function
   Inventory
   Payroll
   Accounting
   6 . 1
   

   View Slide

7. WHY CHANGE MATTERS
   WHY CHANGE MATTERS
   7 . 1
   

   View Slide

8. WHAT'S IN AN APP
   WHAT'S IN AN APP
   Inventory
   User preferences
   Real-time analytics
   8 . 1
   

   View Slide

9. AIM OF THE TALK
   AIM OF THE TALK
   Addressing the tension between process, security, and agility
   9 . 1
   

   View Slide

10. OUTLINE
    OUTLINE
    Process in the Entreprise
    ITIL: the good parts
    Agile operations
    Tooling
    10 . 1
    

    View Slide

11. PROCESS IN THE ENTREPRISE
    PROCESS IN THE ENTREPRISE
    11 . 1
    

    View Slide

12. WHAT IT LOOKS LIKE
    WHAT IT LOOKS LIKE
    12 . 1
    

    View Slide

13. CANONICAL USE-CASE: ADDING
    CANONICAL USE-CASE: ADDING
    OUTBOUND MAIL
    OUTBOUND MAIL
    Please open outbound TCP 25
    Sounds like you want TCP 587 to our internal mailers
    
    13 . 1
    

    View Slide

14. MEANWHILE
    MEANWHILE
    laptop> ssh root@app01
    app01> iptables -A OUTPUT -p tcp --dport 25 -j ACCEPT
    14 . 1
    

    View Slide

15. ABSTRACTING THE PROCESS
    ABSTRACTING THE PROCESS
    15 . 1
    

    View Slide

16. WHAT WE'RE INTERESTED IN
    WHAT WE'RE INTERESTED IN
    Traceability
    Reversibility
    Inventory
    16 . 1
    

    View Slide

17. STANDARDS
    STANDARDS
    ISO 27001 and ISO27018
    CSA: A good basis for IaaS, PaaS and SaaS vendors
    ITIL: Best practices
    17 . 1
    

    View Slide

18. ITIL: THE GOOD PARTS
    ITIL: THE GOOD PARTS
    18 . 1
    

    View Slide

19. ITIL
    ITIL
    Information Technology Infrastructure Library
    Best practices for dealing with (large) IT orgs.
    Common lingo
    No tools
    19 . 1
    

    View Slide

20. ITIL
    ITIL
    20 . 1
    

    View Slide

21. ITIL
    ITIL
    Service design
    Service transition
    Service operation
    21 . 1
    

    View Slide

22. SERVICE DESIGN
    SERVICE DESIGN
    Service catalog
    Capacity Management
    22 . 1
    

    View Slide

23. SERVICE OPERATION
    SERVICE OPERATION
    Problem management
    Incident management
    23 . 1
    

    View Slide

24. SERVICE TRANSITION
    SERVICE TRANSITION
    Configuration Management
    Change management
    24 . 1
    

    View Slide

25. CMDB
    CMDB
    Configuration Management Database
    Holds configuration items and their relationship
    Somewhat conflates Asset Management and Configuration
    Management
    25 . 1
    

    View Slide

26. CHANGE MANAGEMENT
    CHANGE MANAGEMENT
    Defines change lifecycle
    RFCs and classification (standard, emergency, normal)
    Change Acceptance Board (CAB, ECAB)
    Change records
    26 . 1
    

    View Slide

27. AGILE OPERATIONS
    AGILE OPERATIONS
    27 . 1
    

    View Slide

28. SCALING PROCESS
    SCALING PROCESS
    New objectives
    Fast iteration cycle
    Reduced interference
    Not just startups & small orgs
    How do we map valid ITIL concerns with agile orgs ?
    28 . 1
    

    View Slide

29. SERVICE DESIGN
    SERVICE DESIGN
    The service catalog should be self-service
    This is much easier when co-opting IaaS
    29 . 1
    

    View Slide

30. SERVICE OPERATION
    SERVICE OPERATION
    Problem and incident management are most likely already well
    addressed
    Work needed on reducing communication latency
    30 . 1
    

    View Slide

31. SERVICE TRANSITION
    SERVICE TRANSITION
    31 . 1
    

    View Slide

32. CMDB
    CMDB
    Configuration Management
    Standard configuration management frameworks apply
    Cattle vs Pet approach has the underlying notion of service
    catalog
    Asset Management
    Virtualization and Cloud provide dynamic inventory
    32 . 1
    

    View Slide

33. CHANGE MANAGEMENT RECAP
    CHANGE MANAGEMENT RECAP
    RFCs
    Change acceptance board
    Change records
    33 . 1
    

    View Slide

34. CHANGE MANAGEMENT: RFCS
    CHANGE MANAGEMENT: RFCS
    Useful elements
    Motivation & Objective
    Tentative timeframe
    Integration & Non-regression tests
    May be polymorphic
    Text document
    Configuration management update
    Command and control recipe
    34 . 1
    

    View Slide

35. CHANGE MANAGEMENT: CAB
    CHANGE MANAGEMENT: CAB
    Peer review of runbooks
    Breaks long release cycle
    Reaches CAB objectives of traceability
    Recurring tasks can be auto-validated (standard changes)
    For instance adding a vhost
    A good prerequisite: no manual intervention
    35 . 1
    

    View Slide

36. CHANGE MANAGEMENT: CHANGE
    CHANGE MANAGEMENT: CHANGE
    RECORDS
    RECORDS
    Text based log of change
    Accessible by all parties at time of change
    36 . 1
    

    View Slide

37. TOOLING
    TOOLING
    37 . 1
    

    View Slide

38. SERVICE DESIGN
    SERVICE DESIGN
    Cloud hosting platforms
    Kubernetes
    Internal PaaS systems (OpenShi )
    38 . 1
    

    View Slide

39. SERVICE OPERATION
    SERVICE OPERATION
    Co-opt development tools to reduce barriers and friction
    Status pages for critical services
    39 . 1
    

    View Slide

40. SERVICE TRANSITION
    SERVICE TRANSITION
    40 . 1
    

    View Slide

41. CMDB IS EASY
    CMDB IS EASY
    You already have configuration management
    It's already stored in git
    # nodes.yaml
    node "network-lb01" { include network::lb }
    node "portal-front01" { include portal::front }
    node "portal-front02" { include portal::front }
    node "portal-front03" { include portal::front }
    node "portal-db01" { include portal::db }
    41 . 1
    

    View Slide

42. CHANGE MANAGEMENT IS LESS
    CHANGE MANAGEMENT IS LESS
    ADRESSED
    ADRESSED
    No common runbook tool
    No standard execution logging
    42 . 1
    

    View Slide

43. COMMON THEME: REUSE YOUR EXISTING
    COMMON THEME: REUSE YOUR EXISTING
    INFRASTRUCTURE
    INFRASTRUCTURE
    Pull requests are great for peer review
    Git provides a nice way to archive text data
    Introduce as few new tools as possible
    Avoid adding bloat with process
    43 . 1
    

    View Slide

44. EXAMPLE APPROACH:
    EXAMPLE APPROACH:
    RFCs
    Standard Changes: Config Mgmt or Command-and-Control
    recipe
    Normal Changes: Doc update for the corresponding service
    platform
    CAB: Pull requests
    Change records: Archived chat logs
    Hubot is great for this
    44 . 1
    

    View Slide

45. LEVERAGE YOUR CHAT ROOM
    LEVERAGE YOUR CHAT ROOM
    45 . 1
    

    View Slide

46. PARTING WORDS
    PARTING WORDS
    46 . 1
    

    View Slide

47. IT'S ABOUT VISIBILITY
    IT'S ABOUT VISIBILITY
    Finding out what happened 37 days ago is a git log away
    47 . 1
    

    View Slide

48. AVOID DOGMATISM
    AVOID DOGMATISM
    Don't fall in the SCRUM trap
    48 . 1
    

    View Slide

49. CAREFULLY WEIGHT FRICTION TO
    CAREFULLY WEIGHT FRICTION TO
    BENEFIT RATIO
    BENEFIT RATIO
    Don't lose sight of the objectives
    49 . 1
    

    View Slide

50. GO STEP BY STEP
    GO STEP BY STEP
    It's ok to still log-in if you need it
    Consider it's a failure and see how you can remediate
    50 . 1
    

    View Slide

51. QUESTIONS?
    QUESTIONS?
    51 . 1
    

    View Slide

52. PYR
    PYR
    52 . 1
    

    View Slide