Change Management at Scale

Change Management at Scale

Reconciling process and agility

2fcc875f98607b3007909fe4be99160d?s=128

Pierre-Yves Ritschard

May 02, 2018
Tweet

Transcript

  1. CHANGE MANAGEMENT AT CHANGE MANAGEMENT AT SCALE SCALE STAYING ORGANIZED

    IN THE FACE OF AGILITY STAYING ORGANIZED IN THE FACE OF AGILITY 1
  2. CTO at Exoscale: The European Cloud Provider Early devops implementer:

    Orange, Lagardere Early big-data & ML implementer: paper.li @pyr 2 . 1
  3. EXOSCALE EXOSCALE 3 . 1

  4. EXOSCALE EXOSCALE provider "exoscale" { api_key = "${var.exoscale_api_key}" secret_key =

    "${var.exoscale_secret_key}" } resource "exoscale_instance" "web" { template = "Ubuntu 17.04" disk_size = "50g" profile = "medium" ssh_key = "production" } 4 . 1
  5. WHAT'S CHANGE ? WHAT'S CHANGE ? The addition, modification or

    removal of anything that could have an effect on IT services 5 . 1
  6. WHY CHANGE MATTERS WHY CHANGE MATTERS IT used to be

    a support function Inventory Payroll Accounting 6 . 1
  7. WHY CHANGE MATTERS WHY CHANGE MATTERS 7 . 1

  8. WHAT'S IN AN APP WHAT'S IN AN APP Inventory User

    preferences Real-time analytics 8 . 1
  9. AIM OF THE TALK AIM OF THE TALK Addressing the

    tension between process, security, and agility 9 . 1
  10. OUTLINE OUTLINE Process in the Entreprise ITIL: the good parts

    Agile operations Tooling 10 . 1
  11. PROCESS IN THE ENTREPRISE PROCESS IN THE ENTREPRISE 11 .

    1
  12. WHAT IT LOOKS LIKE WHAT IT LOOKS LIKE 12 .

    1
  13. CANONICAL USE-CASE: ADDING CANONICAL USE-CASE: ADDING OUTBOUND MAIL OUTBOUND MAIL

    Please open outbound TCP 25 Sounds like you want TCP 587 to our internal mailers <wait 2 weeks> 13 . 1
  14. MEANWHILE MEANWHILE laptop> ssh root@app01 app01> iptables -A OUTPUT -p

    tcp --dport 25 -j ACCEPT 14 . 1
  15. ABSTRACTING THE PROCESS ABSTRACTING THE PROCESS 15 . 1

  16. WHAT WE'RE INTERESTED IN WHAT WE'RE INTERESTED IN Traceability Reversibility

    Inventory 16 . 1
  17. STANDARDS STANDARDS ISO 27001 and ISO27018 CSA: A good basis

    for IaaS, PaaS and SaaS vendors ITIL: Best practices 17 . 1
  18. ITIL: THE GOOD PARTS ITIL: THE GOOD PARTS 18 .

    1
  19. ITIL ITIL Information Technology Infrastructure Library Best practices for dealing

    with (large) IT orgs. Common lingo No tools 19 . 1
  20. ITIL ITIL 20 . 1

  21. ITIL ITIL Service design Service transition Service operation 21 .

    1
  22. SERVICE DESIGN SERVICE DESIGN Service catalog Capacity Management 22 .

    1
  23. SERVICE OPERATION SERVICE OPERATION Problem management Incident management 23 .

    1
  24. SERVICE TRANSITION SERVICE TRANSITION Configuration Management Change management 24 .

    1
  25. CMDB CMDB Configuration Management Database Holds configuration items and their

    relationship Somewhat conflates Asset Management and Configuration Management 25 . 1
  26. CHANGE MANAGEMENT CHANGE MANAGEMENT Defines change lifecycle RFCs and classification

    (standard, emergency, normal) Change Acceptance Board (CAB, ECAB) Change records 26 . 1
  27. AGILE OPERATIONS AGILE OPERATIONS 27 . 1

  28. SCALING PROCESS SCALING PROCESS New objectives Fast iteration cycle Reduced

    interference Not just startups & small orgs How do we map valid ITIL concerns with agile orgs ? 28 . 1
  29. SERVICE DESIGN SERVICE DESIGN The service catalog should be self-service

    This is much easier when co-opting IaaS 29 . 1
  30. SERVICE OPERATION SERVICE OPERATION Problem and incident management are most

    likely already well addressed Work needed on reducing communication latency 30 . 1
  31. SERVICE TRANSITION SERVICE TRANSITION 31 . 1

  32. CMDB CMDB Configuration Management Standard configuration management frameworks apply Cattle

    vs Pet approach has the underlying notion of service catalog Asset Management Virtualization and Cloud provide dynamic inventory 32 . 1
  33. CHANGE MANAGEMENT RECAP CHANGE MANAGEMENT RECAP RFCs Change acceptance board

    Change records 33 . 1
  34. CHANGE MANAGEMENT: RFCS CHANGE MANAGEMENT: RFCS Useful elements Motivation &

    Objective Tentative timeframe Integration & Non-regression tests May be polymorphic Text document Configuration management update Command and control recipe 34 . 1
  35. CHANGE MANAGEMENT: CAB CHANGE MANAGEMENT: CAB Peer review of runbooks

    Breaks long release cycle Reaches CAB objectives of traceability Recurring tasks can be auto-validated (standard changes) For instance adding a vhost A good prerequisite: no manual intervention 35 . 1
  36. CHANGE MANAGEMENT: CHANGE CHANGE MANAGEMENT: CHANGE RECORDS RECORDS Text based

    log of change Accessible by all parties at time of change 36 . 1
  37. TOOLING TOOLING 37 . 1

  38. SERVICE DESIGN SERVICE DESIGN Cloud hosting platforms Kubernetes Internal PaaS

    systems (OpenShi ) 38 . 1
  39. SERVICE OPERATION SERVICE OPERATION Co-opt development tools to reduce barriers

    and friction Status pages for critical services 39 . 1
  40. SERVICE TRANSITION SERVICE TRANSITION 40 . 1

  41. CMDB IS EASY CMDB IS EASY You already have configuration

    management It's already stored in git # nodes.yaml node "network-lb01" { include network::lb } node "portal-front01" { include portal::front } node "portal-front02" { include portal::front } node "portal-front03" { include portal::front } node "portal-db01" { include portal::db } 41 . 1
  42. CHANGE MANAGEMENT IS LESS CHANGE MANAGEMENT IS LESS ADRESSED ADRESSED

    No common runbook tool No standard execution logging 42 . 1
  43. COMMON THEME: REUSE YOUR EXISTING COMMON THEME: REUSE YOUR EXISTING

    INFRASTRUCTURE INFRASTRUCTURE Pull requests are great for peer review Git provides a nice way to archive text data Introduce as few new tools as possible Avoid adding bloat with process 43 . 1
  44. EXAMPLE APPROACH: EXAMPLE APPROACH: RFCs Standard Changes: Config Mgmt or

    Command-and-Control recipe Normal Changes: Doc update for the corresponding service platform CAB: Pull requests Change records: Archived chat logs Hubot is great for this 44 . 1
  45. LEVERAGE YOUR CHAT ROOM LEVERAGE YOUR CHAT ROOM 45 .

    1
  46. PARTING WORDS PARTING WORDS 46 . 1

  47. IT'S ABOUT VISIBILITY IT'S ABOUT VISIBILITY Finding out what happened

    37 days ago is a git log away 47 . 1
  48. AVOID DOGMATISM AVOID DOGMATISM Don't fall in the SCRUM trap

    48 . 1
  49. CAREFULLY WEIGHT FRICTION TO CAREFULLY WEIGHT FRICTION TO BENEFIT RATIO

    BENEFIT RATIO Don't lose sight of the objectives 49 . 1
  50. GO STEP BY STEP GO STEP BY STEP It's ok

    to still log-in if you need it Consider it's a failure and see how you can remediate 50 . 1
  51. QUESTIONS? QUESTIONS? 51 . 1

  52. PYR PYR 52 . 1