Change Management at Scale

Slide 1

Slide 1 text

CHANGE MANAGEMENT AT CHANGE MANAGEMENT AT SCALE SCALE STAYING ORGANIZED IN THE FACE OF AGILITY STAYING ORGANIZED IN THE FACE OF AGILITY 1

Slide 2

Slide 2 text

CTO at Exoscale: The European Cloud Provider Early devops implementer: Orange, Lagardere Early big-data & ML implementer: paper.li @pyr 2 . 1

Slide 3

Slide 3 text

EXOSCALE EXOSCALE 3 . 1

Slide 4

Slide 4 text

EXOSCALE EXOSCALE provider "exoscale" { api_key = "${var.exoscale_api_key}" secret_key = "${var.exoscale_secret_key}" } resource "exoscale_instance" "web" { template = "Ubuntu 17.04" disk_size = "50g" profile = "medium" ssh_key = "production" } 4 . 1

Slide 5

Slide 5 text

WHAT'S CHANGE ? WHAT'S CHANGE ? The addition, modification or removal of anything that could have an eﬀect on IT services 5 . 1

Slide 6

Slide 6 text

WHY CHANGE MATTERS WHY CHANGE MATTERS IT used to be a support function Inventory Payroll Accounting 6 . 1

Slide 7

Slide 7 text

WHY CHANGE MATTERS WHY CHANGE MATTERS 7 . 1

Slide 8

Slide 8 text

WHAT'S IN AN APP WHAT'S IN AN APP Inventory User preferences Real-time analytics 8 . 1

Slide 9

Slide 9 text

AIM OF THE TALK AIM OF THE TALK Addressing the tension between process, security, and agility 9 . 1

Slide 10

Slide 10 text

OUTLINE OUTLINE Process in the Entreprise ITIL: the good parts Agile operations Tooling 10 . 1

Slide 11

Slide 11 text

PROCESS IN THE ENTREPRISE PROCESS IN THE ENTREPRISE 11 . 1

Slide 12

Slide 12 text

WHAT IT LOOKS LIKE WHAT IT LOOKS LIKE 12 . 1

Slide 13

Slide 13 text

CANONICAL USE-CASE: ADDING CANONICAL USE-CASE: ADDING OUTBOUND MAIL OUTBOUND MAIL Please open outbound TCP 25 Sounds like you want TCP 587 to our internal mailers 13 . 1

Slide 14

Slide 14 text

MEANWHILE MEANWHILE laptop> ssh root@app01 app01> iptables -A OUTPUT -p tcp --dport 25 -j ACCEPT 14 . 1

Slide 15

Slide 15 text

ABSTRACTING THE PROCESS ABSTRACTING THE PROCESS 15 . 1

Slide 16

Slide 16 text

WHAT WE'RE INTERESTED IN WHAT WE'RE INTERESTED IN Traceability Reversibility Inventory 16 . 1

Slide 17

Slide 17 text

STANDARDS STANDARDS ISO 27001 and ISO27018 CSA: A good basis for IaaS, PaaS and SaaS vendors ITIL: Best practices 17 . 1

Slide 18

Slide 18 text

ITIL: THE GOOD PARTS ITIL: THE GOOD PARTS 18 . 1

Slide 19

Slide 19 text

ITIL ITIL Information Technology Infrastructure Library Best practices for dealing with (large) IT orgs. Common lingo No tools 19 . 1

Slide 20

Slide 20 text

ITIL ITIL 20 . 1

Slide 21

Slide 21 text

ITIL ITIL Service design Service transition Service operation 21 . 1

Slide 22

Slide 22 text

SERVICE DESIGN SERVICE DESIGN Service catalog Capacity Management 22 . 1

Slide 23

Slide 23 text

SERVICE OPERATION SERVICE OPERATION Problem management Incident management 23 . 1

Slide 24

Slide 24 text

SERVICE TRANSITION SERVICE TRANSITION Configuration Management Change management 24 . 1

Slide 25

Slide 25 text

CMDB CMDB Configuration Management Database Holds configuration items and their relationship Somewhat conflates Asset Management and Configuration Management 25 . 1

Slide 26

Slide 26 text

CHANGE MANAGEMENT CHANGE MANAGEMENT Defines change lifecycle RFCs and classification (standard, emergency, normal) Change Acceptance Board (CAB, ECAB) Change records 26 . 1

Slide 27

Slide 27 text

AGILE OPERATIONS AGILE OPERATIONS 27 . 1

Slide 28

Slide 28 text

SCALING PROCESS SCALING PROCESS New objectives Fast iteration cycle Reduced interference Not just startups & small orgs How do we map valid ITIL concerns with agile orgs ? 28 . 1

Slide 29

Slide 29 text

SERVICE DESIGN SERVICE DESIGN The service catalog should be self-service This is much easier when co-opting IaaS 29 . 1

Slide 30

Slide 30 text

SERVICE OPERATION SERVICE OPERATION Problem and incident management are most likely already well addressed Work needed on reducing communication latency 30 . 1

Slide 31

Slide 31 text

SERVICE TRANSITION SERVICE TRANSITION 31 . 1

Slide 32

Slide 32 text

CMDB CMDB Configuration Management Standard configuration management frameworks apply Cattle vs Pet approach has the underlying notion of service catalog Asset Management Virtualization and Cloud provide dynamic inventory 32 . 1

Slide 33

Slide 33 text

CHANGE MANAGEMENT RECAP CHANGE MANAGEMENT RECAP RFCs Change acceptance board Change records 33 . 1

Slide 34

Slide 34 text

CHANGE MANAGEMENT: RFCS CHANGE MANAGEMENT: RFCS Useful elements Motivation & Objective Tentative timeframe Integration & Non-regression tests May be polymorphic Text document Configuration management update Command and control recipe 34 . 1

Slide 35

Slide 35 text

CHANGE MANAGEMENT: CAB CHANGE MANAGEMENT: CAB Peer review of runbooks Breaks long release cycle Reaches CAB objectives of traceability Recurring tasks can be auto-validated (standard changes) For instance adding a vhost A good prerequisite: no manual intervention 35 . 1

Slide 36

Slide 36 text

CHANGE MANAGEMENT: CHANGE CHANGE MANAGEMENT: CHANGE RECORDS RECORDS Text based log of change Accessible by all parties at time of change 36 . 1

Slide 37

Slide 37 text

TOOLING TOOLING 37 . 1

Slide 38

Slide 38 text

SERVICE DESIGN SERVICE DESIGN Cloud hosting platforms Kubernetes Internal PaaS systems (OpenShi ) 38 . 1

Slide 39

Slide 39 text

SERVICE OPERATION SERVICE OPERATION Co-opt development tools to reduce barriers and friction Status pages for critical services 39 . 1

Slide 40

Slide 40 text

SERVICE TRANSITION SERVICE TRANSITION 40 . 1

Slide 41

Slide 41 text

CMDB IS EASY CMDB IS EASY You already have configuration management It's already stored in git # nodes.yaml node "network-lb01" { include network::lb } node "portal-front01" { include portal::front } node "portal-front02" { include portal::front } node "portal-front03" { include portal::front } node "portal-db01" { include portal::db } 41 . 1

Slide 42

Slide 42 text

CHANGE MANAGEMENT IS LESS CHANGE MANAGEMENT IS LESS ADRESSED ADRESSED No common runbook tool No standard execution logging 42 . 1

Slide 43

Slide 43 text

COMMON THEME: REUSE YOUR EXISTING COMMON THEME: REUSE YOUR EXISTING INFRASTRUCTURE INFRASTRUCTURE Pull requests are great for peer review Git provides a nice way to archive text data Introduce as few new tools as possible Avoid adding bloat with process 43 . 1

Slide 44

Slide 44 text

EXAMPLE APPROACH: EXAMPLE APPROACH: RFCs Standard Changes: Config Mgmt or Command-and-Control recipe Normal Changes: Doc update for the corresponding service platform CAB: Pull requests Change records: Archived chat logs Hubot is great for this 44 . 1

Slide 45

Slide 45 text

LEVERAGE YOUR CHAT ROOM LEVERAGE YOUR CHAT ROOM 45 . 1

Slide 46

Slide 46 text

PARTING WORDS PARTING WORDS 46 . 1

Slide 47

Slide 47 text

IT'S ABOUT VISIBILITY IT'S ABOUT VISIBILITY Finding out what happened 37 days ago is a git log away 47 . 1

Slide 48

Slide 48 text

AVOID DOGMATISM AVOID DOGMATISM Don't fall in the SCRUM trap 48 . 1

Slide 49

Slide 49 text

CAREFULLY WEIGHT FRICTION TO CAREFULLY WEIGHT FRICTION TO BENEFIT RATIO BENEFIT RATIO Don't lose sight of the objectives 49 . 1

Slide 50

Slide 50 text

GO STEP BY STEP GO STEP BY STEP It's ok to still log-in if you need it Consider it's a failure and see how you can remediate 50 . 1