Slide 1

Slide 1 text

CHANGE MANAGEMENT AT SCALE PIERRE-YVES RITSCHARD @PYR 0

Slide 2

Slide 2 text

@PYR CTO at Exoscale: swiss cloud hosting. Background in both big shops & startups Open source developer

Slide 3

Slide 3 text

WHAT'S CHANGE ? The addition, modification or removal of anything that could have an effect on IT services

Slide 4

Slide 4 text

AIM OF THE TALK Reflect on how to approach change in production.

Slide 5

Slide 5 text

OUTLINE Process in the Entreprise ITIL: the good parts Agile operations Tooling

Slide 6

Slide 6 text

PROCESS IN THE ENTREPRISE

Slide 7

Slide 7 text

WHAT IT LOOKS LIKE

Slide 8

Slide 8 text

CANONICAL USE-CASE: ADDING OUTBOUND MAIL Please open outbound TCP 25 Sounds like you want TCP 587 to our internal mailers < w a i t 2 w e e k s >

Slide 9

Slide 9 text

MEANWHILE l a p t o p > s s h r o o t @ a p p 0 1 a p p 0 1 > i p t a b l e s - A O U T P U T - p t c p - - d p o r t 2 5 - j A C C E P T

Slide 10

Slide 10 text

PROMISES Traceability Reversibility CYAP

Slide 11

Slide 11 text

ABSTRACTING THE PROCESS

Slide 12

Slide 12 text

WHAT WE'RE INTERESTED IN Traceability Reversibility

Slide 13

Slide 13 text

STANDARDS ISO 27001: (mostly crazyness) CSA: A good basis for IaaS, PaaS and SaaS vendors ITIL: Best practices

Slide 14

Slide 14 text

ITIL: THE GOOD PARTS

Slide 15

Slide 15 text

ITIL Information Technology Infrastructure Library Best practices for dealing with (large) IT orgs. Common lingo No tools

Slide 16

Slide 16 text

ITIL

Slide 17

Slide 17 text

ITIL Service design Service transition Service operation

Slide 18

Slide 18 text

SERVICE DESIGN Service catalog Capacity Management

Slide 19

Slide 19 text

SERVICE OPERATION Problem management Incident management

Slide 20

Slide 20 text

SERVICE TRANSITION Configuration Management (CM) Change management

Slide 21

Slide 21 text

CMDB Configuration Management Database Holds configuration items and their relationship Somewhat conflates Asset Management and Configuration Management

Slide 22

Slide 22 text

CHANGE MANAGEMENT Defines change lifecycle RFCs and classification (standard, emergency, normal) Change Acceptance Board (CAB, ECAB) Change records Technical and organizational challenges

Slide 23

Slide 23 text

AGILE OPERATIONS

Slide 24

Slide 24 text

SCALING PROCESS New objectives Fast iteration cycle Reduced interference Not just startups & small orgs How do we map valid ITIL concerns with agile orgs ?

Slide 25

Slide 25 text

CMDB Standard configuration management frameworks apply Cattle vs Pet approach has the underlying notion of service catalog

Slide 26

Slide 26 text

CHANGE MANAGEMENT RECAP RFCs Change acceptance board Change records

Slide 27

Slide 27 text

CHANGE MANAGEMENT: RFCS Hardest step Automation prepared us Useful elements Motivation & Objective Tentative timeframe Functional tests May be polymorphic Text document Configuration management update Command and control recipe

Slide 28

Slide 28 text

CHANGE MANAGEMENT: CAB Peer review of runbooks Breaks long release cycle Reaches CAB objectives of traceability Recurring tasks can be auto-validated (standard changes) For instance adding a vhost A good prerequisite: no manual intervention

Slide 29

Slide 29 text

CHANGE MANAGEMENT: CHANGE RECORDS Text based log of change Accessible by all parties at time of change

Slide 30

Slide 30 text

TOOLING

Slide 31

Slide 31 text

CMDB IS EASY You already have configuration management It's already stored in git # n o d e s . y a m l n o d e " n e t w o r k - l b 0 1 " { i n c l u d e n e t w o r k : : l b } n o d e " p o r t a l - f r o n t 0 1 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - f r o n t 0 2 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - f r o n t 0 3 " { i n c l u d e p o r t a l : : f r o n t } n o d e " p o r t a l - d b 0 1 " { i n c l u d e p o r t a l : : d b }

Slide 32

Slide 32 text

CHANGE MANAGEMENT IS LESS ADRESSED No common runbook tool No standard execution logging

Slide 33

Slide 33 text

COMMON THEME: REUSE YOUR EXISTING INFRASTRUCTURE Pull requests are great for peer review Git provides a nice way to archive text data Introduce as few new tools as possible Avoid adding bloat with process

Slide 34

Slide 34 text

EXAMPLE APPROACH: RFCs Standard Changes: Config Mgmt or Command-and-Control recipe Normal Changes: Doc update for the corresponding service platform CAB: Pull requests Change records: Archived chat logs Hubot is great for this

Slide 35

Slide 35 text

LEVERAGE YOUR CHAT ROOM

Slide 36

Slide 36 text

ADDITIONAL TOOLING: WARP Command and Control. Web + Chat integration. https://github.com/pyr/warp

Slide 37

Slide 37 text

ADDITIONAL TOOLING: WARP # # R u n b o o k : u p d a t e t o l a t e s t w e b s i t e # # = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = # # # # 1 . M a k e s s u r e w e ' r e r u n n i n g a g a i n s t l a t e s t p a c k a g e s # # 2 . U p d a t e s a p p r o p r i a t e p a c k a g e # # 3 . R e l o a d s n g i n x s c r i p t _ n a m e : d e p l o y - w e b s i t e m a t c h : a n d : - { f a c t : e n v i r o n m e n t , v a l u e : s t a g i n g } - { f a c t : p l a t f o r m , v a l u e : w e b i s t e } # # D e f a u l t s t o s t a g i n g , T h i s p r o v i d e s a p r o d u c t i o n p r o f i l e : p r o f i l e s : p r o d u c t i o n : m a t c h : a n d : - { f a c t : e n v i r o n m e n t , v a l u e : p r o d u c t i o n } - { f a c t : p l a t f o r m , v a l u e : w e b s i t e } t i m e o u t : 3 0 0 0 0 # T h i s w i l l r u n w i t h i n 3 0 s

Slide 38

Slide 38 text

ADDITIONAL TOOLING: WARP

Slide 39

Slide 39 text

ADDITIONAL TOOLING: WARP

Slide 40

Slide 40 text

ADDITIONAL TOOLING: WARP

Slide 41

Slide 41 text

PARTING WORDS

Slide 42

Slide 42 text

IT'S ABOUT VISIBILITY Finding out what happened 37 days ago is a g i t l o g away

Slide 43

Slide 43 text

AVOID DOGMATISM Don't fall in the SCRUM trap

Slide 44

Slide 44 text

CAREFULLY WEIGHT FRICTION TO BENEFIT RATIO Don't lose sight of the objectives

Slide 45

Slide 45 text

GO STEP BY STEP It's ok to still log-in if you need it Consider it's a failure and see how you can remediate

Slide 46

Slide 46 text

OPEN SPACE WISHLIST Process tooling: Let's figure a cool name for this Discussion around runbooks, what do you do ? We can extend the discussion to other concerns Building the service catalog Capacity management Problem & Incident management

Slide 47

Slide 47 text

THANK YOU @pyr