Slide 1

Slide 1 text

SRE: the good, the bad, and the ouch Holly Cummins Red Hat WTF is SRE April 28, 2022

Slide 2

Slide 2 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE @holly_cummins

Slide 3

Slide 3 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE @holly_cummins

Slide 4

Slide 4 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE @holly_cummins confession:

Slide 5

Slide 5 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE @holly_cummins confession: i am not an SRE

Slide 6

Slide 6 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE @holly_cummins confession: i am not an SRE Robert Barron Cansu Kavılı Örnek but some of my good friends are

Slide 7

Slide 7 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE Robert Barron IBM Garage Cansu Kavılı Örnek Red Hat Open Innovation Labs

Slide 8

Slide 8 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE thanks for the stories, Robert and Cansu Robert Barron IBM Garage Cansu Kavılı Örnek Red Hat Open Innovation Labs

Slide 9

Slide 9 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE thanks for the stories, Robert and Cansu Robert Barron IBM Garage Cansu Kavılı Örnek Red Hat Open Innovation Labs

Slide 10

Slide 10 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE thanks for the stories, Robert and Cansu Robert Barron IBM Garage Cansu Kavılı Örnek Red Hat Open Innovation Labs

Slide 11

Slide 11 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE poll: who are you? sli.do #886041

Slide 12

Slide 12 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what is SRE?

Slide 13

Slide 13 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what is SRE?

Slide 14

Slide 14 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE no really, WTF is SRE?

Slide 15

Slide 15 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE SRE what ops would be like if it was done by software engineers

Slide 16

Slide 16 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what was wrong with the old way of doing ops?

Slide 17

Slide 17 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE old ops

Slide 18

Slide 18 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE manual old ops

Slide 19

Slide 19 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE manual repetitive old ops

Slide 20

Slide 20 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE manual repetitive siloed old ops

Slide 21

Slide 21 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE manual repetitive siloed not aligned to business goals old ops

Slide 22

Slide 22 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE manual repetitive siloed not aligned to business goals unable to handle complexity of cloud native old ops

Slide 23

Slide 23 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE eliminate repetitive tasks

Slide 24

Slide 24 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE eliminate toil

Slide 25

Slide 25 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE aligned incentives

Slide 26

Slide 26 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE failure is a symptom, not a cause

Slide 27

Slide 27 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE devops?

Slide 28

Slide 28 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE automate everything DevOps SRE

Slide 29

Slide 29 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE holistic + collaborative DevOps SRE

Slide 30

Slide 30 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what could possibly go wrong?

Slide 31

Slide 31 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE poll: where did you come from? sli.do #886041

Slide 32

Slide 32 text

@holly_cummins IBM Garage true story “we’re SRE now” the cunning rebrand

Slide 33

Slide 33 text

@holly_cummins IBM Garage true story “we’re SRE now” the cunning rebrand

Slide 34

Slide 34 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE SRE: what ops would be like if it was done by software engineers software engineer

Slide 35

Slide 35 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE SRE: what ops would be like if it was done by software engineers software engineer

Slide 36

Slide 36 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE SRE: what ops would be like if it was done by software engineers ops software engineer

Slide 37

Slide 37 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE ops SRE: what ops would be like if it was done by ops?

Slide 38

Slide 38 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE ops SRE: what ops would be like if it was done by ops?

Slide 39

Slide 39 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE we’ve read the book!

Slide 40

Slide 40 text

@holly_cummins IBM Garage true story “we are just as good: we have scripts” the cunning rebrand

Slide 41

Slide 41 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what triggers the scripts?

Slide 42

Slide 42 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE dev how much contact do SRE have with dev? SRE

Slide 43

Slide 43 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE dev how much contact do SRE have with dev? SRE

Slide 44

Slide 44 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE dev are there any SRE NFRs in the dev backlog? SRE

Slide 45

Slide 45 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE uh … we never thought of speaking to the developers

Slide 46

Slide 46 text

@holly_cummins IBM Garage true story “we do SRE … in silos” the cunning rebrand

Slide 47

Slide 47 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 48

Slide 48 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 49

Slide 49 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 50

Slide 50 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 51

Slide 51 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE I am not designed for this.

Slide 52

Slide 52 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE two war rooms

Slide 53

Slide 53 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile

Slide 54

Slide 54 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile we’re responsible for stability of the mainframe we’re responsible for stability of the front end

Slide 55

Slide 55 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile the ambassador we’re responsible for stability of the mainframe we’re responsible for stability of the front end we’re responsible for stability of the mainframe … as long as it’s used correctly

Slide 56

Slide 56 text

@holly_cummins IBM Garage true story “we have a ticket per team, not per incident” dots aren’t connected

Slide 57

Slide 57 text

“we want to do SRE but we don’t have enough permissions on our systems”

Slide 58

Slide 58 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “the DBAs don’t trust us”

Slide 59

Slide 59 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “it takes us 15 minutes just to get permission to run a standard set of SQL diagnostic statements”

Slide 60

Slide 60 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “it takes us 15 minutes just to get permission to run a standard set of SQL diagnostic statements”

Slide 61

Slide 61 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE silos cost

Slide 62

Slide 62 text

@holly_cummins IBM Garage true story “we do post-mortems after every incident … maybe” the gap between intent and reality

Slide 63

Slide 63 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 64

Slide 64 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure the number of incidents

Slide 65

Slide 65 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure the number of incidents measure the number of post-mortems

Slide 66

Slide 66 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure the number of incidents measure the number of post-mortems see if they match

Slide 67

Slide 67 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 68

Slide 68 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE advanced metrics:

Slide 69

Slide 69 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE advanced metrics: how many people were in the post-mortem?

Slide 70

Slide 70 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE advanced metrics: how many people were in the post-mortem? does it include more than the people directly involved?

Slide 71

Slide 71 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE advanced metrics: how many people were in the post-mortem? does it include more than the people directly involved? did we invite more than our own team?

Slide 72

Slide 72 text

@holly_cummins IBM Garage true story “no one says anything in our blameless post-mortems”

Slide 73

Slide 73 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE ‘blameless’ post-mortem

Slide 74

Slide 74 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE if involvement in an incident is punished, people will avoid engaging with systems

Slide 75

Slide 75 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “great idea, go build that!” if ideas are punished with extra work, people will try not to have ideas

Slide 76

Slide 76 text

@holly_cummins IBM Garage true story “we have success metrics” the perverse incentive

Slide 77

Slide 77 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE metrics are good

Slide 78

Slide 78 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE SREs are data-driven

Slide 79

Slide 79 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE but …

Slide 80

Slide 80 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE as senior leaders, be careful what you incentivise

Slide 81

Slide 81 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE be careful what behaviours you discourage

Slide 82

Slide 82 text

@holly_cummins IBM Garage true story “we count how many incidents we have; if the number goes down, it means we are working better” the perverse incentive

Slide 83

Slide 83 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE during the holidays, quality is outstanding!

Slide 84

Slide 84 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE delivery excellence!

Slide 85

Slide 85 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE fewer people working → fewer incidents

Slide 86

Slide 86 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE new release → more incidents

Slide 87

Slide 87 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE are you measuring the right things?

Slide 88

Slide 88 text

“management has weaponised the metrics”

Slide 89

Slide 89 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure things, not people

Slide 90

Slide 90 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what should you measure?

Slide 91

Slide 91 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 92

Slide 92 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE make work visible

Slide 93

Slide 93 text

@holly_cummins IBM Garage true story “we never seem to complete the work we planned” the email timesink

Slide 94

Slide 94 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory 1 sprint

Slide 95

Slide 95 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory 1 sprint 50% story points

Slide 96

Slide 96 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory 1 sprint 50% story points 50% unplanned work (tickets)

Slide 97

Slide 97 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory reality 1 sprint 50% story points 50% unplanned work (tickets)

Slide 98

Slide 98 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory reality 1 sprint 50% story points 10% story points 50% unplanned work (tickets)

Slide 99

Slide 99 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory reality 1 sprint 50% story points 10% story points 50% unplanned work (tickets) 50% tickets

Slide 100

Slide 100 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory reality 1 sprint 50% story points 10% story points 50% unplanned work (tickets) 50% tickets 40% ??

Slide 101

Slide 101 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE theory reality 1 sprint 50% story points 10% story points 50% unplanned work (tickets) 50% tickets 40% ??

Slide 102

Slide 102 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 103

Slide 103 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “can you just … “

Slide 104

Slide 104 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “can you just … “ “how do I do this?”

Slide 105

Slide 105 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “can you just … “ “how do I do this?” “where is this documented?”

Slide 106

Slide 106 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “can you just … “ “how do I do this?” “where is this documented?”

Slide 107

Slide 107 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 108

Slide 108 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE this wasn’t a team failure

Slide 109

Slide 109 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE this wasn’t a team failure it was a data quality issue

Slide 110

Slide 110 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE this wasn’t a team failure it was a data quality issue it was a process issue

Slide 111

Slide 111 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE track work use the data to eliminate toil

Slide 112

Slide 112 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure blockers

Slide 113

Slide 113 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE mean time to failure? mean time to detect problems?

Slide 114

Slide 114 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what is failure in a complex system? if a system goes down but user experience is fine, does that count?

Slide 115

Slide 115 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE measure “what have I learned” measure “have I made sure it won’t happen again”

Slide 116

Slide 116 text

@holly_cummins IBM Garage true client story “we can’t actually release this.” value on the shelf

Slide 117

Slide 117 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE are our incentives hurting us?

Slide 118

Slide 118 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what’s stopping more frequent deploys?

Slide 119

Slide 119 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “it costs too much to release”

Slide 120

Slide 120 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “it costs too much to release” you can fix that

Slide 121

Slide 121 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “we can’t ship until we have more confidence in the quality”

Slide 122

Slide 122 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE “we can’t ship until we have more confidence in the quality” you can fix that

Slide 123

Slide 123 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE deferred wiring

Slide 124

Slide 124 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE feature flags

Slide 125

Slide 125 text

@holly_cummins IBM Garage true client story “we can’t release this microservice… we deploy all our microservices at the same time… because otherwise nothing works.” the monolithic microservices

Slide 126

Slide 126 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE let’s talk about microservices

Slide 127

Slide 127 text

@holly_cummins IBM Garage true client story “every time we change code, something breaks” the peril of microservices

Slide 128

Slide 128 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE just because a system runs across 6 containers doesn’t mean it’s decoupled

Slide 129

Slide 129 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 130

Slide 130 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE mars climate explorer

Slide 131

Slide 131 text

@holly_cummins #IBMGarage Courtesy NASA/ JPL-Caltech

Slide 132

Slide 132 text

No content

Slide 133

Slide 133 text

No content

Slide 134

Slide 134 text

distributing did not help

Slide 135

Slide 135 text

metric units distributing did not help

Slide 136

Slide 136 text

metric units imperial units distributing did not help

Slide 137

Slide 137 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE testing

Slide 138

Slide 138 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE $370 million loss https://en.wikipedia.org/wiki/Cluster_(spacecraft) Cluster + Ariane 5

Slide 139

Slide 139 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 140

Slide 140 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE they tested it …

Slide 141

Slide 141 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE they tested it … but stubbed out one component.

Slide 142

Slide 142 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE they tested it … but stubbed out one component. that component was the one that broke.

Slide 143

Slide 143 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE you can’t a/b test a $370 million rocket the ariane failed in 36 seconds

Slide 144

Slide 144 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE testing will always be incomplete aim for recoverability

Slide 145

Slide 145 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 146

Slide 146 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE resilience

Slide 147

Slide 147 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE resilience recoverability

Slide 148

Slide 148 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE observability

Slide 149

Slide 149 text

they often couldn’t see the explorer

Slide 150

Slide 150 text

feedback is good engineering

Slide 151

Slide 151 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE when SRE is right it is great

Slide 152

Slide 152 text

bank

Slide 153

Slide 153 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile remember this bank?

Slide 154

Slide 154 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile we’re responsible for stability of the front end remember this bank?

Slide 155

Slide 155 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE team mainframe team mobile the ambassador we’re responsible for stability of the front end we’re responsible for stability of the mainframe … as long as it’s used correctly remember this bank?

Slide 156

Slide 156 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE another department … web front-end back-end one team

Slide 157

Slide 157 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE another department … web front-end back-end one team mobile front-end

Slide 158

Slide 158 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE one team, range of techniques canary deploys CI/CD pipelines one team CI/CD pipelines big-bang deploys onto AIX

Slide 159

Slide 159 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE by the way …

Slide 160

Slide 160 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE big bang deploys

Slide 161

Slide 161 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE big bang deploys 50% failure rate

Slide 162

Slide 162 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE canary deploys big bang deploys 50% failure rate

Slide 163

Slide 163 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE 10% failure rate canary deploys big bang deploys 50% failure rate

Slide 164

Slide 164 text

industrial

Slide 165

Slide 165 text

remember the suspicious DBAs?

Slide 166

Slide 166 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE two root problems: • automation • trust and transparency

Slide 167

Slide 167 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE trigger automation via slack

Slide 168

Slide 168 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE because it was transparent, DBAs were happy and automated more things

Slide 169

Slide 169 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE what happens when things go wrong?

Slide 170

Slide 170 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE

Slide 171

Slide 171 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE leadership need to provide a safety net.

Slide 172

Slide 172 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE celebrate success celebrate failure

Slide 173

Slide 173 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE celebrate success celebrate learning

Slide 174

Slide 174 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE remember the why

Slide 175

Slide 175 text

@holly_cummins PREVAIL Technical Conference 2021 #WTFisSRE better, safer, faster, happier

Slide 176

Slide 176 text

thank you! #RedHat @holly_cummins