Slide 1
Slide 1 text
No content
Slide 2
Slide 2 text
for whom thE pagEr tolls
TalEs aftEr midnight
Slide 3
Slide 3 text
for whom thE pagEr tolls
TalEs aftEr midnight
An anthology of short cautionary horror storiEs
Slide 4
Slide 4 text
TalEs aftEr Midnight
@cEdmax
WEbmastEr bEforE
it was cool
TEch LEad
Condé Nast IntErnational
Slide 5
Slide 5 text
TalEs aftEr midnight
"ThErE arE 5 typEs of EnginEErs:
ThosE that havE brokEn production
and thosE who arE about
to brEak production"
Matt BoylE
for whom thE pagEr tolls
Slide 6
Slide 6 text
ChaptEr #1
Don't DEPLOY
that CODE
Slide 7
Slide 7 text
Don't DEPLOY that CODE
Slide 8
Slide 8 text
Don't DEPLOY that CODE
Slide 9
Slide 9 text
Don't DEPLOY that CODE
Slide 10
Slide 10 text
Don't DEPLOY that CODE
Slide 11
Slide 11 text
Don't DEPLOY that CODE
Slide 12
Slide 12 text
Don't DEPLOY that CODE
Slide 13
Slide 13 text
Don't DEPLOY that CODE
Slide 14
Slide 14 text
- our rollback stratEgy
WHAT WENT WELL
- thE procEss was pronE to Errors
☠ WHAT WENT WRONG
- AUTOMATE ALL THE THINGS
What wE lEarnt
Don't DEPLOY that CODE
Slide 15
Slide 15 text
ChaptEr #5
THE CSS OF DOOM
Slide 16
Slide 16 text
http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg
THE CSS OF DOOM
Slide 17
Slide 17 text
http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg
THE CSS OF DOOM
Slide 18
Slide 18 text
http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg
THE CSS OF DOOM
Slide 19
Slide 19 text
img {
behavior: url("pngbehavior.htc");
}
THE CSS OF DOOM
Slide 20
Slide 20 text
var supported = (/MSIE (5\.5)|[6789]/
.test(navigator.userAgent)
&& navigator.platform == "Win32");
var blankSrc = "blank.gif";
if (supported) fixImage();
THE CSS OF DOOM
Slide 21
Slide 21 text
function fixImage() {
var src = element.src;
if ( /\.png$/.test( src.toLowerCase() ) ) {
element.src = blankSrc;
element.runtimeStyle.filter =
"progid:DXImageTransform.Microsoft.AlphaImageLoader(src='"
+ src + "',sizingMethod='scale')";
}
}
THE CSS OF DOOM
Slide 22
Slide 22 text
var supported = (/MSIE (5\.5)|[6789]/
.test(navigator.userAgent)
&& navigator.platform == "Win32");
var blankSrc = "blank.gif";
if (supported) fixImage();
THE CSS OF DOOM
Slide 23
Slide 23 text
THE CSS OF DOOM
Slide 24
Slide 24 text
THE CSS OF DOOM
Slide 25
Slide 25 text
- I havE a fun story to tEll
- our rollback stratEgy
WHAT WENT WELL
- I don't EvEn know whErE to start:
- using thE prod db for monitoring
- assuming thE CSS CAn't BREAK anything
- siloEd & BlamE culturE
☠ WHAT WENT WRONG
- You nEEd to undErstand thE systEm
What I lEarnt
THE CSS OF DOOM
Slide 26
Slide 26 text
ChaptEr #3
POLAROID
Slide 27
Slide 27 text
Replica
Replica
Polaroid
Slide 28
Slide 28 text
Replica
Replica
Polaroid
Slide 29
Slide 29 text
Polaroid
Slide 30
Slide 30 text
Polaroid
Slide 31
Slide 31 text
- Run books didn't havE anything on thE issuE
- ThE support Escalation paths wErE wrong
- WE FAILED TO COMMUNICATE THE FIX
Polaroid
Slide 32
Slide 32 text
- wE CAUGHT THE PROBLEM BEFORE IT
BECAME CRITICAL
WHAT WENT WELL
- Monitoring wasn't sEtup for thE rEplicas
- ThE runbooks wErEn't updatEd
☠ WHAT WENT WRONG
- you can't control EvErything
- communication is KEy
What wE lEarnt
Polaroid
Slide 33
Slide 33 text
AppEndix
POST MORTEM
Slide 34
Slide 34 text
Post MortEm
- Summary
THE DOCUMENT
Slide 35
Slide 35 text
- Summary
- TimElinE
THE DOCUMENT
Post MortEm
Slide 36
Slide 36 text
- Summary
- TimElinE
- Impact
THE DOCUMENT
Post MortEm
Slide 37
Slide 37 text
- Summary
- TimElinE
- Impact
- Diagnosis
THE DOCUMENT
Post MortEm
Slide 38
Slide 38 text
- Summary
- TimElinE
- Impact
- Diagnosis
- Actions
THE DOCUMENT
Post MortEm
Slide 39
Slide 39 text
- KEEp thE mEEting small
- KEEp it tEchnical
- Focus on thE whys?
- MakE surE thE actions arE rEalistic
- KEEp it blamElEss
SUGGESTIONS
Post MortEm
Slide 40
Slide 40 text
TakEaways
- Sh*t will go wrong, BE prEparEd
- KEEp your docs updatEd
- Try to undErstand your APP/STACK
- Don't BLAME PEOPLE BUT PROCESSES
for whom thE pagEr tolls
TalEs aftEr midnight