Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Tales after midnight @ webdeldn

Tales after midnight @ webdeldn

Tales after midnight is an anthology of modern horror cautionary stories, including “Don’t deploy that code”, “The CSS of Doom”, “Polaroid” et al.

It’s really important to be ready, in case it was you the pager was tolling for, just after midnight.

“Based on true stories” – me
“Not your typical horror stories” – also me
“The tales are varied as they are interesting and it promises to entertain readers from all types of horror genres” – not me, but also not necessarily about this talk

Marco Cedaro

January 29, 2019

More Decks by Marco Cedaro

Other Decks in Technology


  1. TalEs aftEr Midnight @cEdmax WEbmastEr bEforE it was cool 

    TEch LEad 
 Condé Nast IntErnational
  2. TalEs aftEr midnight "ThErE arE 5 typEs of EnginEErs:

    that havE brokEn production 
 and thosE who arE about 
 to brEak production" Matt BoylE for whom thE pagEr tolls
  3. - our rollback stratEgy WHAT WENT WELL - thE procEss

    was pronE to Errors ☠ WHAT WENT WRONG - AUTOMATE ALL THE THINGS What wE lEarnt Don't DEPLOY that CODE
  4. var supported = (/MSIE (5\.5)|[6789]/ .test(navigator.userAgent) && navigator.platform == "Win32");

    var blankSrc = "blank.gif"; if (supported) fixImage(); THE CSS OF DOOM
  5. function fixImage() { var src = element.src; if ( /\.png$/.test(

    src.toLowerCase() ) ) { element.src = blankSrc; element.runtimeStyle.filter = 
 "progid:DXImageTransform.Microsoft.AlphaImageLoader(src='" + src + "',sizingMethod='scale')"; } } THE CSS OF DOOM
  6. var supported = (/MSIE (5\.5)|[6789]/ .test(navigator.userAgent) && navigator.platform == "Win32");

    var blankSrc = "blank.gif"; if (supported) fixImage(); THE CSS OF DOOM
  7. - I havE a fun story to tEll
 - our

    rollback stratEgy WHAT WENT WELL - I don't EvEn know whErE to start:
 - using thE prod db for monitoring
 - assuming thE CSS CAn't BREAK anything
 - siloEd & BlamE culturE ☠ WHAT WENT WRONG - You nEEd to undErstand thE systEm What I lEarnt THE CSS OF DOOM
  8. - Run books didn't havE anything on thE issuE -

    ThE support Escalation paths wErE wrong - WE FAILED TO COMMUNICATE THE FIX Polaroid

    WENT WELL - Monitoring wasn't sEtup for thE rEplicas
 - ThE runbooks wErEn't updatEd ☠ WHAT WENT WRONG - you can't control EvErything
 - communication is KEy What wE lEarnt Polaroid
  10. - KEEp thE mEEting small - KEEp it tEchnical -

    Focus on thE whys? - MakE surE thE actions arE rEalistic - KEEp it blamElEss SUGGESTIONS Post MortEm
  11. TakEaways - Sh*t will go wrong, BE prEparEd - KEEp

    your docs updatEd - Try to undErstand your APP/STACK - Don't BLAME PEOPLE BUT PROCESSES for whom thE pagEr tolls TalEs aftEr midnight