Tales after midnight @ webdeldn

Tales after midnight is an anthology of modern horror cautionary stories, including “Don’t deploy that code”, “The CSS of Doom”, “Polaroid” et al.

It’s really important to be ready, in case it was you the pager was tolling for, just after midnight.

“Based on true stories” – me
“Not your typical horror stories” – also me
“The tales are varied as they are interesting and it promises to entertain readers from all types of horror genres” – not me, but also not necessarily about this talk

Transcript

1. View Slide

2. for whom thE pagEr tolls
   TalEs aftEr midnight
   

   View Slide

3. for whom thE pagEr tolls
   TalEs aftEr midnight
   An anthology of short cautionary horror storiEs
   

   View Slide

4. TalEs aftEr Midnight
   @cEdmax
   WEbmastEr bEforE
   it was cool
   

   TEch LEad 

   Condé Nast IntErnational
   

   View Slide

5. TalEs aftEr midnight
   "ThErE arE 5 typEs of EnginEErs:

   ThosE that havE brokEn production 

   and thosE who arE about 

   to brEak production"
   Matt BoylE
   for whom thE pagEr tolls
   

   View Slide

6. ChaptEr #1
   Don't DEPLOY

   that CODE
   

   View Slide

7. Don't DEPLOY that CODE
   

   View Slide

8. Don't DEPLOY that CODE
   

   View Slide

9. Don't DEPLOY that CODE
   

   View Slide

10. Don't DEPLOY that CODE
    

    View Slide

11. Don't DEPLOY that CODE
    

    View Slide

12. Don't DEPLOY that CODE
    

    View Slide

13. Don't DEPLOY that CODE
    

    View Slide

14. - our rollback stratEgy
    WHAT WENT WELL
    - thE procEss was pronE to Errors
    ☠ WHAT WENT WRONG
    - AUTOMATE ALL THE THINGS
    What wE lEarnt
    Don't DEPLOY that CODE
    

    View Slide

15. ChaptEr #5
    THE CSS OF DOOM
    

    View Slide

16. http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg
    THE CSS OF DOOM
    

    View Slide

17. http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg
    THE CSS OF DOOM
    

    View Slide

18. http://www.winpix.net/content/images/2010/03/16_png_transparent_ie6.jpg
    THE CSS OF DOOM
    

    View Slide

19. img {
    behavior: url("pngbehavior.htc");
    }
    THE CSS OF DOOM
    

    View Slide

20. var supported = (/MSIE (5\.5)|[6789]/
    .test(navigator.userAgent)
    && navigator.platform == "Win32");

    var blankSrc = "blank.gif";
    if (supported) fixImage();
    THE CSS OF DOOM
    

    View Slide

21. function fixImage() {
    var src = element.src;
    if ( /\.png$/.test( src.toLowerCase() ) ) {
    element.src = blankSrc;
    element.runtimeStyle.filter = 

    "progid:DXImageTransform.Microsoft.AlphaImageLoader(src='"
    + src + "',sizingMethod='scale')";
    }
    }
    THE CSS OF DOOM
    

    View Slide

22. var supported = (/MSIE (5\.5)|[6789]/
    .test(navigator.userAgent)
    && navigator.platform == "Win32");

    var blankSrc = "blank.gif";
    if (supported) fixImage();
    THE CSS OF DOOM
    

    View Slide

23. THE CSS OF DOOM
    

    View Slide

24. THE CSS OF DOOM
    

    View Slide

25. - I havE a fun story to tEll

    - our rollback stratEgy
    WHAT WENT WELL
    - I don't EvEn know whErE to start:

    - using thE prod db for monitoring

    - assuming thE CSS CAn't BREAK anything

    - siloEd & BlamE culturE
    ☠ WHAT WENT WRONG
    - You nEEd to undErstand thE systEm
    What I lEarnt
    THE CSS OF DOOM
    

    View Slide

26. ChaptEr #3
    POLAROID
    

    View Slide

27. Replica
    Replica
    Polaroid
    

    View Slide

28. Replica
    Replica
    Polaroid
    

    View Slide

29. Polaroid
    

    View Slide

30. Polaroid
    

    View Slide

31. - Run books didn't havE anything on thE issuE
    - ThE support Escalation paths wErE wrong
    - WE FAILED TO COMMUNICATE THE FIX
    Polaroid
    

    View Slide

32. - wE CAUGHT THE PROBLEM BEFORE IT
    BECAME CRITICAL
    WHAT WENT WELL
    - Monitoring wasn't sEtup for thE rEplicas

    - ThE runbooks wErEn't updatEd
    ☠ WHAT WENT WRONG
    - you can't control EvErything

    - communication is KEy
    What wE lEarnt
    Polaroid
    

    View Slide

33. AppEndix
    POST MORTEM
    

    View Slide

34. Post MortEm
    - Summary
    THE DOCUMENT
    

    View Slide

35. - Summary
    - TimElinE
    THE DOCUMENT
    Post MortEm
    

    View Slide

36. - Summary
    - TimElinE
    - Impact
    THE DOCUMENT
    Post MortEm
    

    View Slide

37. - Summary
    - TimElinE
    - Impact
    - Diagnosis
    THE DOCUMENT
    Post MortEm
    

    View Slide

38. - Summary
    - TimElinE
    - Impact
    - Diagnosis
    - Actions
    THE DOCUMENT
    Post MortEm
    

    View Slide

39. - KEEp thE mEEting small
    - KEEp it tEchnical
    - Focus on thE whys?
    - MakE surE thE actions arE rEalistic
    - KEEp it blamElEss
    SUGGESTIONS
    Post MortEm
    

    View Slide

40. TakEaways
    - Sh*t will go wrong, BE prEparEd
    - KEEp your docs updatEd
    - Try to undErstand your APP/STACK
    - Don't BLAME PEOPLE BUT PROCESSES
    for whom thE pagEr tolls
    TalEs aftEr midnight
    

    View Slide