Leveraging Failure

Leveraging Failure

This is a talk about constructing good stories around failures to maximize retention and improve critical thinking.


Theo Schlossnagle

January 30, 2014


  1. Theo Schlossnagle @postwait postwait

  2. Date Venue Title 1/30/2014 Monkigras Leveraging Failure 12/4/2013 Nodesummit Moderator:

    two panels 11/14/201 3 Velocity EU (O'Reilly) The Business Bits of Web Operations 11/6/2013 Usenix LISA '13 Observing and Understanding Behavior in Complex Systems 9/13/2013 Surge 2013 Closing Plenary 8/20/2013 Velocity China 2013 (O'Reilly) Simple, common performance tips 6/18/2013 Velocity 2013 (O'Rielly) Monitoring and Observability 5/23/2013 AppNexus Engineering at Scale The ins and outs of shipping a scalable product 4/25/2013 #ChefConf 2013 The InstallShield of the 21st Century 2/26/2013 ApacheCon NA 2013 Scaling: Lessons Learned and Their Applications to Apache Culture 12/4/2012 Velocity China (O'Reilly) Conference Chair 10/3/2012 Velocity EU (O'Reilly) Monitoring and Observability 9/28/2012 Surge 2012 Xtreme Deployment 6/26/2012 Velocity 2012 (O'Reilly) It's all about telemetry 5/16/2012 #ChefConf How can monitoring embrace the future of infrastructure automation? 4/3/2012 dtrace.conf 2012 Better USDT support in the Illumos toolchain 3/9/2012 QCon London Scalable Internet Architectures 3/1/2012 Strata 2012 (O'Reilly) Is this normal? Finding anomalies in real- time data. 2/24/2012 Nodesummit APIs & Endpoints of Scale 1/11/2012 Percona Live MySQL in the Cloud and Virtualized 12/7/2011 Velocity China (O'Reilly) Plenary: A Career in Web Operations 12/6/2011 Velocity China (O'Reilly) What's in a Number? 11/8/2011 Velocity EU (O'Reilly) Monitoring Is Easy, Why Are We So Bad At It? 11/8/2011 Velocity EU (O'Reilly) Plenary: A Career in Web Operations 10/12/201 1 Interact 2011 Organization-wide Monitoring 10/6/2011 Monktoberfest Social Improvements in Monitoring 9/29/2011 Surge 2011 Closing Plenary 9/29/2011 Surge 2011 Architectures for Real-time Data 9/15/2011 nycdevops Meetup What's in a Number? 7/25/2011 OSCON 2011 (O'Reilly) Esperwhispering: get your real-time data game on 6/16/2011 Velocity 2011 (O'Reilly) Actionable Web Performance for Operations 6/15/2011 Velocity 2011 (O'Reilly) Plenary: Career Development 5/26/2011 What's Next 2011 Decoupled Systems 4/30/2011 PICC 2011 Keynote: be a better engineer 4/27/2011 Russian Internet Technology 2011 Asynchronous Queueing Systems 4/25/2011 Russian Internet Technology 2011 High performance web edge systems 3/24/2011 PgEast 2011 Esperwhispering: Using Esper to Find Problems in Real-time Data 3/23/2011 GigaOm Structure BigData 2011 Master Data Wranglers Panel 3/10/2011 Big Web meets Big IT Infrastructure Management Panel 2/2/2011 Strata 2011 (O'Reilly) Esperwhispering: Using Esper to Find Date Venue Title 10/1/2010 Surge 2010 Closing Plenary 9/30/2010 Surge 2010 Scalable Design Patterns 8/2/2010 MilOSS-2 Big Bad PostgreSQL: A Case Study 7/23/2010 OSCON 2010 (O'Reilly) Reconnoiter: Monitoring and Trend Analysis 7/20/2010 OSCON 2010 (O'Reilly) Scalable Internet Architectures 6/25/2010 DevOpsDays (LinkedIn) Infrastructure as Code Panel 6/22/2010 Velocity 2010 (O'Reilly) Scalable Internet Architectures 11/20/200 9 LinkedIn.com Tech Talk Scalable Internet Architectures 11/5/2009 ApacheCon US 2009 Scalable Internet Architectures 7/22/2009 OSCON 2009 (O'Reilly) Reconnoiter: Monitoring and Trend Analysis 7/21/2009 OSCON 2009 (O'Reilly) Scalable Internet Architectures 11/5/2008 ApacheCon US 2007 Apache Projects on DTrace 11/3/2008 ApacheCon US 2007 Advanced Production Troubleshooting 7/23/2008 OSCON 2008 (O'Reilly) Full-stack Introspection Crash Course 3/14/2008 dtrace.conf 2008 PostgreSQL: Looking under the hood with Solaris 11/12/200 7 ApacheCon US 2007 Scalable Internet Architectures 7/25/2007 OSCON 2007 (O'Reilly) Advanced Production Troubleshooting 7/23/2007 OSCON 2007 (O'Reilly) Scalable Internet Architectures 10/9/2006 ApacheCon US 2006 Scalable Internet Architectures 7/26/2006 OSCON 2006 (O'Reilly) Big Bad PostgreSQL: A Case Study 7/24/2006 OSCON 2006 (O'Reilly) Scalable Internet Architectures 6/27/2006 ApacheCon EU 2006 Scalable Internet Architectures 6/26/2006 ApacheCon EU 2006 Advanced Production Troubleshooting 12/12/200 5 ApacheCon US 2005 Clustered Logging with mod_log_spread 12/10/200 5 ApacheCon US 2005 Scalable Internet Architectures 10/17/200 5 EuroOSCON 2005 (O'Reilly) Scalable Internet Architectures 8/3/2005 OSCON 2005 (O'Reilly) MySQL Snapshots and Replication from Oracle 8/1/2005 OSCON 2005 (O'Reilly) Scalable Internet Architectures 7/19/2005 ApacheCon EU 2005 Scalable Internet Architectures 11/17/200 4 ApacheCon US 2004 Clustered Logging with mod_log_spread 11/14/200 4 ApacheCon US 2004 Scalable Internet Architectures 7/29/2004 OSCON 2004 (O'Reilly) Wackamole: Open Source High Availability 11/16/200 3 ApacheCon US 2003 Scalable Internet Architectures 11/19/200 2 ApacheCon US 2002 Scalable Internet Architectures 11/18/200 2 ApacheCon US 2002 Backhand: understanding and building HA/LB clusters 4/6/2001 ApacheCon US 2001 Scalable Internet Architectures 4/5/2001 ApacheCon US 2001 mod_backhand: internals explained 4/3/2001 iEB2001 Managing Large Mass Email Delivery Systems 10/23/200 0 ApacheCon EU 2000 The Backhand Project: load-balancing and monitoring Apache web clusters 3/9/2000 ApacheCon US 2000 mod_backhand: a load balancing module
  3. rant topics • Literally a grouchy,
 sailor-software-engineer • Best

    title thus far:
 “Software Proctologist”
  4. martial arts theme • Bujinkan Budo Taijutsu • Soke Masaaki

  5. ranting on ranting there is an art to ranting it

    can either be poignant with lasting value or drivel leaving listeners thinking you insane
  6. software engineers all learn in different ways

  7. because people all learn in different ways

  8. organizations there be dichotomies in there

  9. interruptions are costly

  10. interruptions are priceless

  11. so similarly two-sided open office environments transparency development methodologies micromanaging

  12. the best environments recognize the appropriate reagents and add them

    carefully but they must be mixed
  13. this is management

  14. the mixing of reagents is an art combining passion, disruption,

    unreasonable demands, high praise, celebration of success, and a culture of leveraging failure
  15. this is leadership

  16. I’ve built 3 companies and consulted with hundreds more

  17. each believes that their approach to all these things is

  18. each is different

  19. despite differences they all must instill a culture of continual

    learning and enable tactical measures that educate
  20. methods galore tech talks training sessions conferences user groups meetups

    book clubs mentorship
  21. the best learning transpires from failure

  22. leveraging failure in a technology world

  23. experience adage making good decisions comes from experience experience comes

    from making bad decisions
  24. failure is expensive

  25. wonderful learning can happen when failures meet good storytelling

  26. storytelling ≯ fiction stories teach better so lies are much

  27. technical storytelling is an art just like any form of

  28. epic tales are useful, but require practice and a venue

    (sorta like this one, despite this not epical tale)
  29. epic tales let’s get back to those

  30. things tend to change when those that can influence

    longer find them acceptable
  31. unacceptable …that word…
 I do not think it means what

    you think it means
  32. driven by either desperation or passion

  33. driven by either desperation or passion spoiler: always aim for

    this one
  34. driven by either desperation or passion spoiler: accept this with

  35. storytelling you should build your stories of despair differently than

    your stories of passion
  36. stories of passion are simple but turn out to be

    full of less valuable lessons
  37. stories of despair are the heart of darkness ! “it

    could be that your sole purpose in life
 is to serve as an example to others.”
  38. stories of despair are the heart of darkness ! “it

    could be that your sole purpose in life
 is to serve as an example to others.”
  39. constructing dissonance is where good storytelling does its heavy lifting

  40. no one pursues failure so framing your story dissonance articulated

    deprives your listener of the journey (and learning)
  41. frame your problem brief and to the point provide a

    topic many details should transpire with the solution ! some facts were unavailable at the journey embarkation
  42. develop the solution take the audience through your problem-solving

  43. most solutions are the organic composition of many smaller solutions

  44. small solutions developed at different points on the timeline of

    your understanding of the problem
  45. catastrophically missing the actual problem ! resulting in…

  46. despair from which real and valid solutions emerge and truly

    valuable learning experiences take place
  47. ultimate tragedy is not supremely educational it requires too much

    “lesson extraction” from the listener
  48. penultimate tragedy provides a context of success around lessons learned

    ! dissonance… and repose
  49. repose shit works great don’t dwell on this take a

    bow and exit stage left