Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Fear Driven Development

B940ce33bce9d6874882f34685d987e0?s=47 Ryan Kennedy
January 28, 2014

Fear Driven Development


Ryan Kennedy

January 28, 2014


  1. Fear Driven Development Confronting our inner daemons this is a

    talk I’ve wanted to give since noticing a lot of what core services did while I ran it was dictated by fear (mostly of BDB/partie)…it struck me that fear caused a lot of unnecessary paralysis, which could be avoided if we considered how to confront fear in software development
  2. What is fear? A feeling of agitation and anxiety caused

    by the presence of imminent danger the recognition of a perceived threat…it’s a survival mechanism that initiates our fight-or-flight response
  3. What causes fear? most people know what fear is but

    perhaps don’t know what actually causes it
  4. Specific fears develop as a result of learning fear can

    be acquired by a frightening traumatic accident…if you were bitten by a dog as a child, it would be natural for you to subsequently fear dogs
  5. Certain fears are more common than others (e.g. animals, heights)

    Also easier to induce in the laboratory…a phenomenon is known as preparedness. Because early humans that were quick to fear dangerous situations were more likely to survive and reproduce, preparedness is theorized to be a genetic effect that is the result of natural selection.
  6. Negative emotions elicit a stronger reaction than positive ones

  7. Emotional contagion the tendency for two individuals to emotionally converge

    if you hang out with people who are afraid of the dark you’re likely to become afraid of the dark
  8. Fear is contagious We learn fear regardless of whether we

    have experienced trauma or if we have observed the fear in others if you see a surfer get eaten by a shark, you’re going to be scared of sharks
  9. Negative emotions are usually more infectious than positive ones people

    with negative emotions may be more likely to drag down people with positive emotions
  10. This is terrifying for a large engineering organization negative emotions,

    like fear, elicit a stronger reaction and are more contagious than positive emotions…in larger organizations it’s like trying to stop the flu from making the rounds at the office
  11. Fear is additive • you can be scared of the

    dark • you can be scared of the bogeyman • if you fear both…how does the bogeyman in the dark make you feel? you’re afraid of sharks…you’re afraid of water…SHARKS IN THE WATER…FUCK THE BEACH! you’re afraid of dogs…you’re afraid of spiders…DOGS WITH 8 LEGS!
  12. Fear compounds even small fears left to linger over long

    periods of time can become large fears urban legends run amok someone dies in a tunnel…cause unknown…mystery case…HAUNTED CAVE
  13. What are humans afraid of? Death, taxes, unpredictability …


  15. I fear databases Ironically I was paid to build one

    here databases are terrifying…they’re complicated and we call on them to store things that would put us out of business if we lost them…we put them on pedestals, believe only experts can build them ! I fear other things as well, but my experience working on partie developed most of my thoughts around fear and software development
  16. What does Yammer fear? why are we actually here? why

    am I giving this talk?
  17. Refactoring how many ways can we convince ourselves out of

    breaking up workfeed? how long have feedie/flatterie lingered on Scala?
  18. Performance raise your hand if something you did backed up

    one or more important queues
  19. Scale raise your hand if you've taken the site down

    by directing too much traffic at something
  20. Upgrades who wants to go upgrade BDB with me after

    this? who's running database migrations today? what about backfills?
  21. Deploying basically roll up every fear we just discussed and

    experience them all at once
  22. So what? What's the downside of fear in software development?

    so what? we’re afraid. is it actually causing problems?
  23. Fear induced loss of agility Analysis paralysis looking for the

    best/safest solution BDB upgrades in core services decomposing work feed
  24. Fear erects barricades around the code and systems that need

    the most attention Fear will keep you from improving things…"that shit's scary, DON'T GO IN THERE!" feedie’s data model workfeed’s data model artie’s ability to DDoS the site
  25. Fear creates local maximums you won't explore areas you're afraid

    of, even if that's where the biggest gains are to be had
  26. How I’ve dealt with my fears if I came only

    with problems and no solutions then this would be a terrible talk
  27. Exposure therapy exposure to the feared object or context, without

    any danger, in order to overcome anxiety
  28. Testing as a safety harness back in the day, partie

    was the best tested code in the company a broken test is cheap and won't hurt anyone/anything most importantly, I had a series of tests that broke BDB every way I could…tests verified that BDB behaved how I expected and that partie could at least identify the failure and potentially fix it
  29. Development and staging are your safe zones I broke partie

    in every way I could conceive before it left my computer…and then I put it back together reduce things to a smaller stage…it’s learning on the practice field vs. learning in Yankee Stadium make a mistake in Yankee Stadium and get sent back to the minors…make a mistake on the practice field and you do the drill 10 more times
  30. Build confidence on non-critical systems we did partie/BDB/Java upgrades on

    services like prankie and whoville first because we knew we could lose all the data and still be fine
  31. Double dispatch when replacing existing functionality we kept writing feeds

    to the old postgresql messaging db while also writing to feedie we slowly increased read traffic to feedie, switching it back when problems arose durable queues meant we could take feedie down entirely without losing deliveries even after switching to 100% reads on feedie, we kept double dispatching for months because we could
  32. Hack Day! it's low risk, even if you can't put

    it back together you still had fun
  33. Sharing my fears Someone else has probably seen or done

    this or something like it before I got in touch with the BDB developers, they reassured you there's nothing to be afraid of
  34. Organizational selective amnesia Let the new person do it (Hi,

    Vivek!) they don't know that repo/library/database is haunted if things aren’t really to be scared, having someone who isn’t scared is invaluable
  35. Decatastrophizing "If the feared event or object happened, what would

    occur then?" partie's 3-way replication and quorum elections/acks was tremendous for my confidence…once I'd seen it work with my own eyes
  36. How can Yammer deal with its fear?

  37. Make the time to test catch problems early nobody's scared

    of well tested code
  38. Do your research ahead of time read up on things

    and perform actual research…like in a laboratory we suck at doing and scheduling research…we have to get better at it
  39. More hack day projects focusing on the things we fear

    get under that hood and start yanking cables…is it as bad as we've made it out to be? best case…it's not bad and we can ship whatever you do worst case…it's bad, but now we know a lot more about the bad
  40. None of this should impact your personal life we need

    to set up the structures that make this part of the expected day-to-day experience heroes can do this today, but heroes aren't sustainable
  41. Don't let anyone bully you out of what you feel

    to be adequate testing or research it’s up to us to address the fear, otherwise we’re going to live in fear forever
  42. Ending the vicious cycle requires us to confront our biggest

    fear… authority we are all empowered to address this fear as we encounter and experience it…if you feel otherwise, remember…engineers have the numbers
  43. Questions?