Fear Driven Development

Transcript

1. Fear Driven Development Confronting our inner daemons this is a talk I’ve wanted to give since noticing a lot of what core services did while I ran it was dictated by fear (mostly of BDB/partie)…it struck me that fear caused a lot of unnecessary paralysis, which could be avoided if we considered how to confront fear in software development
2. What is fear? A feeling of agitation and anxiety caused by the presence of imminent danger the recognition of a perceived threat…it’s a survival mechanism that initiates our fight-or-flight response
3. What causes fear? most people know what fear is but perhaps don’t know what actually causes it
4. Specific fears develop as a result of learning fear can be acquired by a frightening traumatic accident…if you were bitten by a dog as a child, it would be natural for you to subsequently fear dogs
5. Certain fears are more common than others (e.g. animals, heights) Also easier to induce in the laboratory…a phenomenon is known as preparedness. Because early humans that were quick to fear dangerous situations were more likely to survive and reproduce, preparedness is theorized to be a genetic effect that is the result of natural selection.
6. Negative emotions elicit a stronger reaction than positive ones
7. Emotional contagion the tendency for two individuals to emotionally converge if you hang out with people who are afraid of the dark you’re likely to become afraid of the dark
8. Fear is contagious We learn fear regardless of whether we have experienced trauma or if we have observed the fear in others if you see a surfer get eaten by a shark, you’re going to be scared of sharks
9. Negative emotions are usually more infectious than positive ones people with negative emotions may be more likely to drag down people with positive emotions
10. This is terrifying for a large engineering organization negative emotions, like fear, elicit a stronger reaction and are more contagious than positive emotions…in larger organizations it’s like trying to stop the flu from making the rounds at the office
11. Fear is additive • you can be scared of the dark • you can be scared of the bogeyman • if you fear both…how does the bogeyman in the dark make you feel? you’re afraid of sharks…you’re afraid of water…SHARKS IN THE WATER…FUCK THE BEACH! you’re afraid of dogs…you’re afraid of spiders…DOGS WITH 8 LEGS!
12. Fear compounds even small fears left to linger over long periods of time can become large fears urban legends run amok someone dies in a tunnel…cause unknown…mystery case…HAUNTED CAVE
13. What are humans afraid of? Death, taxes, unpredictability …
14. PUBLIC SPEAKING
15. I fear databases Ironically I was paid to build one here databases are terrifying…they’re complicated and we call on them to store things that would put us out of business if we lost them…we put them on pedestals, believe only experts can build them ! I fear other things as well, but my experience working on partie developed most of my thoughts around fear and software development
16. What does Yammer fear? why are we actually here? why am I giving this talk?
17. Refactoring how many ways can we convince ourselves out of breaking up workfeed? how long have feedie/flatterie lingered on Scala?
18. Performance raise your hand if something you did backed up one or more important queues
19. Scale raise your hand if you've taken the site down by directing too much traffic at something
20. Upgrades who wants to go upgrade BDB with me after this? who's running database migrations today? what about backfills?
21. Deploying basically roll up every fear we just discussed and experience them all at once
22. So what? What's the downside of fear in software development? so what? we’re afraid. is it actually causing problems?
23. Fear induced loss of agility Analysis paralysis looking for the best/safest solution BDB upgrades in core services decomposing work feed
24. Fear erects barricades around the code and systems that need the most attention Fear will keep you from improving things…"that shit's scary, DON'T GO IN THERE!" feedie’s data model workfeed’s data model artie’s ability to DDoS the site
25. Fear creates local maximums you won't explore areas you're afraid of, even if that's where the biggest gains are to be had
26. How I’ve dealt with my fears if I came only with problems and no solutions then this would be a terrible talk
27. Exposure therapy exposure to the feared object or context, without any danger, in order to overcome anxiety
28. Testing as a safety harness back in the day, partie was the best tested code in the company a broken test is cheap and won't hurt anyone/anything most importantly, I had a series of tests that broke BDB every way I could…tests verified that BDB behaved how I expected and that partie could at least identify the failure and potentially fix it
29. Development and staging are your safe zones I broke partie in every way I could conceive before it left my computer…and then I put it back together reduce things to a smaller stage…it’s learning on the practice field vs. learning in Yankee Stadium make a mistake in Yankee Stadium and get sent back to the minors…make a mistake on the practice field and you do the drill 10 more times
30. Build confidence on non-critical systems we did partie/BDB/Java upgrades on services like prankie and whoville first because we knew we could lose all the data and still be fine
31. Double dispatch when replacing existing functionality we kept writing feeds to the old postgresql messaging db while also writing to feedie we slowly increased read traffic to feedie, switching it back when problems arose durable queues meant we could take feedie down entirely without losing deliveries even after switching to 100% reads on feedie, we kept double dispatching for months because we could
32. Hack Day! it's low risk, even if you can't put it back together you still had fun
33. Sharing my fears Someone else has probably seen or done this or something like it before I got in touch with the BDB developers, they reassured you there's nothing to be afraid of
34. Organizational selective amnesia Let the new person do it (Hi, Vivek!) they don't know that repo/library/database is haunted if things aren’t really to be scared, having someone who isn’t scared is invaluable
35. Decatastrophizing "If the feared event or object happened, what would occur then?" partie's 3-way replication and quorum elections/acks was tremendous for my confidence…once I'd seen it work with my own eyes
36. How can Yammer deal with its fear?
37. Make the time to test catch problems early nobody's scared of well tested code
38. Do your research ahead of time read up on things and perform actual research…like in a laboratory we suck at doing and scheduling research…we have to get better at it
39. More hack day projects focusing on the things we fear get under that hood and start yanking cables…is it as bad as we've made it out to be? best case…it's not bad and we can ship whatever you do worst case…it's bad, but now we know a lot more about the bad
40. None of this should impact your personal life we need to set up the structures that make this part of the expected day-to-day experience heroes can do this today, but heroes aren't sustainable
41. Don't let anyone bully you out of what you feel to be adequate testing or research it’s up to us to address the fear, otherwise we’re going to live in fear forever
42. Ending the vicious cycle requires us to confront our biggest fear… authority we are all empowered to address this fear as we encounter and experience it…if you feel otherwise, remember…engineers have the numbers
43. Questions?