Fear Driven Development

Slide 1

Slide 1 text

Fear Driven Development Confronting our inner daemons this is a talk I’ve wanted to give since noticing a lot of what core services did while I ran it was dictated by fear (mostly of BDB/partie)…it struck me that fear caused a lot of unnecessary paralysis, which could be avoided if we considered how to confront fear in software development

Slide 2

Slide 2 text

What is fear? A feeling of agitation and anxiety caused by the presence of imminent danger the recognition of a perceived threat…it’s a survival mechanism that initiates our ﬁght-or-ﬂight response

Slide 3

Slide 3 text

What causes fear? most people know what fear is but perhaps don’t know what actually causes it

Slide 4

Slide 4 text

Speciﬁc fears develop as a result of learning fear can be acquired by a frightening traumatic accident…if you were bitten by a dog as a child, it would be natural for you to subsequently fear dogs

Slide 5

Slide 5 text

Certain fears are more common than others (e.g. animals, heights) Also easier to induce in the laboratory…a phenomenon is known as preparedness. Because early humans that were quick to fear dangerous situations were more likely to survive and reproduce, preparedness is theorized to be a genetic effect that is the result of natural selection.

Slide 6

Slide 6 text

Negative emotions elicit a stronger reaction than positive ones

Slide 7

Slide 7 text

Emotional contagion the tendency for two individuals to emotionally converge if you hang out with people who are afraid of the dark you’re likely to become afraid of the dark

Slide 8

Slide 8 text

Fear is contagious We learn fear regardless of whether we have experienced trauma or if we have observed the fear in others if you see a surfer get eaten by a shark, you’re going to be scared of sharks

Slide 9

Slide 9 text

Negative emotions are usually more infectious than positive ones people with negative emotions may be more likely to drag down people with positive emotions

Slide 10

Slide 10 text

This is terrifying for a large engineering organization negative emotions, like fear, elicit a stronger reaction and are more contagious than positive emotions…in larger organizations it’s like trying to stop the ﬂu from making the rounds at the office

Slide 11

Slide 11 text

Fear is additive • you can be scared of the dark • you can be scared of the bogeyman • if you fear both…how does the bogeyman in the dark make you feel? you’re afraid of sharks…you’re afraid of water…SHARKS IN THE WATER…FUCK THE BEACH! you’re afraid of dogs…you’re afraid of spiders…DOGS WITH 8 LEGS!

Slide 12

Slide 12 text

Fear compounds even small fears left to linger over long periods of time can become large fears urban legends run amok someone dies in a tunnel…cause unknown…mystery case…HAUNTED CAVE

Slide 13

Slide 13 text

What are humans afraid of? Death, taxes, unpredictability …

Slide 14

Slide 14 text

PUBLIC SPEAKING

Slide 15

Slide 15 text

I fear databases Ironically I was paid to build one here databases are terrifying…they’re complicated and we call on them to store things that would put us out of business if we lost them…we put them on pedestals, believe only experts can build them ! I fear other things as well, but my experience working on partie developed most of my thoughts around fear and software development

Slide 16

Slide 16 text

What does Yammer fear? why are we actually here? why am I giving this talk?

Slide 17

Slide 17 text

Refactoring how many ways can we convince ourselves out of breaking up workfeed? how long have feedie/ﬂatterie lingered on Scala?

Slide 18

Slide 18 text

Performance raise your hand if something you did backed up one or more important queues

Slide 19

Slide 19 text

Scale raise your hand if you've taken the site down by directing too much traffic at something

Slide 20

Slide 20 text

Upgrades who wants to go upgrade BDB with me after this? who's running database migrations today? what about backﬁlls?

Slide 21

Slide 21 text

Deploying basically roll up every fear we just discussed and experience them all at once

Slide 22

Slide 22 text

So what? What's the downside of fear in software development? so what? we’re afraid. is it actually causing problems?

Slide 23

Slide 23 text

Fear induced loss of agility Analysis paralysis looking for the best/safest solution BDB upgrades in core services decomposing work feed

Slide 24

Slide 24 text

Fear erects barricades around the code and systems that need the most attention Fear will keep you from improving things…"that shit's scary, DON'T GO IN THERE!" feedie’s data model workfeed’s data model artie’s ability to DDoS the site

Slide 25

Slide 25 text

Fear creates local maximums you won't explore areas you're afraid of, even if that's where the biggest gains are to be had

Slide 26

Slide 26 text

How I’ve dealt with my fears if I came only with problems and no solutions then this would be a terrible talk

Slide 27

Slide 27 text

Exposure therapy exposure to the feared object or context, without any danger, in order to overcome anxiety

Slide 28

Slide 28 text

Testing as a safety harness back in the day, partie was the best tested code in the company a broken test is cheap and won't hurt anyone/anything most importantly, I had a series of tests that broke BDB every way I could…tests veriﬁed that BDB behaved how I expected and that partie could at least identify the failure and potentially ﬁx it

Slide 29

Slide 29 text

Development and staging are your safe zones I broke partie in every way I could conceive before it left my computer…and then I put it back together reduce things to a smaller stage…it’s learning on the practice ﬁeld vs. learning in Yankee Stadium make a mistake in Yankee Stadium and get sent back to the minors…make a mistake on the practice ﬁeld and you do the drill 10 more times

Slide 30

Slide 30 text

Build confidence on non-critical systems we did partie/BDB/Java upgrades on services like prankie and whoville first because we knew we could lose all the data and still be fine

Slide 31

Slide 31 text

Double dispatch when replacing existing functionality we kept writing feeds to the old postgresql messaging db while also writing to feedie we slowly increased read traffic to feedie, switching it back when problems arose durable queues meant we could take feedie down entirely without losing deliveries even after switching to 100% reads on feedie, we kept double dispatching for months because we could

Slide 32

Slide 32 text

Hack Day! it's low risk, even if you can't put it back together you still had fun

Slide 33

Slide 33 text

Sharing my fears Someone else has probably seen or done this or something like it before I got in touch with the BDB developers, they reassured you there's nothing to be afraid of

Slide 34

Slide 34 text

Organizational selective amnesia Let the new person do it (Hi, Vivek!) they don't know that repo/library/database is haunted if things aren’t really to be scared, having someone who isn’t scared is invaluable

Slide 35

Slide 35 text

Decatastrophizing "If the feared event or object happened, what would occur then?" partie's 3-way replication and quorum elections/acks was tremendous for my conﬁdence…once I'd seen it work with my own eyes

Slide 36

Slide 36 text

How can Yammer deal with its fear?

Slide 37

Slide 37 text

Make the time to test catch problems early nobody's scared of well tested code

Slide 38

Slide 38 text

Do your research ahead of time read up on things and perform actual research…like in a laboratory we suck at doing and scheduling research…we have to get better at it

Slide 39

Slide 39 text

More hack day projects focusing on the things we fear get under that hood and start yanking cables…is it as bad as we've made it out to be? best case…it's not bad and we can ship whatever you do worst case…it's bad, but now we know a lot more about the bad

Slide 40

Slide 40 text

None of this should impact your personal life we need to set up the structures that make this part of the expected day-to-day experience heroes can do this today, but heroes aren't sustainable

Slide 41

Slide 41 text

Don't let anyone bully you out of what you feel to be adequate testing or research it’s up to us to address the fear, otherwise we’re going to live in fear forever

Slide 42

Slide 42 text

Ending the vicious cycle requires us to confront our biggest fear… authority we are all empowered to address this fear as we encounter and experience it…if you feel otherwise, remember…engineers have the numbers

Slide 43

Slide 43 text

Questions?