Slide 1

Slide 1 text

Channel your
 Fear of the Computer
 into confident engineering Maggie Zhou The Lead Developer New York 2018

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

todo: 
 photo of self Photo of sf Photo of slack

Slide 4

Slide 4 text

We rely on so many so?ware systems to build our products.

Slide 5

Slide 5 text

Change is required to make progress.

Slide 6

Slide 6 text

But sometimes things break photo here.

Slide 7

Slide 7 text

Comic: Alex Norris of webcomicname.com

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

Techniques for confidence • Measure your code. But how? • Ramp up your work incrementally. But why?

Slide 10

Slide 10 text

Case Study 1 How to modify a black box with confidence • Instrument the code to understand your baseline • Ramp up gradually and enable ramping back down

Slide 11

Slide 11 text

Instrument the code func connect_to_db(db_xyz): return connect(xyz) func connect_to_db(db_xyz): time = now() conn = connect(xyz) metric(now() - time, db_xyz) return conn

Slide 12

Slide 12 text

Feature Flags!

Slide 13

Slide 13 text

Why feature flags? • Gate user visible client-side features with server-side configuraOon • Do A/B tesOng • Merge code that isn’t ready for full-feature release • Incremental ramp-ups • Easy, well-socialized place to ramp down and turn it off

Slide 14

Slide 14 text

Why feature flags for infrastructure changes? • Gate user visible client-side features with server-side configuraOon • Do A/B tesOng • Merge code that isn’t ready for full-feature release • Incremental ramp-ups • Easy, well-socialized place to ramp down and turn it off

Slide 15

Slide 15 text

Build confidence From incremental ramp-ups and watching your metrics.

Slide 16

Slide 16 text

Ramp up incrementally func connect_to_db(db_xyz): time = now() conn = connect_old_way(xyz) metric(now() - time, db_xyz) return conn

Slide 17

Slide 17 text

Ramp up incrementally func connect_to_db(db_xyz): time = now() if (Feature::isOn(‘db_new’)): conn = connect_new_way(xyz) else: conn = connect_old_way(xyz) metric(now() - time, db_xyz) return conn 
 
 
 …
 … $config[‘db_new’] = array( ‘on’ => 1 
 # Ramp up to 1% of traffic ) …

Slide 18

Slide 18 text

Ramp up incrementally This triggered out of memory errors!
 
 Why?

Slide 19

Slide 19 text

Actually… A hidden recursive dependency on the ORM inside the Feature library!

Slide 20

Slide 20 text

Ramp up incrementally func connect_to_db(db_xyz): time = now() conn = connect_old_way(xyz) metric(now() - time, db_xyz) return conn

Slide 21

Slide 21 text

Ramp up incrementally func connect_to_db(db_xyz): time = now() if (host in rampup_list): conn = connect_new_way(xyz) else: conn = connect_old_way(xyz) metric(now() - time, db_xyz) return conn

Slide 22

Slide 22 text

Confidence From knowing 
 where the off-switch is.

Slide 23

Slide 23 text

Releasing to 100% should be boring.

Slide 24

Slide 24 text

Minimize user impact by making it easy to ramp down your change.

Slide 25

Slide 25 text

-Allison Kaptur, in ‘Love your Bugs’ “…if you’re wriOng a mobile or a desktop applicaOon: You need server-side feature gaOng and server-side flags. When you discover a problem and you don’t have server- side controls, the resoluOon might take days or weeks as you push out a new release or submit a new version to the app store. That’s a bad situaOon to be in.”

Slide 26

Slide 26 text

Confidence from: • Ramping up my change incrementally • Knowing how to ramp down my change quickly • Knowing that my team has my back and that I’ve socialized how to turn off my change. • Understanding my baseline performance and watching metrics as my change is released.

Slide 27

Slide 27 text

Case Study 2 Build confidence by understanding your changes. How? • Stare really hard at the code?

Slide 28

Slide 28 text

Case Study 2 Build confidence by understanding your changes. How? • Stare really hard at the code? • Reduce risk by dogfooding • Have analyOcs available just for your dogfood group with the changes you made • Measure your yardsOcks

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

one API endpoint ms

Slide 31

Slide 31 text

Segment your metrics by your ramp-up key to catch performance regressions early!

Slide 32

Slide 32 text

But what’s going on? Measurements at the data access layer showed no performance regressions.

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

-Carlos Buenos in Mature OpBmizaBon “Measurement so?ware is just as likely to have bugs as anything else we write, and we should take more care than usual. A bug in user facing code results in a bad experience. A bug in measurement code results in bad decisions.”

Slide 35

Slide 35 text

–Johnny Appleseed “Type a quote here.”

Slide 36

Slide 36 text

Make big infrastructure changes with confidence • Instrument your code. • Understand your baseline. • Check your yardsOcks. • Segment your metrics into populaOons that match your ramp-ups. • Ramp up incrementally • Make it easy and well-understood how to config your changes off.

Slide 37

Slide 37 text

Make big infrastructure changes with confidence.

Slide 38

Slide 38 text

Maggie Zhou The Lead Developer New York 2018 Resources: bit.ly/zmagg_leaddev2018 Find me on the Internet @zmagg ✨Thank you ✨ to my teams at Slack and Etsy for all the lessons learned. Special thanks to Kiran BhaSaram, Kiné Camara, Julia Evans, DureW Hirpa, Andrew Morrison, Bhaskar Mookerji, Tracy Stampfil, Meri Williams and Lydia Wagner for your early feedback and support on dra]s of this talk .