Fear of the Computer

Fear of the Computer

Have you ever worked on a computer system that was so fragile it was frightening to make changes to? Maybe it was challenging to deploy, difficult to delete code, or changing one piece would cause surprising cascading failures.

3d7b72d70ff07f8186126a4464bc6166?s=128

Maggie Zhou

April 24, 2018
Tweet

Transcript

  1. 2.
  2. 8.
  3. 9.

    Techniques for confidence • Measure your code. But how? •

    Ramp up your work incrementally. But why?
  4. 10.

    Case Study 1 How to modify a black box with

    confidence • Instrument the code to understand your baseline • Ramp up gradually and enable ramping back down
  5. 11.

    Instrument the code func connect_to_db(db_xyz): return connect(xyz) func connect_to_db(db_xyz): time

    = now() conn = connect(xyz) metric(now() - time, db_xyz) return conn
  6. 13.

    Why feature flags? • Gate user visible client-side features with

    server-side configuraOon • Do A/B tesOng • Merge code that isn’t ready for full-feature release • Incremental ramp-ups • Easy, well-socialized place to ramp down and turn it off
  7. 14.

    Why feature flags for infrastructure changes? • Gate user visible

    client-side features with server-side configuraOon • Do A/B tesOng • Merge code that isn’t ready for full-feature release • Incremental ramp-ups • Easy, well-socialized place to ramp down and turn it off
  8. 16.

    Ramp up incrementally func connect_to_db(db_xyz): time = now() conn =

    connect_old_way(xyz) metric(now() - time, db_xyz) return conn
  9. 17.

    Ramp up incrementally func connect_to_db(db_xyz): time = now() if (Feature::isOn(‘db_new’)):

    conn = connect_new_way(xyz) else: conn = connect_old_way(xyz) metric(now() - time, db_xyz) return conn 
 
 
 …
 … $config[‘db_new’] = array( ‘on’ => 1 
 # Ramp up to 1% of traffic ) …
  10. 20.

    Ramp up incrementally func connect_to_db(db_xyz): time = now() conn =

    connect_old_way(xyz) metric(now() - time, db_xyz) return conn
  11. 21.

    Ramp up incrementally func connect_to_db(db_xyz): time = now() if (host

    in rampup_list): conn = connect_new_way(xyz) else: conn = connect_old_way(xyz) metric(now() - time, db_xyz) return conn
  12. 25.

    -Allison Kaptur, in ‘Love your Bugs’ “…if you’re wriOng a

    mobile or a desktop applicaOon: You need server-side feature gaOng and server-side flags. When you discover a problem and you don’t have server- side controls, the resoluOon might take days or weeks as you push out a new release or submit a new version to the app store. That’s a bad situaOon to be in.”
  13. 26.

    Confidence from: • Ramping up my change incrementally • Knowing

    how to ramp down my change quickly • Knowing that my team has my back and that I’ve socialized how to turn off my change. • Understanding my baseline performance and watching metrics as my change is released.
  14. 28.

    Case Study 2 Build confidence by understanding your changes. How?

    • Stare really hard at the code? • Reduce risk by dogfooding • Have analyOcs available just for your dogfood group with the changes you made • Measure your yardsOcks
  15. 29.
  16. 32.
  17. 33.
  18. 34.

    -Carlos Buenos in Mature OpBmizaBon “Measurement so?ware is just as

    likely to have bugs as anything else we write, and we should take more care than usual. A bug in user facing code results in a bad experience. A bug in measurement code results in bad decisions.”
  19. 36.

    Make big infrastructure changes with confidence • Instrument your code.

    • Understand your baseline. • Check your yardsOcks. • Segment your metrics into populaOons that match your ramp-ups. • Ramp up incrementally • Make it easy and well-understood how to config your changes off.
  20. 38.

    Maggie Zhou The Lead Developer New York 2018 Resources: bit.ly/zmagg_leaddev2018

    Find me on the Internet @zmagg ✨Thank you ✨ to my teams at Slack and Etsy for all the lessons learned. Special thanks to Kiran BhaSaram, Kiné Camara, Julia Evans, DureW Hirpa, Andrew Morrison, Bhaskar Mookerji, Tracy Stampfil, Meri Williams and Lydia Wagner for your early feedback and support on dra]s of this talk .