Engineering Change

A 10 minute talk I gave at the Brazil Innovators Conference in Sao Paulo. I was asked to cover a topic an entrepreneur could take home and use that day. I covered two.

Transcript

1. Engineering Change:
   Enable rapid experimentation
   Kellan Elliott-McCrea
   @kellan
   CTO, Etsy
   Thursday, May 17, 12
   

   View Slide

2. * A marketplace for handmade and vintage goods.
   In April 2012
   * 875,000 artisans and creative independents
   * $63.6 million USD in sales
   * 1.32 billion page views
   Thursday, May 17, 12
   

   View Slide

3. Engineering Change:
   * Continuous Deployment
   * Metrics driven development
   Thursday, May 17, 12
   

   View Slide

4. Engineering Change:
   * Make mistakes quickly
   * Learn from mistakes
   Thursday, May 17, 12
   

   View Slide

5. Engineering Change:
   * Continuous Deployment
   * Metrics driven development
   Thursday, May 17, 12
   

   View Slide

6. Continuous Deployment:
   a technique for
   rapid iteration
   Thursday, May 17, 12
   

   View Slide

7. Continuous Deployment:
   make failure cheap
   Thursday, May 17, 12
   

   View Slide

8. Continuous Deployment:
   small changes,
   pushed frequently
   Thursday, May 17, 12
   

   View Slide

9. Continuous Deployment:
   small changes,
   pushed frequently.
   To production.
   Thursday, May 17, 12
   

   View Slide

10. Continuous Deployment:
    10,068 releases to
    production in 2011
    @ Etsy
    Thursday, May 17, 12
    

    View Slide

11. Continuous Deployment:
    How?
    Thursday, May 17, 12
    

    View Slide

12. Continuous Deployment:
    No branching.
    “All existing revision control systems were
    built by people who build installed
    software”
    - Paul Hammond,
    Always Ship Trunk, Velocity 2010
    Thursday, March 17, 2011
    Thursday, May 17, 12
    

    View Slide

13. if ($cfg[‘awesome_new_search’]) {
    # new hotness
    $rsp = do_solr();
    } else {
    # boring old stuff
    $rsp = do_grep();
    }
    Continuous Deployment:
    feature flags
    Thursday, May 17, 12
    

    View Slide

14. Continuous Deployment:
    Ramp - ups
    (on top of feature flags)
    1. Launch to staff only
    2. Launch to 1% of all users
    3. Launch to members of a beta group
    Thursday, May 17, 12
    

    View Slide

15. Continuous Deployment:
    any engineer can launch a feature to
    1% of users
    Thursday, May 17, 12
    

    View Slide

16. Continuous Deployment:
    ~200 experiments
    live right now
    Thursday, May 17, 12
    

    View Slide

17. Metrics driven development:
    optimize for learning
    Thursday, May 17, 12
    

    View Slide

18. Metrics driven development:
    measure everything
    Thursday, May 17, 12
    

    View Slide

19. Metrics driven development:
    getting started:
    make it easy
    Thursday, May 17, 12
    

    View Slide

20. Metrics driven development:
    measure everything:
    logs, monitoring, A/B, graphs,
    StatsD
    Thursday, May 17, 12
    

    View Slide

21. Metrics driven development:
    StatsD @
    Instagram, Pinterest, Github,
    Mozilla, LAN.com, Zynga,
    Kickstarter, LivingSocial and 70+
    other companies
    Thursday, May 17, 12
    

    View Slide

22. Metrics driven development:
    StatsD
    “We love statsd at Instagram. Written by Etsy, it’s a
    network daemon that aggregates and rolls-up data into
    Graphite. At its core, it has two types of statistics: counter
    and timers. We use the counters to track everything from
    number of signups per second to number of likes, and we
    use timers to time generation of feeds, how long it takes to
    follow users, and any other major action.”
    - Mike Krieger, Founder and CTO, Instagram.
    Thursday, May 17, 12
    

    View Slide

23. Metrics driven development:
    StatsD
    Open source,
    a daemon for timing and counting things,
    graphs rates, averages and 95th percentile,
    20-30 minute install,
    easy to use
    Thursday, May 17, 12
    

    View Slide

24. Metrics driven development
    StatsD::timing("page.render", $msec);
    Thursday, May 17, 12
    

    View Slide

25. Metrics driven development
    Thursday, May 17, 12
    

    View Slide

26. Metrics driven development
    getting started:
    your 5 core metrics
    Thursday, May 17, 12
    

    View Slide

27. Metrics driven development
    5 core metrics @ Etsy:
    sign ups, logins, checkout, new
    listings, posts in the bugs forums
    Thursday, May 17, 12
    

    View Slide

28. Metrics driven development
    378,000 metrics,
    collected roughly once a
    second
    Thursday, May 17, 12
    

    View Slide

29. Metrics driven development
    the confidence to
    change
    Thursday, May 17, 12
    

    View Slide

30. Adapt, find market fit.
    Engineering change:
    Rapid experimentation
    Thursday, May 17, 12
    

    View Slide

31. Adapt, find market fit,
    addictive to engineers.
    Engineering change:
    Rapid experimentation
    Thursday, May 17, 12
    

    View Slide

32. Thank you!
    Lean more: http://codeascraft.etsy.com
    Thursday, May 17, 12
    

    View Slide