Feature Flagging your Infrastructure for Fun and Profit

Presentation from the September 2013 NYC LOPSA meetup about Chef workflow at Etsy and our feature flagging/whitelisting library for rollouts in Chef

Transcript

1. Feature Flagging your
   Infrastructure for Fun and
   Pro!t
   Daniel Schauenberg
   dschauenberg@etsy.com
   @mrtazz
   Tuesday, September 10, 13
   

   View Slide

2. Tuesday, September 10, 13
   

   View Slide

3. LAMMP
   Item by TheBackPackShoppe
   Tuesday, September 10, 13
   

   View Slide

4. Item by FrankelPhotos
   Monolithic App
   Tuesday, September 10, 13
   

   View Slide

5. Etsy Infrastructure
   •~1400 nodes
   •~30 dev & ops engineers making changes
   regularly
   •Open Source Chef server + GitHub Enterprise
   •Default environment setup (production,
   development, testing)
   Tuesday, September 10, 13
   

   View Slide

6. jonlives/knife-spork
   Tuesday, September 10, 13
   

   View Slide

7. knife-spork
   •Work!ow to manage cookbook and environment
   changes
   •Versioned cookbooks and pinned environments
   •Speci"c work!ow, di#erent ways of using plugins
   Tuesday, September 10, 13
   

   View Slide

8. Tuesday, September 10, 13
   

   View Slide

9. % chef-shell
   chef > recipe_mode
   chef:recipe > echo off
   chef:recipe > include_recipe "apache"
   chef:recipe > run_chef
   Tuesday, September 10, 13
   

   View Slide

10. % review -r jcowie --cc ops
    Tuesday, September 10, 13
    

    View Slide

11. % knife spork check apache
    % knife spork bump apache
    % git commit
    % git push
    % knife spork upload apache
    Staging Deploy
    Tuesday, September 10, 13
    

    View Slide

12. jonlives/knife-!ip
    % knife node flip node.etsy.com testing
    % knife role flip testRole testing
    Tuesday, September 10, 13
    

    View Slide

13. % knife spork promote apache
    % git commit
    % git push
    % knife spork promote apache --remote
    Production Deploy
    Tuesday, September 10, 13
    

    View Slide

14. Monitoring
    Tuesday, September 10, 13
    

    View Slide

15. 19:18:06 irccat | CHEF: Daniel
    Schauenberg promoted apache@0.0.2 to
    development https://
    github.etsycorp.com/gist/12345
    Tuesday, September 10, 13
    

    View Slide

16. etsy/chef-handlers
    19:20:00 irccat | Chef run failed on test.etsy.com
    19:20:00 irccat | https://github.etsycorp.com/
    gist/12347
    jgoulah/knife-lastrun
    % knife node lastrun test.etsy.com
    Tuesday, September 10, 13
    

    View Slide

17. Graphs!
    Tuesday, September 10, 13
    

    View Slide

18. Downsides
    •Longer testing blocks others
    •Staged cookbooks can accidentally be promoted
    •Testing environment a#ects more than one
    cookbook
    •Used “upgrade” environments to circumvent
    Tuesday, September 10, 13
    

    View Slide

19. Feature Flags
    Tuesday, September 10, 13
    

    View Slide

20. Branching in Code
    •Well established pattern for “dark launches”
    •Used in the Etsy Web stack
    •Allows for restricted roll outs
    •http://code.!ickr.net/2009/12/02/!ipping-out/
    Tuesday, September 10, 13
    

    View Slide

21. Tuesday, September 10, 13
    

    View Slide

22. etsy/chef-whitelist
    Tuesday, September 10, 13
    

    View Slide

23. chef-whitelist
    •Data bag driven whitelist
    •Library to include in cookbooks
    •Easy access to feature !ags
    Tuesday, September 10, 13
    

    View Slide

24. whitelist data bag
    {
    "id": "my_whitelist",
    "patterns": [
    "host.example.com",
    "*.subdomain.example.com",
    "prefix*.example.com"
    ]
    }
    Tuesday, September 10, 13
    

    View Slide

25. whitelist data bag
    {
    "id": "my_whitelist",
    "patterns": [
    "host.example.com",
    "*.subdomain.example.com",
    "prefix*.example.com"
    ],
    "roles": [
    "Webserver",
    "DatabaseServer"
    ]
    }
    Tuesday, September 10, 13
    

    View Slide

26. feature !ags in recipe
    if node.is_in_whitelist? "my_whitelist"
    # new hawtness
    else
    # old stuff
    end
    Tuesday, September 10, 13
    

    View Slide

27. Customizable
    node.is_in_whitelist? "my_whitelist", "acl", "hosts"
    Tuesday, September 10, 13
    

    View Slide

28. real world
    example
    {
    "id": "php-5-4-19",
    "patterns": [
    "dschauenberg.vm.dev.etsy.com",
    "web0270.etsy.com",
    "api04.etsy.com",
    "imgcache01.etsy.com",
    "imgwriter01.etsy.com",
    "worker01.etsy.com",
    "beacon01.etsy.com",
    "paymentsweb01.etsy.com"
    ],
    "roles": [
    ]
    }
    Tuesday, September 10, 13
    

    View Slide

29. Advantages
    •Easy to access list of what gets upgrades
    •Upgrades don’t need the spork work!ow
    •Pattern already known by all engineers
    Tuesday, September 10, 13
    

    View Slide

30. Downsides
    •Changes outside the regular work!ow
    •No graphs (yet)
    •Less visible cleanup required
    Tuesday, September 10, 13
    

    View Slide

31. Summary
    •GitHub Enterprise, Dev VMs, chef-shell as
    development environment
    •Chef Server and knife-spork as Deployment
    System
    •Feature !agging with chef-whitelist
    •Monitoring, Noti"cations, Graphs
    Tuesday, September 10, 13
    

    View Slide

32. •http://codeascraft.etsy.com/
    •http://codeascraft.com/2013/08/02/
    infrastructure-upgrades-with-chef/
    •http://www.slideshare.net/jonlives/michelin-
    starred-cooking-with-chef
    •http://www.slideshare.net/mcdonnps/lessons-
    from-etsy-avoiding-kitchen-nightmares-
    chefconf-2012
    •https://github.com/etsy
    Further information
    Tuesday, September 10, 13
    

    View Slide

33. Thank you!
    Questions?
    Tuesday, September 10, 13
    

    View Slide

34. Feature Flagging your
    Infrastructure for Fun and
    Pro!t
    Daniel Schauenberg
    dschauenberg@etsy.com
    @mrtazz
    Tuesday, September 10, 13
    

    View Slide