Feature Flagging your Infrastructure for Fun and Profit

Slide 1

Slide 1 text

Feature Flagging your Infrastructure for Fun and Pro!t Daniel Schauenberg [email protected] @mrtazz Tuesday, September 10, 13

Slide 2

Slide 2 text

Tuesday, September 10, 13

Slide 3

Slide 3 text

LAMMP Item by TheBackPackShoppe Tuesday, September 10, 13

Slide 4

Slide 4 text

Item by FrankelPhotos Monolithic App Tuesday, September 10, 13

Slide 5

Slide 5 text

Etsy Infrastructure •~1400 nodes •~30 dev & ops engineers making changes regularly •Open Source Chef server + GitHub Enterprise •Default environment setup (production, development, testing) Tuesday, September 10, 13

Slide 6

Slide 6 text

jonlives/knife-spork Tuesday, September 10, 13

Slide 7

Slide 7 text

knife-spork •Work!ow to manage cookbook and environment changes •Versioned cookbooks and pinned environments •Speci"c work!ow, di#erent ways of using plugins Tuesday, September 10, 13

Slide 8

Slide 8 text

Tuesday, September 10, 13

Slide 9

Slide 9 text

% chef-shell chef > recipe_mode chef:recipe > echo off chef:recipe > include_recipe "apache" chef:recipe > run_chef Tuesday, September 10, 13

Slide 10

Slide 10 text

% review -r jcowie --cc ops Tuesday, September 10, 13

Slide 11

Slide 11 text

% knife spork check apache % knife spork bump apache % git commit % git push % knife spork upload apache Staging Deploy Tuesday, September 10, 13

Slide 12

Slide 12 text

jonlives/knife-!ip % knife node flip node.etsy.com testing % knife role flip testRole testing Tuesday, September 10, 13

Slide 13

Slide 13 text

% knife spork promote apache % git commit % git push % knife spork promote apache --remote Production Deploy Tuesday, September 10, 13

Slide 14

Slide 14 text

Monitoring Tuesday, September 10, 13

Slide 15

Slide 15 text

19:18:06 irccat | CHEF: Daniel Schauenberg promoted [email protected] to development https:// github.etsycorp.com/gist/12345 Tuesday, September 10, 13

Slide 16

Slide 16 text

etsy/chef-handlers 19:20:00 irccat | Chef run failed on test.etsy.com 19:20:00 irccat | https://github.etsycorp.com/ gist/12347 jgoulah/knife-lastrun % knife node lastrun test.etsy.com Tuesday, September 10, 13

Slide 17

Slide 17 text

Graphs! Tuesday, September 10, 13

Slide 18

Slide 18 text

Downsides •Longer testing blocks others •Staged cookbooks can accidentally be promoted •Testing environment a#ects more than one cookbook •Used “upgrade” environments to circumvent Tuesday, September 10, 13

Slide 19

Slide 19 text

Feature Flags Tuesday, September 10, 13

Slide 20

Slide 20 text

Branching in Code •Well established pattern for “dark launches” •Used in the Etsy Web stack •Allows for restricted roll outs •http://code.!ickr.net/2009/12/02/!ipping-out/ Tuesday, September 10, 13

Slide 21

Slide 21 text

Tuesday, September 10, 13

Slide 22

Slide 22 text

etsy/chef-whitelist Tuesday, September 10, 13

Slide 23

Slide 23 text

chef-whitelist •Data bag driven whitelist •Library to include in cookbooks •Easy access to feature !ags Tuesday, September 10, 13

Slide 24

Slide 24 text

whitelist data bag { "id": "my_whitelist", "patterns": [ "host.example.com", "*.subdomain.example.com", "prefix*.example.com" ] } Tuesday, September 10, 13

Slide 25

Slide 25 text

whitelist data bag { "id": "my_whitelist", "patterns": [ "host.example.com", "*.subdomain.example.com", "prefix*.example.com" ], "roles": [ "Webserver", "DatabaseServer" ] } Tuesday, September 10, 13

Slide 26

Slide 26 text

feature !ags in recipe if node.is_in_whitelist? "my_whitelist" # new hawtness else # old stuff end Tuesday, September 10, 13

Slide 27

Slide 27 text

Customizable node.is_in_whitelist? "my_whitelist", "acl", "hosts" Tuesday, September 10, 13

Slide 28

Slide 28 text

real world example { "id": "php-5-4-19", "patterns": [ "dschauenberg.vm.dev.etsy.com", "web0270.etsy.com", "api04.etsy.com", "imgcache01.etsy.com", "imgwriter01.etsy.com", "worker01.etsy.com", "beacon01.etsy.com", "paymentsweb01.etsy.com" ], "roles": [ ] } Tuesday, September 10, 13

Slide 29

Slide 29 text

Advantages •Easy to access list of what gets upgrades •Upgrades don’t need the spork work!ow •Pattern already known by all engineers Tuesday, September 10, 13

Slide 30

Slide 30 text

Downsides •Changes outside the regular work!ow •No graphs (yet) •Less visible cleanup required Tuesday, September 10, 13

Slide 31

Slide 31 text

Summary •GitHub Enterprise, Dev VMs, chef-shell as development environment •Chef Server and knife-spork as Deployment System •Feature !agging with chef-whitelist •Monitoring, Noti"cations, Graphs Tuesday, September 10, 13

Slide 32

Slide 32 text

•http://codeascraft.etsy.com/ •http://codeascraft.com/2013/08/02/ infrastructure-upgrades-with-chef/ •http://www.slideshare.net/jonlives/michelin- starred-cooking-with-chef •http://www.slideshare.net/mcdonnps/lessons- from-etsy-avoiding-kitchen-nightmares- chefconf-2012 •https://github.com/etsy Further information Tuesday, September 10, 13

Slide 33

Slide 33 text

Thank you! Questions? Tuesday, September 10, 13

Slide 34

Slide 34 text

Feature Flagging your Infrastructure for Fun and Pro!t Daniel Schauenberg [email protected] @mrtazz Tuesday, September 10, 13