Configuration Management Patterns

My talk at RailsConf 2013

Transcript

1. Configuration Management Patterns Beau Harrington github.com/bohford · @bohford
2. First things first
3. This is a conversation about ideas, it is not a prescription for how you have to do it - Chris Kelly, @amateurhuman (paraphrased)
4. 15 second bio • Chief Software Architect @ Kabam • We make mobile and web games • #1 Top Grossing iOS app of 2012 Kingdoms of Camelot • Of course we’re hiring –30–
5. What is configuration?
6. config/database.yml production: adapter: mysql2 host: partytimedb1.colo database: partytime_production username: partytime password: d34db33f timeout: 5000 pool: 10 encoding: utf8 reconnect: true Ways to interact with external systems
7. config/feature_flags.yml production: search_box: true super_cool_new_feature: true append_request_time: false Prescribing the behaviour of our system
8. config/locales/es.yml es: hello_world: “hola mundo” ice_cream: “helado” soccer: “fútbol” football: “fútbol americano” Data that will be used elsewhere in your system, like translations
9. app/models/post.rb class Post < ActiveRecord::Base @@per_page = 30 end Constants and presentation logic sprinkled all over the place; magic numbers This is probably one of the most common magic numbers, and yes that is in completely the wrong place.
10. app/models/quest.rb class Quest < ActiveRecord::Base GOLD_COINS_REWARD = 50 end And here is a business logic constant; how many coins will you get for completing a quest?
11. config/envs/production.rb Partytime::Application.configure config.cache_classes = true config.assets.compile = false config.assets.digest = true end Hm, this one is starting to look like code to me.
12. Configuration is... • everything you want it to be • for our purposes: values that you may want to share, or change Revel in this gloriously ambiguous description
13. Why do we care about configuration so much?
14. Kingdoms of Camelot: Battle for the North
15. Configuration, ahoy • 15+ di erent types of buildings • 10+ levels for each building • 8 di erent requirements per level • 1,200+ values Shared-world strategy game
16. Configuration, ahoy • Research • Knights • Battles • NPCs • Items • Quests • etc...
17. 10,000+ values making up just the business logic configuration of the game
18. Getting serious • Technical QA • Playtesting QA • Deploying new configurations just as rigorously as new code • Bad config can be just as damaging as bad code Technical QA - did we break anything from an engineering sense? Playtesting QA - did we break anything from a game design sense? Anything become OP or get nerfed? we have millions of players, the stakes are high.
19. Agenda • Configuration context • Baking configurations • Deploying configurations • Applications • Further exploration Configuration context - hwo do we make our configurations more easily relevant to an environment, a datacenter, a region, a machine? Baking configurations - how can we make this a serious build process? Deploying configurations - we put a lot of care into deploying code, can we do the same for config? Not all of these are necessarily in use on this game, but they are here in my presentation for you to think about. So let’s dive in to this amazing agenda, we could do all this
20. Let’s talk context
21. Or we could just make a game instead
22. Click this puppy...
23. ...and get 10 gold coins...
24. ...and, sometimes, a prize! the actual code for handling the click is left as an exercise for the viewer.
25. #1 class PuppyClick COINS_PER_CLICK = 10 ODDS_OF_WINNING_PRIZE = 0.1 PRIZES = %w{yo-yo candy} ... end
26. #2: values out of source # config/game.yml game: coins_per_click: 10 odds_of_winning_prize: 0.1 prizes: - ‘candy’ - ‘yo-yo’
27. #3: respect Rails.env development: coins_per_click: 10 odds_of_winning_prize: 1 prizes: - ‘candy’ - ‘yo-yo’ test: coins_per_click: 10 odds_of_winning_prize: 0.1 prizes: - ‘candy’ on dev i always want a prize
28. #4: composite config core.yml coins_per_click: 10 odds_of_winning_prize: 0.1 prizes: - ‘candy’ - ‘yo-yo’ development.yml odds_of_winning_prize: 1 test.yml prizes: - ‘screen_test_256_colors’
29. #4: composite config core = Yaml.load ‘config/core.yml’ env = Yaml.load “config/#{Rails.env}.yml” config = core.deep_merge(env) composite config introduces precedence
30. #4: composite config core.yml coins_per_click: 10 odds_of_winning_prize: 0.1 prizes: - ‘candy’ - ‘yo-yo’ development.yml odds_of_winning_prize: 1.0 test.yml prizes: - ‘screen_test_256_colors’
31. #4: composite config > Rails.env => “development” > config[:coins_per_click] => 10 # from core.yml > config[:odds_of_winning_prize] => 1.0 # from development.yml
32. #5: more context • datacenters • AWS regions • roles • machine names
33. #5: more context @config = {} files = [ “config/core.yml”, “config/envs/#{Rails.env}.yml”, “config/regions/#{AWS_REGION}.yml”, “config/hosts/#{HOSTNAME}.yml” ] files.each do |filename| context = Yaml.load filename @config.deep_merge(context) end
34. #6: tags • apply a config based on ENV • canary or A/B new code • assign role-specific configuration • debugging
35. #6: tags tags = ENV[‘TAGS’].split(‘,’) # load all other configs tags.each do |tag| filename = “config/tags/#{tag}.yml” context = Yaml.load filename @config.deep_merge(context) end
36. rails_config • Settings.coins_per_click • Settings[:coins_per_click] • reloading of settings files • composite configurations • developer-local settings • http://github.com/railsjedi/ rails_config Seemingly the marquee feature for any Rails config library, method notation!
37. And yet $ vim $ rake spec $ git add -u $ git commit -m “...” $ git push origin master $ cap deploy
38. Reset • Our settings are aware of their context and environment! • But still in the source repo • Still tied to a commit and deploy • Still inaccessible to other apps • Still inaccessible to non-engineers
39. Baking configurations
40. Requirements • Decouple generation process entirely from app/repo • Use code deployment principles • Build an artifact (like we used to) • Version and store • Deploy What if we could generate a configuration instead?
41. The power of the internet • A webapp!!! • Single, enforceable process • Access controls, individual auditing • Bounds/sanity checking • Incorporate your particular corporate rules/insanity Anecdote about sanity checking (putting 100000 in the yml file) * sanity checking, etc is outside the bounds of your app but good for generating configs!
42. Wrap an app around this dim name key value core core coins_per_click 10 core core odds_of_winning_prize 0.1 core core prizes [‘candy’, ‘yo-yo’] env dev odds_of_winning_prize 1.0 env test prizes [‘screen_test_256_colors’] Decomposed our configuration data into a table... not normalized at all of course. Yes this works even better with a document store...
43. Dump directly from DB? • slow • dogpile • harder to replicate than static http asset • unnecessary load on your most precious resource • may be OK for your use case
44. Build process • transform config values into a single Yaml doc - building an artifact • take note of: builder, build #, timestamp, changelog • include checksum to prevent alteration, well-intentioned or otherwise
45. EXAMPLE: Double coins!
46. New values dim name key value core core coins_per_click 20 core core odds_of_winning_prize 0.1 core core prizes [‘candy’, ‘yo-yo’] env dev odds_of_winning_prize 1.0 env test prizes [‘screen_test_256_colors’]
47. Our new Yaml doc config: coins_per_click: 20 odds_of_winning_prize: 0.1 prizes: - ‘candy’ - ‘yo-yo’ environments: development: odds_of_winning_prize: 1 test: prizes: - ‘screen_test_256_colors’
48. Artifact • r8471-double_coins_promo- bac8318fcd59afa3912f3c9.yml • http://config/artifacts/8471.yml production.yml - an eventual promotion! this is how we get a config there
49. Promotion • Map a revision to an environment or other dimension • http://config/artifacts/8471.yml • http://config/envs/playtest.yml • Promotion can be end-user accessible production.yml - an eventual promotion! this is how we get a config there
50. Reset • Our settings are aware of their context and environment! • Settings are outside the repo! • Accessible to other apps! • Changes accessible to non-engineers, but not deploys • How do we best deploy config changes?
51. Deploying configuration
52. Release process • run test/CI suites against config • run QA against config • run playtest against config • deploy to production
53. Good... # config/environment.rb [ “envs/#{Rails.env}.yml”, “regions/#{AWS_REGION}.yml”, “hosts/#{HOSTNAME}.yml” ].each do |filename| yaml = Yaml.parse(Http.get “http:// config/#{filename}”) (@config ||= {}).deep_merge!(yaml) end Still have to restart to see changes
54. But I don’t want to restart • Spawn a thread to poll for updates in the background • Celluloid timer is perfect for this • Doesn’t work with GIL (MRI) • Pretty gross multi-process (Unicorn) • Don’t forget to be threadsafe
55. Timer class ConfigFetcher include Celluloid def initialize(sources) refresh @timer = after(30) { refresh; @timer.reset } end def refresh # poll and assemble config... end end
56. Better... > Config.get(‘coins_per_click’) => 10 # ... deploy new config to production, wait for refresh ... > Config.get(‘coins_per_click’) => 20
57. Best Config.on_change ‘coins_per_click’ do |new_value| Logger.warn “Coins changed to # {new_value}” # break cache end
58. Best Config.on_change ‘memcache_servers’ do |new_value| Rails.cache = Dalli::Client.new( *memcache_servers ) end
59. Why poll and not push? • What happens if things go wrong? • Failed push means app carries on with old config • Responsibility in the wrong place • If poll fails, app can shut itself down
60. Reset • config is out of source • non-engineers can build and deploy • config can be shared across apps • update config live without restarts • test configs before production deploy • recorded change control
61. Other applications • rapid A/B testing • I18n and translation management • feature flags
62. Bonus non-Ruby content • Archaius: Netflix’s Java composite configuration library • https://github.com/Netflix/archaius
63. Bonus non-Ruby content • ZooKeeper: cluster synchronization service (strongly consistent) • Nodes with watches • http://zookeeper.apache.org/
64. Bonus non-Ruby content • jones: Python config app • ZooKeeper-backed • Composite configurations (views) • https://github.com/mwhooker/jones
65. Thanks bye