Configuration Management Patterns

Slide 1

Slide 1 text

Conﬁguration Management Patterns Beau Harrington github.com/bohford · @bohford

Slide 2

Slide 2 text

First things ﬁrst

Slide 3

Slide 3 text

This is a conversation about ideas, it is not a prescription for how you have to do it - Chris Kelly, @amateurhuman (paraphrased)

Slide 4

Slide 4 text

15 second bio • Chief Software Architect @ Kabam • We make mobile and web games • #1 Top Grossing iOS app of 2012 Kingdoms of Camelot • Of course we’re hiring –30–

Slide 5

Slide 5 text

What is conﬁguration?

Slide 6

Slide 6 text

conﬁg/database.yml production: adapter: mysql2 host: partytimedb1.colo database: partytime_production username: partytime password: d34db33f timeout: 5000 pool: 10 encoding: utf8 reconnect: true Ways to interact with external systems

Slide 7

Slide 7 text

conﬁg/feature_ﬂags.yml production: search_box: true super_cool_new_feature: true append_request_time: false Prescribing the behaviour of our system

Slide 8

Slide 8 text

conﬁg/locales/es.yml es: hello_world: “hola mundo” ice_cream: “helado” soccer: “fútbol” football: “fútbol americano” Data that will be used elsewhere in your system, like translations

Slide 9

Slide 9 text

app/models/post.rb class Post < ActiveRecord::Base @@per_page = 30 end Constants and presentation logic sprinkled all over the place; magic numbers This is probably one of the most common magic numbers, and yes that is in completely the wrong place.

Slide 10

Slide 10 text

app/models/quest.rb class Quest < ActiveRecord::Base GOLD_COINS_REWARD = 50 end And here is a business logic constant; how many coins will you get for completing a quest?

Slide 11

Slide 11 text

conﬁg/envs/production.rb Partytime::Application.configure config.cache_classes = true config.assets.compile = false config.assets.digest = true end Hm, this one is starting to look like code to me.

Slide 12

Slide 12 text

Conﬁguration is... • everything you want it to be • for our purposes: values that you may want to share, or change Revel in this gloriously ambiguous description

Slide 13

Slide 13 text

Why do we care about conﬁguration so much?

Slide 14

Slide 14 text

Kingdoms of Camelot: Battle for the North

Slide 15

Slide 15 text

Conﬁguration, ahoy • 15+ di erent types of buildings • 10+ levels for each building • 8 di erent requirements per level • 1,200+ values Shared-world strategy game

Slide 16

Slide 16 text

Conﬁguration, ahoy • Research • Knights • Battles • NPCs • Items • Quests • etc...

Slide 17

Slide 17 text

10,000+ values making up just the business logic conﬁguration of the game

Slide 18

Slide 18 text

Getting serious • Technical QA • Playtesting QA • Deploying new conﬁgurations just as rigorously as new code • Bad conﬁg can be just as damaging as bad code Technical QA - did we break anything from an engineering sense? Playtesting QA - did we break anything from a game design sense? Anything become OP or get nerfed? we have millions of players, the stakes are high.

Slide 19

Slide 19 text

Agenda • Configuration context • Baking configurations • Deploying configurations • Applications • Further exploration Configuration context - hwo do we make our configurations more easily relevant to an environment, a datacenter, a region, a machine? Baking configurations - how can we make this a serious build process? Deploying configurations - we put a lot of care into deploying code, can we do the same for config? Not all of these are necessarily in use on this game, but they are here in my presentation for you to think about. So let’s dive in to this amazing agenda, we could do all this

Slide 20

Slide 20 text

Let’s talk context

Slide 21

Slide 21 text

Or we could just make a game instead

Slide 22

Slide 22 text

Click this puppy...

Slide 23

Slide 23 text

...and get 10 gold coins...

Slide 24

Slide 24 text

...and, sometimes, a prize! the actual code for handling the click is left as an exercise for the viewer.

Slide 25

Slide 25 text

#1 class PuppyClick COINS_PER_CLICK = 10 ODDS_OF_WINNING_PRIZE = 0.1 PRIZES = %w{yo-yo candy} ... end

Slide 26

Slide 26 text

#2: values out of source # config/game.yml game: coins_per_click: 10 odds_of_winning_prize: 0.1 prizes: - ‘candy’ - ‘yo-yo’

Slide 27

Slide 27 text

#3: respect Rails.env development: coins_per_click: 10 odds_of_winning_prize: 1 prizes: - ‘candy’ - ‘yo-yo’ test: coins_per_click: 10 odds_of_winning_prize: 0.1 prizes: - ‘candy’ on dev i always want a prize

Slide 28

Slide 28 text

#4: composite conﬁg core.yml coins_per_click: 10 odds_of_winning_prize: 0.1 prizes: - ‘candy’ - ‘yo-yo’ development.yml odds_of_winning_prize: 1 test.yml prizes: - ‘screen_test_256_colors’

Slide 29

Slide 29 text

#4: composite conﬁg core = Yaml.load ‘config/core.yml’ env = Yaml.load “config/#{Rails.env}.yml” config = core.deep_merge(env) composite conﬁg introduces precedence

Slide 30

Slide 30 text

#4: composite conﬁg core.yml coins_per_click: 10 odds_of_winning_prize: 0.1 prizes: - ‘candy’ - ‘yo-yo’ development.yml odds_of_winning_prize: 1.0 test.yml prizes: - ‘screen_test_256_colors’

Slide 31

Slide 31 text

#4: composite conﬁg > Rails.env => “development” > config[:coins_per_click] => 10 # from core.yml > config[:odds_of_winning_prize] => 1.0 # from development.yml

Slide 32

Slide 32 text

#5: more context • datacenters • AWS regions • roles • machine names

Slide 33

Slide 33 text

#5: more context @config = {} files = [ “config/core.yml”, “config/envs/#{Rails.env}.yml”, “config/regions/#{AWS_REGION}.yml”, “config/hosts/#{HOSTNAME}.yml” ] files.each do |filename| context = Yaml.load filename @config.deep_merge(context) end

Slide 34

Slide 34 text

#6: tags • apply a config based on ENV • canary or A/B new code • assign role-specific configuration • debugging

Slide 35

Slide 35 text

#6: tags tags = ENV[‘TAGS’].split(‘,’) # load all other configs tags.each do |tag| filename = “config/tags/#{tag}.yml” context = Yaml.load filename @config.deep_merge(context) end

Slide 36

Slide 36 text

rails_config • Settings.coins_per_click • Settings[:coins_per_click] • reloading of settings files • composite configurations • developer-local settings • http://github.com/railsjedi/ rails_config Seemingly the marquee feature for any Rails config library, method notation!

Slide 37

Slide 37 text

And yet $ vim $ rake spec $ git add -u $ git commit -m “...” $ git push origin master $ cap deploy

Slide 38

Slide 38 text

Reset • Our settings are aware of their context and environment! • But still in the source repo • Still tied to a commit and deploy • Still inaccessible to other apps • Still inaccessible to non-engineers

Slide 39

Slide 39 text

Baking conﬁgurations

Slide 40

Slide 40 text

Requirements • Decouple generation process entirely from app/repo • Use code deployment principles • Build an artifact (like we used to) • Version and store • Deploy What if we could generate a conﬁguration instead?

Slide 41

Slide 41 text

The power of the internet • A webapp!!! • Single, enforceable process • Access controls, individual auditing • Bounds/sanity checking • Incorporate your particular corporate rules/insanity Anecdote about sanity checking (putting 100000 in the yml ﬁle) * sanity checking, etc is outside the bounds of your app but good for generating conﬁgs!

Slide 42

Slide 42 text

Wrap an app around this dim name key value core core coins_per_click 10 core core odds_of_winning_prize 0.1 core core prizes [‘candy’, ‘yo-yo’] env dev odds_of_winning_prize 1.0 env test prizes [‘screen_test_256_colors’] Decomposed our conﬁguration data into a table... not normalized at all of course. Yes this works even better with a document store...

Slide 43

Slide 43 text

Dump directly from DB? • slow • dogpile • harder to replicate than static http asset • unnecessary load on your most precious resource • may be OK for your use case

Slide 44

Slide 44 text

Build process • transform conﬁg values into a single Yaml doc - building an artifact • take note of: builder, build #, timestamp, changelog • include checksum to prevent alteration, well-intentioned or otherwise

Slide 45

Slide 45 text

EXAMPLE: Double coins!

Slide 46

Slide 46 text

New values dim name key value core core coins_per_click 20 core core odds_of_winning_prize 0.1 core core prizes [‘candy’, ‘yo-yo’] env dev odds_of_winning_prize 1.0 env test prizes [‘screen_test_256_colors’]

Slide 47

Slide 47 text

Our new Yaml doc config: coins_per_click: 20 odds_of_winning_prize: 0.1 prizes: - ‘candy’ - ‘yo-yo’ environments: development: odds_of_winning_prize: 1 test: prizes: - ‘screen_test_256_colors’

Slide 48

Slide 48 text

Artifact • r8471-double_coins_promo- bac8318fcd59afa3912f3c9.yml • http://conﬁg/artifacts/8471.yml production.yml - an eventual promotion! this is how we get a conﬁg there

Slide 49

Slide 49 text

Promotion • Map a revision to an environment or other dimension • http://config/artifacts/8471.yml • http://config/envs/playtest.yml • Promotion can be end-user accessible production.yml - an eventual promotion! this is how we get a config there

Slide 50

Slide 50 text

Reset • Our settings are aware of their context and environment! • Settings are outside the repo! • Accessible to other apps! • Changes accessible to non-engineers, but not deploys • How do we best deploy conﬁg changes?

Slide 51

Slide 51 text

Deploying conﬁguration

Slide 52

Slide 52 text

Release process • run test/CI suites against config • run QA against config • run playtest against config • deploy to production

Slide 53

Slide 53 text

Good... # config/environment.rb [ “envs/#{Rails.env}.yml”, “regions/#{AWS_REGION}.yml”, “hosts/#{HOSTNAME}.yml” ].each do |filename| yaml = Yaml.parse(Http.get “http:// config/#{filename}”) (@config ||= {}).deep_merge!(yaml) end Still have to restart to see changes

Slide 54

Slide 54 text

But I don’t want to restart • Spawn a thread to poll for updates in the background • Celluloid timer is perfect for this • Doesn’t work with GIL (MRI) • Pretty gross multi-process (Unicorn) • Don’t forget to be threadsafe

Slide 55

Slide 55 text

Timer class ConfigFetcher include Celluloid def initialize(sources) refresh @timer = after(30) { refresh; @timer.reset } end def refresh # poll and assemble config... end end

Slide 56

Slide 56 text

Better... > Config.get(‘coins_per_click’) => 10 # ... deploy new config to production, wait for refresh ... > Config.get(‘coins_per_click’) => 20

Slide 57

Slide 57 text

Best Config.on_change ‘coins_per_click’ do |new_value| Logger.warn “Coins changed to # {new_value}” # break cache end

Slide 58

Slide 58 text

Best Config.on_change ‘memcache_servers’ do |new_value| Rails.cache = Dalli::Client.new( *memcache_servers ) end

Slide 59

Slide 59 text

Why poll and not push? • What happens if things go wrong? • Failed push means app carries on with old conﬁg • Responsibility in the wrong place • If poll fails, app can shut itself down

Slide 60

Slide 60 text

Reset • config is out of source • non-engineers can build and deploy • config can be shared across apps • update config live without restarts • test configs before production deploy • recorded change control

Slide 61

Slide 61 text

Other applications • rapid A/B testing • I18n and translation management • feature ﬂags

Slide 62

Slide 62 text

Bonus non-Ruby content • Archaius: Netflix’s Java composite configuration library • https://github.com/Netflix/archaius

Slide 63

Slide 63 text

Bonus non-Ruby content • ZooKeeper: cluster synchronization service (strongly consistent) • Nodes with watches • http://zookeeper.apache.org/

Slide 64

Slide 64 text

Bonus non-Ruby content • jones: Python conﬁg app • ZooKeeper-backed • Composite conﬁgurations (views) • https://github.com/mwhooker/jones

Slide 65

Slide 65 text

Thanks bye