Developing at Scale

Talk given at ConFoo 2016 on February 26th, 2016.

Transcript

1. developing at scale
2. cjoudrey @
6. Scale?
9. employee count 275 550 825 1100 2011 2012 2013 2014 2015
10. Journey of a new dev
11. ✓ quick & easy setup
 ✓ reproducible
 ✓ production-like Dev environments
12. pre 2012 — README.md with lots of Bash snippets circa 2013 — Vagrant with shared Chef cookbooks circa 2012 — GitHub's Boxen (Puppet recipes on local) future — ? Dev environments
13. ~ $ git clone git@github.com:Shopify/vagrant.git
 ~ $ cd vagrant && vagrant up
 ~/vagrant $ vagrant ssh (vagrant) ~ $ git clone git@github.com:Shopify/shopify.git
 (vagrant) ~ $ cd shopify && script/setup (vagrant) ~/shopify $ script/server
14. ~ $ git clone git@github.com:Shopify/vagrant.git
 ~ $ cd vagrant && vagrant up
 ~/vagrant $ vagrant ssh (vagrant) ~ $ git clone git@github.com:Shopify/shopify.git
 (vagrant) ~ $ cd shopify && script/setup (vagrant) ~/shopify $ script/server ✓ quick & easy setup
 ✓ reproducible
 ✓ production-like
15. http://localhost:3000
16. https://shop1.myshopify.com https://shop2.myshopify.com https://shop3.myshopify.com
17. https://shop1.myshopify.com https://shop2.myshopify.com https://shop3.myshopify.com /etc/hosts :3000 http:// Rails.env.development?
18. https://shop1.myshopify.com https://shop1.myshopify.io ~ $ whois myshopify.io Domain : myshopify.io Status : Live Expiry : 2016-12-16 ~ $ openssl s_client -connect myshopify.io:443 CONNECTED(00000003) depth=1 /C=US/O=DigiCert Inc/...
19. Code
20. Feature flags class Shop < ActiveRecord::Base include HasFeatureFlags end shop.flags.enabled?('free_ssl') # false shop.flags.enable('free_ssl')
 shop.flags.enabled?('free_ssl') # true FeatureRollout.enable('test_feature', 50) shop.features.enabled?('test_feature') # true shop2.features.enabled?('test_feature') # false
21. A/B Tests Verdict::Experiment.define :my_experiment do # Block returns true if the subject # is qualified to participate. qualify { |subject, context| ... } groups do group :blue_button, 50 group :control, 50 end storage Verdict::Storage::MemoryStorage.new end github.com/shopify/verdict
22. A/B Tests case Verdict[:my_experiment].switch(checkout) when :blue_button # ... when :control # ... else # unqualified subjects end github.com/shopify/verdict
23. Secrets I don’t even know where to start.
24. Secrets circa 2014 — Secrets stored in encrypted data bags in Chef. Devs pinged Ops to get secrets added/changed. Hard to trace changes to secrets. Hard to make changes.
25. Secrets circa 2014 — Secrets stored in encrypted data bags in Chef. Devs pinged Ops to get secrets added/changed. Hard to trace changes to secrets. tl;dr — It was pretty painful!
26. Secrets — EJSON github.com/shopify/ejson now — Secrets can be safely stored in app source. Anyone has access to change/add secrets. Only production servers can decrypt secrets. Secrets change synchronously with app source.
27. Secrets — EJSON github.com/shopify/ejson now — Secrets can be safely stored in app source. Anyone has access to change/add secrets. Only production servers can decrypt secrets. Secrets change synchronously with app source. tl;dr — A lot better!
28. Secrets — EJSON github.com/shopify/ejson $ ejson keygen Public Key: 6623423a9492e68e12eeb1c705888aebdcc0080af7e594fc402b eb24cce9d14f Private Key: 75b80b4a693156eb435f4ed2fe397e583f461f09fd99ec2bd1bd ef0a56cf6e64
29. Secrets — EJSON $ cat secrets.production.ejson { "_public_key": "6623423...", "database_password": "1234password" } github.com/shopify/ejson
30. Secrets — EJSON github.com/shopify/ejson $ ejson encrypt secrets.production.ejson $ cat secrets.production.ejson { "_public_key": "6623423...", "database_password": "EJ[1:WGj...]" }
31. Secrets — EJSON github.com/shopify/ejson $ cat secrets.production.ejson { "_public_key": "6623423...", "database_password": "EJ[1:WGj...]", "some_new_password": "something secret" } $ ejson encrypt secrets.production.ejson
33. Code reviews ✓ learn from peers
 ✓ catch more bugs
 ✓ code consistency
37. @shopify/checkout
39. Style Guides
40. Style Guides Can every language just have a "go fmt" so we can stop arguing and get shit done. Please, I'm begging you.
41. github.com/bbatsov/rubocop
42. RuboCop github.com/bbatsov/rubocop $ rubocop test.rb Inspecting 1 file W Offenses: test.rb:1:5: C: Use snake_case for method names. def badName ^^^^^^^ 1 file inspected, 1 offense detected
43. RuboCop github.com/bbatsov/rubocop Fully configurable via .rubycop.yml …because some things are personal.
44. RuboCop github.com/bbatsov/rubocop Style/StringLiterals: Enabled: false Style/IndentationWidth: Width: 2
45. RuboCop github.com/bbatsov/rubocop $ rubocop --auto-correct test.rb Inspecting 1 file C Offenses: 
 test.rb:2:1: C: [Corrected] Use 2 (not 0) spaces for indentation. def badName
 test.rb:2:7: C: Use snake_case for method names. def badName ^^^^^^^ 1 file inspected, 2 offenses detected, 1 offense corrected
51. Policial octokit = Octokit::Client.new(access_token: 'secret') detective = Policial::Detective.new(octokit) event = Policial::PullRequestEvent.new(webhook_payload) detective.brief(event) detective.investigate detective.violations github.com/volmer/policial
52. Goal: Focus code reviews on logic, design, architecture, the solution itself and less on style.
53. Shipping at scale Locking • SSH access • Audit trail Coordination • CI status
54. pre 2013 — Ping #operations for a deploy (Capistrano) circa 2014 — Shipit circa 2013 — Rackspace's dreadnot Shipping code at Shopify
55. Shipit github.com/shopify/shipit-engine
59. Measuring
60. StatsD github.com/etsy/statsd
61. StatsD::Instrument github.com/shopify/statsd-instrument
62. Liquid::Template.extend StatsD::Instrument Liquid::Template.statsd_measure :render, 'Liquid.Template.render' github.com/shopify/statsd-instrument
63. PaymentProcessingJob.statsd_count :perform, 'PaymentProcessingJob.processed' github.com/shopify/statsd-instrument
66. Store your Datadog
 graphs/alerts in git github.com/bai/doggy
68. Store your <SaaS Name>
 <Things> in git
71. Here's where shit gets super meta
74. Make your logs searchable
75. Query-able production data
78. Bugsnag
79. Lita github.com/litaio/lita
80. github.com/litaio/lita status.shopify.com
81. github.com/litaio/lita Pagerduty
82. github.com/litaio/lita DDOS protection
83. github.com/litaio/lita Lock Shipit deploys
84. github.com/litaio/lita Admin GitHub actions
85. thanks! :) cjoudrey @