Developing at Scale

Developing at Scale

Talk given at ConFoo 2016 on February 26th, 2016.

85b03650a2ec5235376b0b983a49511a?s=128

Christian Joudrey

February 26, 2016
Tweet

Transcript

  1. developing at scale

  2. cjoudrey @

  3. None
  4. None
  5. None
  6. Scale?

  7. None
  8. None
  9. employee count 275 550 825 1100 2011 2012 2013 2014

    2015
  10. Journey of a new dev

  11. ✓ quick & easy setup
 ✓ reproducible
 ✓ production-like Dev

    environments
  12. pre 2012 — README.md with lots of Bash snippets circa

    2013 — Vagrant with shared Chef cookbooks circa 2012 — GitHub's Boxen (Puppet recipes on local) future — ? Dev environments
  13. ~ $ git clone git@github.com:Shopify/vagrant.git
 ~ $ cd vagrant &&

    vagrant up
 ~/vagrant $ vagrant ssh (vagrant) ~ $ git clone git@github.com:Shopify/shopify.git
 (vagrant) ~ $ cd shopify && script/setup (vagrant) ~/shopify $ script/server
  14. ~ $ git clone git@github.com:Shopify/vagrant.git
 ~ $ cd vagrant &&

    vagrant up
 ~/vagrant $ vagrant ssh (vagrant) ~ $ git clone git@github.com:Shopify/shopify.git
 (vagrant) ~ $ cd shopify && script/setup (vagrant) ~/shopify $ script/server ✓ quick & easy setup
 ✓ reproducible
 ✓ production-like
  15. http://localhost:3000

  16. https://shop1.myshopify.com https://shop2.myshopify.com https://shop3.myshopify.com

  17. https://shop1.myshopify.com https://shop2.myshopify.com https://shop3.myshopify.com /etc/hosts :3000 http:// Rails.env.development?

  18. https://shop1.myshopify.com https://shop1.myshopify.io ~ $ whois myshopify.io Domain : myshopify.io Status

    : Live Expiry : 2016-12-16 ~ $ openssl s_client -connect myshopify.io:443 CONNECTED(00000003) depth=1 /C=US/O=DigiCert Inc/...
  19. Code

  20. Feature flags class Shop < ActiveRecord::Base include HasFeatureFlags end shop.flags.enabled?('free_ssl')

    # false shop.flags.enable('free_ssl')
 shop.flags.enabled?('free_ssl') # true FeatureRollout.enable('test_feature', 50) shop.features.enabled?('test_feature') # true shop2.features.enabled?('test_feature') # false
  21. A/B Tests Verdict::Experiment.define :my_experiment do # Block returns true if

    the subject # is qualified to participate. qualify { |subject, context| ... } groups do group :blue_button, 50 group :control, 50 end storage Verdict::Storage::MemoryStorage.new end github.com/shopify/verdict
  22. A/B Tests case Verdict[:my_experiment].switch(checkout) when :blue_button # ... when :control

    # ... else # unqualified subjects end github.com/shopify/verdict
  23. Secrets I don’t even know where to start.

  24. Secrets circa 2014 — Secrets stored in encrypted data bags

    in Chef. Devs pinged Ops to get secrets added/changed. Hard to trace changes to secrets. Hard to make changes.
  25. Secrets circa 2014 — Secrets stored in encrypted data bags

    in Chef. Devs pinged Ops to get secrets added/changed. Hard to trace changes to secrets. tl;dr — It was pretty painful!
  26. Secrets — EJSON github.com/shopify/ejson now — Secrets can be safely

    stored in app source. Anyone has access to change/add secrets. Only production servers can decrypt secrets. Secrets change synchronously with app source.
  27. Secrets — EJSON github.com/shopify/ejson now — Secrets can be safely

    stored in app source. Anyone has access to change/add secrets. Only production servers can decrypt secrets. Secrets change synchronously with app source. tl;dr — A lot better!
  28. Secrets — EJSON github.com/shopify/ejson $ ejson keygen Public Key: 6623423a9492e68e12eeb1c705888aebdcc0080af7e594fc402b

    eb24cce9d14f Private Key: 75b80b4a693156eb435f4ed2fe397e583f461f09fd99ec2bd1bd ef0a56cf6e64
  29. Secrets — EJSON $ cat secrets.production.ejson { "_public_key": "6623423...", "database_password":

    "1234password" } github.com/shopify/ejson
  30. Secrets — EJSON github.com/shopify/ejson $ ejson encrypt secrets.production.ejson $ cat

    secrets.production.ejson { "_public_key": "6623423...", "database_password": "EJ[1:WGj...]" }
  31. Secrets — EJSON github.com/shopify/ejson $ cat secrets.production.ejson { "_public_key": "6623423...",

    "database_password": "EJ[1:WGj...]", "some_new_password": "something secret" } $ ejson encrypt secrets.production.ejson
  32. None
  33. Code reviews ✓ learn from peers
 ✓ catch more bugs


    ✓ code consistency
  34. None
  35. None
  36. None
  37. @shopify/checkout

  38. None
  39. Style Guides

  40. Style Guides Can every language just have a "go fmt"

    so we can stop arguing and get shit done. Please, I'm begging you.
  41. github.com/bbatsov/rubocop

  42. RuboCop github.com/bbatsov/rubocop $ rubocop test.rb Inspecting 1 file W Offenses:

    test.rb:1:5: C: Use snake_case for method names. def badName ^^^^^^^ 1 file inspected, 1 offense detected
  43. RuboCop github.com/bbatsov/rubocop Fully configurable via .rubycop.yml …because some things are

    personal.
  44. RuboCop github.com/bbatsov/rubocop Style/StringLiterals: Enabled: false Style/IndentationWidth: Width: 2

  45. RuboCop github.com/bbatsov/rubocop $ rubocop --auto-correct test.rb Inspecting 1 file C

    Offenses: 
 test.rb:2:1: C: [Corrected] Use 2 (not 0) spaces for indentation. def badName
 test.rb:2:7: C: Use snake_case for method names. def badName ^^^^^^^ 1 file inspected, 2 offenses detected, 1 offense corrected
  46. None
  47. None
  48. None
  49. None
  50. None
  51. Policial octokit = Octokit::Client.new(access_token: 'secret') detective = Policial::Detective.new(octokit) event =

    Policial::PullRequestEvent.new(webhook_payload) detective.brief(event) detective.investigate detective.violations github.com/volmer/policial
  52. Goal: Focus code reviews on logic, design, architecture, the solution

    itself and less on style.
  53. Shipping at scale Locking • SSH access • Audit trail

    Coordination • CI status
  54. pre 2013 — Ping #operations for a deploy (Capistrano) circa

    2014 — Shipit circa 2013 — Rackspace's dreadnot Shipping code at Shopify
  55. Shipit github.com/shopify/shipit-engine

  56. None
  57. None
  58. None
  59. Measuring

  60. StatsD github.com/etsy/statsd

  61. StatsD::Instrument github.com/shopify/statsd-instrument

  62. Liquid::Template.extend StatsD::Instrument Liquid::Template.statsd_measure :render, 'Liquid.Template.render' github.com/shopify/statsd-instrument

  63. PaymentProcessingJob.statsd_count :perform, 'PaymentProcessingJob.processed' github.com/shopify/statsd-instrument

  64. None
  65. None
  66. Store your Datadog
 graphs/alerts in git github.com/bai/doggy

  67. None
  68. Store your <SaaS Name>
 <Things> in git

  69. None
  70. None
  71. Here's where shit gets super meta

  72. None
  73. None
  74. Make your logs searchable

  75. Query-able production data

  76. None
  77. None
  78. Bugsnag

  79. Lita github.com/litaio/lita

  80. github.com/litaio/lita status.shopify.com

  81. github.com/litaio/lita Pagerduty

  82. github.com/litaio/lita DDOS protection

  83. github.com/litaio/lita Lock Shipit deploys

  84. github.com/litaio/lita Admin GitHub actions

  85. thanks! :) cjoudrey @