Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Developing at Scale

Developing at Scale

Talk given at ConFoo 2016 on February 26th, 2016.

Christian Joudrey

February 26, 2016
Tweet

More Decks by Christian Joudrey

Other Decks in Programming

Transcript

  1. developing at scale

  2. cjoudrey @

  3. None
  4. None
  5. None
  6. Scale?

  7. None
  8. None
  9. employee count 275 550 825 1100 2011 2012 2013 2014

    2015
  10. Journey of a new dev

  11. ✓ quick & easy setup
 ✓ reproducible
 ✓ production-like Dev

    environments
  12. pre 2012 — README.md with lots of Bash snippets circa

    2013 — Vagrant with shared Chef cookbooks circa 2012 — GitHub's Boxen (Puppet recipes on local) future — ? Dev environments
  13. ~ $ git clone git@github.com:Shopify/vagrant.git
 ~ $ cd vagrant &&

    vagrant up
 ~/vagrant $ vagrant ssh (vagrant) ~ $ git clone git@github.com:Shopify/shopify.git
 (vagrant) ~ $ cd shopify && script/setup (vagrant) ~/shopify $ script/server
  14. ~ $ git clone git@github.com:Shopify/vagrant.git
 ~ $ cd vagrant &&

    vagrant up
 ~/vagrant $ vagrant ssh (vagrant) ~ $ git clone git@github.com:Shopify/shopify.git
 (vagrant) ~ $ cd shopify && script/setup (vagrant) ~/shopify $ script/server ✓ quick & easy setup
 ✓ reproducible
 ✓ production-like
  15. http://localhost:3000

  16. https://shop1.myshopify.com https://shop2.myshopify.com https://shop3.myshopify.com

  17. https://shop1.myshopify.com https://shop2.myshopify.com https://shop3.myshopify.com /etc/hosts :3000 http:// Rails.env.development?

  18. https://shop1.myshopify.com https://shop1.myshopify.io ~ $ whois myshopify.io Domain : myshopify.io Status

    : Live Expiry : 2016-12-16 ~ $ openssl s_client -connect myshopify.io:443 CONNECTED(00000003) depth=1 /C=US/O=DigiCert Inc/...
  19. Code

  20. Feature flags class Shop < ActiveRecord::Base include HasFeatureFlags end shop.flags.enabled?('free_ssl')

    # false shop.flags.enable('free_ssl')
 shop.flags.enabled?('free_ssl') # true FeatureRollout.enable('test_feature', 50) shop.features.enabled?('test_feature') # true shop2.features.enabled?('test_feature') # false
  21. A/B Tests Verdict::Experiment.define :my_experiment do # Block returns true if

    the subject # is qualified to participate. qualify { |subject, context| ... } groups do group :blue_button, 50 group :control, 50 end storage Verdict::Storage::MemoryStorage.new end github.com/shopify/verdict
  22. A/B Tests case Verdict[:my_experiment].switch(checkout) when :blue_button # ... when :control

    # ... else # unqualified subjects end github.com/shopify/verdict
  23. Secrets I don’t even know where to start.

  24. Secrets circa 2014 — Secrets stored in encrypted data bags

    in Chef. Devs pinged Ops to get secrets added/changed. Hard to trace changes to secrets. Hard to make changes.
  25. Secrets circa 2014 — Secrets stored in encrypted data bags

    in Chef. Devs pinged Ops to get secrets added/changed. Hard to trace changes to secrets. tl;dr — It was pretty painful!
  26. Secrets — EJSON github.com/shopify/ejson now — Secrets can be safely

    stored in app source. Anyone has access to change/add secrets. Only production servers can decrypt secrets. Secrets change synchronously with app source.
  27. Secrets — EJSON github.com/shopify/ejson now — Secrets can be safely

    stored in app source. Anyone has access to change/add secrets. Only production servers can decrypt secrets. Secrets change synchronously with app source. tl;dr — A lot better!
  28. Secrets — EJSON github.com/shopify/ejson $ ejson keygen Public Key: 6623423a9492e68e12eeb1c705888aebdcc0080af7e594fc402b

    eb24cce9d14f Private Key: 75b80b4a693156eb435f4ed2fe397e583f461f09fd99ec2bd1bd ef0a56cf6e64
  29. Secrets — EJSON $ cat secrets.production.ejson { "_public_key": "6623423...", "database_password":

    "1234password" } github.com/shopify/ejson
  30. Secrets — EJSON github.com/shopify/ejson $ ejson encrypt secrets.production.ejson $ cat

    secrets.production.ejson { "_public_key": "6623423...", "database_password": "EJ[1:WGj...]" }
  31. Secrets — EJSON github.com/shopify/ejson $ cat secrets.production.ejson { "_public_key": "6623423...",

    "database_password": "EJ[1:WGj...]", "some_new_password": "something secret" } $ ejson encrypt secrets.production.ejson
  32. None
  33. Code reviews ✓ learn from peers
 ✓ catch more bugs


    ✓ code consistency
  34. None
  35. None
  36. None
  37. @shopify/checkout

  38. None
  39. Style Guides

  40. Style Guides Can every language just have a "go fmt"

    so we can stop arguing and get shit done. Please, I'm begging you.
  41. github.com/bbatsov/rubocop

  42. RuboCop github.com/bbatsov/rubocop $ rubocop test.rb Inspecting 1 file W Offenses:

    test.rb:1:5: C: Use snake_case for method names. def badName ^^^^^^^ 1 file inspected, 1 offense detected
  43. RuboCop github.com/bbatsov/rubocop Fully configurable via .rubycop.yml …because some things are

    personal.
  44. RuboCop github.com/bbatsov/rubocop Style/StringLiterals: Enabled: false Style/IndentationWidth: Width: 2

  45. RuboCop github.com/bbatsov/rubocop $ rubocop --auto-correct test.rb Inspecting 1 file C

    Offenses: 
 test.rb:2:1: C: [Corrected] Use 2 (not 0) spaces for indentation. def badName
 test.rb:2:7: C: Use snake_case for method names. def badName ^^^^^^^ 1 file inspected, 2 offenses detected, 1 offense corrected
  46. None
  47. None
  48. None
  49. None
  50. None
  51. Policial octokit = Octokit::Client.new(access_token: 'secret') detective = Policial::Detective.new(octokit) event =

    Policial::PullRequestEvent.new(webhook_payload) detective.brief(event) detective.investigate detective.violations github.com/volmer/policial
  52. Goal: Focus code reviews on logic, design, architecture, the solution

    itself and less on style.
  53. Shipping at scale Locking • SSH access • Audit trail

    Coordination • CI status
  54. pre 2013 — Ping #operations for a deploy (Capistrano) circa

    2014 — Shipit circa 2013 — Rackspace's dreadnot Shipping code at Shopify
  55. Shipit github.com/shopify/shipit-engine

  56. None
  57. None
  58. None
  59. Measuring

  60. StatsD github.com/etsy/statsd

  61. StatsD::Instrument github.com/shopify/statsd-instrument

  62. Liquid::Template.extend StatsD::Instrument Liquid::Template.statsd_measure :render, 'Liquid.Template.render' github.com/shopify/statsd-instrument

  63. PaymentProcessingJob.statsd_count :perform, 'PaymentProcessingJob.processed' github.com/shopify/statsd-instrument

  64. None
  65. None
  66. Store your Datadog
 graphs/alerts in git github.com/bai/doggy

  67. None
  68. Store your <SaaS Name>
 <Things> in git

  69. None
  70. None
  71. Here's where shit gets super meta

  72. None
  73. None
  74. Make your logs searchable

  75. Query-able production data

  76. None
  77. None
  78. Bugsnag

  79. Lita github.com/litaio/lita

  80. github.com/litaio/lita status.shopify.com

  81. github.com/litaio/lita Pagerduty

  82. github.com/litaio/lita DDOS protection

  83. github.com/litaio/lita Lock Shipit deploys

  84. github.com/litaio/lita Admin GitHub actions

  85. thanks! :) cjoudrey @