$30 off During Our Annual Pro Sale. View Details »

Developing at Scale

Developing at Scale

Talk given at ConFoo 2016 on February 26th, 2016.

Christian Joudrey

February 26, 2016
Tweet

More Decks by Christian Joudrey

Other Decks in Programming

Transcript

  1. developing
    at
    scale

    View Slide

  2. cjoudrey @

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. Scale?

    View Slide

  7. View Slide

  8. View Slide

  9. employee count
    275
    550
    825
    1100
    2011 2012 2013 2014 2015

    View Slide

  10. Journey of a new dev

    View Slide

  11. ✓ quick & easy setup

    ✓ reproducible

    ✓ production-like
    Dev environments

    View Slide

  12. pre 2012 — README.md with lots of Bash snippets
    circa 2013 — Vagrant with shared Chef cookbooks
    circa 2012 — GitHub's Boxen (Puppet recipes on local)
    future — ?
    Dev environments

    View Slide

  13. ~ $ git clone [email protected]:Shopify/vagrant.git

    ~ $ cd vagrant && vagrant up

    ~/vagrant $ vagrant ssh
    (vagrant) ~ $ git clone [email protected]:Shopify/shopify.git

    (vagrant) ~ $ cd shopify && script/setup
    (vagrant) ~/shopify $ script/server

    View Slide

  14. ~ $ git clone [email protected]:Shopify/vagrant.git

    ~ $ cd vagrant && vagrant up

    ~/vagrant $ vagrant ssh
    (vagrant) ~ $ git clone [email protected]:Shopify/shopify.git

    (vagrant) ~ $ cd shopify && script/setup
    (vagrant) ~/shopify $ script/server
    ✓ quick & easy setup

    ✓ reproducible

    ✓ production-like

    View Slide

  15. http://localhost:3000

    View Slide

  16. https://shop1.myshopify.com
    https://shop2.myshopify.com
    https://shop3.myshopify.com

    View Slide

  17. https://shop1.myshopify.com
    https://shop2.myshopify.com
    https://shop3.myshopify.com
    /etc/hosts
    :3000
    http://
    Rails.env.development?

    View Slide

  18. https://shop1.myshopify.com
    https://shop1.myshopify.io
    ~ $ whois myshopify.io
    Domain : myshopify.io
    Status : Live
    Expiry : 2016-12-16
    ~ $ openssl s_client -connect myshopify.io:443
    CONNECTED(00000003)
    depth=1 /C=US/O=DigiCert Inc/...

    View Slide

  19. Code

    View Slide

  20. Feature flags
    class Shop < ActiveRecord::Base
    include HasFeatureFlags
    end
    shop.flags.enabled?('free_ssl') # false
    shop.flags.enable('free_ssl')

    shop.flags.enabled?('free_ssl') # true
    FeatureRollout.enable('test_feature', 50)
    shop.features.enabled?('test_feature') # true
    shop2.features.enabled?('test_feature') # false

    View Slide

  21. A/B Tests
    Verdict::Experiment.define :my_experiment do
    # Block returns true if the subject
    # is qualified to participate.
    qualify { |subject, context| ... }
    groups do
    group :blue_button, 50
    group :control, 50
    end
    storage Verdict::Storage::MemoryStorage.new
    end
    github.com/shopify/verdict

    View Slide

  22. A/B Tests
    case Verdict[:my_experiment].switch(checkout)
    when :blue_button
    # ...
    when :control
    # ...
    else
    # unqualified subjects
    end
    github.com/shopify/verdict

    View Slide

  23. Secrets
    I don’t even know where to start.

    View Slide

  24. Secrets
    circa 2014 —
    Secrets stored in encrypted data bags in Chef.
    Devs pinged Ops to get secrets added/changed.
    Hard to trace changes to secrets.
    Hard to make changes.

    View Slide

  25. Secrets
    circa 2014 —
    Secrets stored in encrypted data bags in Chef.
    Devs pinged Ops to get secrets added/changed.
    Hard to trace changes to secrets.
    tl;dr — It was pretty painful!

    View Slide

  26. Secrets — EJSON
    github.com/shopify/ejson
    now —
    Secrets can be safely stored in app source.
    Anyone has access to change/add secrets.
    Only production servers can decrypt secrets.
    Secrets change synchronously with app source.

    View Slide

  27. Secrets — EJSON
    github.com/shopify/ejson
    now —
    Secrets can be safely stored in app source.
    Anyone has access to change/add secrets.
    Only production servers can decrypt secrets.
    Secrets change synchronously with app source.
    tl;dr — A lot better!

    View Slide

  28. Secrets — EJSON
    github.com/shopify/ejson
    $ ejson keygen
    Public Key:
    6623423a9492e68e12eeb1c705888aebdcc0080af7e594fc402b
    eb24cce9d14f
    Private Key:
    75b80b4a693156eb435f4ed2fe397e583f461f09fd99ec2bd1bd
    ef0a56cf6e64

    View Slide

  29. Secrets — EJSON
    $ cat secrets.production.ejson
    {
    "_public_key": "6623423...",
    "database_password": "1234password"
    }
    github.com/shopify/ejson

    View Slide

  30. Secrets — EJSON
    github.com/shopify/ejson
    $ ejson encrypt secrets.production.ejson
    $ cat secrets.production.ejson
    {
    "_public_key": "6623423...",
    "database_password": "EJ[1:WGj...]"
    }

    View Slide

  31. Secrets — EJSON
    github.com/shopify/ejson
    $ cat secrets.production.ejson
    {
    "_public_key": "6623423...",
    "database_password": "EJ[1:WGj...]",
    "some_new_password": "something secret"
    }
    $ ejson encrypt secrets.production.ejson

    View Slide

  32. View Slide

  33. Code reviews
    ✓ learn from peers

    ✓ catch more bugs

    ✓ code consistency

    View Slide

  34. View Slide

  35. View Slide

  36. View Slide

  37. @shopify/checkout

    View Slide

  38. View Slide

  39. Style Guides

    View Slide

  40. Style Guides
    Can every language just have a "go fmt" so
    we can stop arguing and get shit done.
    Please, I'm begging you.

    View Slide

  41. github.com/bbatsov/rubocop

    View Slide

  42. RuboCop
    github.com/bbatsov/rubocop
    $ rubocop test.rb
    Inspecting 1 file
    W
    Offenses:
    test.rb:1:5: C: Use snake_case for method names.
    def badName
    ^^^^^^^
    1 file inspected, 1 offense detected

    View Slide

  43. RuboCop
    github.com/bbatsov/rubocop
    Fully configurable via .rubycop.yml
    …because some things are personal.

    View Slide

  44. RuboCop
    github.com/bbatsov/rubocop
    Style/StringLiterals:
    Enabled: false
    Style/IndentationWidth:
    Width: 2

    View Slide

  45. RuboCop
    github.com/bbatsov/rubocop
    $ rubocop --auto-correct test.rb
    Inspecting 1 file
    C
    Offenses:

    test.rb:2:1: C: [Corrected] Use 2 (not 0) spaces for indentation.
    def badName

    test.rb:2:7: C: Use snake_case for method names.
    def badName
    ^^^^^^^
    1 file inspected, 2 offenses detected, 1 offense corrected

    View Slide

  46. View Slide

  47. View Slide

  48. View Slide

  49. View Slide

  50. View Slide

  51. Policial
    octokit = Octokit::Client.new(access_token: 'secret')
    detective = Policial::Detective.new(octokit)
    event = Policial::PullRequestEvent.new(webhook_payload)
    detective.brief(event)
    detective.investigate
    detective.violations
    github.com/volmer/policial

    View Slide

  52. Goal: Focus code reviews on logic,
    design, architecture, the solution
    itself and less on style.

    View Slide

  53. Shipping at scale
    Locking • SSH access • Audit trail
    Coordination • CI status

    View Slide

  54. pre 2013 — Ping #operations for a deploy (Capistrano)
    circa 2014 — Shipit
    circa 2013 — Rackspace's dreadnot
    Shipping code at Shopify

    View Slide

  55. Shipit
    github.com/shopify/shipit-engine

    View Slide

  56. View Slide

  57. View Slide

  58. View Slide

  59. Measuring

    View Slide

  60. StatsD
    github.com/etsy/statsd

    View Slide

  61. StatsD::Instrument
    github.com/shopify/statsd-instrument

    View Slide

  62. Liquid::Template.extend StatsD::Instrument
    Liquid::Template.statsd_measure :render,
    'Liquid.Template.render'
    github.com/shopify/statsd-instrument

    View Slide

  63. PaymentProcessingJob.statsd_count :perform,
    'PaymentProcessingJob.processed'
    github.com/shopify/statsd-instrument

    View Slide

  64. View Slide

  65. View Slide

  66. Store your Datadog

    graphs/alerts in git
    github.com/bai/doggy

    View Slide

  67. View Slide

  68. Store your 

    in git

    View Slide

  69. View Slide

  70. View Slide

  71. Here's where shit gets super meta

    View Slide

  72. View Slide

  73. View Slide

  74. Make your logs searchable

    View Slide

  75. Query-able production data

    View Slide

  76. View Slide

  77. View Slide

  78. Bugsnag

    View Slide

  79. Lita
    github.com/litaio/lita

    View Slide

  80. github.com/litaio/lita
    status.shopify.com

    View Slide

  81. github.com/litaio/lita
    Pagerduty

    View Slide

  82. github.com/litaio/lita
    DDOS protection

    View Slide

  83. github.com/litaio/lita
    Lock Shipit deploys

    View Slide

  84. github.com/litaio/lita
    Admin GitHub actions

    View Slide

  85. thanks! :)
    cjoudrey @

    View Slide