Developing at Scale

Talk given at ConFoo 2016 on February 26th, 2016.

Transcript

1. developing
   at
   scale
   

   View Slide

2. cjoudrey @
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. Scale?
   

   View Slide

7. View Slide

8. View Slide

9. employee count
   275
   550
   825
   1100
   2011 2012 2013 2014 2015
   

   View Slide

10. Journey of a new dev
    

    View Slide

11. ✓ quick & easy setup

    ✓ reproducible

    ✓ production-like
    Dev environments
    

    View Slide

12. pre 2012 — README.md with lots of Bash snippets
    circa 2013 — Vagrant with shared Chef cookbooks
    circa 2012 — GitHub's Boxen (Puppet recipes on local)
    future — ?
    Dev environments
    

    View Slide

13. ~ $ git clone git@github.com:Shopify/vagrant.git

    ~ $ cd vagrant && vagrant up

    ~/vagrant $ vagrant ssh
    (vagrant) ~ $ git clone git@github.com:Shopify/shopify.git

    (vagrant) ~ $ cd shopify && script/setup
    (vagrant) ~/shopify $ script/server
    

    View Slide

14. ~ $ git clone git@github.com:Shopify/vagrant.git

    ~ $ cd vagrant && vagrant up

    ~/vagrant $ vagrant ssh
    (vagrant) ~ $ git clone git@github.com:Shopify/shopify.git

    (vagrant) ~ $ cd shopify && script/setup
    (vagrant) ~/shopify $ script/server
    ✓ quick & easy setup

    ✓ reproducible

    ✓ production-like
    

    View Slide

15. http://localhost:3000
    

    View Slide

16. https://shop1.myshopify.com
    https://shop2.myshopify.com
    https://shop3.myshopify.com
    

    View Slide

17. https://shop1.myshopify.com
    https://shop2.myshopify.com
    https://shop3.myshopify.com
    /etc/hosts
    :3000
    http://
    Rails.env.development?
    

    View Slide

18. https://shop1.myshopify.com
    https://shop1.myshopify.io
    ~ $ whois myshopify.io
    Domain : myshopify.io
    Status : Live
    Expiry : 2016-12-16
    ~ $ openssl s_client -connect myshopify.io:443
    CONNECTED(00000003)
    depth=1 /C=US/O=DigiCert Inc/...
    

    View Slide

19. Code
    

    View Slide

20. Feature flags
    class Shop < ActiveRecord::Base
    include HasFeatureFlags
    end
    shop.flags.enabled?('free_ssl') # false
    shop.flags.enable('free_ssl')

    shop.flags.enabled?('free_ssl') # true
    FeatureRollout.enable('test_feature', 50)
    shop.features.enabled?('test_feature') # true
    shop2.features.enabled?('test_feature') # false
    

    View Slide

21. A/B Tests
    Verdict::Experiment.define :my_experiment do
    # Block returns true if the subject
    # is qualified to participate.
    qualify { |subject, context| ... }
    groups do
    group :blue_button, 50
    group :control, 50
    end
    storage Verdict::Storage::MemoryStorage.new
    end
    github.com/shopify/verdict
    

    View Slide

22. A/B Tests
    case Verdict[:my_experiment].switch(checkout)
    when :blue_button
    # ...
    when :control
    # ...
    else
    # unqualified subjects
    end
    github.com/shopify/verdict
    

    View Slide

23. Secrets
    I don’t even know where to start.
    

    View Slide

24. Secrets
    circa 2014 —
    Secrets stored in encrypted data bags in Chef.
    Devs pinged Ops to get secrets added/changed.
    Hard to trace changes to secrets.
    Hard to make changes.
    

    View Slide

25. Secrets
    circa 2014 —
    Secrets stored in encrypted data bags in Chef.
    Devs pinged Ops to get secrets added/changed.
    Hard to trace changes to secrets.
    tl;dr — It was pretty painful!
    

    View Slide

26. Secrets — EJSON
    github.com/shopify/ejson
    now —
    Secrets can be safely stored in app source.
    Anyone has access to change/add secrets.
    Only production servers can decrypt secrets.
    Secrets change synchronously with app source.
    

    View Slide

27. Secrets — EJSON
    github.com/shopify/ejson
    now —
    Secrets can be safely stored in app source.
    Anyone has access to change/add secrets.
    Only production servers can decrypt secrets.
    Secrets change synchronously with app source.
    tl;dr — A lot better!
    

    View Slide

28. Secrets — EJSON
    github.com/shopify/ejson
    $ ejson keygen
    Public Key:
    6623423a9492e68e12eeb1c705888aebdcc0080af7e594fc402b
    eb24cce9d14f
    Private Key:
    75b80b4a693156eb435f4ed2fe397e583f461f09fd99ec2bd1bd
    ef0a56cf6e64
    

    View Slide

29. Secrets — EJSON
    $ cat secrets.production.ejson
    {
    "_public_key": "6623423...",
    "database_password": "1234password"
    }
    github.com/shopify/ejson
    

    View Slide

30. Secrets — EJSON
    github.com/shopify/ejson
    $ ejson encrypt secrets.production.ejson
    $ cat secrets.production.ejson
    {
    "_public_key": "6623423...",
    "database_password": "EJ[1:WGj...]"
    }
    

    View Slide

31. Secrets — EJSON
    github.com/shopify/ejson
    $ cat secrets.production.ejson
    {
    "_public_key": "6623423...",
    "database_password": "EJ[1:WGj...]",
    "some_new_password": "something secret"
    }
    $ ejson encrypt secrets.production.ejson
    

    View Slide

32. View Slide

33. Code reviews
    ✓ learn from peers

    ✓ catch more bugs

    ✓ code consistency
    

    View Slide

34. View Slide

35. View Slide

36. View Slide

37. @shopify/checkout
    

    View Slide

38. View Slide

39. Style Guides
    

    View Slide

40. Style Guides
    Can every language just have a "go fmt" so
    we can stop arguing and get shit done.
    Please, I'm begging you.
    

    View Slide

41. github.com/bbatsov/rubocop
    

    View Slide

42. RuboCop
    github.com/bbatsov/rubocop
    $ rubocop test.rb
    Inspecting 1 file
    W
    Offenses:
    test.rb:1:5: C: Use snake_case for method names.
    def badName
    ^^^^^^^
    1 file inspected, 1 offense detected
    

    View Slide

43. RuboCop
    github.com/bbatsov/rubocop
    Fully configurable via .rubycop.yml
    …because some things are personal.
    

    View Slide

44. RuboCop
    github.com/bbatsov/rubocop
    Style/StringLiterals:
    Enabled: false
    Style/IndentationWidth:
    Width: 2
    

    View Slide

45. RuboCop
    github.com/bbatsov/rubocop
    $ rubocop --auto-correct test.rb
    Inspecting 1 file
    C
    Offenses:
    

    test.rb:2:1: C: [Corrected] Use 2 (not 0) spaces for indentation.
    def badName

    test.rb:2:7: C: Use snake_case for method names.
    def badName
    ^^^^^^^
    1 file inspected, 2 offenses detected, 1 offense corrected
    

    View Slide

46. View Slide

47. View Slide

48. View Slide

49. View Slide

50. View Slide

51. Policial
    octokit = Octokit::Client.new(access_token: 'secret')
    detective = Policial::Detective.new(octokit)
    event = Policial::PullRequestEvent.new(webhook_payload)
    detective.brief(event)
    detective.investigate
    detective.violations
    github.com/volmer/policial
    

    View Slide

52. Goal: Focus code reviews on logic,
    design, architecture, the solution
    itself and less on style.
    

    View Slide

53. Shipping at scale
    Locking • SSH access • Audit trail
    Coordination • CI status
    

    View Slide

54. pre 2013 — Ping #operations for a deploy (Capistrano)
    circa 2014 — Shipit
    circa 2013 — Rackspace's dreadnot
    Shipping code at Shopify
    

    View Slide

55. Shipit
    github.com/shopify/shipit-engine
    

    View Slide

56. View Slide

57. View Slide

58. View Slide

59. Measuring
    

    View Slide

60. StatsD
    github.com/etsy/statsd
    

    View Slide

61. StatsD::Instrument
    github.com/shopify/statsd-instrument
    

    View Slide

62. Liquid::Template.extend StatsD::Instrument
    Liquid::Template.statsd_measure :render,
    'Liquid.Template.render'
    github.com/shopify/statsd-instrument
    

    View Slide

63. PaymentProcessingJob.statsd_count :perform,
    'PaymentProcessingJob.processed'
    github.com/shopify/statsd-instrument
    

    View Slide

64. View Slide

65. View Slide

66. Store your Datadog

    graphs/alerts in git
    github.com/bai/doggy
    

    View Slide

67. View Slide

68. Store your 

    in git
    

    View Slide

69. View Slide

70. View Slide

71. Here's where shit gets super meta
    

    View Slide

72. View Slide

73. View Slide

74. Make your logs searchable
    

    View Slide

75. Query-able production data
    

    View Slide

76. View Slide

77. View Slide

78. Bugsnag
    

    View Slide

79. Lita
    github.com/litaio/lita
    

    View Slide

80. github.com/litaio/lita
    status.shopify.com
    

    View Slide

81. github.com/litaio/lita
    Pagerduty
    

    View Slide

82. github.com/litaio/lita
    DDOS protection
    

    View Slide

83. github.com/litaio/lita
    Lock Shipit deploys
    

    View Slide

84. github.com/litaio/lita
    Admin GitHub actions
    

    View Slide

85. thanks! :)
    cjoudrey @
    

    View Slide