Developing at Scale - Speaker Deck

Slide 1

Slide 1 text

developing at scale

Slide 2

Slide 2 text

cjoudrey @

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Scale?

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

employee count 275 550 825 1100 2011 2012 2013 2014 2015

Slide 10

Slide 10 text

Journey of a new dev

Slide 11

Slide 11 text

✓ quick & easy setup  ✓ reproducible  ✓ production-like Dev environments

Slide 12

Slide 12 text

pre 2012 — README.md with lots of Bash snippets circa 2013 — Vagrant with shared Chef cookbooks circa 2012 — GitHub's Boxen (Puppet recipes on local) future — ? Dev environments

Slide 13

Slide 13 text

~ $ git clone [email protected]:Shopify/vagrant.git  ~ $ cd vagrant && vagrant up  ~/vagrant $ vagrant ssh (vagrant) ~ $ git clone [email protected]:Shopify/shopify.git  (vagrant) ~ $ cd shopify && script/setup (vagrant) ~/shopify $ script/server

Slide 14

Slide 14 text

Slide 15

Slide 15 text

http://localhost:3000

Slide 16

Slide 16 text

https://shop1.myshopify.com https://shop2.myshopify.com https://shop3.myshopify.com

Slide 17

Slide 17 text

https://shop1.myshopify.com https://shop2.myshopify.com https://shop3.myshopify.com /etc/hosts :3000 http:// Rails.env.development?

Slide 18

Slide 18 text

https://shop1.myshopify.com https://shop1.myshopify.io ~ $ whois myshopify.io Domain : myshopify.io Status : Live Expiry : 2016-12-16 ~ $ openssl s_client -connect myshopify.io:443 CONNECTED(00000003) depth=1 /C=US/O=DigiCert Inc/...

Slide 19

Slide 19 text

Code

Slide 20

Slide 20 text

Feature flags class Shop < ActiveRecord::Base include HasFeatureFlags end shop.flags.enabled?('free_ssl') # false shop.flags.enable('free_ssl')  shop.flags.enabled?('free_ssl') # true FeatureRollout.enable('test_feature', 50) shop.features.enabled?('test_feature') # true shop2.features.enabled?('test_feature') # false

Slide 21

Slide 21 text

A/B Tests Verdict::Experiment.define :my_experiment do # Block returns true if the subject # is qualified to participate. qualify { |subject, context| ... } groups do group :blue_button, 50 group :control, 50 end storage Verdict::Storage::MemoryStorage.new end github.com/shopify/verdict

Slide 22

Slide 22 text

A/B Tests case Verdict[:my_experiment].switch(checkout) when :blue_button # ... when :control # ... else # unqualified subjects end github.com/shopify/verdict

Slide 23

Slide 23 text

Secrets I don’t even know where to start.

Slide 24

Slide 24 text

Secrets circa 2014 — Secrets stored in encrypted data bags in Chef. Devs pinged Ops to get secrets added/changed. Hard to trace changes to secrets. Hard to make changes.

Slide 25

Slide 25 text

Secrets circa 2014 — Secrets stored in encrypted data bags in Chef. Devs pinged Ops to get secrets added/changed. Hard to trace changes to secrets. tl;dr — It was pretty painful!

Slide 26

Slide 26 text

Secrets — EJSON github.com/shopify/ejson now — Secrets can be safely stored in app source. Anyone has access to change/add secrets. Only production servers can decrypt secrets. Secrets change synchronously with app source.

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Secrets — EJSON github.com/shopify/ejson $ ejson keygen Public Key: 6623423a9492e68e12eeb1c705888aebdcc0080af7e594fc402b eb24cce9d14f Private Key: 75b80b4a693156eb435f4ed2fe397e583f461f09fd99ec2bd1bd ef0a56cf6e64

Slide 29

Slide 29 text

Secrets — EJSON $ cat secrets.production.ejson { "_public_key": "6623423...", "database_password": "1234password" } github.com/shopify/ejson

Slide 30

Slide 30 text

Secrets — EJSON github.com/shopify/ejson $ ejson encrypt secrets.production.ejson $ cat secrets.production.ejson { "_public_key": "6623423...", "database_password": "EJ[1:WGj...]" }

Slide 31

Slide 31 text

Secrets — EJSON github.com/shopify/ejson $ cat secrets.production.ejson { "_public_key": "6623423...", "database_password": "EJ[1:WGj...]", "some_new_password": "something secret" } $ ejson encrypt secrets.production.ejson

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

Code reviews ✓ learn from peers  ✓ catch more bugs  ✓ code consistency

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

@shopify/checkout

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

Style Guides

Slide 40

Slide 40 text

Style Guides Can every language just have a "go fmt" so we can stop arguing and get shit done. Please, I'm begging you.

Slide 41

Slide 41 text

github.com/bbatsov/rubocop

Slide 42

Slide 42 text

RuboCop github.com/bbatsov/rubocop $ rubocop test.rb Inspecting 1 file W Offenses: test.rb:1:5: C: Use snake_case for method names. def badName ^^^^^^^ 1 file inspected, 1 offense detected

Slide 43

Slide 43 text

RuboCop github.com/bbatsov/rubocop Fully configurable via .rubycop.yml …because some things are personal.

Slide 44

Slide 44 text

RuboCop github.com/bbatsov/rubocop Style/StringLiterals: Enabled: false Style/IndentationWidth: Width: 2

Slide 45

Slide 45 text

RuboCop github.com/bbatsov/rubocop $ rubocop --auto-correct test.rb Inspecting 1 file C Offenses:   test.rb:2:1: C: [Corrected] Use 2 (not 0) spaces for indentation. def badName  test.rb:2:7: C: Use snake_case for method names. def badName ^^^^^^^ 1 file inspected, 2 offenses detected, 1 offense corrected

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

Policial octokit = Octokit::Client.new(access_token: 'secret') detective = Policial::Detective.new(octokit) event = Policial::PullRequestEvent.new(webhook_payload) detective.brief(event) detective.investigate detective.violations github.com/volmer/policial

Slide 52

Slide 52 text

Goal: Focus code reviews on logic, design, architecture, the solution itself and less on style.

Slide 53

Slide 53 text

Shipping at scale Locking • SSH access • Audit trail Coordination • CI status

Slide 54

Slide 54 text

pre 2013 — Ping #operations for a deploy (Capistrano) circa 2014 — Shipit circa 2013 — Rackspace's dreadnot Shipping code at Shopify

Slide 55

Slide 55 text

Shipit github.com/shopify/shipit-engine

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

Measuring

Slide 60

Slide 60 text

StatsD github.com/etsy/statsd

Slide 61

Slide 61 text

StatsD::Instrument github.com/shopify/statsd-instrument

Slide 62

Slide 62 text

Liquid::Template.extend StatsD::Instrument Liquid::Template.statsd_measure :render, 'Liquid.Template.render' github.com/shopify/statsd-instrument

Slide 63

Slide 63 text

PaymentProcessingJob.statsd_count :perform, 'PaymentProcessingJob.processed' github.com/shopify/statsd-instrument