Slide 1

Slide 1 text

The wind of change by @dschenkelman

Slide 2

Slide 2 text

Logins

Slide 3

Slide 3 text

Embrace risk

Slide 4

Slide 4 text

SLOs https://landing.google.com/sre/book/chapters/service- level-objectives.html • Latency: 99th percentile at edge (measured every 5 minutes) < 500 ms • Reliability: 99.99% of requests succeed (200-499 status code)

Slide 5

Slide 5 text

Error Budgets

Slide 6

Slide 6 text

Consumption Uniform/simplified model https://goo.gl/7kWoX4 duration[s] * requests impacted[%] / 100 (60*60*24*30)[s] * (error budget[%]/100[%]) * 100

Slide 7

Slide 7 text

Take me to the magic of the moment On a glory night Where the children of tomorrow dream away (dream away) In the wind of change

Slide 8

Slide 8 text

Rolling updates

Slide 9

Slide 9 text

duration[s] * requests impacted[%] / 100 (60*60*24*30)[s] * (error budget[%]/100[%]) * 100

Slide 10

Slide 10 text

Instances

Slide 11

Slide 11 text

Environments

Slide 12

Slide 12 text

Canary

Slide 13

Slide 13 text

Blue/Green

Slide 14

Slide 14 text

duration[s] * requests impacted[%] / 100 (60*60*24*30)[s] * (error budget[%]/100[%]) * 100

Slide 15

Slide 15 text

Usage patterns

Slide 16

Slide 16 text

duration[s] * requests impacted[%] * (period weight) / 100 (60*60*24*30)[s] * (error budget[%]/100[%]) * 100 req during period length avg req period length during month period weight = Consumption Real Model

Slide 17

Slide 17 text

Schedule Region/Time Start End US 12 AM UTC 10 AM UTC EU 9 PM UTC 9 AM UTC AU 11 AM UTC 9 PM UTC

Slide 18

Slide 18 text

Feature flags

Slide 19

Slide 19 text

Belgrano

Slide 20

Slide 20 text

Usage const belgrano = require('belgrano'); belgrano.init({ /* config */ }); const getFlag = belgrano.entities.getFlag; if (await getFlag(entityId, flag)) { // enabled case } else { // disabled case }

Slide 21

Slide 21 text

duration[s] * requests impacted[%] / 100 (60*60*24*30)[s] * (error budget[%]/100[%]) * 100

Slide 22

Slide 22 text

Topologies

Slide 23

Slide 23 text

DB only Belgrano SDK MongoDB

Slide 24

Slide 24 text

+ Abstraction Belgrano SDK Belgrano Server MongoDB

Slide 25

Slide 25 text

Faster

Slide 26

Slide 26 text

+ Cache Belgrano SDK Belgrano Server MongoDB Redis

Slide 27

Slide 27 text

Circuit Breakers

Slide 28

Slide 28 text

Manage

Slide 29

Slide 29 text

Go fast

Slide 30

Slide 30 text

Hope is not a strategy

Slide 31

Slide 31 text

Experiments

Slide 32

Slide 32 text

Shadowing Fire and forget

Slide 33

Slide 33 text

Real

Slide 34

Slide 34 text

duration[s] * requests impacted[%] / 100 (60*60*24*30)[s] * (error budget[%]/100[%]) * 100

Slide 35

Slide 35 text

Rate limiting

Slide 36

Slide 36 text

Rate limiting limitd-leveldb Client Client Client { key space 2} { key space 3} { key space 1} level level level limitd limitd limitd

Slide 37

Slide 37 text

Cluster Rate limiting limitd-redis Client Client Client { key space 2} { key space 3} { key space 1}

Slide 38

Slide 38 text

Tap Compare

Slide 39

Slide 39 text

feature-change const feature_change = require('feature-change'); var options = { expected: function(cb){ search_v2(query, cb); }, actual: function(cb){ search_v3(query, cb); }, logAction: function(current_result, new_result){ // invoked when there is a difference in the results // (useful for logging) } }; feature_change(options, function(err, result){ // this is the original callback you were using for search v2 // err and result always come from search_v2 });

Slide 40

Slide 40 text

duration[s] * requests impacted[%] / 100 (60*60*24*30)[s] * (error budget[%]/100[%]) * 100

Slide 41

Slide 41 text

Iron out differences

Slide 42

Slide 42 text

Future

Slide 43

Slide 43 text

Platform • Pick resources (CPU, memory, etc.) • Pick deployment method • Control routed traffic • Spinnaker and automatic anomaly detection

Slide 44

Slide 44 text

Feature flags • % based flags • User Interface for managing • New stores (not just mongodb)

Slide 45

Slide 45 text

Rate limiting • Dynamic configuration • Support for “concurrent requests”

Slide 46

Slide 46 text

We’re hiring https://auth0.com/careers

Slide 47

Slide 47 text

Links • https://medium.com/@copyconstruct/testing-in- production-the-safe-way-18ca102d0ef1 • https://landing.google.com/sre/book/index.html • https://martinfowler.com/articles/feature- toggles.html • https://redis.io/topics/cluster-tutorial • https://stripe.com/blog/rate-limiters

Slide 48

Slide 48 text

Links • http://zachholman.com/talk/move-fast-break- nothing/ • https://auth0.com/blog/2015/10/27/feature- changes-at-auth0/ • https://github.com/dschenkelman/feature- change

Slide 49

Slide 49 text

Gracias! https://github.com/dschenkelman/wind-of-change-talk @dschenkelman