Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
Knobs, buttons & switches Operating your application at scale
Slide 2
Slide 2 text
Got Options?
Slide 3
Slide 3 text
Application Resilience
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
1. Maintenance Mode 2. Read-Only Mode 3. Feature Flags 4. Rate Limits 5. Non-critical work 6. New code paths 7. Circuit Breakers
Slide 10
Slide 10 text
1 Maintenance Mode
Slide 11
Slide 11 text
Environment Variable MAINTENANCE_MODE=ON
Slide 12
Slide 12 text
2 Read-only Mode
Slide 13
Slide 13 text
READONLY_MODE=ON Environment Variable
Slide 14
Slide 14 text
3 Feature Flags
Slide 15
Slide 15 text
Feature Flags Granularity ● User ● Global ● Group
Slide 16
Slide 16 text
ApplicationSetting.get(‘billing-enabled’) Datastore
Slide 17
Slide 17 text
ApplicationSetting.get(‘billing-enabled-eu’) Datastore
Slide 18
Slide 18 text
4 Rate Limits
Slide 19
Slide 19 text
Surge pricing Gray Rate Limits
Slide 20
Slide 20 text
Preferred traffic Gray Rate Limits
Slide 21
Slide 21 text
Default & modifiers Gray Rate Limits
Slide 22
Slide 22 text
ApplicationSetting.set(‘default-rate’) = 100 Datastore
Slide 23
Slide 23 text
> customer.rate_limit_modifier => 1.0 Datastore
Slide 24
Slide 24 text
> customer.rate_limit => 100 Datastore
Slide 25
Slide 25 text
> customer.rate_limit_modifier = 2.0 Datastore
Slide 26
Slide 26 text
> customer.rate_limit => 200 Datastore
Slide 27
Slide 27 text
ApplicationSetting.set(‘default-rate’) = 50 Datastore
Slide 28
Slide 28 text
> customer.rate_limit => 100 Datastore
Slide 29
Slide 29 text
Cost-based limits Gray Rate Limits
Slide 30
Slide 30 text
(D)DoS Concerns Gray Rate Limits
Slide 31
Slide 31 text
5 Stop non-critical work
Slide 32
Slide 32 text
> ReportSetting.get(“monthly_user_report”) => false Datastore
Slide 33
Slide 33 text
class MonthlyUserReport end
Slide 34
Slide 34 text
class MonthlyUserReport def run end end
Slide 35
Slide 35 text
class MonthlyUserReport def run do_something end end
Slide 36
Slide 36 text
class MonthlyUserReport def run return unless enabled? do_something end def enabled? ReportSetting.get(“monthly_user_report”) end end
Slide 37
Slide 37 text
class MonthlyUserReport < Report def run return unless enabled? do_something end def enabled? ReportSetting.get(“monthly_user_report”) end end
Slide 38
Slide 38 text
class MonthlyUserReport < Report def build do_something end end
Slide 39
Slide 39 text
class Report def run return unless enabled? build end def enabled? ReportSetting.get(self.class.underscore) end end
Slide 40
Slide 40 text
6 Known Unknowns
Slide 41
Slide 41 text
Scientist! https://github.com/github/scientist Datastore
Slide 42
Slide 42 text
7 Circuit Breakers
Slide 43
Slide 43 text
Responsive shut-offs Gray Circuit Breakers
Slide 44
Slide 44 text
Hard shut-offs Gray Circuit Breakers
Slide 45
Slide 45 text
> BillingServiceClient.circuit.open => > BillingServiceClient.circuit.close => Caching layer or Datastore
Slide 46
Slide 46 text
Consider implementation
Slide 47
Slide 47 text
● Database ● Data caching layer ● Environment variables ● Code Gray Consider Implementation
Slide 48
Slide 48 text
Caveats
Slide 49
Slide 49 text
Visibility Gray Caveats
Slide 50
Slide 50 text
Got Options?
Slide 51
Slide 51 text
Does it actually work? Gray Caveats
Slide 52
Slide 52 text
Knowledge vs. Control Gray Caveats
Slide 53
Slide 53 text
“I’d still take this deal any day
Slide 54
Slide 54 text
Thank you! Amy Unger @cdwort
Slide 55
Slide 55 text
Tomorrow 10:50 So You’ve Got Yourself a Kafka: Event-Powered Rails Services Stella Cotton Room 301-302 Tomorrow 11:40 Postgres 10, Performance, and You Gabe Enslein Room 306-307
Slide 56
Slide 56 text
Thank you! Amy Unger @cdwort