Slide 1
Slide 1 text
No content
Slide 2
Slide 2 text
Michael Heap
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
No content
Slide 7
Slide 7 text
No content
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
Fun with Country Flags!
Presented by @mheap at #phpsrb
Slide 11
Slide 11 text
Fun with Country Flags!
Presented by @mheap at #phpsrb
Slide 12
Slide 12 text
Fun with Country Flags!
Presented by @mheap at #phpsrb
Slide 13
Slide 13 text
Fun with Country Flags!
Presented by @mheap at #phpsrb
Slide 14
Slide 14 text
Fun with Feature Flags!
Presented by @mheap at #phpsrb
Slide 15
Slide 15 text
Michael
@mheap
Developer Relations
NexmoDev
Slide 16
Slide 16 text
Fun with Feature Flags!
Presented by @mheap at #phpsrb
Slide 17
Slide 17 text
Today’s Plan
Slide 18
Slide 18 text
1. What are feature flags?
2. Implementation
3. Flag management
4. Tips and tricks
5. Conclusion
Slide 19
Slide 19 text
1. What are feature flags?
2. Implementation
3. Flag management
4. Tips and tricks
5. Conclusion
Slide 20
Slide 20 text
1. What are feature flags?
2. Implementation
3. Flag management
4. Tips and tricks
5. Conclusion
Slide 21
Slide 21 text
1. What are feature flags?
2. Implementation
3. Flag management
4. Tips and tricks
5. Conclusion
Slide 22
Slide 22 text
1. What are feature flags?
2. Implementation
3. Flag management
4. Tips and tricks
5. Conclusion
Slide 23
Slide 23 text
Feature Flags?
Slide 24
Slide 24 text
Feature Toggles?
Slide 25
Slide 25 text
Feature Flip?
Slide 26
Slide 26 text
Feature Gate?
Slide 27
Slide 27 text
Feature Switches?
Slide 28
Slide 28 text
Separate feature release
from code deployment
Slide 29
Slide 29 text
http://code.flickr.net/2009/12/02/flipping-out/
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
No content
Slide 32
Slide 32 text
Feature flags aren’t new
Slide 33
Slide 33 text
#ifdef NEW_ALGORITHM
NEW_ALGORITHM(input)
#else
ALGORITHM(input)
#endif
$ make build -DNEW_ALGORITHM
Slide 34
Slide 34 text
./make_call.sh
--use-new-algorithm
./make_call.sh
--use-original-algorithm
Slide 35
Slide 35 text
if (in_array($user->id, $whitelistedUsers)) {
return $service->useNewAlgorithm($input);
}
return $service->useAlgorithm($input);
Slide 36
Slide 36 text
No content
Slide 37
Slide 37 text
Use Cases?
Slide 38
Slide 38 text
1. Permissions
Slide 39
Slide 39 text
1. Permissions
2. Early-access
Slide 40
Slide 40 text
1. Permissions
2. Early-access
3. Opt-in
Slide 41
Slide 41 text
1. Permissions
2. Early-access
3. Opt-in
4. Incremental roll-out
Slide 42
Slide 42 text
1. Permissions
2. Early-access
3. Opt-in
4. Incremental roll-out
5. Block users
Slide 43
Slide 43 text
1. Permissions
2. Early-access
3. Opt-in
4. Incremental roll-out
5. Block users
6. Calendar driven
Slide 44
Slide 44 text
1. Permissions
2. Early-access
3. Opt-in
4. Incremental roll-out
5. Block users
6. Calendar driven
7. Kill switch
Slide 45
Slide 45 text
1. Permissions
2. Early-access
3. Opt-in
4. Incremental roll-out
5. Block users
6. Calendar driven
7. Kill switch
8. Power user
Slide 46
Slide 46 text
1. Permissions
2. Early-access
3. Opt-in
4. Incremental roll-out
5. Block users
6. Calendar driven
7. Kill switch
8. Power user
9. Maintenance
Slide 47
Slide 47 text
1. Permissions
2. Early-access
3. Opt-in
4. Incremental roll-out
5. Block users
6. Calendar driven
7. Kill switch
8. Power user
9. Maintenance
10. Sunset
Slide 48
Slide 48 text
No content
Slide 49
Slide 49 text
Flag Types
Slide 50
Slide 50 text
Release
Toggles
Ops
Toggles
Flag Classes
Permission
Toggles
Experiment
Toggles
Slide 51
Slide 51 text
Release
Toggles
Ops
Toggles
Flag Classes
Permission
Toggles
Experiment
Toggles
Lifetime
Days
Dynamism
Per deployment
Slide 52
Slide 52 text
Release
Toggles
Ops
Toggles
Flag Classes
Permission
Toggles
Experiment
Toggles
Lifetime
Days - Weeks
Dynamism
Human Controlled
Slide 53
Slide 53 text
Release
Toggles
Ops
Toggles
Flag Classes
Permission
Toggles
Experiment
Toggles
Lifetime
Years
Dynamism
Per Request
Slide 54
Slide 54 text
Release
Toggles
Ops
Toggles
Flag Classes
Permission
Toggles
Experiment
Toggles
Lifetime
Weeks - Months
Dynamism
Per Request
Slide 55
Slide 55 text
https://martinfowler.com/articles/feature-toggles.html
Slide 56
Slide 56 text
Ownership
Release toggles owned by devs
Ops toggles owned by ops
Permission toggles
Experiment toggles
owned by product
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
Flag Scopes
Slide 59
Slide 59 text
On/Off
Switch
Enable or
disable
entirely
Slide 60
Slide 60 text
On/Off
Switch
Percentage
of users
Enable or
disable
entirely
Toggle
for a
percentage
of users
Slide 61
Slide 61 text
On/Off
Switch
Percentage
of users
Individuals
Enable or
disable
entirely
Toggle
for a
percentage
of users
Toggle for
specific
users
Slide 62
Slide 62 text
On/Off
Switch
Percentage
of users
Individuals Groups
Enable or
disable
entirely
Toggle
for a
percentage
of users
Toggle for
specific
users
Enable for
specific
groups
e.g.
Power Users
Slide 63
Slide 63 text
On/Off
Switch
Percentage
of users
Individuals Groups Request
information
Enable or
disable
entirely
Toggle
for a
percentage
of users
Toggle for
specific
users
Enable for
specific
groups
e.g.
Power Users
Enable for
specific
request
headers or
bodies
Slide 64
Slide 64 text
No content
Slide 65
Slide 65 text
Implementation
Slide 66
Slide 66 text
Build vs Buy
Slide 67
Slide 67 text
1. Dashboard
Slide 68
Slide 68 text
1. Dashboard
2. Access control
Slide 69
Slide 69 text
1. Dashboard
2. Access control
3. Complex rules
Slide 70
Slide 70 text
1. Dashboard
2. Access control
3. Complex rules
4. Cascading flags
Slide 71
Slide 71 text
1. Dashboard
2. Access control
3. Complex rules
4. Cascading flags
5. Multivariate rules
Slide 72
Slide 72 text
1. Dashboard
2. Access control
3. Complex rules
4. Cascading flags
5. Multivariate rules
6. Auditing
Slide 73
Slide 73 text
1. Dashboard
2. Access control
3. Complex rules
4. Cascading flags
5. Multivariate rules
6. Auditing
7. Speed
Slide 74
Slide 74 text
1. Dashboard
2. Access control
3. Complex rules
4. Cascading flags
5. Multivariate rules
6. Auditing
7. Speed
8. Scale
Slide 75
Slide 75 text
1. Dashboard
2. Access control
3. Complex rules
4. Cascading flags
5. Multivariate rules
6. Auditing
7. Speed
8. Scale
9. 100% uptime
Slide 76
Slide 76 text
Build
Slide 77
Slide 77 text
Rollout (Ruby)
# Is a feature active?
$rollout.active?(:chat)
# Activate a specific user
$rollout.activate_user(:chat, @user)
# Define a group
$rollout.define_group(:admins) do |user|
user.admin?
end
# Roll out to a percentage of users
$rollout.activate_percentage(:chat, 20)
Slide 78
Slide 78 text
Rollout (PHP)
# Is a feature active?
$rollout->isActive('chat');
# Activate a specific user
$rollout->activateUser('chat', $user);
# Define a group
$rollout->defineGroup('admins', function($user = null) {
return $user->isAdmin();
});
# Roll out to a percentage of users
$rollout->activatePercentage('chat', 20);
Slide 79
Slide 79 text
Qandidate Toggle
$manager = new ToggleManager(new InMemoryCollection());
$afterSix = new GreaterThanEqual(18);
$beforeTen = new LessThanEqual(22);
$conditions = [];
$conditions[] = new OperatorCondition('time', $afterSix);
$conditions[] = new OperatorCondition('time', $beforeTen);
$toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
$manager->add($toggle);
$context = new Context();
$context->set('time', (int) date('G'));
echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
? ‘happy!' : 'sad :(';
Slide 80
Slide 80 text
Qandidate Toggle
$manager = new ToggleManager(new InMemoryCollection());
$afterSix = new GreaterThanEqual(18);
$beforeTen = new LessThanEqual(22);
$conditions = [];
$conditions[] = new OperatorCondition('time', $afterSix);
$conditions[] = new OperatorCondition('time', $beforeTen);
$toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
$manager->add($toggle);
$context = new Context();
$context->set('time', (int) date('G'));
echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
? ‘happy!' : 'sad :(';
Slide 81
Slide 81 text
Qandidate Toggle
$manager = new ToggleManager(new InMemoryCollection());
$afterSix = new GreaterThanEqual(18);
$beforeTen = new LessThanEqual(22);
$conditions = [];
$conditions[] = new OperatorCondition('time', $afterSix);
$conditions[] = new OperatorCondition('time', $beforeTen);
$toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
$manager->add($toggle);
$context = new Context();
$context->set('time', (int) date('G'));
echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
? ‘happy!' : 'sad :(';
Slide 82
Slide 82 text
Qandidate Toggle
$manager = new ToggleManager(new InMemoryCollection());
$afterSix = new GreaterThanEqual(18);
$beforeTen = new LessThanEqual(22);
$conditions = [];
$conditions[] = new OperatorCondition('time', $afterSix);
$conditions[] = new OperatorCondition('time', $beforeTen);
$toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
$manager->add($toggle);
$context = new Context();
$context->set('time', (int) date('G'));
echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
? ‘happy!' : 'sad :(';
Slide 83
Slide 83 text
Qandidate Toggle
$manager = new ToggleManager(new InMemoryCollection());
$afterSix = new GreaterThanEqual(18);
$beforeTen = new LessThanEqual(22);
$conditions = [];
$conditions[] = new OperatorCondition('time', $afterSix);
$conditions[] = new OperatorCondition('time', $beforeTen);
$toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
$manager->add($toggle);
$context = new Context();
$context->set('time', (int) date('G'));
echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
? ‘happy!' : 'sad :(';
Slide 84
Slide 84 text
Qandidate Toggle
$manager = new ToggleManager(new InMemoryCollection());
$afterSix = new GreaterThanEqual(18);
$beforeTen = new LessThanEqual(22);
$conditions = [];
$conditions[] = new OperatorCondition('time', $afterSix);
$conditions[] = new OperatorCondition('time', $beforeTen);
$toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_MAJORITY);
$manager->add($toggle);
$context = new Context();
$context->set('time', (int) date('G'));
echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
? ‘happy!' : 'sad :(';
Slide 85
Slide 85 text
Qandidate Toggle
$manager = new ToggleManager(new InMemoryCollection());
$afterSix = new GreaterThanEqual(18);
$beforeTen = new LessThanEqual(22);
$conditions = [];
$conditions[] = new OperatorCondition('time', $afterSix);
$conditions[] = new OperatorCondition('time', $beforeTen);
$toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
$manager->add($toggle);
$context = new Context();
$context->set('time', (int) date('G'));
echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
? ‘happy!' : 'sad :(';
Slide 86
Slide 86 text
Qandidate Toggle
$manager = new ToggleManager(new InMemoryCollection());
$afterSix = new GreaterThanEqual(18);
$beforeTen = new LessThanEqual(22);
$conditions = [];
$conditions[] = new OperatorCondition('time', $afterSix);
$conditions[] = new OperatorCondition('time', $beforeTen);
$toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
$manager->add($toggle);
$context = new Context();
$context->set('time', (int) date('G'));
echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
? ‘happy!' : 'sad :(';
Slide 87
Slide 87 text
Qandidate Toggle
$manager = new ToggleManager(new InMemoryCollection());
$afterSix = new GreaterThanEqual(18);
$beforeTen = new LessThanEqual(22);
$conditions = [];
$conditions[] = new OperatorCondition('time', $afterSix);
$conditions[] = new OperatorCondition('time', $beforeTen);
$toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
$manager->add($toggle);
$context = new Context();
$context->set('time', (int) date('G'));
echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
? ‘happy!' : 'sad :(';
Slide 88
Slide 88 text
Qandidate Toggle
$manager = new ToggleManager(new InMemoryCollection());
$afterSix = new GreaterThanEqual(18);
$beforeTen = new LessThanEqual(22);
$conditions = [];
$conditions[] = new OperatorCondition('time', $afterSix);
$conditions[] = new OperatorCondition('time', $beforeTen);
$toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
$manager->add($toggle);
$context = new Context();
$context->set('time', (int) date('G'));
echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
? ‘happy!' : 'sad :(';
Slide 89
Slide 89 text
$manager = new ToggleManager(new InMemoryCollection());
$toggle = new Toggle('some-feature', [
new OperatorCondition('user_id', new GreaterThan(41))
]);
$manager->add($toggle);
Configuration: Code
Slide 90
Slide 90 text
Configuration: Array
$data = array(
'some-feature' => array(
'name' => 'toggling',
'conditions' => array(
array(
'name' => 'operator-condition',
'key' => 'user_id',
'operator' => array('name' => 'greater-than', 'value' => 41),
),
),
),
);
$serializer = new InMemoryCollectionSerializer();
$collection = $serializer->deserialize($data);
$manager = new ToggleManager($collection);
Slide 91
Slide 91 text
Configuration: JSON/YAML
$yaml = <<deserialize($array);
$manager = new ToggleManager($collection);
Slide 92
Slide 92 text
Configuration: Redis
$predis = new Predis\Client();
$collection = new PredisCollection('toggle_demo', $predis);
$manager = new ToggleManager($collection);
Slide 93
Slide 93 text
Qandidate Toggle API
{
"conditions" : [
{
"name" : "operator-condition",
"operator" : {
"name" : "less-than",
"value" : "1337"
},
"key" : "user_id"
}
],
"name" : "foo",
"status" : "conditionally-active",
"originalName" : "foo"
}
PUT /toggles/{name}
https://github.com/qandidate-labs/qandidate-toggle-api
Slide 94
Slide 94 text
Qandidate Toggle UI
https://github.com/qandidate-labs/qandidate-toggle-ui
Slide 95
Slide 95 text
Build (Pros)
1. Free!
2. Total control
3. Complex rules
4. API and UI available
Slide 96
Slide 96 text
Build (Cons)
1. Self hosted
2. PHP only
3. Complex rules
Slide 97
Slide 97 text
Buy
Slide 98
Slide 98 text
No content
Slide 99
Slide 99 text
Multi-Language
Slide 100
Slide 100 text
Prebuilt Dashboard
Slide 101
Slide 101 text
Environment Aware
Slide 102
Slide 102 text
Scaleable
Slide 103
Slide 103 text
Audit Log
Slide 104
Slide 104 text
A/B Testing
Slide 105
Slide 105 text
ld-relay
Slide 106
Slide 106 text
Buy (Pros)
1. Lovely dashboard
2. Multi-language
3. Built in audit log
4. Environment aware
5. A/B testing conversion tracking
Slide 107
Slide 107 text
Buy (Cons)
1. It costs (at least) $79/month
2. Can’t modify the product
3. Dependent on 3rd party
Slide 108
Slide 108 text
Build vs Buy
Slide 109
Slide 109 text
No content
Slide 110
Slide 110 text
It depends
Slide 111
Slide 111 text
Show me code
Slide 112
Slide 112 text
Slide 113
Slide 113 text
Slide 114
Slide 114 text
public function onKernelRequest($event) {
$flag = $event->getRequest->attributes->get('_flag');
$alt = $event->getRequest->attributes->get('_alt');
if ($toggleManager->active($flag, $context)) {
$event->getRequest->attributes->set('controller', $alt);
}
}
profile:
path: /profile/{user}
defaults:
_controller: AcmeBundle:Profile:update
_alt: AcmeBundle:NewProfile:update
_flag: new-profile-controller
Slide 115
Slide 115 text
{% if feature_is_active('new-profile-image-picker') %}
{# Show new picker #}
{% else %}
{# Use old flash-based picker #}
{% endif %}
Slide 116
Slide 116 text
toggleManager->active('new-profile', $context)) {
return new ShinyNewProfile($args);
}
return new Profile($args);
}
}
Slide 117
Slide 117 text
Controller View Factory
Enable a
route for
specific users
Show different
UI elements
Branch by
abstraction for
refactoring
Slide 118
Slide 118 text
No content
Slide 119
Slide 119 text
Toggle Configuratin
Slide 120
Slide 120 text
Hard
Coded
Change
the code
Dynamic
Static
Slide 121
Slide 121 text
Hard
Coded
Deploy
Config
Change
the code
Deploy
your
application
Dynamic
Static
Slide 122
Slide 122 text
Hard
Coded
Deploy
Config
Parameterised
Config
Change
the code
Deploy
your
application
Edit config,
restart app
Dynamic
Static
Slide 123
Slide 123 text
Hard
Coded
Deploy
Config
Parameterised
Config
Database
Config
Change
the code
Deploy
your
application
Edit config,
restart app
Edit via
UI
Dynamic
Static
Slide 124
Slide 124 text
Hard
Coded
Deploy
Config
Parameterised
Config
Database
Config
Distributed
Config
Change
the code
Deploy
your
application
Edit config,
restart app
Edit via
UI
e.g. Consul
or
Zookeeper
Dynamic
Static
Slide 125
Slide 125 text
Feature Dashboard
Slide 126
Slide 126 text
Flag State
Slide 127
Slide 127 text
Role Based Access
Slide 128
Slide 128 text
Audit Log
Slide 129
Slide 129 text
Non-technical Users
Slide 130
Slide 130 text
Technical Debt
Slide 131
Slide 131 text
DON’T
Slide 132
Slide 132 text
Cleanup based on activity
Slide 133
Slide 133 text
Cleanup based on activity
Slide 134
Slide 134 text
Create & Cleanup
Slide 135
Slide 135 text
WIP limits
Slide 136
Slide 136 text
WIP limits
Slide 137
Slide 137 text
Expiration date
Slide 138
Slide 138 text
Explosion date
Slide 139
Slide 139 text
Tips & Tricks
Slide 140
Slide 140 text
Testing Flags
Slide 141
Slide 141 text
Testing Flags
Feature A Feature B
✅ ✅
✅ ☠
☠ ✅
☠ ☠
Slide 142
Slide 142 text
Testing Flags
Feature A Feature B Feature C
✅ ✅ ✅
✅ ☠ ✅
✅ ✅ ☠
☠ ✅ ✅
✅ ☠ ☠
☠ ✅ ☠
☠ ☠ ✅
☠ ☠ ☠
Slide 143
Slide 143 text
Testing Flags
Feature A Feature B Feature C Feature D
✅ ✅ ✅ ✅
✅ ✅ ✅ ☠
✅ ✅ ☠ ✅
✅ ☠ ✅ ✅
☠ ✅ ✅ ✅
✅ ✅ ☠ ☠
✅ ☠ ✅ ☠
✅ ☠ ☠ ✅
☠ ✅ ✅ ☠
☠ ✅ ☠ ✅
☠ ☠ ✅ ✅
✅ ☠ ☠ ☠
☠ ✅ ☠ ☠
☠ ☠ ✅ ☠
☠ ☠ ☠ ✅
☠ ☠ ☠ ☠
Slide 144
Slide 144 text
Testing Flags
Number of flags Possible permutations
2 4
3 8
4 16
5 32
10 1024
25 33,554,432
50 1.12589991E+15
100 1.2676506E+30
Slide 145
Slide 145 text
Homepage
Carousel
Homepage CTA
✅ ✅
✅ ☠
☠ ✅
☠ ☠
Forum Edit
Permission
Forum Delete
Permission
✅ ✅
✅ ☠
☠ ✅
☠ ☠
Upload Profile
Image
✅
☠
New Interesting
Algorithm
✅
☠
High Post Length
✅
☠
Show Debug
Info?
✅
☠
Enable
Audit Log
✅
☠
Auto Logout
✅
☠
API Access API v2
✅ ✅
✅ ☠
☠ ✅
☠ ☠
Slide 146
Slide 146 text
Expose State
{
"api_v2": true,
"audit_log": true,
"new_recommendation_algo": false
}
GET /toggle_status
Slide 147
Slide 147 text
Flag Descriptions
basic-rec-algo: true
Slide 148
Slide 148 text
Flag Descriptions
basic-rec-algo:
enabled: true
description: Use a simplistic recommendation algorithm. This is fast and
produces less load on backend systems, but is way less accurate than
our standard algorithm.
Slide 149
Slide 149 text
Clear Naming
allow_user_delete and allow_user_edit
not
user_control
Slide 150
Slide 150 text
Audit Log
Feature State User Date
allow_user_edit
✅ Sarah 2018-05-11 12:22:08
enhanced_rec_algo
✅ Ashley 2018-05-04 15:33:16
allow_user_signup
☠ Ashley 2018-05-04 15:27:33
enhanced_rec_algo
☠ system:load_manager 2018-05-04 15:16:30
allow_user_signup
✅ Bob 2018-05-04 15:12:01
enable_api
✅ Sarah 2018-04-30 09:56:59
Slide 151
Slide 151 text
Fail Safe
Do positively enable features e.g. enable-ibm-
analysis, enhanced-recomendation-algo
Don’t use flags for suppressing new features e.g.
disable-shiny-new-feature
Slide 152
Slide 152 text
Conclusion
Slide 153
Slide 153 text
No content
Slide 154
Slide 154 text
Flags are great!
Slide 155
Slide 155 text
Ops
Toggles
Release
Toggles
Flag Classes
Permission
Toggles
Experiment
Toggles
Slide 156
Slide 156 text
Build vs Buy
Slide 157
Slide 157 text
Qandidate Toggle
Slide 158
Slide 158 text
No content
Slide 159
Slide 159 text
One More Thing
Slide 160
Slide 160 text
No Reuse
Slide 161
Slide 161 text
Knight Capital
Slide 162
Slide 162 text
High frequency trader
Slide 163
Slide 163 text
No content
Slide 164
Slide 164 text
No content
Slide 165
Slide 165 text
Mistakes were made
Slide 166
Slide 166 text
Knight Capital
1. Reused flags
Slide 167
Slide 167 text
Knight Capital
1. Reused flags
2. Didn’t delete obsolete code
Slide 168
Slide 168 text
Knight Capital
1. Reused flags
2. Didn’t delete obsolete code
3. Couldn’t be disabled without a deployment
Slide 169
Slide 169 text
Knight Capital
1. Reused flags
2. Didn’t delete obsolete code
3. Couldn’t be disabled without a deployment
4. The deploy actually enabled the old code on
all 8 servers, instead of disabling it
Slide 170
Slide 170 text
$460m Gone