Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dr Sheldon Cooper Presents: Fun with Flags

Dr Sheldon Cooper Presents: Fun with Flags

No no, not country flags, feature flags! Feature flags are a powerful technique that allows teams to modify a system’s behaviour without changing code. They can be used several reasons – canary releases and A/B testing to name a few.

This talk will show you how you’re already using feature flags in your application without realising it. Next, we’ll take a look at some of the best tooling out there to help you take feature flags to the next level. Finally, we’ll cover strategies for removing feature flags before they become technical debt that you have to manage.

Michael Heap

May 11, 2018
Tweet

More Decks by Michael Heap

Other Decks in Technology

Transcript

  1. View Slide

  2. Michael Heap

    View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. Fun with Country Flags!
    Presented by @mheap at @phpday

    View Slide

  10. Fun with Country Flags!
    Presented by @mheap at @phpday

    View Slide

  11. Fun with Country Flags!
    Presented by @mheap at @phpday

    View Slide

  12. Fun with Country Flags!
    Presented by @mheap at @phpday

    View Slide

  13. Fun with Feature Flags!
    Presented by @mheap at @phpday

    View Slide

  14. Michael
    @mheap
    Developer Relations
    NexmoDev

    View Slide

  15. Fun with Feature Flags!
    Presented by @mheap at @phpday

    View Slide

  16. Today’s Plan

    View Slide

  17. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View Slide

  18. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View Slide

  19. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View Slide

  20. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View Slide

  21. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View Slide

  22. Feature Flags?

    View Slide

  23. Feature Toggles?

    View Slide

  24. Feature Flip?

    View Slide

  25. Feature Gate?

    View Slide

  26. Feature Switches?

    View Slide

  27. Separates feature release
    from code deployment

    View Slide

  28. http://code.flickr.net/2009/12/02/flipping-out/

    View Slide

  29. View Slide

  30. View Slide

  31. Feature flags aren’t new

    View Slide

  32. #ifdef IBM_ANALYSIS
    IBM_ANALYSIS(audio)
    #else
    INTERNAL_ANALYSIS(audio)
    #endif
    $ make build -DIBM_ANALYSIS

    View Slide

  33. ./make_call.sh
    —-use-internal-analysis
    ./make_call.sh
    —-use-ibm-analysis

    View Slide

  34. if (in_array($user->id, $whitelistedUsers)) {
    return $ibm->analyzeAudio($audio);
    }
    return $sentiment->analyze($audio);

    View Slide

  35. Use Cases?

    View Slide

  36. 1. Permissions

    View Slide

  37. 1. Permissions
    2. Early-access

    View Slide

  38. 1. Permissions
    2. Early-access
    3. Opt-in

    View Slide

  39. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out

    View Slide

  40. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users

    View Slide

  41. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven

    View Slide

  42. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch

    View Slide

  43. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user

    View Slide

  44. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    9. Maintenance

    View Slide

  45. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    9. Maintenance
    10. Sunset

    View Slide

  46. Flag Classes

    View Slide

  47. Ops
    Toggles
    Release
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles

    View Slide

  48. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Days
    Dynamism
    Per deployment

    View Slide

  49. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Days - Weeks
    Dynamism
    Human Controlled

    View Slide

  50. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Years
    Dynamism
    Per Request

    View Slide

  51. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Weeks - Months
    Dynamism
    Per Request

    View Slide

  52. https://martinfowler.com/articles/feature-toggles.html

    View Slide

  53. Ownership
    Release toggles owned by devs
    Ops toggles owned by ops
    Permission toggles
    Experiment toggles
    owned by product

    View Slide

  54. Flag Scopes

    View Slide

  55. On/Off
    Switch
    Enable or
    disable
    entirely

    View Slide

  56. On/Off
    Switch
    Percentage
    of users
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users

    View Slide

  57. On/Off
    Switch
    Percentage
    of users
    Individuals
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users

    View Slide

  58. On/Off
    Switch
    Percentage
    of users
    Individuals Groups
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    Enable for
    specific
    groups
    e.g.
    Power Users

    View Slide

  59. On/Off
    Switch
    Percentage
    of users
    Individuals Groups Request
    information
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    Enable for
    specific
    groups
    e.g.
    Power Users
    Enable for
    specific
    request
    headers or
    bodies

    View Slide

  60. Implementation

    View Slide

  61. Build vs Buy

    View Slide

  62. 1. Dashboard

    View Slide

  63. 1. Dashboard
    2. Access control

    View Slide

  64. 1. Dashboard
    2. Access control
    3. Complex rules

    View Slide

  65. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags

    View Slide

  66. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules

    View Slide

  67. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing

    View Slide

  68. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed

    View Slide

  69. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    8. Scale

    View Slide

  70. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    8. Scale
    9. 100% uptime

    View Slide

  71. Build

    View Slide

  72. Rollout (Ruby)
    # Is a feature active?
    $rollout.active?(:chat)
    # Activate a specific user
    $rollout.activate_user(:chat, @user)
    # Define a group
    $rollout.define_group(:admins) do |user|
    user.admin?
    end
    # Roll out to a percentage of users
    $rollout.activate_percentage(:chat, 20)

    View Slide

  73. Rollout (PHP)
    # Is a feature active?
    $rollout->isActive('chat');
    # Activate a specific user
    $rollout->activateUser('chat', $user);
    # Define a group
    $rollout->defineGroup('admins', function($user = null) {
    return $user->isAdmin();
    });
    # Roll out to a percentage of users
    $rollout->activatePercentage('chat', 20);

    View Slide

  74. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active(‘disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  75. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active(‘disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  76. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  77. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  78. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  79. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  80. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_MAJORITY);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  81. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  82. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  83. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  84. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  85. $manager = new ToggleManager(new InMemoryCollection());
    $toggle = new Toggle('some-feature', [
    new OperatorCondition('user_id', new GreaterThan(41))
    ]);
    $manager->add($toggle);
    Configuration: Code

    View Slide

  86. Configuration: Array
    $data = array(
    'some-feature' => array(
    'name' => 'toggling',
    'conditions' => array(
    array(
    'name' => 'operator-condition',
    'key' => 'user_id',
    'operator' => array('name' => 'greater-than', 'value' => 41),
    ),
    ),
    ),
    );
    $serializer = new InMemoryCollectionSerializer();
    $collection = $serializer->deserialize($data);
    $manager = new ToggleManager($collection);

    View Slide

  87. Configuration: JSON/YAML
    $yaml = <<toggles:
    - name: some-feature
    conditions:
    - name: operator-condition
    key: user_id
    operator:
    name: greater-than
    value: 41
    YML;
    $array = Yaml::parse($yaml);
    $serializer = new InMemoryCollectionSerializer();
    $collection = $serializer->deserialize($array);
    $manager = new ToggleManager($collection);

    View Slide

  88. Configuration: Redis
    $predis = new Predis\Client();
    $collection = new PredisCollection('toggle_demo', $predis);
    $manager = new ToggleManager($collection);

    View Slide

  89. Qandidate Toggle API
    {
    "conditions" : [
    {
    "name" : "operator-condition",
    "operator" : {
    "name" : "less-than",
    "value" : "1337"
    },
    "key" : "user_id"
    }
    ],
    "name" : "foo",
    "status" : "conditionally-active",
    "originalName" : "foo"
    }
    PUT /toggles/{name}
    https://github.com/qandidate-labs/qandidate-toggle-api

    View Slide

  90. Qandidate Toggle UI
    https://github.com/qandidate-labs/qandidate-toggle-ui

    View Slide

  91. Build (Pros)
    1. Free!
    2. Total control
    3. Complex rules
    4. API and UI available

    View Slide

  92. Build (Cons)
    1. Self hosted
    2. PHP only
    3. Complex rules

    View Slide

  93. Buy

    View Slide

  94. View Slide

  95. Multi-Language

    View Slide

  96. Prebuilt Dashboard

    View Slide

  97. Environment Aware

    View Slide

  98. Scaleable

    View Slide

  99. Audit Log

    View Slide

  100. A/B Testing

    View Slide

  101. Buy (Pros)
    1. Lovely dashboard
    2. Multi-language
    3. Built in audit log
    4. Environment aware
    5. A/B testing conversion tracking

    View Slide

  102. Buy (Cons)
    1. It costs $79/month
    2. Can’t modify the product
    3. Dependent on 3rd party

    View Slide

  103. Build vs Buy

    View Slide

  104. It depends

    View Slide

  105. Flag Management

    View Slide

  106. Toggle Configuratin

    View Slide

  107. Hard
    Coded
    Change
    the code
    Dynamic
    Static

    View Slide

  108. Hard
    Coded
    Deploy
    Config
    Change
    the code
    Deploy
    your
    application
    Dynamic
    Static

    View Slide

  109. Hard
    Coded
    Deploy
    Config
    Parameterised
    Config
    Change
    the code
    Deploy
    your
    application
    Edit config,
    restart app
    Dynamic
    Static

    View Slide

  110. Hard
    Coded
    Deploy
    Config
    Parameterised
    Config
    Database
    Config
    Change
    the code
    Deploy
    your
    application
    Edit config,
    restart app
    Edit via
    UI
    Dynamic
    Static

    View Slide

  111. Hard
    Coded
    Deploy
    Config
    Parameterised
    Config
    Database
    Config
    Distributed
    Config
    Change
    the code
    Deploy
    your
    application
    Edit config,
    restart app
    Edit via
    UI
    e.g. Consul
    or
    Zookeeper
    Dynamic
    Static

    View Slide

  112. Feature Dashboard

    View Slide

  113. Flag State

    View Slide

  114. Role Based Access

    View Slide

  115. Audit Log

    View Slide

  116. Non-technical Users

    View Slide

  117. Technical Debt

    View Slide

  118. Cleanup based on activity

    View Slide

  119. Cleanup based on activity

    View Slide

  120. Create & Cleanup

    View Slide

  121. WIP limits

    View Slide

  122. WIP limits

    View Slide

  123. Expiration date

    View Slide

  124. Explosion date

    View Slide

  125. Tips & Tricks

    View Slide

  126. Testing Flags

    View Slide

  127. Testing Flags
    Feature A Feature B
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠

    View Slide

  128. Testing Flags
    Feature A Feature B Feature C
    ✅ ✅ ✅
    ✅ ☠ ✅
    ✅ ✅ ☠
    ☠ ✅ ✅
    ✅ ☠ ☠
    ☠ ✅ ☠
    ☠ ☠ ✅
    ☠ ☠ ☠

    View Slide

  129. Testing Flags
    Feature A Feature B Feature C Feature D
    ✅ ✅ ✅ ✅
    ✅ ✅ ✅ ☠
    ✅ ✅ ☠ ✅
    ✅ ☠ ✅ ✅
    ☠ ✅ ✅ ✅
    ✅ ✅ ☠ ☠
    ✅ ☠ ✅ ☠
    ✅ ☠ ☠ ✅
    ☠ ✅ ✅ ☠
    ☠ ✅ ☠ ✅
    ☠ ☠ ✅ ✅
    ✅ ☠ ☠ ☠
    ☠ ✅ ☠ ☠
    ☠ ☠ ✅ ☠
    ☠ ☠ ☠ ✅
    ☠ ☠ ☠ ☠

    View Slide

  130. Testing Flags
    Number of flags Possible permutations
    2 4
    3 8
    4 16
    5 32
    10 1024
    25 33,554,432
    50 1.12589991E+15
    100 1.2676506E+30

    View Slide

  131. Homepage
    Carousel
    Homepage CTA
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠
    Forum Edit
    Permission
    Forum Delete
    Permission
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠
    Upload Profile
    Image


    New Interesting
    Algorithm


    High Post Length


    Show Debug
    Info?


    Enable
    Audit Log


    Auto Logout


    API Access API v2
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠

    View Slide

  132. Expose State
    {
    "api_v2": true,
    "audit_log": true,
    "new_recommendation_algo": false
    }
    GET /toggle_status

    View Slide

  133. Flag Descriptions
    basic-rec-algo: true



    View Slide

  134. Flag Descriptions
    basic-rec-algo:

    enabled: true

    description: Use a simplistic recommendation algorithm. This is fast and
    produces less load on backend systems, but is way less accurate than
    our standard algorithm.

    View Slide

  135. Clear Naming
    allow_user_delete and allow_user_edit
    not
    user_control

    View Slide

  136. Audit Log
    Feature State User Date
    allow_user_edit
    ✅ Sarah 2018-05-11 12:22:08
    enhanced_rec_algo
    ✅ Ashley 2018-05-04 15:33:16
    allow_user_signup
    ☠ Ashley 2018-05-04 15:27:33
    enhanced_rec_algo
    ☠ system:load_manager 2018-05-04 15:16:30
    allow_user_signup
    ✅ Bob 2018-05-04 15:12:01
    enable_api
    ✅ Sarah 2018-04-30 09:56:59

    View Slide

  137. Fail Safe
    Do positively enable features e.g. enable-ibm-
    analysis, enhanced-recomendation-algo

    Don’t use flags for suppressing new features e.g.

    disable-shiny-new-feature

    View Slide

  138. Conclusion

    View Slide

  139. Flags are great!

    View Slide

  140. Ops
    Toggles
    Release
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles

    View Slide

  141. Build vs Buy

    View Slide

  142. Qandidate Toggle

    View Slide

  143. View Slide

  144. One More Thing

    View Slide

  145. No Reuse

    View Slide

  146. Knight Capital

    View Slide

  147. High frequency trader

    View Slide

  148. Mistakes were made

    View Slide

  149. Knight Capital
    1. Reused flags

    View Slide

  150. Knight Capital
    1. Reused flags
    2. Didn’t delete obsolete code

    View Slide

  151. Knight Capital
    1. Reused flags
    2. Didn’t delete obsolete code
    3. Couldn’t be disabled without a deployment

    View Slide

  152. Knight Capital
    1. Reused flags
    2. Didn’t delete obsolete code
    3. Couldn’t be disabled without a deployment
    4. The deploy actually enabled the old code on
    all 8 servers, instead of disabling it

    View Slide

  153. $460m Gone

    View Slide

  154. @mheap / @NexmoDev
    [email protected]
    https://joind.in/talk/298e7

    View Slide