$30 off During Our Annual Pro Sale. View Details »

Dr Sheldon Cooper Presents: Fun with Flags

Dr Sheldon Cooper Presents: Fun with Flags

No no, not country flags, feature flags! Feature flags are a powerful technique that allows teams to modify a system’s behaviour without changing code. They can be used several reasons – canary releases and A/B testing to name a few.

This talk will show you how you’re already using feature flags in your application without realising it. Next, we’ll take a look at some of the best tooling out there to help you take feature flags to the next level. Finally, we’ll cover strategies for removing feature flags before they become technical debt that you have to manage.

Michael Heap

June 09, 2018
Tweet

More Decks by Michael Heap

Other Decks in Technology

Transcript

  1. View Slide

  2. Michael Heap

    View Slide

  3. Don’t leave,
    pleeeaaaasseee

    View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. Fun with Country Flags!
    Presented by @mheap at #dpc18

    View Slide

  10. Fun with Country Flags!
    Presented by @mheap at #dpc18

    View Slide

  11. Fun with Country Flags!
    Presented by @mheap at #dpc18

    View Slide

  12. Fun with Country Flags!
    Presented by @mheap at #dpc18

    View Slide

  13. Fun with Feature Flags!
    Presented by @mheap at #dpc18

    View Slide

  14. Michael
    @mheap
    Developer Relations
    NexmoDev

    View Slide

  15. Fun with Feature Flags!
    Presented by @mheap at #dpc18

    View Slide

  16. Today’s Plan

    View Slide

  17. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View Slide

  18. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View Slide

  19. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View Slide

  20. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View Slide

  21. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View Slide

  22. Feature Flags?

    View Slide

  23. Feature Toggles?

    View Slide

  24. Feature Flip?

    View Slide

  25. Feature Gate?

    View Slide

  26. Feature Switches?

    View Slide

  27. Separate feature release
    from code deployment

    View Slide

  28. http://code.flickr.net/2009/12/02/flipping-out/

    View Slide

  29. View Slide

  30. View Slide

  31. Feature flags aren’t new

    View Slide

  32. #ifdef NEW_ALGORITHM
    NEW_ALGORITHM(input)
    #else
    ALGORITHM(input)
    #endif
    $ make build -DNEW_ALGORITHM

    View Slide

  33. ./make_call.sh
    --use-new-algorithm
    ./make_call.sh
    --use-original-algorithm

    View Slide

  34. if (in_array($user->id, $whitelistedUsers)) {
    return $service->useNewAlgorithm($input);
    }
    return $service->useAlgorithm($input);

    View Slide

  35. I love a good history lesson

    View Slide

  36. Use Cases?

    View Slide

  37. 1. Permissions

    View Slide

  38. 1. Permissions
    2. Early-access

    View Slide

  39. 1. Permissions
    2. Early-access
    3. Opt-in

    View Slide

  40. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out

    View Slide

  41. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users

    View Slide

  42. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven

    View Slide

  43. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch

    View Slide

  44. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user

    View Slide

  45. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    9. Maintenance

    View Slide

  46. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    9. Maintenance
    10. Sunset

    View Slide

  47. View Slide

  48. Flag Types

    View Slide

  49. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles

    View Slide

  50. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Days
    Dynamism
    Per deployment

    View Slide

  51. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Days - Weeks
    Dynamism
    Human Controlled

    View Slide

  52. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Years
    Dynamism
    Per Request

    View Slide

  53. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Weeks - Months
    Dynamism
    Per Request

    View Slide

  54. https://martinfowler.com/articles/feature-toggles.html

    View Slide

  55. Ownership
    Release toggles owned by devs
    Ops toggles owned by ops
    Permission toggles
    Experiment toggles
    owned by product

    View Slide

  56. View Slide

  57. Flag Granularity

    View Slide

  58. On/Off
    Switch
    Enable or
    disable
    entirely

    View Slide

  59. On/Off
    Switch
    Percentage
    of users
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users

    View Slide

  60. On/Off
    Switch
    Percentage
    of users
    Individuals
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users

    View Slide

  61. On/Off
    Switch
    Percentage
    of users
    Individuals Groups
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    Enable for
    specific
    groups
    e.g.
    Power Users

    View Slide

  62. On/Off
    Switch
    Percentage
    of users
    Individuals Groups Request
    information
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    Enable for
    specific
    groups
    e.g.
    Power Users
    Enable for
    specific
    request
    headers or
    bodies

    View Slide

  63. View Slide

  64. Implementation

    View Slide

  65. Build vs Buy

    View Slide

  66. 1. Dashboard

    View Slide

  67. 1. Dashboard
    2. Access control

    View Slide

  68. 1. Dashboard
    2. Access control
    3. Complex rules

    View Slide

  69. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags

    View Slide

  70. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules

    View Slide

  71. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing

    View Slide

  72. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed

    View Slide

  73. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    8. Scale

    View Slide

  74. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    8. Scale
    9. 100% uptime

    View Slide

  75. Build

    View Slide

  76. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  77. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  78. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  79. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  80. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  81. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_MAJORITY);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  82. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  83. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  84. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  85. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View Slide

  86. $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions);
    $manager->add($toggle);
    Configuration: Code

    View Slide

  87. Configuration: Array
    $data = array(
    'some-feature' => array(
    'name' => 'toggling',
    'conditions' => array(
    array(
    'name' => 'operator-condition',
    'key' => 'user_id',
    'operator' => array('name' => 'greater-than', 'value' => 41),
    ),
    ),
    ),
    );
    $serializer = new InMemoryCollectionSerializer();
    $collection = $serializer->deserialize($data);
    $manager = new ToggleManager($collection);

    View Slide

  88. Configuration: JSON/YAML
    $yaml = <<toggles:
    - name: some-feature
    conditions:
    - name: operator-condition
    key: user_id
    operator:
    name: greater-than
    value: 41
    YML;
    $array = Yaml::parse($yaml);
    $serializer = new InMemoryCollectionSerializer();
    $collection = $serializer->deserialize($array);
    $manager = new ToggleManager($collection);

    View Slide

  89. Configuration: Redis
    $predis = new Predis\Client();
    $collection = new PredisCollection('toggle_demo', $predis);
    $manager = new ToggleManager($collection);

    View Slide

  90. Qandidate Toggle API
    {
    "conditions" : [
    {
    "name" : "operator-condition",
    "operator" : {
    "name" : "less-than",
    "value" : "1337"
    },
    "key" : "user_id"
    }
    ],
    "name" : "foo",
    "status" : "conditionally-active",
    "originalName" : "foo"
    }
    PUT /toggles/{name}
    https://github.com/qandidate-labs/qandidate-toggle-api

    View Slide

  91. Qandidate Toggle UI
    https://github.com/qandidate-labs/qandidate-toggle-ui

    View Slide

  92. Build (Pros)
    1. Free!
    2. Total control
    3. Complex rules
    4. API and UI available

    View Slide

  93. Build (Cons)
    1. Self hosted
    2. PHP only
    3. Complex rules

    View Slide

  94. Buy

    View Slide

  95. View Slide

  96. Multi-Language

    View Slide

  97. Prebuilt Dashboard

    View Slide

  98. Environment Aware

    View Slide

  99. Scaleable

    View Slide

  100. Audit Log

    View Slide

  101. A/B Testing

    View Slide

  102. ld-relay

    View Slide

  103. Buy (Pros)
    1. Lovely dashboard
    2. Multi-language
    3. Built in audit log
    4. Environment aware
    5. A/B testing conversion tracking

    View Slide

  104. Buy (Cons)
    1. It costs (at least) $79/month
    2. Can’t modify the product
    3. Dependent on 3rd party

    View Slide

  105. Build vs Buy

    View Slide

  106. So. Much. Choice

    View Slide

  107. It depends

    View Slide

  108. Show me code

    View Slide

  109. use Qandidate\Bundle\ToggleBundle\Annotations\Toggle;
    /**
    * @Toggle("profile-management")
    */
    class ProfileController
    {
    public function updatePassword()
    {
    }
    public function updateEmail()
    {
    }
    }

    View Slide

  110. use Qandidate\Bundle\ToggleBundle\Annotations\Toggle;
    class ProfileController
    {
    /**
    * @Toggle("profile-edit-password")
    */
    public function updatePassword()
    {
    }
    public function updateEmail()
    {
    }
    }

    View Slide

  111. public function onKernelRequest($event) {
    $flag = $event->getRequest->attributes->get('_flag');
    $alt = $event->getRequest->attributes->get('_alt');
    if ($toggleManager->active($flag, $context)) {
    $event->getRequest->attributes->set('controller', $alt);
    }
    }
    profile:
    path: /profile/{user}
    defaults:
    _controller: AcmeBundle:Profile:update
    _alt: AcmeBundle:NewProfile:update
    _flag: new-profile-controller

    View Slide

  112. {% if feature_is_active('new-profile-image-picker') %}
    {# Show new picker #}
    {% else %}
    {# Use old flash-based picker #}
    {% endif %}

    View Slide

  113. class AudioProcessorFactory {
    public function build($args, $context) {
    if ($this->toggleManager->active(‘enhanced-audio', $context)) {
    return new EnhancedAudioStreamProcessor($args);
    }
    return new BasicAudioStreamProcessor($args);
    }
    }

    View Slide

  114. Controller View Factory
    Enable a
    route for
    specific users
    Show different
    UI elements
    Branch by
    abstraction for
    refactoring

    View Slide

  115. View Slide

  116. Toggle Configuratin

    View Slide

  117. Hard
    Coded
    Change
    the code
    Dynamic
    Static

    View Slide

  118. Hard
    Coded
    Deploy
    Config
    Change
    the code
    Deploy
    your
    application
    Dynamic
    Static

    View Slide

  119. Hard
    Coded
    Deploy
    Config
    Parameterised
    Config
    Change
    the code
    Deploy
    your
    application
    Edit config,
    restart app
    Dynamic
    Static

    View Slide

  120. Hard
    Coded
    Deploy
    Config
    Parameterised
    Config
    Database
    Config
    Change
    the code
    Deploy
    your
    application
    Edit config,
    restart app
    Edit via
    UI
    Dynamic
    Static

    View Slide

  121. Hard
    Coded
    Deploy
    Config
    Parameterised
    Config
    Database
    Config
    Distributed
    Config
    Change
    the code
    Deploy
    your
    application
    Edit config,
    restart app
    Edit via
    UI
    e.g. Consul
    or
    Zookeeper
    Dynamic
    Static

    View Slide

  122. Feature Dashboard

    View Slide

  123. Flag State

    View Slide

  124. Role Based Access

    View Slide

  125. Audit Log

    View Slide

  126. Non-technical Users

    View Slide

  127. Technical Debt

    View Slide

  128. DON’T

    View Slide

  129. Cleanup based on activity

    View Slide

  130. Cleanup based on activity

    View Slide

  131. Create & Cleanup

    View Slide

  132. WIP limits

    View Slide

  133. Expiration date

    View Slide

  134. Explosion date

    View Slide

  135. Tips & Tricks

    View Slide

  136. Testing Flags

    View Slide

  137. Testing Flags
    Feature A Feature B
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠

    View Slide

  138. Testing Flags
    Feature A Feature B Feature C
    ✅ ✅ ✅
    ✅ ☠ ✅
    ✅ ✅ ☠
    ☠ ✅ ✅
    ✅ ☠ ☠
    ☠ ✅ ☠
    ☠ ☠ ✅
    ☠ ☠ ☠

    View Slide

  139. Testing Flags
    Feature A Feature B Feature C Feature D
    ✅ ✅ ✅ ✅
    ✅ ✅ ✅ ☠
    ✅ ✅ ☠ ✅
    ✅ ☠ ✅ ✅
    ☠ ✅ ✅ ✅
    ✅ ✅ ☠ ☠
    ✅ ☠ ✅ ☠
    ✅ ☠ ☠ ✅
    ☠ ✅ ✅ ☠
    ☠ ✅ ☠ ✅
    ☠ ☠ ✅ ✅
    ✅ ☠ ☠ ☠
    ☠ ✅ ☠ ☠
    ☠ ☠ ✅ ☠
    ☠ ☠ ☠ ✅
    ☠ ☠ ☠ ☠

    View Slide

  140. Testing Flags
    Number of flags Possible permutations
    2 4
    3 8
    4 16
    5 32
    10 1024
    25 33,554,432
    50 1.12589991E+15
    100 1.2676506E+30

    View Slide

  141. Homepage
    Carousel
    Homepage CTA
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠
    Forum Edit
    Permission
    Forum Delete
    Permission
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠
    Upload Profile
    Image


    New Interesting
    Algorithm


    High Post Length


    Show Debug
    Info?


    Enable
    Audit Log


    Auto Logout


    API Access API v2
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠

    View Slide

  142. Expose State
    {
    "api_v2": true,
    "audit_log": true,
    "new_recommendation_algo": false
    }
    GET /toggle_status

    View Slide

  143. Flag Descriptions
    basic-rec-algo: true



    View Slide

  144. Flag Descriptions
    basic-rec-algo:

    enabled: true

    description: Use a simplistic recommendation algorithm. This is fast and
    produces less load on backend systems, but is way less accurate than
    our standard algorithm.

    View Slide

  145. Clear Naming
    allow_user_delete and allow_user_edit
    not
    user_control

    View Slide

  146. Audit Log
    Feature State User Date
    allow_user_edit
    ✅ Sarah 2018-05-11 12:22:08
    enhanced_rec_algo
    ✅ Ashley 2018-05-04 15:33:16
    allow_user_signup
    ☠ Ashley 2018-05-04 15:27:33
    enhanced_rec_algo
    ☠ system:load_manager 2018-05-04 15:16:30
    allow_user_signup
    ✅ Bob 2018-05-04 15:12:01
    enable_api
    ✅ Sarah 2018-04-30 09:56:59

    View Slide

  147. Fail Safe
    Do positively enable features e.g. enable-ibm-
    analysis, enhanced-recomendation-algo

    Don’t use flags for suppressing new features e.g.

    disable-shiny-new-feature

    View Slide

  148. Conclusion

    View Slide

  149. Woooooooo

    View Slide

  150. Flags are great!

    View Slide

  151. Ops
    Toggles
    Release
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles

    View Slide

  152. Build vs Buy

    View Slide

  153. Qandidate Toggle

    View Slide

  154. View Slide

  155. One More Thing

    View Slide

  156. No Reuse

    View Slide

  157. Knight Capital

    View Slide

  158. High frequency trader

    View Slide

  159. View Slide

  160. View Slide

  161. Mistakes were made

    View Slide

  162. Knight Capital
    1. Reused flags

    View Slide

  163. Knight Capital
    1. Reused flags
    2. Didn’t delete obsolete code

    View Slide

  164. Knight Capital
    1. Reused flags
    2. Didn’t delete obsolete code
    3. Couldn’t be disabled without a deployment

    View Slide

  165. Knight Capital
    1. Reused flags
    2. Didn’t delete obsolete code
    3. Couldn’t be disabled without a deployment
    4. The deploy actually enabled the old code on
    all 8 servers, instead of disabling it

    View Slide

  166. $460m Gone

    View Slide

  167. @mheap / @NexmoDev
    [email protected]
    https://joind.in/talk/253a0

    View Slide