Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)

Michael Heap
September 20, 2018

Dr Sheldon Cooper Presents: Fun with Flags (NEPHP)

No no, not country flags, feature flags! Feature flags are a powerful technique that allows teams to modify a system’s behaviour without changing code. They can be used several reasons – canary releases and A/B testing to name a few.

This talk will show you how you’re already using feature flags in your application without realising it. Next, we’ll take a look at some of the best tooling out there to help you take feature flags to the next level. Finally, we’ll cover strategies for removing feature flags before they become technical debt that you have to manage.

Michael Heap

September 20, 2018
Tweet

More Decks by Michael Heap

Other Decks in Technology

Transcript

  1. Michael Heap

    View full-size slide

  2. Don’t leave,
    pleeeaaaasseee

    View full-size slide

  3. Fun with Country Flags!
    Presented by @mheap at #nephp

    View full-size slide

  4. Fun with Country Flags!
    Presented by @mheap at #nephp

    View full-size slide

  5. Fun with Country Flags!
    Presented by @mheap at #nephp

    View full-size slide

  6. Fun with Country Flags!
    Presented by @mheap at #nephp

    View full-size slide

  7. Fun with Feature Flags!
    Presented by @mheap at #nephp

    View full-size slide

  8. Michael
    @mheap
    Developer Relations
    NexmoDev

    View full-size slide

  9. Fun with Feature Flags!
    Presented by @mheap at #nephp

    View full-size slide

  10. Today’s Plan

    View full-size slide

  11. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View full-size slide

  12. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View full-size slide

  13. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View full-size slide

  14. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View full-size slide

  15. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion

    View full-size slide

  16. Feature Flags?

    View full-size slide

  17. Feature Toggles?

    View full-size slide

  18. Feature Flip?

    View full-size slide

  19. Feature Gate?

    View full-size slide

  20. Feature Switches?

    View full-size slide

  21. Separate feature release
    from code deployment

    View full-size slide

  22. http://code.flickr.net/2009/12/02/flipping-out/

    View full-size slide

  23. Feature flags aren’t new

    View full-size slide

  24. #ifdef NEW_ALGORITHM
    NEW_ALGORITHM(input)
    #else
    ALGORITHM(input)
    #endif
    $ make build -DNEW_ALGORITHM

    View full-size slide

  25. ./make_call.sh
    --use-new-algorithm
    ./make_call.sh
    --use-original-algorithm

    View full-size slide

  26. if (in_array($user->id, $whitelistedUsers)) {
    return $service->useNewAlgorithm($input);
    }
    return $service->useAlgorithm($input);

    View full-size slide

  27. I love a good history lesson

    View full-size slide

  28. 1. Permissions

    View full-size slide

  29. 1. Permissions
    2. Early-access

    View full-size slide

  30. 1. Permissions
    2. Early-access
    3. Opt-in

    View full-size slide

  31. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out

    View full-size slide

  32. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users

    View full-size slide

  33. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven

    View full-size slide

  34. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch

    View full-size slide

  35. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user

    View full-size slide

  36. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    9. Maintenance

    View full-size slide

  37. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    9. Maintenance
    10. Sunset

    View full-size slide

  38. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles

    View full-size slide

  39. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Days
    Dynamism
    Per deployment

    View full-size slide

  40. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Days - Weeks
    Dynamism
    Human Controlled

    View full-size slide

  41. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Years
    Dynamism
    Per Request

    View full-size slide

  42. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Weeks - Months
    Dynamism
    Per Request

    View full-size slide

  43. https://martinfowler.com/articles/feature-toggles.html

    View full-size slide

  44. Ownership
    Release toggles owned by devs
    Ops toggles owned by ops
    Permission toggles
    Experiment toggles
    owned by product

    View full-size slide

  45. Flag Granularity

    View full-size slide

  46. On/Off
    Switch
    Enable or
    disable
    entirely

    View full-size slide

  47. On/Off
    Switch
    Percentage
    of users
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users

    View full-size slide

  48. On/Off
    Switch
    Percentage
    of users
    Individuals
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users

    View full-size slide

  49. On/Off
    Switch
    Percentage
    of users
    Individuals Groups
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    Enable for
    specific
    groups
    e.g.
    Power Users

    View full-size slide

  50. On/Off
    Switch
    Percentage
    of users
    Individuals Groups Request
    information
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    Enable for
    specific
    groups
    e.g.
    Power Users
    Enable for
    specific
    request
    headers or
    bodies

    View full-size slide

  51. Implementation

    View full-size slide

  52. Build vs Buy

    View full-size slide

  53. 1. Dashboard

    View full-size slide

  54. 1. Dashboard
    2. Access control

    View full-size slide

  55. 1. Dashboard
    2. Access control
    3. Complex rules

    View full-size slide

  56. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags

    View full-size slide

  57. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules

    View full-size slide

  58. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing

    View full-size slide

  59. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed

    View full-size slide

  60. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    8. Scale

    View full-size slide

  61. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    8. Scale
    9. 100% uptime

    View full-size slide

  62. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View full-size slide

  63. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View full-size slide

  64. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View full-size slide

  65. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View full-size slide

  66. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View full-size slide

  67. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_MAJORITY);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View full-size slide

  68. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View full-size slide

  69. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View full-size slide

  70. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View full-size slide

  71. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';

    View full-size slide

  72. $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions);
    $manager->add($toggle);
    Configuration: Code

    View full-size slide

  73. Configuration: Array
    $data = array(
    'some-feature' => array(
    'name' => 'toggling',
    'conditions' => array(
    array(
    'name' => 'operator-condition',
    'key' => 'user_id',
    'operator' => array('name' => 'greater-than', 'value' => 41),
    ),
    ),
    ),
    );
    $serializer = new InMemoryCollectionSerializer();
    $collection = $serializer->deserialize($data);
    $manager = new ToggleManager($collection);

    View full-size slide

  74. Configuration: JSON/YAML
    $yaml = <<toggles:
    - name: some-feature
    conditions:
    - name: operator-condition
    key: user_id
    operator:
    name: greater-than
    value: 41
    YML;
    $array = Yaml::parse($yaml);
    $serializer = new InMemoryCollectionSerializer();
    $collection = $serializer->deserialize($array);
    $manager = new ToggleManager($collection);

    View full-size slide

  75. Configuration: Redis
    $predis = new Predis\Client();
    $collection = new PredisCollection('toggle_demo', $predis);
    $manager = new ToggleManager($collection);

    View full-size slide

  76. Qandidate Toggle API
    {
    "conditions" : [
    {
    "name" : "operator-condition",
    "operator" : {
    "name" : "less-than",
    "value" : "1337"
    },
    "key" : "user_id"
    }
    ],
    "name" : "foo",
    "status" : "conditionally-active",
    "originalName" : "foo"
    }
    PUT /toggles/{name}
    https://github.com/qandidate-labs/qandidate-toggle-api

    View full-size slide

  77. Qandidate Toggle UI
    https://github.com/qandidate-labs/qandidate-toggle-ui

    View full-size slide

  78. Build (Pros)
    1. Free!
    2. Total control
    3. Complex rules
    4. API and UI available

    View full-size slide

  79. Build (Cons)
    1. Self hosted
    2. PHP only
    3. Complex rules

    View full-size slide

  80. Multi-Language

    View full-size slide

  81. Prebuilt Dashboard

    View full-size slide

  82. Environment Aware

    View full-size slide

  83. Buy (Pros)
    1. Lovely dashboard
    2. Multi-language
    3. Built in audit log
    4. Environment aware
    5. A/B testing conversion tracking

    View full-size slide

  84. Buy (Cons)
    1. It costs (at least) $79/month
    2. Can’t modify the product
    3. Dependent on 3rd party

    View full-size slide

  85. Build vs Buy

    View full-size slide

  86. So. Much. Choice

    View full-size slide

  87. Show me code

    View full-size slide

  88. use Qandidate\Bundle\ToggleBundle\Annotations\Toggle;
    /**
    * @Toggle("profile-management")
    */
    class ProfileController
    {
    public function updatePassword()
    {
    }
    public function updateEmail()
    {
    }
    }

    View full-size slide

  89. use Qandidate\Bundle\ToggleBundle\Annotations\Toggle;
    class ProfileController
    {
    /**
    * @Toggle("profile-edit-password")
    */
    public function updatePassword()
    {
    }
    public function updateEmail()
    {
    }
    }

    View full-size slide

  90. public function onKernelRequest($event) {
    $flag = $event->getRequest->attributes->get('_flag');
    $alt = $event->getRequest->attributes->get('_alt');
    if ($toggleManager->active($flag, $context)) {
    $event->getRequest->attributes->set('controller', $alt);
    }
    }
    profile:
    path: /profile/{user}
    defaults:
    _controller: AcmeBundle:Profile:update
    _alt: AcmeBundle:NewProfile:update
    _flag: new-profile-controller

    View full-size slide

  91. {% if feature_is_active('new-profile-image-picker') %}
    {# Show new picker #}
    {% else %}
    {# Use old flash-based picker #}
    {% endif %}

    View full-size slide

  92. class AudioProcessorFactory {
    public function build($args, $context) {
    if ($this->toggleManager->active('enhanced-audio', $context)) {
    return new EnhancedAudioStreamProcessor($args);
    }
    return new BasicAudioStreamProcessor($args);
    }
    }

    View full-size slide

  93. Controller View Factory
    Enable a
    route for
    specific users
    Show different
    UI elements
    Branch by
    abstraction for
    refactoring

    View full-size slide

  94. Toggle Configuratin

    View full-size slide

  95. Hard
    Coded
    Change
    the code
    Dynamic
    Static

    View full-size slide

  96. Hard
    Coded
    Deploy
    Config
    Change
    the code
    Deploy
    your
    application
    Dynamic
    Static

    View full-size slide

  97. Hard
    Coded
    Deploy
    Config
    Parameterised
    Config
    Change
    the code
    Deploy
    your
    application
    Edit config,
    restart app
    Dynamic
    Static

    View full-size slide

  98. Hard
    Coded
    Deploy
    Config
    Parameterised
    Config
    Database
    Config
    Change
    the code
    Deploy
    your
    application
    Edit config,
    restart app
    Edit via
    UI
    Dynamic
    Static

    View full-size slide

  99. Hard
    Coded
    Deploy
    Config
    Parameterised
    Config
    Database
    Config
    Distributed
    Config
    Change
    the code
    Deploy
    your
    application
    Edit config,
    restart app
    Edit via
    UI
    e.g. Consul
    or
    Zookeeper
    Dynamic
    Static

    View full-size slide

  100. Feature Dashboard

    View full-size slide

  101. Role Based Access

    View full-size slide

  102. Non-technical Users

    View full-size slide

  103. Technical Debt

    View full-size slide

  104. Cleanup based on activity

    View full-size slide

  105. Cleanup based on activity

    View full-size slide

  106. Create & Cleanup

    View full-size slide

  107. Expiration date

    View full-size slide

  108. Explosion date

    View full-size slide

  109. Tips & Tricks

    View full-size slide

  110. Testing Flags

    View full-size slide

  111. Testing Flags
    Feature A Feature B
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠

    View full-size slide

  112. Testing Flags
    Feature A Feature B Feature C
    ✅ ✅ ✅
    ✅ ☠ ✅
    ✅ ✅ ☠
    ☠ ✅ ✅
    ✅ ☠ ☠
    ☠ ✅ ☠
    ☠ ☠ ✅
    ☠ ☠ ☠

    View full-size slide

  113. Testing Flags
    Feature A Feature B Feature C Feature D
    ✅ ✅ ✅ ✅
    ✅ ✅ ✅ ☠
    ✅ ✅ ☠ ✅
    ✅ ☠ ✅ ✅
    ☠ ✅ ✅ ✅
    ✅ ✅ ☠ ☠
    ✅ ☠ ✅ ☠
    ✅ ☠ ☠ ✅
    ☠ ✅ ✅ ☠
    ☠ ✅ ☠ ✅
    ☠ ☠ ✅ ✅
    ✅ ☠ ☠ ☠
    ☠ ✅ ☠ ☠
    ☠ ☠ ✅ ☠
    ☠ ☠ ☠ ✅
    ☠ ☠ ☠ ☠

    View full-size slide

  114. Testing Flags
    Number of flags Possible permutations
    2 4
    3 8
    4 16
    5 32
    10 1024
    25 33,554,432
    50 1.12589991E+15
    100 1.2676506E+30

    View full-size slide

  115. Homepage
    Carousel
    Homepage CTA
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠
    Forum Edit
    Permission
    Forum Delete
    Permission
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠
    Upload Profile
    Image


    New Interesting
    Algorithm


    High Post Length


    Show Debug
    Info?


    Enable
    Audit Log


    Auto Logout


    API Access API v2
    ✅ ✅
    ✅ ☠
    ☠ ✅
    ☠ ☠

    View full-size slide

  116. Expose State
    {
    "api_v2": true,
    "audit_log": true,
    "new_recommendation_algo": false
    }
    GET /toggle_status

    View full-size slide

  117. Flag Descriptions
    basic-rec-algo: true



    View full-size slide

  118. Flag Descriptions
    basic-rec-algo:

    enabled: true

    description: Use a simplistic recommendation algorithm. This is fast and
    produces less load on backend systems, but is way less accurate than
    our standard algorithm.

    View full-size slide

  119. Clear Naming
    allow_user_delete and allow_user_edit
    not
    user_control

    View full-size slide

  120. Audit Log
    Feature State User Date
    allow_user_edit
    ✅ Sarah 2018-05-11 12:22:08
    enhanced_rec_algo
    ✅ Ashley 2018-05-04 15:33:16
    allow_user_signup
    ☠ Ashley 2018-05-04 15:27:33
    enhanced_rec_algo
    ☠ system:load_manager 2018-05-04 15:16:30
    allow_user_signup
    ✅ Bob 2018-05-04 15:12:01
    enable_api
    ✅ Sarah 2018-04-30 09:56:59

    View full-size slide

  121. Fail Safe
    Do positively enable features e.g. enable-ibm-
    analysis, enhanced-recomendation-algo

    Don’t use flags for suppressing new features e.g.

    disable-shiny-new-feature

    View full-size slide

  122. Flags are great!

    View full-size slide

  123. Ops
    Toggles
    Release
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles

    View full-size slide

  124. Build vs Buy

    View full-size slide

  125. Qandidate Toggle

    View full-size slide

  126. One More Thing

    View full-size slide

  127. Knight Capital

    View full-size slide

  128. High frequency trader

    View full-size slide

  129. Mistakes were made

    View full-size slide

  130. Knight Capital
    1. Reused flags

    View full-size slide

  131. Knight Capital
    1. Reused flags
    2. Didn’t delete obsolete code

    View full-size slide

  132. Knight Capital
    1. Reused flags
    2. Didn’t delete obsolete code
    3. Couldn’t be disabled without a deployment

    View full-size slide

  133. Knight Capital
    1. Reused flags
    2. Didn’t delete obsolete code
    3. Couldn’t be disabled without a deployment
    4. The deploy actually enabled the old code on
    all 8 servers, instead of disabling it

    View full-size slide