Dr Sheldon Cooper Presents: Fun with Flags

No no, not country flags, feature flags! Feature flags are a powerful technique that allows teams to modify a system’s behaviour without changing code. They can be used several reasons – canary releases and A/B testing to name a few.

This talk will show you how you’re already using feature flags in your application without realising it. Next, we’ll take a look at some of the best tooling out there to help you take feature flags to the next level. Finally, we’ll cover strategies for removing feature flags before they become technical debt that you have to manage.

Transcript

1. View Slide

2. Michael Heap
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. Fun with Country Flags!
   Presented by @mheap at @phpday
   

   View Slide

10. Fun with Country Flags!
    Presented by @mheap at @phpday
    

    View Slide

11. Fun with Country Flags!
    Presented by @mheap at @phpday
    

    View Slide

12. Fun with Country Flags!
    Presented by @mheap at @phpday
    

    View Slide

13. Fun with Feature Flags!
    Presented by @mheap at @phpday
    

    View Slide

14. Michael
    @mheap
    Developer Relations
    NexmoDev
    

    View Slide

15. Fun with Feature Flags!
    Presented by @mheap at @phpday
    

    View Slide

16. Today’s Plan
    

    View Slide

17. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion
    

    View Slide

18. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion
    

    View Slide

19. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion
    

    View Slide

20. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion
    

    View Slide

21. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion
    

    View Slide

22. Feature Flags?
    

    View Slide

23. Feature Toggles?
    

    View Slide

24. Feature Flip?
    

    View Slide

25. Feature Gate?
    

    View Slide

26. Feature Switches?
    

    View Slide

27. Separates feature release
    from code deployment
    

    View Slide

28. http://code.flickr.net/2009/12/02/flipping-out/
    

    View Slide

29. View Slide

30. View Slide

31. Feature flags aren’t new
    

    View Slide

32. #ifdef IBM_ANALYSIS
    IBM_ANALYSIS(audio)
    #else
    INTERNAL_ANALYSIS(audio)
    #endif
    $ make build -DIBM_ANALYSIS
    

    View Slide

33. ./make_call.sh
    —-use-internal-analysis
    ./make_call.sh
    —-use-ibm-analysis
    

    View Slide

34. if (in_array($user->id, $whitelistedUsers)) {
    return $ibm->analyzeAudio($audio);
    }
    return $sentiment->analyze($audio);
    

    View Slide

35. Use Cases?
    

    View Slide

36. 1. Permissions
    

    View Slide

37. 1. Permissions
    2. Early-access
    

    View Slide

38. 1. Permissions
    2. Early-access
    3. Opt-in
    

    View Slide

39. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    

    View Slide

40. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    

    View Slide

41. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    

    View Slide

42. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    

    View Slide

43. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    

    View Slide

44. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    9. Maintenance
    

    View Slide

45. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    9. Maintenance
    10. Sunset
    

    View Slide

46. Flag Classes
    

    View Slide

47. Ops
    Toggles
    Release
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    

    View Slide

48. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Days
    Dynamism
    Per deployment
    

    View Slide

49. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Days - Weeks
    Dynamism
    Human Controlled
    

    View Slide

50. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Years
    Dynamism
    Per Request
    

    View Slide

51. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Weeks - Months
    Dynamism
    Per Request
    

    View Slide

52. https://martinfowler.com/articles/feature-toggles.html
    

    View Slide

53. Ownership
    Release toggles owned by devs
    Ops toggles owned by ops
    Permission toggles
    Experiment toggles
    owned by product
    

    View Slide

54. Flag Scopes
    

    View Slide

55. On/Off
    Switch
    Enable or
    disable
    entirely
    

    View Slide

56. On/Off
    Switch
    Percentage
    of users
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    

    View Slide

57. On/Off
    Switch
    Percentage
    of users
    Individuals
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    

    View Slide

58. On/Off
    Switch
    Percentage
    of users
    Individuals Groups
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    Enable for
    specific
    groups
    e.g.
    Power Users
    

    View Slide

59. On/Off
    Switch
    Percentage
    of users
    Individuals Groups Request
    information
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    Enable for
    specific
    groups
    e.g.
    Power Users
    Enable for
    specific
    request
    headers or
    bodies
    

    View Slide

60. Implementation
    

    View Slide

61. Build vs Buy
    

    View Slide

62. 1. Dashboard
    

    View Slide

63. 1. Dashboard
    2. Access control
    

    View Slide

64. 1. Dashboard
    2. Access control
    3. Complex rules
    

    View Slide

65. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    

    View Slide

66. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    

    View Slide

67. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    

    View Slide

68. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    

    View Slide

69. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    8. Scale
    

    View Slide

70. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    8. Scale
    9. 100% uptime
    

    View Slide

71. Build
    

    View Slide

72. Rollout (Ruby)
    # Is a feature active?
    $rollout.active?(:chat)
    # Activate a specific user
    $rollout.activate_user(:chat, @user)
    # Define a group
    $rollout.define_group(:admins) do |user|
    user.admin?
    end
    # Roll out to a percentage of users
    $rollout.activate_percentage(:chat, 20)
    

    View Slide

73. Rollout (PHP)
    # Is a feature active?
    $rollout->isActive('chat');
    # Activate a specific user
    $rollout->activateUser('chat', $user);
    # Define a group
    $rollout->defineGroup('admins', function($user = null) {
    return $user->isAdmin();
    });
    # Roll out to a percentage of users
    $rollout->activatePercentage('chat', 20);
    

    View Slide

74. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active(‘disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

75. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active(‘disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

76. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

77. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

78. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

79. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

80. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_MAJORITY);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

81. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

82. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

83. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

84. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

85. $manager = new ToggleManager(new InMemoryCollection());
    $toggle = new Toggle('some-feature', [
    new OperatorCondition('user_id', new GreaterThan(41))
    ]);
    $manager->add($toggle);
    Configuration: Code
    

    View Slide

86. Configuration: Array
    $data = array(
    'some-feature' => array(
    'name' => 'toggling',
    'conditions' => array(
    array(
    'name' => 'operator-condition',
    'key' => 'user_id',
    'operator' => array('name' => 'greater-than', 'value' => 41),
    ),
    ),
    ),
    );
    $serializer = new InMemoryCollectionSerializer();
    $collection = $serializer->deserialize($data);
    $manager = new ToggleManager($collection);
    

    View Slide

87. Configuration: JSON/YAML
    $yaml = <<toggles:
    - name: some-feature
    conditions:
    - name: operator-condition
    key: user_id
    operator:
    name: greater-than
    value: 41
    YML;
    $array = Yaml::parse($yaml);
    $serializer = new InMemoryCollectionSerializer();
    $collection = $serializer->deserialize($array);
    $manager = new ToggleManager($collection);
    

    View Slide

88. Configuration: Redis
    $predis = new Predis\Client();
    $collection = new PredisCollection('toggle_demo', $predis);
    $manager = new ToggleManager($collection);
    

    View Slide

89. Qandidate Toggle API
    {
    "conditions" : [
    {
    "name" : "operator-condition",
    "operator" : {
    "name" : "less-than",
    "value" : "1337"
    },
    "key" : "user_id"
    }
    ],
    "name" : "foo",
    "status" : "conditionally-active",
    "originalName" : "foo"
    }
    PUT /toggles/{name}
    https://github.com/qandidate-labs/qandidate-toggle-api
    

    View Slide

90. Qandidate Toggle UI
    https://github.com/qandidate-labs/qandidate-toggle-ui
    

    View Slide

91. Build (Pros)
    1. Free!
    2. Total control
    3. Complex rules
    4. API and UI available
    

    View Slide

92. Build (Cons)
    1. Self hosted
    2. PHP only
    3. Complex rules
    

    View Slide

93. Buy
    

    View Slide

94. View Slide

95. Multi-Language
    

    View Slide

96. Prebuilt Dashboard
    

    View Slide

97. Environment Aware
    

    View Slide

98. Scaleable
    

    View Slide

99. Audit Log
    

    View Slide

100. A/B Testing
     

     View Slide

101. Buy (Pros)
     1. Lovely dashboard
     2. Multi-language
     3. Built in audit log
     4. Environment aware
     5. A/B testing conversion tracking
     

     View Slide

102. Buy (Cons)
     1. It costs $79/month
     2. Can’t modify the product
     3. Dependent on 3rd party
     

     View Slide

103. Build vs Buy
     

     View Slide

104. It depends
     

     View Slide

105. Flag Management
     

     View Slide

106. Toggle Configuratin
     

     View Slide

107. Hard
     Coded
     Change
     the code
     Dynamic
     Static
     

     View Slide

108. Hard
     Coded
     Deploy
     Config
     Change
     the code
     Deploy
     your
     application
     Dynamic
     Static
     

     View Slide

109. Hard
     Coded
     Deploy
     Config
     Parameterised
     Config
     Change
     the code
     Deploy
     your
     application
     Edit config,
     restart app
     Dynamic
     Static
     

     View Slide

110. Hard
     Coded
     Deploy
     Config
     Parameterised
     Config
     Database
     Config
     Change
     the code
     Deploy
     your
     application
     Edit config,
     restart app
     Edit via
     UI
     Dynamic
     Static
     

     View Slide

111. Hard
     Coded
     Deploy
     Config
     Parameterised
     Config
     Database
     Config
     Distributed
     Config
     Change
     the code
     Deploy
     your
     application
     Edit config,
     restart app
     Edit via
     UI
     e.g. Consul
     or
     Zookeeper
     Dynamic
     Static
     

     View Slide

112. Feature Dashboard
     

     View Slide

113. Flag State
     

     View Slide

114. Role Based Access
     

     View Slide

115. Audit Log
     

     View Slide

116. Non-technical Users
     

     View Slide

117. Technical Debt
     

     View Slide

118. Cleanup based on activity
     

     View Slide

119. Cleanup based on activity
     

     View Slide

120. Create & Cleanup
     

     View Slide

121. WIP limits
     

     View Slide

122. WIP limits
     

     View Slide

123. Expiration date
     

     View Slide

124. Explosion date
     

     View Slide

125. Tips & Tricks
     

     View Slide

126. Testing Flags
     

     View Slide

127. Testing Flags
     Feature A Feature B
     ✅ ✅
     ✅ ☠
     ☠ ✅
     ☠ ☠
     

     View Slide

128. Testing Flags
     Feature A Feature B Feature C
     ✅ ✅ ✅
     ✅ ☠ ✅
     ✅ ✅ ☠
     ☠ ✅ ✅
     ✅ ☠ ☠
     ☠ ✅ ☠
     ☠ ☠ ✅
     ☠ ☠ ☠
     

     View Slide

129. Testing Flags
     Feature A Feature B Feature C Feature D
     ✅ ✅ ✅ ✅
     ✅ ✅ ✅ ☠
     ✅ ✅ ☠ ✅
     ✅ ☠ ✅ ✅
     ☠ ✅ ✅ ✅
     ✅ ✅ ☠ ☠
     ✅ ☠ ✅ ☠
     ✅ ☠ ☠ ✅
     ☠ ✅ ✅ ☠
     ☠ ✅ ☠ ✅
     ☠ ☠ ✅ ✅
     ✅ ☠ ☠ ☠
     ☠ ✅ ☠ ☠
     ☠ ☠ ✅ ☠
     ☠ ☠ ☠ ✅
     ☠ ☠ ☠ ☠
     

     View Slide

130. Testing Flags
     Number of flags Possible permutations
     2 4
     3 8
     4 16
     5 32
     10 1024
     25 33,554,432
     50 1.12589991E+15
     100 1.2676506E+30
     

     View Slide

131. Homepage
     Carousel
     Homepage CTA
     ✅ ✅
     ✅ ☠
     ☠ ✅
     ☠ ☠
     Forum Edit
     Permission
     Forum Delete
     Permission
     ✅ ✅
     ✅ ☠
     ☠ ✅
     ☠ ☠
     Upload Profile
     Image
     ✅
     ☠
     New Interesting
     Algorithm
     ✅
     ☠
     High Post Length
     ✅
     ☠
     Show Debug
     Info?
     ✅
     ☠
     Enable
     Audit Log
     ✅
     ☠
     Auto Logout
     ✅
     ☠
     API Access API v2
     ✅ ✅
     ✅ ☠
     ☠ ✅
     ☠ ☠
     

     View Slide

132. Expose State
     {
     "api_v2": true,
     "audit_log": true,
     "new_recommendation_algo": false
     }
     GET /toggle_status
     

     View Slide

133. Flag Descriptions
     basic-rec-algo: true
     
     
     
     

     View Slide

134. Flag Descriptions
     basic-rec-algo:
     
     enabled: true
     
     description: Use a simplistic recommendation algorithm. This is fast and
     produces less load on backend systems, but is way less accurate than
     our standard algorithm.
     

     View Slide

135. Clear Naming
     allow_user_delete and allow_user_edit
     not
     user_control
     

     View Slide

136. Audit Log
     Feature State User Date
     allow_user_edit
     ✅ Sarah 2018-05-11 12:22:08
     enhanced_rec_algo
     ✅ Ashley 2018-05-04 15:33:16
     allow_user_signup
     ☠ Ashley 2018-05-04 15:27:33
     enhanced_rec_algo
     ☠ system:load_manager 2018-05-04 15:16:30
     allow_user_signup
     ✅ Bob 2018-05-04 15:12:01
     enable_api
     ✅ Sarah 2018-04-30 09:56:59
     

     View Slide

137. Fail Safe
     Do positively enable features e.g. enable-ibm-
     analysis, enhanced-recomendation-algo
     
     Don’t use flags for suppressing new features e.g.
     
     disable-shiny-new-feature
     

     View Slide

138. Conclusion
     

     View Slide

139. Flags are great!
     

     View Slide

140. Ops
     Toggles
     Release
     Toggles
     Flag Classes
     Permission
     Toggles
     Experiment
     Toggles
     

     View Slide

141. Build vs Buy
     

     View Slide

142. Qandidate Toggle
     

     View Slide

143. View Slide

144. One More Thing
     

     View Slide

145. No Reuse
     

     View Slide

146. Knight Capital
     

     View Slide

147. High frequency trader
     

     View Slide

148. Mistakes were made
     

     View Slide

149. Knight Capital
     1. Reused flags
     

     View Slide

150. Knight Capital
     1. Reused flags
     2. Didn’t delete obsolete code
     

     View Slide

151. Knight Capital
     1. Reused flags
     2. Didn’t delete obsolete code
     3. Couldn’t be disabled without a deployment
     

     View Slide

152. Knight Capital
     1. Reused flags
     2. Didn’t delete obsolete code
     3. Couldn’t be disabled without a deployment
     4. The deploy actually enabled the old code on
     all 8 servers, instead of disabling it
     

     View Slide

153. $460m Gone
     

     View Slide

154. @mheap / @NexmoDev
     m@michaelheap.com
     https://joind.in/talk/298e7
     

     View Slide