Dr Sheldon Cooper Presents: Fun with Flags

No no, not country flags, feature flags! Feature flags are a powerful technique that allows teams to modify a system’s behaviour without changing code. They can be used several reasons – canary releases and A/B testing to name a few.

This talk will show you how you’re already using feature flags in your application without realising it. Next, we’ll take a look at some of the best tooling out there to help you take feature flags to the next level. Finally, we’ll cover strategies for removing feature flags before they become technical debt that you have to manage.

Transcript

1. View Slide

2. Michael Heap
   

   View Slide

3. Don’t leave,
   pleeeaaaasseee
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. Fun with Country Flags!
   Presented by @mheap at #dpc18
   

   View Slide

10. Fun with Country Flags!
    Presented by @mheap at #dpc18
    

    View Slide

11. Fun with Country Flags!
    Presented by @mheap at #dpc18
    

    View Slide

12. Fun with Country Flags!
    Presented by @mheap at #dpc18
    

    View Slide

13. Fun with Feature Flags!
    Presented by @mheap at #dpc18
    

    View Slide

14. Michael
    @mheap
    Developer Relations
    NexmoDev
    

    View Slide

15. Fun with Feature Flags!
    Presented by @mheap at #dpc18
    

    View Slide

16. Today’s Plan
    

    View Slide

17. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion
    

    View Slide

18. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion
    

    View Slide

19. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion
    

    View Slide

20. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion
    

    View Slide

21. 1. What are feature flags?
    2. Implementation
    3. Flag management
    4. Tips and tricks
    5. Conclusion
    

    View Slide

22. Feature Flags?
    

    View Slide

23. Feature Toggles?
    

    View Slide

24. Feature Flip?
    

    View Slide

25. Feature Gate?
    

    View Slide

26. Feature Switches?
    

    View Slide

27. Separate feature release
    from code deployment
    

    View Slide

28. http://code.flickr.net/2009/12/02/flipping-out/
    

    View Slide

29. View Slide

30. View Slide

31. Feature flags aren’t new
    

    View Slide

32. #ifdef NEW_ALGORITHM
    NEW_ALGORITHM(input)
    #else
    ALGORITHM(input)
    #endif
    $ make build -DNEW_ALGORITHM
    

    View Slide

33. ./make_call.sh
    --use-new-algorithm
    ./make_call.sh
    --use-original-algorithm
    

    View Slide

34. if (in_array($user->id, $whitelistedUsers)) {
    return $service->useNewAlgorithm($input);
    }
    return $service->useAlgorithm($input);
    

    View Slide

35. I love a good history lesson
    

    View Slide

36. Use Cases?
    

    View Slide

37. 1. Permissions
    

    View Slide

38. 1. Permissions
    2. Early-access
    

    View Slide

39. 1. Permissions
    2. Early-access
    3. Opt-in
    

    View Slide

40. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    

    View Slide

41. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    

    View Slide

42. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    

    View Slide

43. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    

    View Slide

44. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    

    View Slide

45. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    9. Maintenance
    

    View Slide

46. 1. Permissions
    2. Early-access
    3. Opt-in
    4. Incremental roll-out
    5. Block users
    6. Calendar driven
    7. Kill switch
    8. Power user
    9. Maintenance
    10. Sunset
    

    View Slide

47. View Slide

48. Flag Types
    

    View Slide

49. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    

    View Slide

50. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Days
    Dynamism
    Per deployment
    

    View Slide

51. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Days - Weeks
    Dynamism
    Human Controlled
    

    View Slide

52. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Years
    Dynamism
    Per Request
    

    View Slide

53. Release
    Toggles
    Ops
    Toggles
    Flag Classes
    Permission
    Toggles
    Experiment
    Toggles
    Lifetime
    Weeks - Months
    Dynamism
    Per Request
    

    View Slide

54. https://martinfowler.com/articles/feature-toggles.html
    

    View Slide

55. Ownership
    Release toggles owned by devs
    Ops toggles owned by ops
    Permission toggles
    Experiment toggles
    owned by product
    

    View Slide

56. View Slide

57. Flag Granularity
    

    View Slide

58. On/Off
    Switch
    Enable or
    disable
    entirely
    

    View Slide

59. On/Off
    Switch
    Percentage
    of users
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    

    View Slide

60. On/Off
    Switch
    Percentage
    of users
    Individuals
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    

    View Slide

61. On/Off
    Switch
    Percentage
    of users
    Individuals Groups
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    Enable for
    specific
    groups
    e.g.
    Power Users
    

    View Slide

62. On/Off
    Switch
    Percentage
    of users
    Individuals Groups Request
    information
    Enable or
    disable
    entirely
    Toggle
    for a
    percentage
    of users
    Toggle for
    specific
    users
    Enable for
    specific
    groups
    e.g.
    Power Users
    Enable for
    specific
    request
    headers or
    bodies
    

    View Slide

63. View Slide

64. Implementation
    

    View Slide

65. Build vs Buy
    

    View Slide

66. 1. Dashboard
    

    View Slide

67. 1. Dashboard
    2. Access control
    

    View Slide

68. 1. Dashboard
    2. Access control
    3. Complex rules
    

    View Slide

69. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    

    View Slide

70. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    

    View Slide

71. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    

    View Slide

72. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    

    View Slide

73. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    8. Scale
    

    View Slide

74. 1. Dashboard
    2. Access control
    3. Complex rules
    4. Cascading flags
    5. Multivariate rules
    6. Auditing
    7. Speed
    8. Scale
    9. 100% uptime
    

    View Slide

75. Build
    

    View Slide

76. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

77. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

78. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

79. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

80. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

81. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_MAJORITY);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

82. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_AFFIRMATIVE);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

83. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

84. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

85. Qandidate Toggle
    $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions, Toggle::STRATEGY_UNANIMOUS);
    $manager->add($toggle);
    $context = new Context();
    $context->set('time', (int) date('G'));
    echo 'Everyone is ' . ($manager->active('disable-work-email', $context))
    ? ‘happy!' : 'sad :(';
    

    View Slide

86. $manager = new ToggleManager(new InMemoryCollection());
    $afterSix = new GreaterThanEqual(18);
    $beforeTen = new LessThanEqual(22);
    $conditions = [];
    $conditions[] = new OperatorCondition('time', $afterSix);
    $conditions[] = new OperatorCondition('time', $beforeTen);
    $toggle = new Toggle('disable-work-email', $conditions);
    $manager->add($toggle);
    Configuration: Code
    

    View Slide

87. Configuration: Array
    $data = array(
    'some-feature' => array(
    'name' => 'toggling',
    'conditions' => array(
    array(
    'name' => 'operator-condition',
    'key' => 'user_id',
    'operator' => array('name' => 'greater-than', 'value' => 41),
    ),
    ),
    ),
    );
    $serializer = new InMemoryCollectionSerializer();
    $collection = $serializer->deserialize($data);
    $manager = new ToggleManager($collection);
    

    View Slide

88. Configuration: JSON/YAML
    $yaml = <<toggles:
    - name: some-feature
    conditions:
    - name: operator-condition
    key: user_id
    operator:
    name: greater-than
    value: 41
    YML;
    $array = Yaml::parse($yaml);
    $serializer = new InMemoryCollectionSerializer();
    $collection = $serializer->deserialize($array);
    $manager = new ToggleManager($collection);
    

    View Slide

89. Configuration: Redis
    $predis = new Predis\Client();
    $collection = new PredisCollection('toggle_demo', $predis);
    $manager = new ToggleManager($collection);
    

    View Slide

90. Qandidate Toggle API
    {
    "conditions" : [
    {
    "name" : "operator-condition",
    "operator" : {
    "name" : "less-than",
    "value" : "1337"
    },
    "key" : "user_id"
    }
    ],
    "name" : "foo",
    "status" : "conditionally-active",
    "originalName" : "foo"
    }
    PUT /toggles/{name}
    https://github.com/qandidate-labs/qandidate-toggle-api
    

    View Slide

91. Qandidate Toggle UI
    https://github.com/qandidate-labs/qandidate-toggle-ui
    

    View Slide

92. Build (Pros)
    1. Free!
    2. Total control
    3. Complex rules
    4. API and UI available
    

    View Slide

93. Build (Cons)
    1. Self hosted
    2. PHP only
    3. Complex rules
    

    View Slide

94. Buy
    

    View Slide

95. View Slide

96. Multi-Language
    

    View Slide

97. Prebuilt Dashboard
    

    View Slide

98. Environment Aware
    

    View Slide

99. Scaleable
    

    View Slide

100. Audit Log
     

     View Slide

101. A/B Testing
     

     View Slide

102. ld-relay
     

     View Slide

103. Buy (Pros)
     1. Lovely dashboard
     2. Multi-language
     3. Built in audit log
     4. Environment aware
     5. A/B testing conversion tracking
     

     View Slide

104. Buy (Cons)
     1. It costs (at least) $79/month
     2. Can’t modify the product
     3. Dependent on 3rd party
     

     View Slide

105. Build vs Buy
     

     View Slide

106. So. Much. Choice
     

     View Slide

107. It depends
     

     View Slide

108. Show me code
     

     View Slide

109. use Qandidate\Bundle\ToggleBundle\Annotations\Toggle;
     /**
     * @Toggle("profile-management")
     */
     class ProfileController
     {
     public function updatePassword()
     {
     }
     public function updateEmail()
     {
     }
     }
     

     View Slide

110. use Qandidate\Bundle\ToggleBundle\Annotations\Toggle;
     class ProfileController
     {
     /**
     * @Toggle("profile-edit-password")
     */
     public function updatePassword()
     {
     }
     public function updateEmail()
     {
     }
     }
     

     View Slide

111. public function onKernelRequest($event) {
     $flag = $event->getRequest->attributes->get('_flag');
     $alt = $event->getRequest->attributes->get('_alt');
     if ($toggleManager->active($flag, $context)) {
     $event->getRequest->attributes->set('controller', $alt);
     }
     }
     profile:
     path: /profile/{user}
     defaults:
     _controller: AcmeBundle:Profile:update
     _alt: AcmeBundle:NewProfile:update
     _flag: new-profile-controller
     

     View Slide

112. {% if feature_is_active('new-profile-image-picker') %}
     {# Show new picker #}
     {% else %}
     {# Use old flash-based picker #}
     {% endif %}
     

     View Slide

113. class AudioProcessorFactory {
     public function build($args, $context) {
     if ($this->toggleManager->active(‘enhanced-audio', $context)) {
     return new EnhancedAudioStreamProcessor($args);
     }
     return new BasicAudioStreamProcessor($args);
     }
     }
     

     View Slide

114. Controller View Factory
     Enable a
     route for
     specific users
     Show different
     UI elements
     Branch by
     abstraction for
     refactoring
     

     View Slide

115. View Slide

116. Toggle Configuratin
     

     View Slide

117. Hard
     Coded
     Change
     the code
     Dynamic
     Static
     

     View Slide

118. Hard
     Coded
     Deploy
     Config
     Change
     the code
     Deploy
     your
     application
     Dynamic
     Static
     

     View Slide

119. Hard
     Coded
     Deploy
     Config
     Parameterised
     Config
     Change
     the code
     Deploy
     your
     application
     Edit config,
     restart app
     Dynamic
     Static
     

     View Slide

120. Hard
     Coded
     Deploy
     Config
     Parameterised
     Config
     Database
     Config
     Change
     the code
     Deploy
     your
     application
     Edit config,
     restart app
     Edit via
     UI
     Dynamic
     Static
     

     View Slide

121. Hard
     Coded
     Deploy
     Config
     Parameterised
     Config
     Database
     Config
     Distributed
     Config
     Change
     the code
     Deploy
     your
     application
     Edit config,
     restart app
     Edit via
     UI
     e.g. Consul
     or
     Zookeeper
     Dynamic
     Static
     

     View Slide

122. Feature Dashboard
     

     View Slide

123. Flag State
     

     View Slide

124. Role Based Access
     

     View Slide

125. Audit Log
     

     View Slide

126. Non-technical Users
     

     View Slide

127. Technical Debt
     

     View Slide

128. DON’T
     

     View Slide

129. Cleanup based on activity
     

     View Slide

130. Cleanup based on activity
     

     View Slide

131. Create & Cleanup
     

     View Slide

132. WIP limits
     

     View Slide

133. Expiration date
     

     View Slide

134. Explosion date
     

     View Slide

135. Tips & Tricks
     

     View Slide

136. Testing Flags
     

     View Slide

137. Testing Flags
     Feature A Feature B
     ✅ ✅
     ✅ ☠
     ☠ ✅
     ☠ ☠
     

     View Slide

138. Testing Flags
     Feature A Feature B Feature C
     ✅ ✅ ✅
     ✅ ☠ ✅
     ✅ ✅ ☠
     ☠ ✅ ✅
     ✅ ☠ ☠
     ☠ ✅ ☠
     ☠ ☠ ✅
     ☠ ☠ ☠
     

     View Slide

139. Testing Flags
     Feature A Feature B Feature C Feature D
     ✅ ✅ ✅ ✅
     ✅ ✅ ✅ ☠
     ✅ ✅ ☠ ✅
     ✅ ☠ ✅ ✅
     ☠ ✅ ✅ ✅
     ✅ ✅ ☠ ☠
     ✅ ☠ ✅ ☠
     ✅ ☠ ☠ ✅
     ☠ ✅ ✅ ☠
     ☠ ✅ ☠ ✅
     ☠ ☠ ✅ ✅
     ✅ ☠ ☠ ☠
     ☠ ✅ ☠ ☠
     ☠ ☠ ✅ ☠
     ☠ ☠ ☠ ✅
     ☠ ☠ ☠ ☠
     

     View Slide

140. Testing Flags
     Number of flags Possible permutations
     2 4
     3 8
     4 16
     5 32
     10 1024
     25 33,554,432
     50 1.12589991E+15
     100 1.2676506E+30
     

     View Slide

141. Homepage
     Carousel
     Homepage CTA
     ✅ ✅
     ✅ ☠
     ☠ ✅
     ☠ ☠
     Forum Edit
     Permission
     Forum Delete
     Permission
     ✅ ✅
     ✅ ☠
     ☠ ✅
     ☠ ☠
     Upload Profile
     Image
     ✅
     ☠
     New Interesting
     Algorithm
     ✅
     ☠
     High Post Length
     ✅
     ☠
     Show Debug
     Info?
     ✅
     ☠
     Enable
     Audit Log
     ✅
     ☠
     Auto Logout
     ✅
     ☠
     API Access API v2
     ✅ ✅
     ✅ ☠
     ☠ ✅
     ☠ ☠
     

     View Slide

142. Expose State
     {
     "api_v2": true,
     "audit_log": true,
     "new_recommendation_algo": false
     }
     GET /toggle_status
     

     View Slide

143. Flag Descriptions
     basic-rec-algo: true
     
     
     
     

     View Slide

144. Flag Descriptions
     basic-rec-algo:
     
     enabled: true
     
     description: Use a simplistic recommendation algorithm. This is fast and
     produces less load on backend systems, but is way less accurate than
     our standard algorithm.
     

     View Slide

145. Clear Naming
     allow_user_delete and allow_user_edit
     not
     user_control
     

     View Slide

146. Audit Log
     Feature State User Date
     allow_user_edit
     ✅ Sarah 2018-05-11 12:22:08
     enhanced_rec_algo
     ✅ Ashley 2018-05-04 15:33:16
     allow_user_signup
     ☠ Ashley 2018-05-04 15:27:33
     enhanced_rec_algo
     ☠ system:load_manager 2018-05-04 15:16:30
     allow_user_signup
     ✅ Bob 2018-05-04 15:12:01
     enable_api
     ✅ Sarah 2018-04-30 09:56:59
     

     View Slide

147. Fail Safe
     Do positively enable features e.g. enable-ibm-
     analysis, enhanced-recomendation-algo
     
     Don’t use flags for suppressing new features e.g.
     
     disable-shiny-new-feature
     

     View Slide

148. Conclusion
     

     View Slide

149. Woooooooo
     

     View Slide

150. Flags are great!
     

     View Slide

151. Ops
     Toggles
     Release
     Toggles
     Flag Classes
     Permission
     Toggles
     Experiment
     Toggles
     

     View Slide

152. Build vs Buy
     

     View Slide

153. Qandidate Toggle
     

     View Slide

154. View Slide

155. One More Thing
     

     View Slide

156. No Reuse
     

     View Slide

157. Knight Capital
     

     View Slide

158. High frequency trader
     

     View Slide

159. View Slide

160. View Slide

161. Mistakes were made
     

     View Slide

162. Knight Capital
     1. Reused flags
     

     View Slide

163. Knight Capital
     1. Reused flags
     2. Didn’t delete obsolete code
     

     View Slide

164. Knight Capital
     1. Reused flags
     2. Didn’t delete obsolete code
     3. Couldn’t be disabled without a deployment
     

     View Slide

165. Knight Capital
     1. Reused flags
     2. Didn’t delete obsolete code
     3. Couldn’t be disabled without a deployment
     4. The deploy actually enabled the old code on
     all 8 servers, instead of disabling it
     

     View Slide

166. $460m Gone
     

     View Slide

167. @mheap / @NexmoDev
     m@michaelheap.com
     https://joind.in/talk/253a0
     

     View Slide