Crafting an Effective Security Organisation (KiwiCon 8)

Talk given at KiwiCon 8 about the approaches taken by the security team at Etsy to build an effective security organisation and spread a progressive security culture.
The presentations shows approaches that we have found work for us but should not be seen as a one size fits all solution. Every organisation is different and has it's own cultural needs, it is hoped people will be able to adapt our learnings to best meet their organisation and in doing so share back their learnings with the community.

Transcript

1. Crafting An Effective
   Security Organisation
   Kiwicon 8
   Rich Smith (@iodboi)
   

   View Slide

2. @iodboi
   $whoami
   • Rich Smith - Brooklyn, NYC
   • Director of Security at Etsy
   • Co-Founder of Syndis in Reykjavík, Iceland
   • Background in offense: pen-testing, attack
   frameworks, post-exploitation, Goal Oriented Attack
   methodologies…
   

   View Slide

3. @iodboi
   Now (FY 2013)
   • Gross Marketplace Sales (GMS) $1.35 Billion
   • 40 million members, 1 million active sellers
   • 26 million active listings
   • 200+ Countries Performing Transactions
   • >615 Employees
   • Offices in 8 countries
   

   View Slide

4. @iodboi
   Focus Of Today
   • Lessons learnt building the Etsy security organization
   

   View Slide

5. @iodboi
   Focus Of Today
   • Lessons learnt building the Etsy security organization
   • How we foster & grow our security culture
   

   View Slide

6. @iodboi
   Focus Of Today
   • Lessons learnt building the Etsy security organization
   • How we foster & grow our security culture
   • The motivations driving how we think about security
   

   View Slide

7. @iodboi
   Focus Of Today
   • Lessons learnt building the Etsy security organization
   • How we foster & grow our security culture
   • The motivations driving how we think about security
   

   View Slide

8. @iodboi
   Focus Of Today
   • Lessons learnt building the Etsy security organization
   • How we foster & grow our security culture
   • The motivations driving how we think about security
   All work in progress !
   

   View Slide

9. Disclaimer
   A + B != Culture
   

   View Slide

10. Security from 50,000 ft
    

    View Slide

11. Technology
    

    View Slide

12. Society
    

    View Slide

13. Society
    Technology Security
    

    View Slide

14. Security
    

    View Slide

15. @iodboi
    From this perspective it’s easy
    to see that people need to be
    considered alongside
    technology for effective
    security
    

    View Slide

16. Etsy Engineering Culture
    

    View Slide

17. @iodboi
    (Some) Core Engineering Principles
    • Empower the edges
    • Trust but verify
    • ‘Just Ship’ - Get things done
    • ‘If it moves graph it’ - Let the data lead you
    • Every engineer can push to prod at any time
    

    View Slide

18. What does Continuous
    Deployment look like?
    

    View Slide

19. @iodboi
    Very end of 2009 Today
    Pushes Per Day
    

    View Slide

20. Continuous Deployment Continuous Delivery
    Frequent checkins
    directly to mainline
    ✓ ✓
    Automated build & test
    cycle
    ✓ ✓
    Keep the build green,
    always ready to release
    ✓ ✓
    One button deploys ✓ ✓
    Business dictates when
    to deploy
    ✓
    Every passing build
    deployed to prod
    ✓
    All enhancements gated
    by feature flag
    ✓ ?
    

    View Slide

21. @iodboi
    Why Do This ?
    • Continuous Deployment/Delivery/Integration
    • Build your apps in a reproducible way after each push to git
    • Identify bugs, missing dependencies early & often
    • Integrate security testing throughout lifecycle
    • Developers iterate in production, A/B experimentation
    • Improve Mean Time To Recovery
    

    View Slide

22. @iodboi
    But doesn’t this CD stuff
    mean the stuff breaks all
    the time?
    

    View Slide

23. Single
    release
    Many
    releases
    50K LOC/month
    Few opportunities for failure

    Wide surface area (50,000 LOC)
    High MTTR
    !
    All of the bugs we’ve written
    More opportunities for failure
    Narrow surface area (< 100 LOC)
    Low MTTR
    !
    A fraction of the bugs we’ve

    written per release
    Imagine that we’ll write
    

    View Slide

24. View Slide

25. View Slide

26. View Slide

27. @iodboi
    But how do you ‘security’
    the anarchy?
    

    View Slide

28. @iodboi
    In such an environment
    classical security approaches
    don’t work well
    

    View Slide

29. @iodboi
    Security trying to reintroduce
    blocking to the org will mean
    it is ignored, not embraced
    

    View Slide

30. @iodboi
    Continuous Deployment & Security
    • The lessons & tools from DevOps are directly applicable
    

    View Slide

31. @iodboi
    Continuous Deployment & Security
    • The lessons & tools from DevOps are directly applicable
    • Apply the same ‘if it moves graph it’ for security events
    

    View Slide

32. @iodboi
    Continuous Deployment & Security
    • The lessons & tools from DevOps are directly applicable
    • Apply the same ‘if it moves graph it’ for security events
    • Makes security related data available to everyone
    

    View Slide

33. @iodboi
    Continuous Deployment & Security
    • The lessons & tools from DevOps are directly applicable
    • Apply the same ‘if it moves graph it’ for security events
    • Makes security related data available to everyone
    • No such things as ‘out of cycle’ patches
    

    View Slide

34. @iodboi
    Continuous Deployment & Security
    • The lessons & tools from DevOps are directly applicable
    • Apply the same ‘if it moves graph it’ for security events
    • Makes security related data available to everyone
    • No such things as ‘out of cycle’ patches
    • Security engineers push fixes directly to production
    

    View Slide

35. Principles of Effective
    Security
    

    View Slide

36. @iodboi
    3 Principles of Effective Security
    1. Enabling
    

    View Slide

37. @iodboi
    3 Principles of Effective Security
    1. Enabling
    2. Transparent
    

    View Slide

38. @iodboi
    3 Principles of Effective Security
    1. Enabling
    2. Transparent
    3. Blameless
    

    View Slide

39. @iodboi
    A security team’s success
    should be measured by what
    they enable not by what
    they block
    Enabling
    

    View Slide

40. @iodboi
    A security team that is open
    as to what it does, and why,
    spreads understanding and
    is embraced
    Transparent
    

    View Slide

41. @iodboi
    Security failures will
    happen, only without blame
    will you be able to
    understand the true causes
    Blameless
    

    View Slide

42. ‘DevOpsSec’
    

    View Slide

43. @iodboi
    or ….
    ‘Lessons Security can
    learn from DevOps’
    

    View Slide

44. @iodboi
    DevOps
    • ‘DevOps’ has become somewhat overloaded
    

    View Slide

45. @iodboi
    DevOps
    • ‘DevOps’ has become somewhat overloaded
    • Aim: Remove silos & organizational blockers between
    Ops and Developers
    

    View Slide

46. @iodboi
    DevOps
    • ‘DevOps’ has become somewhat overloaded
    • Aim: Remove silos & organizational blockers between
    Ops and Developers
    • Central to this focus on good Communication &
    Collaboration
    

    View Slide

47. @iodboi
    ‘DevOpsSec’
    Dev Ops
    • Natural extension of DevOps
    

    View Slide

48. @iodboi
    ‘DevOpsSec’
    Dev Ops
    Sec
    • Natural extension of DevOps
    • Security faces many of the same
    challenges as Ops did
    • Removing barriers between
    Security, Developers and
    Operations
    

    View Slide

49. @iodboi
    Security as a blocker
    • Lazy and plain ‘bad’ security teams default to blocking
    

    View Slide

50. @iodboi
    Security as a blocker
    • Lazy and plain ‘bad’ security teams default to blocking
    • Blocking makes Security a NOP in the CD world
    

    View Slide

51. @iodboi
    Security as a blocker
    • Lazy and plain ‘bad’ security teams default to blocking
    • Blocking makes Security a NOP in the CD world
    • You will be ignored and teams will work around you
    

    View Slide

52. @iodboi
    Security as a blocker
    • Lazy and plain ‘bad’ security teams default to blocking
    • Blocking makes Security a NOP in the CD world
    • You will be ignored and teams will work around you
    • Save your ‘No’s’ as the very last resort
    

    View Slide

53. @iodboi
    Security as a enabler
    • Assist teams to do their new awesome ideas securely
    

    View Slide

54. @iodboi
    Security as a enabler
    • Assist teams to do their new awesome ideas securely
    • Incentivizes proactive engagement with Security
    

    View Slide

55. @iodboi
    Security as a enabler
    • Assist teams to do their new awesome ideas securely
    • Incentivizes proactive engagement with Security
    • Chase solutions to difficult challenges
    

    View Slide

56. @iodboi
    Designated Hackers
    • Security engineers assist multiple teams
    

    View Slide

57. @iodboi
    Designated Hackers
    • Security engineers assist multiple teams
    • ‘Designated’ not ‘Dedicated’
    

    View Slide

58. @iodboi
    Designated Hackers
    • Security engineers assist multiple teams
    • ‘Designated’ not ‘Dedicated’
    • Breaks down barriers, build trust & relationships
    

    View Slide

59. @iodboi
    Designated Hackers
    • Security engineers assist multiple teams
    • ‘Designated’ not ‘Dedicated’
    • Breaks down barriers, build trust & relationships
    • Represent teams back to security
    

    View Slide

60. @iodboi
    Designated Hackers
    • Security engineers assist multiple teams
    • ‘Designated’ not ‘Dedicated’
    • Breaks down barriers, build trust & relationships
    • Represent teams back to security
    • Early visibility, input & deeper insight
    

    View Slide

61. ‘You’re only a blocker if
    you’re the last to know’
    John Allspaw, !
    Some meeting room ,somewhere at Etsy
    

    View Slide

62. Progressive Security
    Culture
    

    View Slide

63. @iodboi
    Progressive Security Culture
    • Understanding that security is as much of a people
    problem as a technology problem
    

    View Slide

64. @iodboi
    Progressive Security Culture
    • Understanding that security is as much of a people
    problem as a technology problem
    • As an industry, security has done a poor job of
    discussing the need for positive security culture
    

    View Slide

65. @iodboi
    Progressive Security Culture
    • Understanding that security is as much of a people
    problem as a technology problem
    • As an industry, security has done a poor job of
    discussing the need for positive security culture
    • Often approaches focussed on are entirely technical
    

    View Slide

66. @iodboi
    Progressive Security Culture
    • Understanding that security is as much of a people
    problem as a technology problem
    • As an industry, security has done a poor job of
    discussing the need for positive security culture
    • Often approaches focussed on are entirely technical
    • Great culture depends on great people
    

    View Slide

67. @iodboi
    Security Team Hiring
    Number 1 rule ……
    

    View Slide

68. @iodboi
    Security Team Hiring
    Number 1 rule ……
    Don’t Hire Assholes
    

    View Slide

69. @iodboi
    Security Team Hiring
    If you inadvertently do, or you inherit one……
    

    View Slide

70. @iodboi
    Security Team Hiring
    If you inadvertently do, or you inherit one……
    Remove them ASAP
    

    View Slide

71. @iodboi
    Great culture needs great people
    • Abrasive members will be the single biggest factor
    undermining all your progressive security efforts
    

    View Slide

72. @iodboi
    Great culture needs great people
    • Abrasive members will be the single biggest factor
    undermining all your progressive security efforts
    • Value social skills as highly as technical skills when
    making your security hires
    

    View Slide

73. @iodboi
    Great culture needs great people
    • Abrasive members will be the single biggest factor
    undermining all your progressive security efforts
    • Value social skills as highly as technical skills when
    making your security hires
    • ‘Cultural fit’ critically important
    

    View Slide

74. @iodboi
    The more diverse a security team is, the
    more approachable it is to more people
    

    View Slide

75. @iodboi
    Security Outreach
    • Distinct from security education
    

    View Slide

76. @iodboi
    Security Outreach
    • Distinct from security education
    • Focus on building relationships
    

    View Slide

77. @iodboi
    Security Outreach
    • Distinct from security education
    • Focus on building relationships
    • Removes barriers / reduces intimidation
    

    View Slide

78. @iodboi
    Security Outreach
    • Distinct from security education
    • Focus on building relationships
    • Removes barriers / reduces intimidation
    • Can be as simple as picking up some bar tabs
    

    View Slide

79. @iodboi
    Security Outreach
    • Distinct from security education
    • Focus on building relationships
    • Removes barriers / reduces intimidation
    • Can be as simple as picking up some bar tabs
    • Assign budget to this, it will be the best ROI you see
    

    View Slide

80. @iodboi
    Bootcamps
    • Have people come and ‘bootcamp’ with security
    

    View Slide

81. @iodboi
    Bootcamps
    • Have people come and ‘bootcamp’ with security
    • Embracing transparency
    

    View Slide

82. @iodboi
    Bootcamps
    • Have people come and ‘bootcamp’ with security
    • Embracing transparency
    • Deep insight to daily security issues and concerns
    

    View Slide

83. @iodboi
    Bootcamps
    • Have people come and ‘bootcamp’ with security
    • Embracing transparency
    • Deep insight to daily security issues and concerns
    • Build strong personal relationships
    

    View Slide

84. @iodboi
    Bootcamps
    • Have people come and ‘bootcamp’ with security
    • Embracing transparency
    • Deep insight to daily security issues and concerns
    • Build strong personal relationships
    • Seed champions back out to the organization
    

    View Slide

85. @iodboi
    Security Candy!
    • Biggest source of security pod
    ‘drive bys’
    • IRC bot command so people
    can see what’s in stock
    • Graph consumption!
    

    View Slide

86. Securgonomics!
    

    View Slide

87. er·go·nom·ics
    ˌərɡəˈnämiks/
    noun
    the study of people's efficiency in their working
    environment.
    

    View Slide

88. secur·go·nom·ics
    /səˈkyo͝or/ ɡəˈnämiks/
    noun
    the study of people's efficiency of security interactions
    in their working environment.
    

    View Slide

89. @iodboi
    Securgonomics
    • Lowering the barrier to interact with security
    

    View Slide

90. @iodboi
    Securgonomics
    • Lowering the barrier to interact with security
    • Too often security teams lock themselves away
    

    View Slide

91. @iodboi
    Securgonomics
    • Lowering the barrier to interact with security
    • Too often security teams lock themselves away
    • Being accessible & visible to everyone is invaluable
    

    View Slide

92. @iodboi
    Securgonomics
    • Lowering the barrier to interact with security
    • Too often security teams lock themselves away
    • Being accessible & visible to everyone is invaluable
    • Sit on the busiest office pathway you can
    

    View Slide

93. @iodboi
    Securgonomics
    • Lowering the barrier to interact with security
    • Too often security teams lock themselves away
    • Being accessible & visible to everyone is invaluable
    • Sit on the busiest office pathway you can
    • Have your security dashboards front & centre
    

    View Slide

94. @iodboi
    ‘We must strive to understand that accidents don’t
    happen because people gamble and loose. Accidents
    happen because the person believes that what is
    about to happen:
    - Is not possible
    - Has no connection to what they are doing
    - The intended outcome is worth the risk’
    !
    Erik Hollnagel
    Blameless Postmortems
    

    View Slide

95. @iodboi
    Blameless Postmortems
    • Comes from our desire to have Just Culture
    

    View Slide

96. @iodboi
    Blameless Postmortems
    • Comes from our desire to have Just Culture
    • Aim to learn from failings not to target blame
    

    View Slide

97. @iodboi
    Blameless Postmortems
    • Comes from our desire to have Just Culture
    • Aim to learn from failings not to target blame
    • Share detailed accounts of actions, decisions and
    circumstances without fear of punishment or retribution
    

    View Slide

98. @iodboi
    Blameless Postmortems
    • Comes from our desire to have Just Culture
    • Aim to learn from failings not to target blame
    • Share detailed accounts of actions, decisions and
    circumstances without fear of punishment or retribution
    • Empower engineers to own their own stories
    

    View Slide

99. @iodboi
    Blameless Postmortems
    • Comes from our desire to have Just Culture
    • Aim to learn from failings not to target blame
    • Share detailed accounts of actions, decisions and
    circumstances without fear of punishment or retribution
    • Empower engineers to own their own stories
    • Applies to security failures as much as Ops failures
    

    View Slide

100. Indicators of an effective
     security team
     

     View Slide

101. @iodboi
     Is Data Driven
     • Too often security is explained with religious conviction
     

     View Slide

102. @iodboi
     Is Data Driven
     • Too often security is explained with religious conviction
     • Security is not black and white, many shades of grey
     

     View Slide

103. @iodboi
     Is Data Driven
     • Too often security is explained with religious conviction
     • Security is not black and white, many shades of grey
     • Security is not a point but a vector
     

     View Slide

104. @iodboi
     Is Data Driven
     • Too often security is explained with religious conviction
     • Security is not black and white, many shades of grey
     • Security is not a point but a vector
     • Gather data to support security decisions and let it lead
     you to the correct shade of grey
     

     View Slide

105. @iodboi
     Runs a Bug Bounty
     • Continuous Assessment of your security program
     • D’ya you think you’re not under attack 24/7 anyway …….
     

     View Slide

106. @iodboi
     Runs a Bug Bounty
     • Continuous Assessment of your security program
     • D’ya you think you’re not under attack 24/7 anyway …….
     • Raises cost of attack for real adversaries
     

     View Slide

107. @iodboi
     Runs a Bug Bounty
     • Continuous Assessment of your security program
     • D’ya you think you’re not under attack 24/7 anyway …….
     • Raises cost of attack for real adversaries
     • Increases value from focused pentests/red teaming
     

     View Slide

108. @iodboi
     Runs a Bug Bounty
     • Continuous Assessment of your security program
     • D’ya you think you’re not under attack 24/7 anyway …….
     • Raises cost of attack for real adversaries
     • Increases value from focused pentests/red teaming
     • Generates good metric sets about security (data driven)
     

     View Slide

109. @iodboi
     Doesn’t Cry Wolf
     • Verify issues before raising them to developers
     

     View Slide

110. @iodboi
     Doesn’t Cry Wolf
     • Verify issues before raising them to developers
     • They will only chase their tail a few times before ignoring
     

     View Slide

111. @iodboi
     Doesn’t Cry Wolf
     • Verify issues before raising them to developers
     • They will only chase their tail a few times before ignoring
     • Security engineers should be in amongst the codebase
     • Aim to own the entire fix process themselves
     

     View Slide

112. @iodboi
     Makes Realistic Tradeoffs
     • Not everything is critical
     

     View Slide

113. @iodboi
     Makes Realistic Tradeoffs
     • Not everything is critical
     • Let low risk things ship along with commitments to a
     reasonable remediation window buys you lots
     

     View Slide

114. @iodboi
     Makes Realistic Tradeoffs
     • Not everything is critical
     • Let low risk things ship along with commitments to a
     reasonable remediation window buys you lots
     • Save your NOs for when you need them - they are a
     finite resource
     

     View Slide

115. @iodboi
     Provides Context & Impact
     • Explaining why something is an issue and what it may
     result in to the team affected
     

     View Slide

116. @iodboi
     Provides Context & Impact
     • Explaining why something is an issue and what it may
     result in to the team affected
     • Provides security education and garners understanding
     

     View Slide

117. @iodboi
     Provides Context & Impact
     • Explaining why something is an issue and what it may
     result in to the team affected
     • Provides security education and garners understanding
     • ‘This would allow an attacker to impersonate another
     user & read their mail’ is useful and starts dialogue ….
     

     View Slide

118. @iodboi
     Provides Context & Impact
     • Explaining why something is an issue and what it may
     result in to the team affected
     • Provides security education and garners understanding
     • ‘This would allow an attacker to impersonate another
     user & read their mail’ is useful and starts dialogue ….
     • ‘Input validation was insufficiently applied’ does not
     

     View Slide

119. @iodboi
     Recognises & Rewards
     • Rewarding folks in the org who reach out to Security
     • We do this is a number of ways:
     • Pins and patches
     • T-Shirts
     • Etsy gift vouchers
     • IRC Pluses & Value Awards
     

     View Slide

120. @iodboi
     Etsy Value Awards
     

     View Slide

121. @iodboi
     Treats Security as a BRAND
     • Your security culture has real value
     

     View Slide

122. @iodboi
     Treats Security as a BRAND
     • Your security culture has real value
     • Work long & hard to build it up
     

     View Slide

123. @iodboi
     Treats Security as a BRAND
     • Your security culture has real value
     • Work long & hard to build it up
     • Can be damaged in the blink of an eye
     

     View Slide

124. @iodboi
     Treats Security as a BRAND
     • Your security culture has real value
     • Work long & hard to build it up
     • Can be damaged in the blink of an eye
     • Aim to build strong, positive, long term associations
     with the security team
     

     View Slide

125. @iodboi
     Treats Security as a BRAND
     • Your security culture has real value
     • Work long & hard to build it up
     • Can be damaged in the blink of an eye
     • Aim to build strong, positive, long term associations
     with the security team
     • Get your consumers to buy in to this security shit
     

     View Slide

126. Wrap up
     

     View Slide

127. @iodboi
     Final thoughts
     • Building an effective security organisation takes effort
     

     View Slide

128. @iodboi
     Final thoughts
     • Building an effective security organisation takes effort
     • Requires a focus on people as much as technology
     

     View Slide

129. @iodboi
     Final thoughts
     • Building an effective security organisation takes effort
     • Requires a focus on people as much as technology
     • Learn from DevOps & move to a DevOpsSec mindset
     

     View Slide

130. @iodboi
     Final thoughts
     • Building an effective security organisation takes effort
     • Requires a focus on people as much as technology
     • Learn from DevOps & move to a DevOpsSec mindset
     • Enable don’t block, else you’ll make security a NOP
     

     View Slide

131. @iodboi
     Enabling. Transparent. Blameless
     

     View Slide

132. @iodboi
     Thanks! Questions ?
     

     View Slide