Crafting an Effective Security Organisation (KiwiCon 8)

Talk given at KiwiCon 8 about the approaches taken by the security team at Etsy to build an effective security organisation and spread a progressive security culture.
The presentations shows approaches that we have found work for us but should not be seen as a one size fits all solution. Every organisation is different and has it's own cultural needs, it is hoped people will be able to adapt our learnings to best meet their organisation and in doing so share back their learnings with the community.

Transcript

1. Crafting An Effective Security Organisation Kiwicon 8 Rich Smith (@iodboi)
2. @iodboi $whoami • Rich Smith - Brooklyn, NYC • Director of Security at Etsy • Co-Founder of Syndis in Reykjavík, Iceland • Background in offense: pen-testing, attack frameworks, post-exploitation, Goal Oriented Attack methodologies…
3. @iodboi Now (FY 2013) • Gross Marketplace Sales (GMS) $1.35 Billion • 40 million members, 1 million active sellers • 26 million active listings • 200+ Countries Performing Transactions • >615 Employees • Offices in 8 countries
4. @iodboi Focus Of Today • Lessons learnt building the Etsy security organization
5. @iodboi Focus Of Today • Lessons learnt building the Etsy security organization • How we foster & grow our security culture
6. @iodboi Focus Of Today • Lessons learnt building the Etsy security organization • How we foster & grow our security culture • The motivations driving how we think about security
7. @iodboi Focus Of Today • Lessons learnt building the Etsy security organization • How we foster & grow our security culture • The motivations driving how we think about security
8. @iodboi Focus Of Today • Lessons learnt building the Etsy security organization • How we foster & grow our security culture • The motivations driving how we think about security All work in progress !
9. Disclaimer A + B != Culture
10. Security from 50,000 ft
11. Technology
12. Society
13. Society Technology Security
14. Security
15. @iodboi From this perspective it’s easy to see that people need to be considered alongside technology for effective security
16. Etsy Engineering Culture
17. @iodboi (Some) Core Engineering Principles • Empower the edges • Trust but verify • ‘Just Ship’ - Get things done • ‘If it moves graph it’ - Let the data lead you • Every engineer can push to prod at any time
18. What does Continuous Deployment look like?
19. @iodboi Very end of 2009 Today Pushes Per Day
20. Continuous Deployment Continuous Delivery Frequent checkins directly to mainline ✓ ✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ?
21. @iodboi Why Do This ? • Continuous Deployment/Delivery/Integration • Build your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Developers iterate in production, A/B experimentation • Improve Mean Time To Recovery
22. @iodboi But doesn’t this CD stuff mean the stuff breaks all the time?
23. Single release Many releases 50K LOC/month Few opportunities for failure
 Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve
 written per release Imagine that we’ll write
27. @iodboi But how do you ‘security’ the anarchy?
28. @iodboi In such an environment classical security approaches don’t work well
29. @iodboi Security trying to reintroduce blocking to the org will mean it is ignored, not embraced
30. @iodboi Continuous Deployment & Security • The lessons & tools from DevOps are directly applicable
31. @iodboi Continuous Deployment & Security • The lessons & tools from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events
32. @iodboi Continuous Deployment & Security • The lessons & tools from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone
33. @iodboi Continuous Deployment & Security • The lessons & tools from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone • No such things as ‘out of cycle’ patches
34. @iodboi Continuous Deployment & Security • The lessons & tools from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone • No such things as ‘out of cycle’ patches • Security engineers push fixes directly to production
35. Principles of Effective Security
36. @iodboi 3 Principles of Effective Security 1. Enabling
37. @iodboi 3 Principles of Effective Security 1. Enabling 2. Transparent
38. @iodboi 3 Principles of Effective Security 1. Enabling 2. Transparent 3. Blameless
39. @iodboi A security team’s success should be measured by what they enable not by what they block Enabling
40. @iodboi A security team that is open as to what it does, and why, spreads understanding and is embraced Transparent
41. @iodboi Security failures will happen, only without blame will you be able to understand the true causes Blameless
42. ‘DevOpsSec’
43. @iodboi or …. ‘Lessons Security can learn from DevOps’
44. @iodboi DevOps • ‘DevOps’ has become somewhat overloaded
45. @iodboi DevOps • ‘DevOps’ has become somewhat overloaded • Aim: Remove silos & organizational blockers between Ops and Developers
46. @iodboi DevOps • ‘DevOps’ has become somewhat overloaded • Aim: Remove silos & organizational blockers between Ops and Developers • Central to this focus on good Communication & Collaboration
47. @iodboi ‘DevOpsSec’ Dev Ops • Natural extension of DevOps
48. @iodboi ‘DevOpsSec’ Dev Ops Sec • Natural extension of DevOps • Security faces many of the same challenges as Ops did • Removing barriers between Security, Developers and Operations
49. @iodboi Security as a blocker • Lazy and plain ‘bad’ security teams default to blocking
50. @iodboi Security as a blocker • Lazy and plain ‘bad’ security teams default to blocking • Blocking makes Security a NOP in the CD world
51. @iodboi Security as a blocker • Lazy and plain ‘bad’ security teams default to blocking • Blocking makes Security a NOP in the CD world • You will be ignored and teams will work around you
52. @iodboi Security as a blocker • Lazy and plain ‘bad’ security teams default to blocking • Blocking makes Security a NOP in the CD world • You will be ignored and teams will work around you • Save your ‘No’s’ as the very last resort
53. @iodboi Security as a enabler • Assist teams to do their new awesome ideas securely
54. @iodboi Security as a enabler • Assist teams to do their new awesome ideas securely • Incentivizes proactive engagement with Security
55. @iodboi Security as a enabler • Assist teams to do their new awesome ideas securely • Incentivizes proactive engagement with Security • Chase solutions to difficult challenges
56. @iodboi Designated Hackers • Security engineers assist multiple teams
57. @iodboi Designated Hackers • Security engineers assist multiple teams • ‘Designated’ not ‘Dedicated’
58. @iodboi Designated Hackers • Security engineers assist multiple teams • ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships
59. @iodboi Designated Hackers • Security engineers assist multiple teams • ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships • Represent teams back to security
60. @iodboi Designated Hackers • Security engineers assist multiple teams • ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships • Represent teams back to security • Early visibility, input & deeper insight
61. ‘You’re only a blocker if you’re the last to know’ John Allspaw, ! Some meeting room ,somewhere at Etsy
62. Progressive Security Culture
63. @iodboi Progressive Security Culture • Understanding that security is as much of a people problem as a technology problem
64. @iodboi Progressive Security Culture • Understanding that security is as much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture
65. @iodboi Progressive Security Culture • Understanding that security is as much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture • Often approaches focussed on are entirely technical
66. @iodboi Progressive Security Culture • Understanding that security is as much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture • Often approaches focussed on are entirely technical • Great culture depends on great people
67. @iodboi Security Team Hiring Number 1 rule ……
68. @iodboi Security Team Hiring Number 1 rule …… Don’t Hire Assholes
69. @iodboi Security Team Hiring If you inadvertently do, or you inherit one……
70. @iodboi Security Team Hiring If you inadvertently do, or you inherit one…… Remove them ASAP
71. @iodboi Great culture needs great people • Abrasive members will be the single biggest factor undermining all your progressive security efforts
72. @iodboi Great culture needs great people • Abrasive members will be the single biggest factor undermining all your progressive security efforts • Value social skills as highly as technical skills when making your security hires
73. @iodboi Great culture needs great people • Abrasive members will be the single biggest factor undermining all your progressive security efforts • Value social skills as highly as technical skills when making your security hires • ‘Cultural fit’ critically important
74. @iodboi The more diverse a security team is, the more approachable it is to more people
75. @iodboi Security Outreach • Distinct from security education
76. @iodboi Security Outreach • Distinct from security education • Focus on building relationships
77. @iodboi Security Outreach • Distinct from security education • Focus on building relationships • Removes barriers / reduces intimidation
78. @iodboi Security Outreach • Distinct from security education • Focus on building relationships • Removes barriers / reduces intimidation • Can be as simple as picking up some bar tabs
79. @iodboi Security Outreach • Distinct from security education • Focus on building relationships • Removes barriers / reduces intimidation • Can be as simple as picking up some bar tabs • Assign budget to this, it will be the best ROI you see
80. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security
81. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security • Embracing transparency
82. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security • Embracing transparency • Deep insight to daily security issues and concerns
83. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security • Embracing transparency • Deep insight to daily security issues and concerns • Build strong personal relationships
84. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security • Embracing transparency • Deep insight to daily security issues and concerns • Build strong personal relationships • Seed champions back out to the organization
85. @iodboi Security Candy! • Biggest source of security pod ‘drive bys’ • IRC bot command so people can see what’s in stock • Graph consumption!
86. Securgonomics!
87. er·go·nom·ics ˌərɡəˈnämiks/ noun the study of people's efficiency in their working environment.
88. secur·go·nom·ics /səˈkyo͝or/ ɡəˈnämiks/ noun the study of people's efficiency of security interactions in their working environment.
89. @iodboi Securgonomics • Lowering the barrier to interact with security
90. @iodboi Securgonomics • Lowering the barrier to interact with security • Too often security teams lock themselves away
91. @iodboi Securgonomics • Lowering the barrier to interact with security • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable
92. @iodboi Securgonomics • Lowering the barrier to interact with security • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable • Sit on the busiest office pathway you can
93. @iodboi Securgonomics • Lowering the barrier to interact with security • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable • Sit on the busiest office pathway you can • Have your security dashboards front & centre
94. @iodboi ‘We must strive to understand that accidents don’t happen because people gamble and loose. Accidents happen because the person believes that what is about to happen: - Is not possible - Has no connection to what they are doing - The intended outcome is worth the risk’ ! Erik Hollnagel Blameless Postmortems
95. @iodboi Blameless Postmortems • Comes from our desire to have Just Culture
96. @iodboi Blameless Postmortems • Comes from our desire to have Just Culture • Aim to learn from failings not to target blame
97. @iodboi Blameless Postmortems • Comes from our desire to have Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution
98. @iodboi Blameless Postmortems • Comes from our desire to have Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution • Empower engineers to own their own stories
99. @iodboi Blameless Postmortems • Comes from our desire to have Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution • Empower engineers to own their own stories • Applies to security failures as much as Ops failures
100. Indicators of an effective security team
101. @iodboi Is Data Driven • Too often security is explained with religious conviction
102. @iodboi Is Data Driven • Too often security is explained with religious conviction • Security is not black and white, many shades of grey
103. @iodboi Is Data Driven • Too often security is explained with religious conviction • Security is not black and white, many shades of grey • Security is not a point but a vector
104. @iodboi Is Data Driven • Too often security is explained with religious conviction • Security is not black and white, many shades of grey • Security is not a point but a vector • Gather data to support security decisions and let it lead you to the correct shade of grey
105. @iodboi Runs a Bug Bounty • Continuous Assessment of your security program • D’ya you think you’re not under attack 24/7 anyway …….
106. @iodboi Runs a Bug Bounty • Continuous Assessment of your security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries
107. @iodboi Runs a Bug Bounty • Continuous Assessment of your security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries • Increases value from focused pentests/red teaming
108. @iodboi Runs a Bug Bounty • Continuous Assessment of your security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries • Increases value from focused pentests/red teaming • Generates good metric sets about security (data driven)
109. @iodboi Doesn’t Cry Wolf • Verify issues before raising them to developers
110. @iodboi Doesn’t Cry Wolf • Verify issues before raising them to developers • They will only chase their tail a few times before ignoring
111. @iodboi Doesn’t Cry Wolf • Verify issues before raising them to developers • They will only chase their tail a few times before ignoring • Security engineers should be in amongst the codebase • Aim to own the entire fix process themselves
112. @iodboi Makes Realistic Tradeoffs • Not everything is critical
113. @iodboi Makes Realistic Tradeoffs • Not everything is critical • Let low risk things ship along with commitments to a reasonable remediation window buys you lots
114. @iodboi Makes Realistic Tradeoffs • Not everything is critical • Let low risk things ship along with commitments to a reasonable remediation window buys you lots • Save your NOs for when you need them - they are a finite resource
115. @iodboi Provides Context & Impact • Explaining why something is an issue and what it may result in to the team affected
116. @iodboi Provides Context & Impact • Explaining why something is an issue and what it may result in to the team affected • Provides security education and garners understanding
117. @iodboi Provides Context & Impact • Explaining why something is an issue and what it may result in to the team affected • Provides security education and garners understanding • ‘This would allow an attacker to impersonate another user & read their mail’ is useful and starts dialogue ….
118. @iodboi Provides Context & Impact • Explaining why something is an issue and what it may result in to the team affected • Provides security education and garners understanding • ‘This would allow an attacker to impersonate another user & read their mail’ is useful and starts dialogue …. • ‘Input validation was insufficiently applied’ does not
119. @iodboi Recognises & Rewards • Rewarding folks in the org who reach out to Security • We do this is a number of ways: • Pins and patches • T-Shirts • Etsy gift vouchers • IRC Pluses & Value Awards
120. @iodboi Etsy Value Awards
121. @iodboi Treats Security as a BRAND • Your security culture has real value
122. @iodboi Treats Security as a BRAND • Your security culture has real value • Work long & hard to build it up
123. @iodboi Treats Security as a BRAND • Your security culture has real value • Work long & hard to build it up • Can be damaged in the blink of an eye
124. @iodboi Treats Security as a BRAND • Your security culture has real value • Work long & hard to build it up • Can be damaged in the blink of an eye • Aim to build strong, positive, long term associations with the security team
125. @iodboi Treats Security as a BRAND • Your security culture has real value • Work long & hard to build it up • Can be damaged in the blink of an eye • Aim to build strong, positive, long term associations with the security team • Get your consumers to buy in to this security shit
126. Wrap up
127. @iodboi Final thoughts • Building an effective security organisation takes effort
128. @iodboi Final thoughts • Building an effective security organisation takes effort • Requires a focus on people as much as technology
129. @iodboi Final thoughts • Building an effective security organisation takes effort • Requires a focus on people as much as technology • Learn from DevOps & move to a DevOpsSec mindset
130. @iodboi Final thoughts • Building an effective security organisation takes effort • Requires a focus on people as much as technology • Learn from DevOps & move to a DevOpsSec mindset • Enable don’t block, else you’ll make security a NOP
131. @iodboi Enabling. Transparent. Blameless
132. @iodboi Thanks! Questions ?