Crafting an Effective Security Organisation (KiwiCon 8)

Aa6b7d3a12a61bfe0ce9c8a12405bd75?s=47 Rich Smith
December 11, 2014

Crafting an Effective Security Organisation (KiwiCon 8)

Talk given at KiwiCon 8 about the approaches taken by the security team at Etsy to build an effective security organisation and spread a progressive security culture.
The presentations shows approaches that we have found work for us but should not be seen as a one size fits all solution. Every organisation is different and has it's own cultural needs, it is hoped people will be able to adapt our learnings to best meet their organisation and in doing so share back their learnings with the community.

Aa6b7d3a12a61bfe0ce9c8a12405bd75?s=128

Rich Smith

December 11, 2014
Tweet

Transcript

  1. Crafting An Effective Security Organisation Kiwicon 8 Rich Smith (@iodboi)

  2. @iodboi $whoami • Rich Smith - Brooklyn, NYC • Director

    of Security at Etsy • Co-Founder of Syndis in Reykjavík, Iceland • Background in offense: pen-testing, attack frameworks, post-exploitation, Goal Oriented Attack methodologies…
  3. @iodboi Now (FY 2013) • Gross Marketplace Sales (GMS) $1.35

    Billion • 40 million members, 1 million active sellers • 26 million active listings • 200+ Countries Performing Transactions • >615 Employees • Offices in 8 countries
  4. @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization
  5. @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture
  6. @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture • The motivations driving how we think about security
  7. @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture • The motivations driving how we think about security
  8. @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture • The motivations driving how we think about security All work in progress !
  9. Disclaimer A + B != Culture

  10. Security from 50,000 ft

  11. Technology

  12. Society

  13. Society Technology Security

  14. Security

  15. @iodboi From this perspective it’s easy to see that people

    need to be considered alongside technology for effective security
  16. Etsy Engineering Culture

  17. @iodboi (Some) Core Engineering Principles • Empower the edges •

    Trust but verify • ‘Just Ship’ - Get things done • ‘If it moves graph it’ - Let the data lead you • Every engineer can push to prod at any time
  18. What does Continuous Deployment look like?

  19. @iodboi Very end of 2009 Today Pushes Per Day

  20. Continuous Deployment Continuous Delivery Frequent checkins directly to mainline ✓

    ✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ?
  21. @iodboi Why Do This ? • Continuous Deployment/Delivery/Integration • Build

    your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Developers iterate in production, A/B experimentation • Improve Mean Time To Recovery
  22. @iodboi But doesn’t this CD stuff mean the stuff breaks

    all the time?
  23. Single release Many releases 50K LOC/month Few opportunities for failure


    Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve
 written per release Imagine that we’ll write
  24. None
  25. None
  26. None
  27. @iodboi But how do you ‘security’ the anarchy?

  28. @iodboi In such an environment classical security approaches don’t work

    well
  29. @iodboi Security trying to reintroduce blocking to the org will

    mean it is ignored, not embraced
  30. @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable
  31. @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events
  32. @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone
  33. @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone • No such things as ‘out of cycle’ patches
  34. @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone • No such things as ‘out of cycle’ patches • Security engineers push fixes directly to production
  35. Principles of Effective Security

  36. @iodboi 3 Principles of Effective Security 1. Enabling

  37. @iodboi 3 Principles of Effective Security 1. Enabling 2. Transparent

  38. @iodboi 3 Principles of Effective Security 1. Enabling 2. Transparent

    3. Blameless
  39. @iodboi A security team’s success should be measured by what

    they enable not by what they block Enabling
  40. @iodboi A security team that is open as to what

    it does, and why, spreads understanding and is embraced Transparent
  41. @iodboi Security failures will happen, only without blame will you

    be able to understand the true causes Blameless
  42. ‘DevOpsSec’

  43. @iodboi or …. ‘Lessons Security can learn from DevOps’

  44. @iodboi DevOps • ‘DevOps’ has become somewhat overloaded

  45. @iodboi DevOps • ‘DevOps’ has become somewhat overloaded • Aim:

    Remove silos & organizational blockers between Ops and Developers
  46. @iodboi DevOps • ‘DevOps’ has become somewhat overloaded • Aim:

    Remove silos & organizational blockers between Ops and Developers • Central to this focus on good Communication & Collaboration
  47. @iodboi ‘DevOpsSec’ Dev Ops • Natural extension of DevOps

  48. @iodboi ‘DevOpsSec’ Dev Ops Sec • Natural extension of DevOps

    • Security faces many of the same challenges as Ops did • Removing barriers between Security, Developers and Operations
  49. @iodboi Security as a blocker • Lazy and plain ‘bad’

    security teams default to blocking
  50. @iodboi Security as a blocker • Lazy and plain ‘bad’

    security teams default to blocking • Blocking makes Security a NOP in the CD world
  51. @iodboi Security as a blocker • Lazy and plain ‘bad’

    security teams default to blocking • Blocking makes Security a NOP in the CD world • You will be ignored and teams will work around you
  52. @iodboi Security as a blocker • Lazy and plain ‘bad’

    security teams default to blocking • Blocking makes Security a NOP in the CD world • You will be ignored and teams will work around you • Save your ‘No’s’ as the very last resort
  53. @iodboi Security as a enabler • Assist teams to do

    their new awesome ideas securely
  54. @iodboi Security as a enabler • Assist teams to do

    their new awesome ideas securely • Incentivizes proactive engagement with Security
  55. @iodboi Security as a enabler • Assist teams to do

    their new awesome ideas securely • Incentivizes proactive engagement with Security • Chase solutions to difficult challenges
  56. @iodboi Designated Hackers • Security engineers assist multiple teams

  57. @iodboi Designated Hackers • Security engineers assist multiple teams •

    ‘Designated’ not ‘Dedicated’
  58. @iodboi Designated Hackers • Security engineers assist multiple teams •

    ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships
  59. @iodboi Designated Hackers • Security engineers assist multiple teams •

    ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships • Represent teams back to security
  60. @iodboi Designated Hackers • Security engineers assist multiple teams •

    ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships • Represent teams back to security • Early visibility, input & deeper insight
  61. ‘You’re only a blocker if you’re the last to know’

    John Allspaw, ! Some meeting room ,somewhere at Etsy
  62. Progressive Security Culture

  63. @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem
  64. @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture
  65. @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture • Often approaches focussed on are entirely technical
  66. @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture • Often approaches focussed on are entirely technical • Great culture depends on great people
  67. @iodboi Security Team Hiring Number 1 rule ……

  68. @iodboi Security Team Hiring Number 1 rule …… Don’t Hire

    Assholes
  69. @iodboi Security Team Hiring If you inadvertently do, or you

    inherit one……
  70. @iodboi Security Team Hiring If you inadvertently do, or you

    inherit one…… Remove them ASAP
  71. @iodboi Great culture needs great people • Abrasive members will

    be the single biggest factor undermining all your progressive security efforts
  72. @iodboi Great culture needs great people • Abrasive members will

    be the single biggest factor undermining all your progressive security efforts • Value social skills as highly as technical skills when making your security hires
  73. @iodboi Great culture needs great people • Abrasive members will

    be the single biggest factor undermining all your progressive security efforts • Value social skills as highly as technical skills when making your security hires • ‘Cultural fit’ critically important
  74. @iodboi The more diverse a security team is, the more

    approachable it is to more people
  75. @iodboi Security Outreach • Distinct from security education

  76. @iodboi Security Outreach • Distinct from security education • Focus

    on building relationships
  77. @iodboi Security Outreach • Distinct from security education • Focus

    on building relationships • Removes barriers / reduces intimidation
  78. @iodboi Security Outreach • Distinct from security education • Focus

    on building relationships • Removes barriers / reduces intimidation • Can be as simple as picking up some bar tabs
  79. @iodboi Security Outreach • Distinct from security education • Focus

    on building relationships • Removes barriers / reduces intimidation • Can be as simple as picking up some bar tabs • Assign budget to this, it will be the best ROI you see
  80. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security

  81. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security

    • Embracing transparency
  82. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security

    • Embracing transparency • Deep insight to daily security issues and concerns
  83. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security

    • Embracing transparency • Deep insight to daily security issues and concerns • Build strong personal relationships
  84. @iodboi Bootcamps • Have people come and ‘bootcamp’ with security

    • Embracing transparency • Deep insight to daily security issues and concerns • Build strong personal relationships • Seed champions back out to the organization
  85. @iodboi Security Candy! • Biggest source of security pod ‘drive

    bys’ • IRC bot command so people can see what’s in stock • Graph consumption!
  86. Securgonomics!

  87. er·go·nom·ics ˌərɡəˈnämiks/ noun the study of people's efficiency in their

    working environment.
  88. secur·go·nom·ics /səˈkyo͝or/ ɡəˈnämiks/ noun the study of people's efficiency of

    security interactions in their working environment.
  89. @iodboi Securgonomics • Lowering the barrier to interact with security

  90. @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away
  91. @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable
  92. @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable • Sit on the busiest office pathway you can
  93. @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable • Sit on the busiest office pathway you can • Have your security dashboards front & centre
  94. @iodboi ‘We must strive to understand that accidents don’t happen

    because people gamble and loose. Accidents happen because the person believes that what is about to happen: - Is not possible - Has no connection to what they are doing - The intended outcome is worth the risk’ ! Erik Hollnagel Blameless Postmortems
  95. @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture
  96. @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame
  97. @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution
  98. @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution • Empower engineers to own their own stories
  99. @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution • Empower engineers to own their own stories • Applies to security failures as much as Ops failures
  100. Indicators of an effective security team

  101. @iodboi Is Data Driven • Too often security is explained

    with religious conviction
  102. @iodboi Is Data Driven • Too often security is explained

    with religious conviction • Security is not black and white, many shades of grey
  103. @iodboi Is Data Driven • Too often security is explained

    with religious conviction • Security is not black and white, many shades of grey • Security is not a point but a vector
  104. @iodboi Is Data Driven • Too often security is explained

    with religious conviction • Security is not black and white, many shades of grey • Security is not a point but a vector • Gather data to support security decisions and let it lead you to the correct shade of grey
  105. @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway …….
  106. @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries
  107. @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries • Increases value from focused pentests/red teaming
  108. @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries • Increases value from focused pentests/red teaming • Generates good metric sets about security (data driven)
  109. @iodboi Doesn’t Cry Wolf • Verify issues before raising them

    to developers
  110. @iodboi Doesn’t Cry Wolf • Verify issues before raising them

    to developers • They will only chase their tail a few times before ignoring
  111. @iodboi Doesn’t Cry Wolf • Verify issues before raising them

    to developers • They will only chase their tail a few times before ignoring • Security engineers should be in amongst the codebase • Aim to own the entire fix process themselves
  112. @iodboi Makes Realistic Tradeoffs • Not everything is critical

  113. @iodboi Makes Realistic Tradeoffs • Not everything is critical •

    Let low risk things ship along with commitments to a reasonable remediation window buys you lots
  114. @iodboi Makes Realistic Tradeoffs • Not everything is critical •

    Let low risk things ship along with commitments to a reasonable remediation window buys you lots • Save your NOs for when you need them - they are a finite resource
  115. @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected
  116. @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected • Provides security education and garners understanding
  117. @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected • Provides security education and garners understanding • ‘This would allow an attacker to impersonate another user & read their mail’ is useful and starts dialogue ….
  118. @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected • Provides security education and garners understanding • ‘This would allow an attacker to impersonate another user & read their mail’ is useful and starts dialogue …. • ‘Input validation was insufficiently applied’ does not
  119. @iodboi Recognises & Rewards • Rewarding folks in the org

    who reach out to Security • We do this is a number of ways: • Pins and patches • T-Shirts • Etsy gift vouchers • IRC Pluses & Value Awards
  120. @iodboi Etsy Value Awards

  121. @iodboi Treats Security as a BRAND • Your security culture

    has real value
  122. @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up
  123. @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up • Can be damaged in the blink of an eye
  124. @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up • Can be damaged in the blink of an eye • Aim to build strong, positive, long term associations with the security team
  125. @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up • Can be damaged in the blink of an eye • Aim to build strong, positive, long term associations with the security team • Get your consumers to buy in to this security shit
  126. Wrap up

  127. @iodboi Final thoughts • Building an effective security organisation takes

    effort
  128. @iodboi Final thoughts • Building an effective security organisation takes

    effort • Requires a focus on people as much as technology
  129. @iodboi Final thoughts • Building an effective security organisation takes

    effort • Requires a focus on people as much as technology • Learn from DevOps & move to a DevOpsSec mindset
  130. @iodboi Final thoughts • Building an effective security organisation takes

    effort • Requires a focus on people as much as technology • Learn from DevOps & move to a DevOpsSec mindset • Enable don’t block, else you’ll make security a NOP
  131. @iodboi Enabling. Transparent. Blameless

  132. @iodboi Thanks! Questions ?