$30 off During Our Annual Pro Sale. View Details »

Crafting an Effective Security Organisation (KiwiCon 8)

Rich Smith
December 11, 2014

Crafting an Effective Security Organisation (KiwiCon 8)

Talk given at KiwiCon 8 about the approaches taken by the security team at Etsy to build an effective security organisation and spread a progressive security culture.
The presentations shows approaches that we have found work for us but should not be seen as a one size fits all solution. Every organisation is different and has it's own cultural needs, it is hoped people will be able to adapt our learnings to best meet their organisation and in doing so share back their learnings with the community.

Rich Smith

December 11, 2014
Tweet

More Decks by Rich Smith

Other Decks in Technology

Transcript

  1. Crafting An Effective
    Security Organisation
    Kiwicon 8
    Rich Smith (@iodboi)

    View Slide

  2. @iodboi
    $whoami
    • Rich Smith - Brooklyn, NYC
    • Director of Security at Etsy
    • Co-Founder of Syndis in Reykjavík, Iceland
    • Background in offense: pen-testing, attack
    frameworks, post-exploitation, Goal Oriented Attack
    methodologies…

    View Slide

  3. @iodboi
    Now (FY 2013)
    • Gross Marketplace Sales (GMS) $1.35 Billion
    • 40 million members, 1 million active sellers
    • 26 million active listings
    • 200+ Countries Performing Transactions
    • >615 Employees
    • Offices in 8 countries

    View Slide

  4. @iodboi
    Focus Of Today
    • Lessons learnt building the Etsy security organization

    View Slide

  5. @iodboi
    Focus Of Today
    • Lessons learnt building the Etsy security organization
    • How we foster & grow our security culture

    View Slide

  6. @iodboi
    Focus Of Today
    • Lessons learnt building the Etsy security organization
    • How we foster & grow our security culture
    • The motivations driving how we think about security

    View Slide

  7. @iodboi
    Focus Of Today
    • Lessons learnt building the Etsy security organization
    • How we foster & grow our security culture
    • The motivations driving how we think about security

    View Slide

  8. @iodboi
    Focus Of Today
    • Lessons learnt building the Etsy security organization
    • How we foster & grow our security culture
    • The motivations driving how we think about security
    All work in progress !

    View Slide

  9. Disclaimer
    A + B != Culture

    View Slide

  10. Security from 50,000 ft

    View Slide

  11. Technology

    View Slide

  12. Society

    View Slide

  13. Society
    Technology Security

    View Slide

  14. Security

    View Slide

  15. @iodboi
    From this perspective it’s easy
    to see that people need to be
    considered alongside
    technology for effective
    security

    View Slide

  16. Etsy Engineering Culture

    View Slide

  17. @iodboi
    (Some) Core Engineering Principles
    • Empower the edges
    • Trust but verify
    • ‘Just Ship’ - Get things done
    • ‘If it moves graph it’ - Let the data lead you
    • Every engineer can push to prod at any time

    View Slide

  18. What does Continuous
    Deployment look like?

    View Slide

  19. @iodboi
    Very end of 2009 Today
    Pushes Per Day

    View Slide

  20. Continuous Deployment Continuous Delivery
    Frequent checkins
    directly to mainline
    ✓ ✓
    Automated build & test
    cycle
    ✓ ✓
    Keep the build green,
    always ready to release
    ✓ ✓
    One button deploys ✓ ✓
    Business dictates when
    to deploy

    Every passing build
    deployed to prod

    All enhancements gated
    by feature flag
    ✓ ?

    View Slide

  21. @iodboi
    Why Do This ?
    • Continuous Deployment/Delivery/Integration
    • Build your apps in a reproducible way after each push to git
    • Identify bugs, missing dependencies early & often
    • Integrate security testing throughout lifecycle
    • Developers iterate in production, A/B experimentation
    • Improve Mean Time To Recovery

    View Slide

  22. @iodboi
    But doesn’t this CD stuff
    mean the stuff breaks all
    the time?

    View Slide

  23. Single
    release
    Many
    releases
    50K LOC/month
    Few opportunities for failure

    Wide surface area (50,000 LOC)
    High MTTR
    !
    All of the bugs we’ve written
    More opportunities for failure
    Narrow surface area (< 100 LOC)
    Low MTTR
    !
    A fraction of the bugs we’ve

    written per release
    Imagine that we’ll write

    View Slide

  24. View Slide

  25. View Slide

  26. View Slide

  27. @iodboi
    But how do you ‘security’
    the anarchy?

    View Slide

  28. @iodboi
    In such an environment
    classical security approaches
    don’t work well

    View Slide

  29. @iodboi
    Security trying to reintroduce
    blocking to the org will mean
    it is ignored, not embraced

    View Slide

  30. @iodboi
    Continuous Deployment & Security
    • The lessons & tools from DevOps are directly applicable

    View Slide

  31. @iodboi
    Continuous Deployment & Security
    • The lessons & tools from DevOps are directly applicable
    • Apply the same ‘if it moves graph it’ for security events

    View Slide

  32. @iodboi
    Continuous Deployment & Security
    • The lessons & tools from DevOps are directly applicable
    • Apply the same ‘if it moves graph it’ for security events
    • Makes security related data available to everyone

    View Slide

  33. @iodboi
    Continuous Deployment & Security
    • The lessons & tools from DevOps are directly applicable
    • Apply the same ‘if it moves graph it’ for security events
    • Makes security related data available to everyone
    • No such things as ‘out of cycle’ patches

    View Slide

  34. @iodboi
    Continuous Deployment & Security
    • The lessons & tools from DevOps are directly applicable
    • Apply the same ‘if it moves graph it’ for security events
    • Makes security related data available to everyone
    • No such things as ‘out of cycle’ patches
    • Security engineers push fixes directly to production

    View Slide

  35. Principles of Effective
    Security

    View Slide

  36. @iodboi
    3 Principles of Effective Security
    1. Enabling

    View Slide

  37. @iodboi
    3 Principles of Effective Security
    1. Enabling
    2. Transparent

    View Slide

  38. @iodboi
    3 Principles of Effective Security
    1. Enabling
    2. Transparent
    3. Blameless

    View Slide

  39. @iodboi
    A security team’s success
    should be measured by what
    they enable not by what
    they block
    Enabling

    View Slide

  40. @iodboi
    A security team that is open
    as to what it does, and why,
    spreads understanding and
    is embraced
    Transparent

    View Slide

  41. @iodboi
    Security failures will
    happen, only without blame
    will you be able to
    understand the true causes
    Blameless

    View Slide

  42. ‘DevOpsSec’

    View Slide

  43. @iodboi
    or ….
    ‘Lessons Security can
    learn from DevOps’

    View Slide

  44. @iodboi
    DevOps
    • ‘DevOps’ has become somewhat overloaded

    View Slide

  45. @iodboi
    DevOps
    • ‘DevOps’ has become somewhat overloaded
    • Aim: Remove silos & organizational blockers between
    Ops and Developers

    View Slide

  46. @iodboi
    DevOps
    • ‘DevOps’ has become somewhat overloaded
    • Aim: Remove silos & organizational blockers between
    Ops and Developers
    • Central to this focus on good Communication &
    Collaboration

    View Slide

  47. @iodboi
    ‘DevOpsSec’
    Dev Ops
    • Natural extension of DevOps

    View Slide

  48. @iodboi
    ‘DevOpsSec’
    Dev Ops
    Sec
    • Natural extension of DevOps
    • Security faces many of the same
    challenges as Ops did
    • Removing barriers between
    Security, Developers and
    Operations

    View Slide

  49. @iodboi
    Security as a blocker
    • Lazy and plain ‘bad’ security teams default to blocking

    View Slide

  50. @iodboi
    Security as a blocker
    • Lazy and plain ‘bad’ security teams default to blocking
    • Blocking makes Security a NOP in the CD world

    View Slide

  51. @iodboi
    Security as a blocker
    • Lazy and plain ‘bad’ security teams default to blocking
    • Blocking makes Security a NOP in the CD world
    • You will be ignored and teams will work around you

    View Slide

  52. @iodboi
    Security as a blocker
    • Lazy and plain ‘bad’ security teams default to blocking
    • Blocking makes Security a NOP in the CD world
    • You will be ignored and teams will work around you
    • Save your ‘No’s’ as the very last resort

    View Slide

  53. @iodboi
    Security as a enabler
    • Assist teams to do their new awesome ideas securely

    View Slide

  54. @iodboi
    Security as a enabler
    • Assist teams to do their new awesome ideas securely
    • Incentivizes proactive engagement with Security

    View Slide

  55. @iodboi
    Security as a enabler
    • Assist teams to do their new awesome ideas securely
    • Incentivizes proactive engagement with Security
    • Chase solutions to difficult challenges

    View Slide

  56. @iodboi
    Designated Hackers
    • Security engineers assist multiple teams

    View Slide

  57. @iodboi
    Designated Hackers
    • Security engineers assist multiple teams
    • ‘Designated’ not ‘Dedicated’

    View Slide

  58. @iodboi
    Designated Hackers
    • Security engineers assist multiple teams
    • ‘Designated’ not ‘Dedicated’
    • Breaks down barriers, build trust & relationships

    View Slide

  59. @iodboi
    Designated Hackers
    • Security engineers assist multiple teams
    • ‘Designated’ not ‘Dedicated’
    • Breaks down barriers, build trust & relationships
    • Represent teams back to security

    View Slide

  60. @iodboi
    Designated Hackers
    • Security engineers assist multiple teams
    • ‘Designated’ not ‘Dedicated’
    • Breaks down barriers, build trust & relationships
    • Represent teams back to security
    • Early visibility, input & deeper insight

    View Slide

  61. ‘You’re only a blocker if
    you’re the last to know’
    John Allspaw, !
    Some meeting room ,somewhere at Etsy

    View Slide

  62. Progressive Security
    Culture

    View Slide

  63. @iodboi
    Progressive Security Culture
    • Understanding that security is as much of a people
    problem as a technology problem

    View Slide

  64. @iodboi
    Progressive Security Culture
    • Understanding that security is as much of a people
    problem as a technology problem
    • As an industry, security has done a poor job of
    discussing the need for positive security culture

    View Slide

  65. @iodboi
    Progressive Security Culture
    • Understanding that security is as much of a people
    problem as a technology problem
    • As an industry, security has done a poor job of
    discussing the need for positive security culture
    • Often approaches focussed on are entirely technical

    View Slide

  66. @iodboi
    Progressive Security Culture
    • Understanding that security is as much of a people
    problem as a technology problem
    • As an industry, security has done a poor job of
    discussing the need for positive security culture
    • Often approaches focussed on are entirely technical
    • Great culture depends on great people

    View Slide

  67. @iodboi
    Security Team Hiring
    Number 1 rule ……

    View Slide

  68. @iodboi
    Security Team Hiring
    Number 1 rule ……
    Don’t Hire Assholes

    View Slide

  69. @iodboi
    Security Team Hiring
    If you inadvertently do, or you inherit one……

    View Slide

  70. @iodboi
    Security Team Hiring
    If you inadvertently do, or you inherit one……
    Remove them ASAP

    View Slide

  71. @iodboi
    Great culture needs great people
    • Abrasive members will be the single biggest factor
    undermining all your progressive security efforts

    View Slide

  72. @iodboi
    Great culture needs great people
    • Abrasive members will be the single biggest factor
    undermining all your progressive security efforts
    • Value social skills as highly as technical skills when
    making your security hires

    View Slide

  73. @iodboi
    Great culture needs great people
    • Abrasive members will be the single biggest factor
    undermining all your progressive security efforts
    • Value social skills as highly as technical skills when
    making your security hires
    • ‘Cultural fit’ critically important

    View Slide

  74. @iodboi
    The more diverse a security team is, the
    more approachable it is to more people

    View Slide

  75. @iodboi
    Security Outreach
    • Distinct from security education

    View Slide

  76. @iodboi
    Security Outreach
    • Distinct from security education
    • Focus on building relationships

    View Slide

  77. @iodboi
    Security Outreach
    • Distinct from security education
    • Focus on building relationships
    • Removes barriers / reduces intimidation

    View Slide

  78. @iodboi
    Security Outreach
    • Distinct from security education
    • Focus on building relationships
    • Removes barriers / reduces intimidation
    • Can be as simple as picking up some bar tabs

    View Slide

  79. @iodboi
    Security Outreach
    • Distinct from security education
    • Focus on building relationships
    • Removes barriers / reduces intimidation
    • Can be as simple as picking up some bar tabs
    • Assign budget to this, it will be the best ROI you see

    View Slide

  80. @iodboi
    Bootcamps
    • Have people come and ‘bootcamp’ with security

    View Slide

  81. @iodboi
    Bootcamps
    • Have people come and ‘bootcamp’ with security
    • Embracing transparency

    View Slide

  82. @iodboi
    Bootcamps
    • Have people come and ‘bootcamp’ with security
    • Embracing transparency
    • Deep insight to daily security issues and concerns

    View Slide

  83. @iodboi
    Bootcamps
    • Have people come and ‘bootcamp’ with security
    • Embracing transparency
    • Deep insight to daily security issues and concerns
    • Build strong personal relationships

    View Slide

  84. @iodboi
    Bootcamps
    • Have people come and ‘bootcamp’ with security
    • Embracing transparency
    • Deep insight to daily security issues and concerns
    • Build strong personal relationships
    • Seed champions back out to the organization

    View Slide

  85. @iodboi
    Security Candy!
    • Biggest source of security pod
    ‘drive bys’
    • IRC bot command so people
    can see what’s in stock
    • Graph consumption!

    View Slide

  86. Securgonomics!

    View Slide

  87. er·go·nom·ics
    ˌərɡəˈnämiks/
    noun
    the study of people's efficiency in their working
    environment.

    View Slide

  88. secur·go·nom·ics
    /səˈkyo͝or/ ɡəˈnämiks/
    noun
    the study of people's efficiency of security interactions
    in their working environment.

    View Slide

  89. @iodboi
    Securgonomics
    • Lowering the barrier to interact with security

    View Slide

  90. @iodboi
    Securgonomics
    • Lowering the barrier to interact with security
    • Too often security teams lock themselves away

    View Slide

  91. @iodboi
    Securgonomics
    • Lowering the barrier to interact with security
    • Too often security teams lock themselves away
    • Being accessible & visible to everyone is invaluable

    View Slide

  92. @iodboi
    Securgonomics
    • Lowering the barrier to interact with security
    • Too often security teams lock themselves away
    • Being accessible & visible to everyone is invaluable
    • Sit on the busiest office pathway you can

    View Slide

  93. @iodboi
    Securgonomics
    • Lowering the barrier to interact with security
    • Too often security teams lock themselves away
    • Being accessible & visible to everyone is invaluable
    • Sit on the busiest office pathway you can
    • Have your security dashboards front & centre

    View Slide

  94. @iodboi
    ‘We must strive to understand that accidents don’t
    happen because people gamble and loose. Accidents
    happen because the person believes that what is
    about to happen:
    - Is not possible
    - Has no connection to what they are doing
    - The intended outcome is worth the risk’
    !
    Erik Hollnagel
    Blameless Postmortems

    View Slide

  95. @iodboi
    Blameless Postmortems
    • Comes from our desire to have Just Culture

    View Slide

  96. @iodboi
    Blameless Postmortems
    • Comes from our desire to have Just Culture
    • Aim to learn from failings not to target blame

    View Slide

  97. @iodboi
    Blameless Postmortems
    • Comes from our desire to have Just Culture
    • Aim to learn from failings not to target blame
    • Share detailed accounts of actions, decisions and
    circumstances without fear of punishment or retribution

    View Slide

  98. @iodboi
    Blameless Postmortems
    • Comes from our desire to have Just Culture
    • Aim to learn from failings not to target blame
    • Share detailed accounts of actions, decisions and
    circumstances without fear of punishment or retribution
    • Empower engineers to own their own stories

    View Slide

  99. @iodboi
    Blameless Postmortems
    • Comes from our desire to have Just Culture
    • Aim to learn from failings not to target blame
    • Share detailed accounts of actions, decisions and
    circumstances without fear of punishment or retribution
    • Empower engineers to own their own stories
    • Applies to security failures as much as Ops failures

    View Slide

  100. Indicators of an effective
    security team

    View Slide

  101. @iodboi
    Is Data Driven
    • Too often security is explained with religious conviction

    View Slide

  102. @iodboi
    Is Data Driven
    • Too often security is explained with religious conviction
    • Security is not black and white, many shades of grey

    View Slide

  103. @iodboi
    Is Data Driven
    • Too often security is explained with religious conviction
    • Security is not black and white, many shades of grey
    • Security is not a point but a vector

    View Slide

  104. @iodboi
    Is Data Driven
    • Too often security is explained with religious conviction
    • Security is not black and white, many shades of grey
    • Security is not a point but a vector
    • Gather data to support security decisions and let it lead
    you to the correct shade of grey

    View Slide

  105. @iodboi
    Runs a Bug Bounty
    • Continuous Assessment of your security program
    • D’ya you think you’re not under attack 24/7 anyway …….

    View Slide

  106. @iodboi
    Runs a Bug Bounty
    • Continuous Assessment of your security program
    • D’ya you think you’re not under attack 24/7 anyway …….
    • Raises cost of attack for real adversaries

    View Slide

  107. @iodboi
    Runs a Bug Bounty
    • Continuous Assessment of your security program
    • D’ya you think you’re not under attack 24/7 anyway …….
    • Raises cost of attack for real adversaries
    • Increases value from focused pentests/red teaming

    View Slide

  108. @iodboi
    Runs a Bug Bounty
    • Continuous Assessment of your security program
    • D’ya you think you’re not under attack 24/7 anyway …….
    • Raises cost of attack for real adversaries
    • Increases value from focused pentests/red teaming
    • Generates good metric sets about security (data driven)

    View Slide

  109. @iodboi
    Doesn’t Cry Wolf
    • Verify issues before raising them to developers

    View Slide

  110. @iodboi
    Doesn’t Cry Wolf
    • Verify issues before raising them to developers
    • They will only chase their tail a few times before ignoring

    View Slide

  111. @iodboi
    Doesn’t Cry Wolf
    • Verify issues before raising them to developers
    • They will only chase their tail a few times before ignoring
    • Security engineers should be in amongst the codebase
    • Aim to own the entire fix process themselves

    View Slide

  112. @iodboi
    Makes Realistic Tradeoffs
    • Not everything is critical

    View Slide

  113. @iodboi
    Makes Realistic Tradeoffs
    • Not everything is critical
    • Let low risk things ship along with commitments to a
    reasonable remediation window buys you lots

    View Slide

  114. @iodboi
    Makes Realistic Tradeoffs
    • Not everything is critical
    • Let low risk things ship along with commitments to a
    reasonable remediation window buys you lots
    • Save your NOs for when you need them - they are a
    finite resource

    View Slide

  115. @iodboi
    Provides Context & Impact
    • Explaining why something is an issue and what it may
    result in to the team affected

    View Slide

  116. @iodboi
    Provides Context & Impact
    • Explaining why something is an issue and what it may
    result in to the team affected
    • Provides security education and garners understanding

    View Slide

  117. @iodboi
    Provides Context & Impact
    • Explaining why something is an issue and what it may
    result in to the team affected
    • Provides security education and garners understanding
    • ‘This would allow an attacker to impersonate another
    user & read their mail’ is useful and starts dialogue ….

    View Slide

  118. @iodboi
    Provides Context & Impact
    • Explaining why something is an issue and what it may
    result in to the team affected
    • Provides security education and garners understanding
    • ‘This would allow an attacker to impersonate another
    user & read their mail’ is useful and starts dialogue ….
    • ‘Input validation was insufficiently applied’ does not

    View Slide

  119. @iodboi
    Recognises & Rewards
    • Rewarding folks in the org who reach out to Security
    • We do this is a number of ways:
    • Pins and patches
    • T-Shirts
    • Etsy gift vouchers
    • IRC Pluses & Value Awards

    View Slide

  120. @iodboi
    Etsy Value Awards

    View Slide

  121. @iodboi
    Treats Security as a BRAND
    • Your security culture has real value

    View Slide

  122. @iodboi
    Treats Security as a BRAND
    • Your security culture has real value
    • Work long & hard to build it up

    View Slide

  123. @iodboi
    Treats Security as a BRAND
    • Your security culture has real value
    • Work long & hard to build it up
    • Can be damaged in the blink of an eye

    View Slide

  124. @iodboi
    Treats Security as a BRAND
    • Your security culture has real value
    • Work long & hard to build it up
    • Can be damaged in the blink of an eye
    • Aim to build strong, positive, long term associations
    with the security team

    View Slide

  125. @iodboi
    Treats Security as a BRAND
    • Your security culture has real value
    • Work long & hard to build it up
    • Can be damaged in the blink of an eye
    • Aim to build strong, positive, long term associations
    with the security team
    • Get your consumers to buy in to this security shit

    View Slide

  126. Wrap up

    View Slide

  127. @iodboi
    Final thoughts
    • Building an effective security organisation takes effort

    View Slide

  128. @iodboi
    Final thoughts
    • Building an effective security organisation takes effort
    • Requires a focus on people as much as technology

    View Slide

  129. @iodboi
    Final thoughts
    • Building an effective security organisation takes effort
    • Requires a focus on people as much as technology
    • Learn from DevOps & move to a DevOpsSec mindset

    View Slide

  130. @iodboi
    Final thoughts
    • Building an effective security organisation takes effort
    • Requires a focus on people as much as technology
    • Learn from DevOps & move to a DevOpsSec mindset
    • Enable don’t block, else you’ll make security a NOP

    View Slide

  131. @iodboi
    Enabling. Transparent. Blameless

    View Slide

  132. @iodboi
    Thanks! Questions ?

    View Slide