Crafting an Effective Security Organisation (KiwiCon 8)

Aa6b7d3a12a61bfe0ce9c8a12405bd75?s=47 Rich Smith
December 11, 2014

Crafting an Effective Security Organisation (KiwiCon 8)

Talk given at KiwiCon 8 about the approaches taken by the security team at Etsy to build an effective security organisation and spread a progressive security culture.
The presentations shows approaches that we have found work for us but should not be seen as a one size fits all solution. Every organisation is different and has it's own cultural needs, it is hoped people will be able to adapt our learnings to best meet their organisation and in doing so share back their learnings with the community.

Aa6b7d3a12a61bfe0ce9c8a12405bd75?s=128

Rich Smith

December 11, 2014
Tweet

Transcript

  1. 2.

    @iodboi $whoami • Rich Smith - Brooklyn, NYC • Director

    of Security at Etsy • Co-Founder of Syndis in Reykjavík, Iceland • Background in offense: pen-testing, attack frameworks, post-exploitation, Goal Oriented Attack methodologies…
  2. 3.

    @iodboi Now (FY 2013) • Gross Marketplace Sales (GMS) $1.35

    Billion • 40 million members, 1 million active sellers • 26 million active listings • 200+ Countries Performing Transactions • >615 Employees • Offices in 8 countries
  3. 5.

    @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture
  4. 6.

    @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture • The motivations driving how we think about security
  5. 7.

    @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture • The motivations driving how we think about security
  6. 8.

    @iodboi Focus Of Today • Lessons learnt building the Etsy

    security organization • How we foster & grow our security culture • The motivations driving how we think about security All work in progress !
  7. 12.
  8. 14.
  9. 15.

    @iodboi From this perspective it’s easy to see that people

    need to be considered alongside technology for effective security
  10. 17.

    @iodboi (Some) Core Engineering Principles • Empower the edges •

    Trust but verify • ‘Just Ship’ - Get things done • ‘If it moves graph it’ - Let the data lead you • Every engineer can push to prod at any time
  11. 20.

    Continuous Deployment Continuous Delivery Frequent checkins directly to mainline ✓

    ✓ Automated build & test cycle ✓ ✓ Keep the build green, always ready to release ✓ ✓ One button deploys ✓ ✓ Business dictates when to deploy ✓ Every passing build deployed to prod ✓ All enhancements gated by feature flag ✓ ?
  12. 21.

    @iodboi Why Do This ? • Continuous Deployment/Delivery/Integration • Build

    your apps in a reproducible way after each push to git • Identify bugs, missing dependencies early & often • Integrate security testing throughout lifecycle • Developers iterate in production, A/B experimentation • Improve Mean Time To Recovery
  13. 23.

    Single release Many releases 50K LOC/month Few opportunities for failure


    Wide surface area (50,000 LOC) High MTTR ! All of the bugs we’ve written More opportunities for failure Narrow surface area (< 100 LOC) Low MTTR ! A fraction of the bugs we’ve
 written per release Imagine that we’ll write
  14. 24.
  15. 25.
  16. 26.
  17. 31.

    @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events
  18. 32.

    @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone
  19. 33.

    @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone • No such things as ‘out of cycle’ patches
  20. 34.

    @iodboi Continuous Deployment & Security • The lessons & tools

    from DevOps are directly applicable • Apply the same ‘if it moves graph it’ for security events • Makes security related data available to everyone • No such things as ‘out of cycle’ patches • Security engineers push fixes directly to production
  21. 39.

    @iodboi A security team’s success should be measured by what

    they enable not by what they block Enabling
  22. 40.

    @iodboi A security team that is open as to what

    it does, and why, spreads understanding and is embraced Transparent
  23. 41.

    @iodboi Security failures will happen, only without blame will you

    be able to understand the true causes Blameless
  24. 45.

    @iodboi DevOps • ‘DevOps’ has become somewhat overloaded • Aim:

    Remove silos & organizational blockers between Ops and Developers
  25. 46.

    @iodboi DevOps • ‘DevOps’ has become somewhat overloaded • Aim:

    Remove silos & organizational blockers between Ops and Developers • Central to this focus on good Communication & Collaboration
  26. 48.

    @iodboi ‘DevOpsSec’ Dev Ops Sec • Natural extension of DevOps

    • Security faces many of the same challenges as Ops did • Removing barriers between Security, Developers and Operations
  27. 49.
  28. 50.

    @iodboi Security as a blocker • Lazy and plain ‘bad’

    security teams default to blocking • Blocking makes Security a NOP in the CD world
  29. 51.

    @iodboi Security as a blocker • Lazy and plain ‘bad’

    security teams default to blocking • Blocking makes Security a NOP in the CD world • You will be ignored and teams will work around you
  30. 52.

    @iodboi Security as a blocker • Lazy and plain ‘bad’

    security teams default to blocking • Blocking makes Security a NOP in the CD world • You will be ignored and teams will work around you • Save your ‘No’s’ as the very last resort
  31. 53.

    @iodboi Security as a enabler • Assist teams to do

    their new awesome ideas securely
  32. 54.

    @iodboi Security as a enabler • Assist teams to do

    their new awesome ideas securely • Incentivizes proactive engagement with Security
  33. 55.

    @iodboi Security as a enabler • Assist teams to do

    their new awesome ideas securely • Incentivizes proactive engagement with Security • Chase solutions to difficult challenges
  34. 58.

    @iodboi Designated Hackers • Security engineers assist multiple teams •

    ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships
  35. 59.

    @iodboi Designated Hackers • Security engineers assist multiple teams •

    ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships • Represent teams back to security
  36. 60.

    @iodboi Designated Hackers • Security engineers assist multiple teams •

    ‘Designated’ not ‘Dedicated’ • Breaks down barriers, build trust & relationships • Represent teams back to security • Early visibility, input & deeper insight
  37. 61.

    ‘You’re only a blocker if you’re the last to know’

    John Allspaw, ! Some meeting room ,somewhere at Etsy
  38. 63.

    @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem
  39. 64.

    @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture
  40. 65.

    @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture • Often approaches focussed on are entirely technical
  41. 66.

    @iodboi Progressive Security Culture • Understanding that security is as

    much of a people problem as a technology problem • As an industry, security has done a poor job of discussing the need for positive security culture • Often approaches focussed on are entirely technical • Great culture depends on great people
  42. 70.
  43. 71.

    @iodboi Great culture needs great people • Abrasive members will

    be the single biggest factor undermining all your progressive security efforts
  44. 72.

    @iodboi Great culture needs great people • Abrasive members will

    be the single biggest factor undermining all your progressive security efforts • Value social skills as highly as technical skills when making your security hires
  45. 73.

    @iodboi Great culture needs great people • Abrasive members will

    be the single biggest factor undermining all your progressive security efforts • Value social skills as highly as technical skills when making your security hires • ‘Cultural fit’ critically important
  46. 74.

    @iodboi The more diverse a security team is, the more

    approachable it is to more people
  47. 77.

    @iodboi Security Outreach • Distinct from security education • Focus

    on building relationships • Removes barriers / reduces intimidation
  48. 78.

    @iodboi Security Outreach • Distinct from security education • Focus

    on building relationships • Removes barriers / reduces intimidation • Can be as simple as picking up some bar tabs
  49. 79.

    @iodboi Security Outreach • Distinct from security education • Focus

    on building relationships • Removes barriers / reduces intimidation • Can be as simple as picking up some bar tabs • Assign budget to this, it will be the best ROI you see
  50. 82.

    @iodboi Bootcamps • Have people come and ‘bootcamp’ with security

    • Embracing transparency • Deep insight to daily security issues and concerns
  51. 83.

    @iodboi Bootcamps • Have people come and ‘bootcamp’ with security

    • Embracing transparency • Deep insight to daily security issues and concerns • Build strong personal relationships
  52. 84.

    @iodboi Bootcamps • Have people come and ‘bootcamp’ with security

    • Embracing transparency • Deep insight to daily security issues and concerns • Build strong personal relationships • Seed champions back out to the organization
  53. 85.

    @iodboi Security Candy! • Biggest source of security pod ‘drive

    bys’ • IRC bot command so people can see what’s in stock • Graph consumption!
  54. 90.

    @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away
  55. 91.

    @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable
  56. 92.

    @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable • Sit on the busiest office pathway you can
  57. 93.

    @iodboi Securgonomics • Lowering the barrier to interact with security

    • Too often security teams lock themselves away • Being accessible & visible to everyone is invaluable • Sit on the busiest office pathway you can • Have your security dashboards front & centre
  58. 94.

    @iodboi ‘We must strive to understand that accidents don’t happen

    because people gamble and loose. Accidents happen because the person believes that what is about to happen: - Is not possible - Has no connection to what they are doing - The intended outcome is worth the risk’ ! Erik Hollnagel Blameless Postmortems
  59. 96.

    @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame
  60. 97.

    @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution
  61. 98.

    @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution • Empower engineers to own their own stories
  62. 99.

    @iodboi Blameless Postmortems • Comes from our desire to have

    Just Culture • Aim to learn from failings not to target blame • Share detailed accounts of actions, decisions and circumstances without fear of punishment or retribution • Empower engineers to own their own stories • Applies to security failures as much as Ops failures
  63. 102.

    @iodboi Is Data Driven • Too often security is explained

    with religious conviction • Security is not black and white, many shades of grey
  64. 103.

    @iodboi Is Data Driven • Too often security is explained

    with religious conviction • Security is not black and white, many shades of grey • Security is not a point but a vector
  65. 104.

    @iodboi Is Data Driven • Too often security is explained

    with religious conviction • Security is not black and white, many shades of grey • Security is not a point but a vector • Gather data to support security decisions and let it lead you to the correct shade of grey
  66. 105.

    @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway …….
  67. 106.

    @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries
  68. 107.

    @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries • Increases value from focused pentests/red teaming
  69. 108.

    @iodboi Runs a Bug Bounty • Continuous Assessment of your

    security program • D’ya you think you’re not under attack 24/7 anyway ……. • Raises cost of attack for real adversaries • Increases value from focused pentests/red teaming • Generates good metric sets about security (data driven)
  70. 110.

    @iodboi Doesn’t Cry Wolf • Verify issues before raising them

    to developers • They will only chase their tail a few times before ignoring
  71. 111.

    @iodboi Doesn’t Cry Wolf • Verify issues before raising them

    to developers • They will only chase their tail a few times before ignoring • Security engineers should be in amongst the codebase • Aim to own the entire fix process themselves
  72. 113.

    @iodboi Makes Realistic Tradeoffs • Not everything is critical •

    Let low risk things ship along with commitments to a reasonable remediation window buys you lots
  73. 114.

    @iodboi Makes Realistic Tradeoffs • Not everything is critical •

    Let low risk things ship along with commitments to a reasonable remediation window buys you lots • Save your NOs for when you need them - they are a finite resource
  74. 115.

    @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected
  75. 116.

    @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected • Provides security education and garners understanding
  76. 117.

    @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected • Provides security education and garners understanding • ‘This would allow an attacker to impersonate another user & read their mail’ is useful and starts dialogue ….
  77. 118.

    @iodboi Provides Context & Impact • Explaining why something is

    an issue and what it may result in to the team affected • Provides security education and garners understanding • ‘This would allow an attacker to impersonate another user & read their mail’ is useful and starts dialogue …. • ‘Input validation was insufficiently applied’ does not
  78. 119.

    @iodboi Recognises & Rewards • Rewarding folks in the org

    who reach out to Security • We do this is a number of ways: • Pins and patches • T-Shirts • Etsy gift vouchers • IRC Pluses & Value Awards
  79. 122.

    @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up
  80. 123.

    @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up • Can be damaged in the blink of an eye
  81. 124.

    @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up • Can be damaged in the blink of an eye • Aim to build strong, positive, long term associations with the security team
  82. 125.

    @iodboi Treats Security as a BRAND • Your security culture

    has real value • Work long & hard to build it up • Can be damaged in the blink of an eye • Aim to build strong, positive, long term associations with the security team • Get your consumers to buy in to this security shit
  83. 126.
  84. 128.

    @iodboi Final thoughts • Building an effective security organisation takes

    effort • Requires a focus on people as much as technology
  85. 129.

    @iodboi Final thoughts • Building an effective security organisation takes

    effort • Requires a focus on people as much as technology • Learn from DevOps & move to a DevOpsSec mindset
  86. 130.

    @iodboi Final thoughts • Building an effective security organisation takes

    effort • Requires a focus on people as much as technology • Learn from DevOps & move to a DevOpsSec mindset • Enable don’t block, else you’ll make security a NOP