Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Buying Security

Buying Security

You can’t buy security, but vendors play a key role in effective security programs. This talk will provide a comprehensive guide to buying and getting value, based on experiences on both sides of the marketplace, a comprehensive literature review, and a survey of clients and vendors of all stripes.

Rami McCarthy

June 04, 2022
Tweet

More Decks by Rami McCarthy

Other Decks in Technology

Transcript

  1. @ramimacisabird
    BSidesSF 2022
    Buying Security
    A Client’s Guide
    Rami McCarthy

    View full-size slide

  2. @ramimacisabird
    👋 hello
    • Security @ series d health-tech
    • Reformed security consultant
    • with thanks to
    I’m Rami McCarthy

    View full-size slide

  3. @ramimacisabird

    View full-size slide

  4. @ramimacisabird
    Why are we here?
    Security services are important and
    buying them is hard

    View full-size slide

  5. @ramimacisabird
    Challenges
    “There is too little reliable information
    about the type and quality of services
    offered, and too little knowledge about how
    to sell and buy them.”

    View full-size slide

  6. @ramimacisabird

    View full-size slide

  7. @ramimacisabird
    Security Assessment
    > a consulting engagement focused on evaluating security design,
    architecture, and/or implemented controls to identify whether they
    are operating as intended and helping the organization to meet its
    security requirements
    NIST

    View full-size slide

  8. @ramimacisabird
    What will you get?
    ● ~200 resources synthesized
    ● ~100 security professionals surveyed
    ● A comprehensive guide to buying and
    getting value from security services

    View full-size slide

  9. @ramimacisabird

    View full-size slide

  10. @ramimacisabird

    View full-size slide

  11. @ramimacisabird

    View full-size slide

  12. @ramimacisabird

    View full-size slide

  13. @ramimacisabird

    View full-size slide

  14. @ramimacisabird
    Why are we here?

    View full-size slide

  15. @ramimacisabird
    Penetration Testing Considered Harmful Today (2012)
    Thinkst’s Haroon Meer, 44CON 2011 [video]

    View full-size slide

  16. @ramimacisabird
    Point/Counterpoint: Penetration Testing
    Counterpoint: Marcus Ranum, 2007
    Application Security Tools: Good or Bad?
    Gary McGraw, Freedom to Tinker, 2006

    View full-size slide

  17. @ramimacisabird
    The average security services vendor delivers
    a quality assessment

    View full-size slide

  18. @ramimacisabird
    The average security services vendor delivers
    a quality assessment
    Buyers Sellers
    2.64
    3.14

    View full-size slide

  19. @ramimacisabird
    Quality

    View full-size slide

  20. @ramimacisabird
    Security Assessments

    View full-size slide

  21. @ramimacisabird
    Patrick Thomas
    Types of Security Assessments

    View full-size slide

  22. @ramimacisabird
    Types of Security
    Assessments
    • Vulnerability assessment
    • Penetration testing
    • White-box
    (“full knowledge”)
    • Grey-box
    (“partial knowledge”)
    • Black-box
    (“no knowledge”)
    • Code review
    • Threat Model
    • Red Team
    (Adversary Simulation)
    • Social Engineering (Phishing,
    Vishing, Smishing)
    • Technical Specialty (hardware,
    cryptography, cryptocurrency)
    ● The Difference Between a Vulnerability Assessment and a Penetration Test, Daniel Miessler
    ● What Are The Different Types Of Penetration Testing?, Jason Firch, PurpleSec.us

    View full-size slide

  23. @ramimacisabird
    On newer models…

    View full-size slide

  24. @ramimacisabird
    “Getting tough to
    find something
    that's not just a
    dressed up Nessus
    scan.”

    View full-size slide

  25. @ramimacisabird
    GOTO 1
    Security Assessment Process
    8 The Readout
    6 Contracting
    Gathering Proposals
    4
    2 Finding Vendors
    9 Ingestion
    7 Preparation & Delivery
    5 Vendor scoping
    Client scoping and requirements
    3
    1 Defining Motivation
    1
    0
    After the assessment
    1
    1
    Now what?

    View full-size slide

  26. @ramimacisabird
    1 Defining Motivation

    View full-size slide

  27. @ramimacisabird
    • Risk Reduction
    • Compliance
    • Internal attestation
    • Investment or M&A
    • Sales
    • Post-breach
    1 Defining Motivation

    View full-size slide

  28. @ramimacisabird
    • Risk Reduction
    • Compliance
    • Internal attestation
    • Investment or M&A
    • Sales
    • Post-breach
    1 Defining Motivation

    View full-size slide

  29. @ramimacisabird
    2 Finding Vendors
    https://twitter.com/cure53berlin/status/976463307942088706

    View full-size slide

  30. @ramimacisabird
    Taxonomy of Vendors

    View full-size slide

  31. @ramimacisabird
    Taxonomy Types of Vendors
    •Global/enterprise
    consulting
    •Cybersecurity services
    •Boutique
    •Specialty
    •Sole practitioner
    •Researcher/bug
    hunter
    •Low cost
    •Managed Security
    Service Provider
    •Value Added Reseller

    View full-size slide

  32. @ramimacisabird
    What is the greatest challenge in buying security
    services?
    ● Finding good vendors, who provide significant value
    ○ who are available when you need them
    ○ who can support specific systems or architecture
    ○ who provide consistent quality staff
    2 Finding Vendors

    View full-size slide

  33. @ramimacisabird
    •Network
    recommendations
    •Follow-the-leader
    •Research
    •Conference
    speakers
    •Published research
    •Prominent staff
    •Public reports
    •Compliance
    approved
    •Assessment
    standards work
    •Certifications
    •Analyst
    recommendations
    ● https://github.com/juliocesarfort/public-pentesting-reports
    ● http://www.pentest-standard.org/index.php/FAQ#Q:_Who_is_involved_with_this_standard.3F
    2 Finding Vendors

    View full-size slide

  34. @ramimacisabird
    ● https://github.com/juliocesarfort/public-pentesting-reports
    ● http://www.pentest-standard.org/index.php/FAQ#Q:_Who_is_involved_with_this_standard.3F
    2 Finding Vendors

    View full-size slide

  35. @ramimacisabird
    Client scoping and requirements
    3

    View full-size slide

  36. @ramimacisabird
    Client scoping and requirements
    3
    > Scope management is the process of defining what work is
    required, and then making sure that all of that work, and only that
    work, is done.
    SANS
    Scoping Security Assessments - A Project Management Approach

    View full-size slide

  37. @ramimacisabird
    Client scoping and requirements
    3

    View full-size slide

  38. @ramimacisabird
    > Strike a balance between performing a comprehensive set of tests
    and evaluating functionality and features that present the greatest
    risk.
    GSA IT Security Procedural Guide:
    Conducting Penetration Test Exercises
    Client scoping and requirements
    3

    View full-size slide

  39. @ramimacisabird
    Client scoping and requirements
    3

    View full-size slide

  40. @ramimacisabird
    • Budget
    ● https://web.archive.org/web/20071207150024/http://securitybuddha.com/2007/08/22/the-art-of-scoping-application-security-reviews-part-1-the-business/
    Client scoping and requirements
    3

    View full-size slide

  41. @ramimacisabird
    • Budget
    • Motivations
    • Documentation needs
    • Measurement goals
    • Breadth vs. Depth
    • Review your:
    • Risk assessment
    • Threat model
    • Data classification
    Client scoping and requirements
    3
    • Mark Curphey, The Art of Scoping Application Security Reviews
    • 4Armed, Scoping a penetration test
    • PTES, Pre-engagement Interactions
    • Trustwave, Missing Critical Vulnerabilities Through Narrow Scoping

    View full-size slide

  42. @ramimacisabird
    • Follow-on requirements
    ● https://web.archive.org/web/20071207150024/http://securitybuddha.com/2007/08/22/the-art-of-scoping-application-security-reviews-part-1-the-business/
    • Remediation Assistance
    • Assessor requirements
    • Onsite
    • Citizenship
    • Clearance
    • Specific methodologies
    • Certification
    Client scoping and requirements
    3

    View full-size slide

  43. @ramimacisabird
    Gathering Proposals
    4

    View full-size slide

  44. @ramimacisabird
    1. Request for proposals
    2. Shortlisting (3-5)
    3. Initial call
    Gathering Proposals
    4

    View full-size slide

  45. @ramimacisabird
    • Risk Reduction
    -> Flexibility. Focus on collaboration and business risk
    • Compliance
    -> Certification, balance of substance, auditor relationship
    • Internal attestation
    -> Audience, executive summary quality
    • Investment or M&A
    -> Speed to engage, experience with M&A, not your call
    • Sales
    -> Client relationship, brand name, deliverables
    • Post-breach
    -> Incident experience, legal counsel, advisory work
    1 Defining Motivation
    Gathering Proposals
    4

    View full-size slide

  46. @ramimacisabird
    ● Question bank:
    ○ How soon would you be able to staff this engagement?
    ○ What experience do you have with organizations like ours?
    ○ What is your engagement model?
    ■ Collaboration
    ■ Staffing
    ■ Project management
    ■ Methodology and tools
    ● https://owasp.org/www-project-application-security-verification-standard/
    Gathering Proposals
    4

    View full-size slide

  47. @ramimacisabird
    5 Vendor scoping

    View full-size slide

  48. @ramimacisabird
    5 Vendor scoping
    > The penetration test team should identify what testing they
    believe will give a full picture of the vulnerability status of the estate.
    Advice on how to get the most from penetration testing
    National Cybersecurity Centre

    View full-size slide

  49. @ramimacisabird
    Scoping by naive metrics

    View full-size slide

  50. @ramimacisabird
    How Vendors Scope
    1. Questionnaire
    a. or Scan
    b. or Code
    2. Conversation
    3. Demonstration

    View full-size slide

  51. @ramimacisabird
    ● Fixed price or time and materials
    ● Detailed pricing
    ● Earmarked discounts
    ● Payment terms
    ○ Net 30 (or 60, 180, 365, with penalties for late payment)
    ○ Percent upfront (commonly half, either as a deposit or delivered at
    kickoff)
    ○ Milestone based
    Quotes
    5 Vendor scoping

    View full-size slide

  52. @ramimacisabird
    5 Vendor scoping
    Engagement Economics and Security Assessments
    The Guerilla CISO, ryblov

    View full-size slide

  53. @ramimacisabird
    6 Contracting

    View full-size slide

  54. @ramimacisabird
    • Rate
    • Scope
    • Overall Level of Effort
    • Trade depth for breadth
    • Reporting
    • Relationship (volume)
    Negotiation axis
    6 Contracting

    View full-size slide

  55. @ramimacisabird
    6 Contracting

    View full-size slide

  56. @ramimacisabird
    What is the greatest challenge in buying security
    services?
    ● Balancing quality/price/availability
    ○ and differentiating quality
    ○ and justifying spend to management
    ■ comparing oranges and apples
    ○ or even affording it at all
    6 Contracting

    View full-size slide

  57. @ramimacisabird
    • Explicit proposals
    • Like-for-like
    • Reference checks
    • Long-term needs
    • Vet the consultants
    Vetting
    6 Contracting

    View full-size slide

  58. @ramimacisabird
    The paperwork
    • (Mutual) Non-Disclosure Agreement - (m)NDA
    • Master Service Agreement - MSA
    • Statement of Work - SOW
    • Rules of Engagement
    • Common clauses
    • Service Fees; Taxes; Invoicing and Payment
    • Termination
    • Proprietary Rights
    • Confidentiality
    • Warranties; Limitation of Liability; Insurance
    • Indemnification
    6 Contracting

    View full-size slide

  59. @ramimacisabird
    The paperwork
    What to Look For in a Penetration Testing Statement of Work?
    CUSTOMER MASTER SERVICES AGREEMENT
    Checklist: Starting a Security Consulting Firm
    6 Contracting

    View full-size slide

  60. @ramimacisabird
    7 Preparation & Delivery

    View full-size slide

  61. @ramimacisabird
    Logistics Tips
    • “Need about double the resource to manage than you think!”
    • “Always have technical staff work with your procurement team -- and
    that's on both sides (vendor and client).”
    • “Have one primary person for all contact with vendor”
    • “Leverage business initiatives (e.g. new product launches) to fund pentest
    procurement as a capital expenditure, leverage ROI to bring it to routine
    operational budget for the rest of the landscape.”
    7 Preparation & Delivery

    View full-size slide

  62. @ramimacisabird
    Internal alignment
    • Authorization
    • Buy-in
    • Blue team collaboration
    Logistics
    Communication channels
    • Track progress
    • Respond to questions
    • Dispatch
    • Escalation policy
    Known risks
    • Risk assessments
    • Threat models
    • Previous reports
    7 Preparation & Delivery

    View full-size slide

  63. @ramimacisabird
    Technical Preparation
    • Resolve outstanding issues
    • Test environment
    • Integration
    • Configuration
    • Feature flags
    • Roles
    • Seed data
    • Change freeze
    • Out-of-scope controls
    ● APPSEC Cali 2018 - Hunter – Optimize your Pentesters Time
    ● NCC Group - Jerome Smith - The Why Behind Web Application Penetration Test Prerequisites
    7 Preparation & Delivery

    View full-size slide

  64. @ramimacisabird
    Onboarding
    • Hardware
    • Software
    • Remote access
    • Legal, HR, IT
    • Demos, documentation, code
    7 Preparation & Delivery

    View full-size slide

  65. @ramimacisabird
    THE ASSESSMENT

    View full-size slide

  66. @ramimacisabird
    8 The Readout

    View full-size slide

  67. @ramimacisabird
    8 The Readout
    The Report
    • Assessment details: Scope, level of effort, tools and methodology,
    vendor and consultant information
    • Executive summary: overall outcome, including risk posture,
    findings of note, and executive or meta recommendations
    • Findings: details, impact and risk, reproduction information, and
    remediation guidance
    • Appendix: additional information on bug classes, detailed
    remediation steps, custom tools or scripts developed, or raw data from
    testing

    View full-size slide

  68. @ramimacisabird
    •Question bank:
    • If you had more time, where would you dig further or look next?
    • What would you recommend we do differently for our next
    engagement?
    • Were there any trends you observed? Are there any systemic
    mitigations you’d recommend?
    • Were there any areas that were particularly well hardened?
    • How does our posture compare to the
    (industry/benchmark/average engagement)?
    8 The Readout

    View full-size slide

  69. @ramimacisabird
    > Security consulting firms are the only way you have to know how
    you compare to others in your field as only a consulting firm can
    combine trust-based data acquisition with identity-protecting
    pooling of that otherwise unobtainable comparability data.
    Penetration testing: a duet
    Dan Geer & John Harthorne
    8 The Readout

    View full-size slide

  70. @ramimacisabird
    Offboarding
    8 The Readout

    View full-size slide

  71. @ramimacisabird
    Offboarding
    8 The Readout

    View full-size slide

  72. @ramimacisabird
    No findings? No problem!
    •“Manage client expectations”
    • Note limitations
    •Detailed test plan and test coverage
    • “assurance that we have looked diligently”
    • “internal investigation / quality control”
    • “follow up with client for a sanity check”
    •“A bit more conversation on other security related observations and
    best practices that can be deployed given the lack of findings”
    • Highlight true negatives
    8 The Readout

    View full-size slide

  73. @ramimacisabird
    9 Ingestion

    View full-size slide

  74. @ramimacisabird
    9 Ingestion
    ● Use your standard processes
    ● Triage

    View full-size slide

  75. @ramimacisabird
    9 Ingestion
    ● Use your standard processes
    ● Triage
    ○ Root cause analysis
    ○ Variant analysis

    View full-size slide

  76. @ramimacisabird
    9 Ingestion
    ● Use your standard processes
    ● Triage
    ○ Root cause analysis
    ○ Variant analysis
    ● Remediation planning
    ○ Level of effort
    ○ Fix, mitigate, or accept
    ● Parsable reporting
    ● 2020 Global Appsec SF - Clint Gibler & Isaac Evans - Eradicating Vulnerability Classes

    View full-size slide

  77. @ramimacisabird
    10 After the assessment

    View full-size slide

  78. @ramimacisabird
    Remediation
    1
    0
    After the assessment

    View full-size slide

  79. @ramimacisabird
    Retrospective
    1
    0
    After the assessment
    •Question bank:
    • How were their answers to the questions in the readout?
    • How was the report quality?
    • Were there false positives or false negatives?
    • Are there canary bugs?
    • How do you feel about value for price given the category and quality
    of the vendor chosen?
    • How was the vendor’s communication?

    View full-size slide

  80. @ramimacisabird
    1
    1
    Now what?

    View full-size slide

  81. @ramimacisabird
    1
    1
    Now what?
    Testing Cadence
    ● Annual
    ● Quarterly
    ● Development cycle aligned
    ● Compliance aligned
    > “I do them as ongoing events because I dont feel like punching someone in
    the face once a year qualifies as a means for improving ones ability to dodge or
    block a punch.”

    View full-size slide

  82. @ramimacisabird
    Scope and Vendor
    ● Retro breadth and depth
    ○ Listed limitations
    ○ Retargeting can enhance one or the other
    ● Vendor rotation
    1
    1
    Now what?

    View full-size slide

  83. @ramimacisabird
    Scope and Vendor
    ● “switch vendors and sometimes overlap for A/B
    testing”

    ● “go back to the same person for at least 3 years
    running.”
    1
    1
    Now what?

    View full-size slide

  84. @ramimacisabird
    Scope and Vendor
    ● Vendor rotation - Pros
    ■ Cross-vendor comparison
    ■ “vendors work hard on new clients”
    ■ Firms are fungible quality
    ■ Recommended or required by policy or auditors
    ■ Different firms have specialties and methodologies
    ● Penetration Testing in the Financial Services Industry (2010)
    1
    1
    Now what?

    View full-size slide

  85. @ramimacisabird
    Scope and Vendor
    ● Vendor repetition - Pros
    ■ Decreased ramp up time
    ■ Improved project management and comms
    ■ Improved understanding of business risk
    ■ Cost-savings possible on volume or
    relationship
    ■ Expectation of consistency on performance
    ● Penetration Testing in the Financial Services Industry (2010)
    1
    1
    Now what?

    View full-size slide

  86. @ramimacisabird
    Scaling your program
    ● Maximize advantage of leverage
    ○ price, scheduling, and consultant selection
    ● Optimize for project management
    ● Standardize
    ○ Procurement
    ○ Internal customer experience
    ○ Ingestion
    ● Decide when to staff in-house
    ● Define ROI
    1
    1
    Now what?

    View full-size slide

  87. @ramimacisabird
    Scaling your program
    “Select a handful of companies and work on a contract framework with
    them. This lowers the amount of work for procuring individual pentests
    (everything is pre-approved internally & less paperwork).”
    “Make friends with your supplier management people. They can make
    your life easy or difficult.”
    1
    1
    Now what?

    View full-size slide

  88. @ramimacisabird
    Return on Investment
    How are companies calculating…
    1
    1
    Now what?

    View full-size slide

  89. @ramimacisabird
    Return on Investment
    ● “We don't” - verbatim… x3
    ● Quality:
    ○ “Look at the overall quality from the pentest provider over time
    (can't do it for an individual assessment)”
    ○ “Depth of analysis and quality of analysis that goes beyond scanning
    tools.”
    ○ Quality of findings, specifically those that are scalable across our
    company.
    ○ “quality of the assessment, quality of the findings”
    1
    1
    Now what?

    View full-size slide

  90. @ramimacisabird
    Return on Investment
    ● “identify *and* close critical or high bugs … a general sentiment from
    those who hear about it. “
    ● “Risk reduction” / “Aggregate organizational risk identified”
    ● “Value in contributing to sales success” / “$ business lost from
    potential risks”
    ● “1. grading the visibility to areas needing improvement, 2. grading the
    efficacy of monitoring and our response capabilities"”
    1
    1
    Now what?

    View full-size slide

  91. @ramimacisabird
    If I can only offer three pieces of advice…
    1. Be aware of the market for lemons, and think critically about
    how you’ll know whether a proposal is good, and how you’ll
    tell if the assessment delivers value
    2. Structure your assessment carefully based on key motivations,
    to deliver measurable business value
    3. Use assessments not to kill bugs, but to kill bug classes

    View full-size slide

  92. @ramimacisabird
    http://tldrsec.com/guides/buying-security
    https://speakerdeck.com/ramimac/buying-security

    View full-size slide

  93. @ramimacisabird
    with thanks to
    • Adrian ("Time to kill the pen test") Sanabria, Tenchi Security
    • Edward Farrell, Mercury Information Security Services
    • Robert Postill, Privay
    • Elliot Murphy, KindlyOps.com
    • John Cannady, Palo Alto Networks
    • Robert Shala, Sentry Cybersecurity
    • Javier Hijas, Efficience
    • Damien Wilson, Mindglob.com
    • Dan Guido, Trail of Bits
    • Mick Douglas (@BetterSafetyNet), InfoSec Innovations
    • lvh, Latacora
    • Travis McPeak
    • Cristiano Maruti
    • Joel St. John
    • Emil Vaagland
    And anonymous friends from:
    • MTX
    • Prompt Security
    • Carve Systems, an iVision company

    View full-size slide

  94. @ramimacisabird
    Topics that didn’t fit into the
    main slides

    View full-size slide

  95. @ramimacisabird
    “It's fucking difficult and nothing is
    consistent across the industry.”

    View full-size slide

  96. @ramimacisabird
    5 Vendor scoping

    View full-size slide

  97. @ramimacisabird
    Buyer Statistics

    View full-size slide

  98. @ramimacisabird
    Quality

    View full-size slide

  99. @ramimacisabird
    On Red Teaming
    •External Pressures:
    • “Regulatory requirements, insurance requirements”
    • “management said so”
    • “Company board couldn’t be convinced otherwise”
    • “It's just something we do periodically”
    • “Regulatory drivers”
    • “Internal rules demand one Red team per year”

    View full-size slide

  100. @ramimacisabird
    On Red Teaming
    •Maturity curve:
    • “known risks were mitigated with preventative or detective
    controls”
    • “Self assessments indicated we were mature enough”
    • “once defensive controls are in a strong position and are ready to be
    put to the test”
    • “Results from prior tests/assessments showing maturity”

    View full-size slide

  101. @ramimacisabird
    On Red Teaming
    •Chained threats and external validation:
    • “We were concerned about abuse cases of an internal CI/CD service
    based on our own experience, but the company continued heavy
    investments in this internal tool. We wanted external validation of
    the criticality of the issues, so we could use it for making our case
    internally. It helped.”
    • “We had a large call center and wanted to clue in leadership that our
    controls around it were insufficient.”
    • “individual pentest failed to show the big picture about how
    vulnerable we were as a whole”

    View full-size slide

  102. @ramimacisabird
    Scaling your program
    > I ran a trial at Facebook where 10 security consulting companies
    audited the same code. Code my team had already carefully audited. All
    10 found the same pool of shallow bugs (about half) but the remaining
    issues were all over the map, including one we ourselves had missed. Each
    person brings their own long tail of security knowledge to bear. Contrast
    this with something like performance (another attribute of “quality” in
    software) where it is trivial to measure progress.
    Why Product Security is Hard
    Collin Greene
    1
    1
    Now what?

    View full-size slide

  103. @ramimacisabird
    Challenges
    ● Collaborating with engineering to prepare
    ○ including a reasonable test environment with reasonable
    level of effort
    ● Managing logistics across the overall program
    ● Lack of standardization
    ● Scoping
    ● Remediation
    ○ And infosec team bandwidth

    View full-size slide

  104. 💰
    Buying Security
    A Client’s Guide
    Keep up with security research
    tldrsec.com
    By Rami McCarthy
    @ramimacisabird

    View full-size slide