Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Buying Security

Buying Security

You can’t buy security, but vendors play a key role in effective security programs. This talk will provide a comprehensive guide to buying and getting value, based on experiences on both sides of the marketplace, a comprehensive literature review, and a survey of clients and vendors of all stripes.

Rami McCarthy

June 04, 2022

More Decks by Rami McCarthy

Other Decks in Technology


  1. @ramimacisabird 👋 hello • Security @ series d health-tech •

    Reformed security consultant • with thanks to I’m Rami McCarthy
  2. @ramimacisabird Challenges “There is too little reliable information about the

    type and quality of services offered, and too little knowledge about how to sell and buy them.”
  3. @ramimacisabird Security Assessment > a consulting engagement focused on evaluating

    security design, architecture, and/or implemented controls to identify whether they are operating as intended and helping the organization to meet its security requirements NIST
  4. @ramimacisabird What will you get? • ~200 resources synthesized •

    ~100 security professionals surveyed • A comprehensive guide to buying and getting value from security services
  5. @ramimacisabird Types of Security Assessments • Vulnerability assessment • Penetration

    testing • White-box (“full knowledge”) • Grey-box (“partial knowledge”) • Black-box (“no knowledge”) • Code review • Threat Model • Red Team (Adversary Simulation) • Social Engineering (Phishing, Vishing, Smishing) • Technical Specialty (hardware, cryptography, cryptocurrency) • The Difference Between a Vulnerability Assessment and a Penetration Test, Daniel Miessler • What Are The Different Types Of Penetration Testing?, Jason Firch, PurpleSec.us
  6. @ramimacisabird GOTO 1 Security Assessment Process 8 The Readout 6

    Contracting Gathering Proposals 4 2 Finding Vendors 9 Ingestion 7 Preparation & Delivery 5 Vendor scoping Client scoping and requirements 3 1 Defining Motivation 1 0 After the assessment 1 1 Now what?
  7. @ramimacisabird • Risk Reduction • Compliance • Internal attestation •

    Investment or M&A • Sales • Post-breach 1 Defining Motivation
  8. @ramimacisabird • Risk Reduction • Compliance • Internal attestation •

    Investment or M&A • Sales • Post-breach 1 Defining Motivation
  9. @ramimacisabird Taxonomy Types of Vendors •Global/enterprise consulting •Cybersecurity services •Boutique

    •Specialty •Sole practitioner •Researcher/bug hunter •Low cost •Managed Security Service Provider •Value Added Reseller
  10. @ramimacisabird What is the greatest challenge in buying security services?

    • Finding good vendors, who provide significant value ◦ who are available when you need them ◦ who can support specific systems or architecture ◦ who provide consistent quality staff 2 Finding Vendors
  11. @ramimacisabird •Network recommendations •Follow-the-leader •Research •Conference speakers •Published research •Prominent

    staff •Public reports •Compliance approved •Assessment standards work •Certifications •Analyst recommendations • https://github.com/juliocesarfort/public-pentesting-reports • http://www.pentest-standard.org/index.php/FAQ#Q:_Who_is_involved_with_this_standard.3F 2 Finding Vendors
  12. @ramimacisabird Client scoping and requirements 3 > Scope management is

    the process of defining what work is required, and then making sure that all of that work, and only that work, is done. SANS Scoping Security Assessments - A Project Management Approach
  13. @ramimacisabird > Strike a balance between performing a comprehensive set

    of tests and evaluating functionality and features that present the greatest risk. GSA IT Security Procedural Guide: Conducting Penetration Test Exercises Client scoping and requirements 3
  14. @ramimacisabird • Budget • Motivations • Documentation needs • Measurement

    goals • Breadth vs. Depth • Review your: • Risk assessment • Threat model • Data classification Client scoping and requirements 3 • Mark Curphey, The Art of Scoping Application Security Reviews • 4Armed, Scoping a penetration test • PTES, Pre-engagement Interactions • Trustwave, Missing Critical Vulnerabilities Through Narrow Scoping
  15. @ramimacisabird • Risk Reduction -> Flexibility. Focus on collaboration and

    business risk • Compliance -> Certification, balance of substance, auditor relationship • Internal attestation -> Audience, executive summary quality • Investment or M&A -> Speed to engage, experience with M&A, not your call • Sales -> Client relationship, brand name, deliverables • Post-breach -> Incident experience, legal counsel, advisory work 1 Defining Motivation Gathering Proposals 4
  16. @ramimacisabird • Question bank: ◦ How soon would you be

    able to staff this engagement? ◦ What experience do you have with organizations like ours? ◦ What is your engagement model? ▪ Collaboration ▪ Staffing ▪ Project management ▪ Methodology and tools • https://owasp.org/www-project-application-security-verification-standard/ Gathering Proposals 4
  17. @ramimacisabird 5 Vendor scoping > The penetration test team should

    identify what testing they believe will give a full picture of the vulnerability status of the estate. Advice on how to get the most from penetration testing National Cybersecurity Centre
  18. @ramimacisabird How Vendors Scope 1. Questionnaire a. or Scan b.

    or Code 2. Conversation 3. Demonstration
  19. @ramimacisabird • Fixed price or time and materials • Detailed

    pricing • Earmarked discounts • Payment terms ◦ Net 30 (or 60, 180, 365, with penalties for late payment) ◦ Percent upfront (commonly half, either as a deposit or delivered at kickoff) ◦ Milestone based Quotes 5 Vendor scoping
  20. @ramimacisabird • Rate • Scope • Overall Level of Effort

    • Trade depth for breadth • Reporting • Relationship (volume) Negotiation axis 6 Contracting
  21. @ramimacisabird What is the greatest challenge in buying security services?

    • Balancing quality/price/availability ◦ and differentiating quality ◦ and justifying spend to management ▪ comparing oranges and apples ◦ or even affording it at all 6 Contracting
  22. @ramimacisabird • Explicit proposals • Like-for-like • Reference checks •

    Long-term needs • Vet the consultants Vetting 6 Contracting
  23. @ramimacisabird The paperwork • (Mutual) Non-Disclosure Agreement - (m)NDA •

    Master Service Agreement - MSA • Statement of Work - SOW • Rules of Engagement • Common clauses • Service Fees; Taxes; Invoicing and Payment • Termination • Proprietary Rights • Confidentiality • Warranties; Limitation of Liability; Insurance • Indemnification 6 Contracting
  24. @ramimacisabird The paperwork What to Look For in a Penetration

    Testing Statement of Work? CUSTOMER MASTER SERVICES AGREEMENT Checklist: Starting a Security Consulting Firm 6 Contracting
  25. @ramimacisabird Logistics Tips • “Need about double the resource to

    manage than you think!” • “Always have technical staff work with your procurement team -- and that's on both sides (vendor and client).” • “Have one primary person for all contact with vendor” • “Leverage business initiatives (e.g. new product launches) to fund pentest procurement as a capital expenditure, leverage ROI to bring it to routine operational budget for the rest of the landscape.” 7 Preparation & Delivery
  26. @ramimacisabird Internal alignment • Authorization • Buy-in • Blue team

    collaboration Logistics Communication channels • Track progress • Respond to questions • Dispatch • Escalation policy Known risks • Risk assessments • Threat models • Previous reports 7 Preparation & Delivery
  27. @ramimacisabird Technical Preparation • Resolve outstanding issues • Test environment

    • Integration • Configuration • Feature flags • Roles • Seed data • Change freeze • Out-of-scope controls • APPSEC Cali 2018 - Hunter – Optimize your Pentesters Time • NCC Group - Jerome Smith - The Why Behind Web Application Penetration Test Prerequisites 7 Preparation & Delivery
  28. @ramimacisabird Onboarding • Hardware • Software • Remote access •

    Legal, HR, IT • Demos, documentation, code 7 Preparation & Delivery
  29. @ramimacisabird 8 The Readout The Report • Assessment details: Scope,

    level of effort, tools and methodology, vendor and consultant information • Executive summary: overall outcome, including risk posture, findings of note, and executive or meta recommendations • Findings: details, impact and risk, reproduction information, and remediation guidance • Appendix: additional information on bug classes, detailed remediation steps, custom tools or scripts developed, or raw data from testing
  30. @ramimacisabird •Question bank: • If you had more time, where

    would you dig further or look next? • What would you recommend we do differently for our next engagement? • Were there any trends you observed? Are there any systemic mitigations you’d recommend? • Were there any areas that were particularly well hardened? • How does our posture compare to the (industry/benchmark/average engagement)? 8 The Readout
  31. @ramimacisabird > Security consulting firms are the only way you

    have to know how you compare to others in your field as only a consulting firm can combine trust-based data acquisition with identity-protecting pooling of that otherwise unobtainable comparability data. Penetration testing: a duet Dan Geer & John Harthorne 8 The Readout
  32. @ramimacisabird No findings? No problem! •“Manage client expectations” • Note

    limitations •Detailed test plan and test coverage • “assurance that we have looked diligently” • “internal investigation / quality control” • “follow up with client for a sanity check” •“A bit more conversation on other security related observations and best practices that can be deployed given the lack of findings” • Highlight true negatives 8 The Readout
  33. @ramimacisabird 9 Ingestion • Use your standard processes • Triage

    ◦ Root cause analysis ◦ Variant analysis
  34. @ramimacisabird 9 Ingestion • Use your standard processes • Triage

    ◦ Root cause analysis ◦ Variant analysis • Remediation planning ◦ Level of effort ◦ Fix, mitigate, or accept • Parsable reporting • 2020 Global Appsec SF - Clint Gibler & Isaac Evans - Eradicating Vulnerability Classes
  35. @ramimacisabird Retrospective 1 0 After the assessment •Question bank: •

    How were their answers to the questions in the readout? • How was the report quality? • Were there false positives or false negatives? • Are there canary bugs? • How do you feel about value for price given the category and quality of the vendor chosen? • How was the vendor’s communication?
  36. @ramimacisabird 1 1 Now what? Testing Cadence • Annual •

    Quarterly • Development cycle aligned • Compliance aligned > “I do them as ongoing events because I dont feel like punching someone in the face once a year qualifies as a means for improving ones ability to dodge or block a punch.”
  37. @ramimacisabird Scope and Vendor • Retro breadth and depth ◦

    Listed limitations ◦ Retargeting can enhance one or the other • Vendor rotation 1 1 Now what?
  38. @ramimacisabird Scope and Vendor • “switch vendors and sometimes overlap

    for A/B testing” … • “go back to the same person for at least 3 years running.” 1 1 Now what?
  39. @ramimacisabird Scope and Vendor • Vendor rotation - Pros ▪

    Cross-vendor comparison ▪ “vendors work hard on new clients” ▪ Firms are fungible quality ▪ Recommended or required by policy or auditors ▪ Different firms have specialties and methodologies • Penetration Testing in the Financial Services Industry (2010) 1 1 Now what?
  40. @ramimacisabird Scope and Vendor • Vendor repetition - Pros ▪

    Decreased ramp up time ▪ Improved project management and comms ▪ Improved understanding of business risk ▪ Cost-savings possible on volume or relationship ▪ Expectation of consistency on performance • Penetration Testing in the Financial Services Industry (2010) 1 1 Now what?
  41. @ramimacisabird Scaling your program • Maximize advantage of leverage ◦

    price, scheduling, and consultant selection • Optimize for project management • Standardize ◦ Procurement ◦ Internal customer experience ◦ Ingestion • Decide when to staff in-house • Define ROI 1 1 Now what?
  42. @ramimacisabird Scaling your program “Select a handful of companies and

    work on a contract framework with them. This lowers the amount of work for procuring individual pentests (everything is pre-approved internally & less paperwork).” “Make friends with your supplier management people. They can make your life easy or difficult.” 1 1 Now what?
  43. @ramimacisabird Return on Investment • “We don't” - verbatim… x3

    • Quality: ◦ “Look at the overall quality from the pentest provider over time (can't do it for an individual assessment)” ◦ “Depth of analysis and quality of analysis that goes beyond scanning tools.” ◦ Quality of findings, specifically those that are scalable across our company. ◦ “quality of the assessment, quality of the findings” 1 1 Now what?
  44. @ramimacisabird Return on Investment • “identify *and* close critical or

    high bugs … a general sentiment from those who hear about it. “ • “Risk reduction” / “Aggregate organizational risk identified” • “Value in contributing to sales success” / “$ business lost from potential risks” • “1. grading the visibility to areas needing improvement, 2. grading the efficacy of monitoring and our response capabilities"” 1 1 Now what?
  45. @ramimacisabird If I can only offer three pieces of advice…

    1. Be aware of the market for lemons, and think critically about how you’ll know whether a proposal is good, and how you’ll tell if the assessment delivers value 2. Structure your assessment carefully based on key motivations, to deliver measurable business value 3. Use assessments not to kill bugs, but to kill bug classes
  46. @ramimacisabird with thanks to • Adrian ("Time to kill the

    pen test") Sanabria, Tenchi Security • Edward Farrell, Mercury Information Security Services • Robert Postill, Privay • Elliot Murphy, KindlyOps.com • John Cannady, Palo Alto Networks • Robert Shala, Sentry Cybersecurity • Javier Hijas, Efficience • Damien Wilson, Mindglob.com • Dan Guido, Trail of Bits • Mick Douglas (@BetterSafetyNet), InfoSec Innovations • lvh, Latacora • Travis McPeak • Cristiano Maruti • Joel St. John • Emil Vaagland And anonymous friends from: • MTX • Prompt Security • Carve Systems, an iVision company
  47. @ramimacisabird On Red Teaming •External Pressures: • “Regulatory requirements, insurance

    requirements” • “management said so” • “Company board couldn’t be convinced otherwise” • “It's just something we do periodically” • “Regulatory drivers” • “Internal rules demand one Red team per year”
  48. @ramimacisabird On Red Teaming •Maturity curve: • “known risks were

    mitigated with preventative or detective controls” • “Self assessments indicated we were mature enough” • “once defensive controls are in a strong position and are ready to be put to the test” • “Results from prior tests/assessments showing maturity”
  49. @ramimacisabird On Red Teaming •Chained threats and external validation: •

    “We were concerned about abuse cases of an internal CI/CD service based on our own experience, but the company continued heavy investments in this internal tool. We wanted external validation of the criticality of the issues, so we could use it for making our case internally. It helped.” • “We had a large call center and wanted to clue in leadership that our controls around it were insufficient.” • “individual pentest failed to show the big picture about how vulnerable we were as a whole”
  50. @ramimacisabird Scaling your program > I ran a trial at

    Facebook where 10 security consulting companies audited the same code. Code my team had already carefully audited. All 10 found the same pool of shallow bugs (about half) but the remaining issues were all over the map, including one we ourselves had missed. Each person brings their own long tail of security knowledge to bear. Contrast this with something like performance (another attribute of “quality” in software) where it is trivial to measure progress. Why Product Security is Hard Collin Greene 1 1 Now what?
  51. @ramimacisabird Challenges • Collaborating with engineering to prepare ◦ including

    a reasonable test environment with reasonable level of effort • Managing logistics across the overall program • Lack of standardization • Scoping • Remediation ◦ And infosec team bandwidth
  52. 💰 Buying Security A Client’s Guide Keep up with security

    research tldrsec.com By Rami McCarthy @ramimacisabird