Buying Security

@ramimacisabird BSidesSF 2022 Buying Security A Client’s Guide Rami McCarthy

@ramimacisabird 👋 hello • Security @ series d health-tech •
Reformed security consultant • with thanks to I’m Rami McCarthy

@ramimacisabird

@ramimacisabird Why are we here? Security services are important and
buying them is hard

@ramimacisabird Challenges “There is too little reliable information about the
type and quality of services offered, and too little knowledge about how to sell and buy them.”

@ramimacisabird

@ramimacisabird Security Assessment > a consulting engagement focused on evaluating
security design, architecture, and/or implemented controls to identify whether they are operating as intended and helping the organization to meet its security requirements NIST

@ramimacisabird What will you get? • ~200 resources synthesized •
~100 security professionals surveyed • A comprehensive guide to buying and getting value from security services

@ramimacisabird

@ramimacisabird Why are we here?

@ramimacisabird Penetration Testing Considered Harmful Today (2012) Thinkst’s Haroon Meer,
44CON 2011 [video]

@ramimacisabird Point/Counterpoint: Penetration Testing Counterpoint: Marcus Ranum, 2007 Application Security
Tools: Good or Bad? Gary McGraw, Freedom to Tinker, 2006

@ramimacisabird The average security services vendor delivers a quality assessment

@ramimacisabird The average security services vendor delivers a quality assessment
Buyers Sellers 2.64 3.14

@ramimacisabird Quality

@ramimacisabird Security Assessments

@ramimacisabird Patrick Thomas Types of Security Assessments

@ramimacisabird Types of Security Assessments • Vulnerability assessment • Penetration
testing • White-box (“full knowledge”) • Grey-box (“partial knowledge”) • Black-box (“no knowledge”) • Code review • Threat Model • Red Team (Adversary Simulation) • Social Engineering (Phishing, Vishing, Smishing) • Technical Specialty (hardware, cryptography, cryptocurrency) • The Difference Between a Vulnerability Assessment and a Penetration Test, Daniel Miessler • What Are The Different Types Of Penetration Testing?, Jason Firch, PurpleSec.us

@ramimacisabird On newer models…

@ramimacisabird “Getting tough to find something that's not just a
dressed up Nessus scan.”

@ramimacisabird GOTO 1 Security Assessment Process 8 The Readout 6
Contracting Gathering Proposals 4 2 Finding Vendors 9 Ingestion 7 Preparation & Delivery 5 Vendor scoping Client scoping and requirements 3 1 Defining Motivation 1 0 After the assessment 1 1 Now what?

@ramimacisabird 1 Defining Motivation

@ramimacisabird • Risk Reduction • Compliance • Internal attestation •
Investment or M&A • Sales • Post-breach 1 Defining Motivation

@ramimacisabird 2 Finding Vendors https://twitter.com/cure53berlin/status/976463307942088706

@ramimacisabird Taxonomy of Vendors

@ramimacisabird Taxonomy Types of Vendors •Global/enterprise consulting •Cybersecurity services •Boutique
•Specialty •Sole practitioner •Researcher/bug hunter •Low cost •Managed Security Service Provider •Value Added Reseller

@ramimacisabird What is the greatest challenge in buying security services?
• Finding good vendors, who provide significant value ◦ who are available when you need them ◦ who can support specific systems or architecture ◦ who provide consistent quality staff 2 Finding Vendors

@ramimacisabird •Network recommendations •Follow-the-leader •Research •Conference speakers •Published research •Prominent
staff •Public reports •Compliance approved •Assessment standards work •Certifications •Analyst recommendations • https://github.com/juliocesarfort/public-pentesting-reports • http://www.pentest-standard.org/index.php/FAQ#Q:_Who_is_involved_with_this_standard.3F 2 Finding Vendors

@ramimacisabird • https://github.com/juliocesarfort/public-pentesting-reports • http://www.pentest-standard.org/index.php/FAQ#Q:_Who_is_involved_with_this_standard.3F 2 Finding Vendors

@ramimacisabird Client scoping and requirements 3

@ramimacisabird Client scoping and requirements 3 > Scope management is
the process of defining what work is required, and then making sure that all of that work, and only that work, is done. SANS Scoping Security Assessments - A Project Management Approach

@ramimacisabird > Strike a balance between performing a comprehensive set
of tests and evaluating functionality and features that present the greatest risk. GSA IT Security Procedural Guide: Conducting Penetration Test Exercises Client scoping and requirements 3

@ramimacisabird • Budget • https://web.archive.org/web/20071207150024/http://securitybuddha.com/2007/08/22/the-art-of-scoping-application-security-reviews-part-1-the-business/ Client scoping and requirements 3

@ramimacisabird • Budget • Motivations • Documentation needs • Measurement
goals • Breadth vs. Depth • Review your: • Risk assessment • Threat model • Data classification Client scoping and requirements 3 • Mark Curphey, The Art of Scoping Application Security Reviews • 4Armed, Scoping a penetration test • PTES, Pre-engagement Interactions • Trustwave, Missing Critical Vulnerabilities Through Narrow Scoping

@ramimacisabird • Follow-on requirements • https://web.archive.org/web/20071207150024/http://securitybuddha.com/2007/08/22/the-art-of-scoping-application-security-reviews-part-1-the-business/ • Remediation Assistance •
Assessor requirements • Onsite • Citizenship • Clearance • Specific methodologies • Certification Client scoping and requirements 3

@ramimacisabird Gathering Proposals 4

@ramimacisabird 1. Request for proposals 2. Shortlisting (3-5) 3. Initial
call Gathering Proposals 4

@ramimacisabird • Risk Reduction -> Flexibility. Focus on collaboration and
business risk • Compliance -> Certification, balance of substance, auditor relationship • Internal attestation -> Audience, executive summary quality • Investment or M&A -> Speed to engage, experience with M&A, not your call • Sales -> Client relationship, brand name, deliverables • Post-breach -> Incident experience, legal counsel, advisory work 1 Defining Motivation Gathering Proposals 4

@ramimacisabird • Question bank: ◦ How soon would you be
able to staff this engagement? ◦ What experience do you have with organizations like ours? ◦ What is your engagement model? ▪ Collaboration ▪ Staffing ▪ Project management ▪ Methodology and tools • https://owasp.org/www-project-application-security-verification-standard/ Gathering Proposals 4

@ramimacisabird 5 Vendor scoping

@ramimacisabird 5 Vendor scoping > The penetration test team should
identify what testing they believe will give a full picture of the vulnerability status of the estate. Advice on how to get the most from penetration testing National Cybersecurity Centre

@ramimacisabird Scoping by naive metrics

@ramimacisabird How Vendors Scope 1. Questionnaire a. or Scan b.
or Code 2. Conversation 3. Demonstration

@ramimacisabird • Fixed price or time and materials • Detailed
pricing • Earmarked discounts • Payment terms ◦ Net 30 (or 60, 180, 365, with penalties for late payment) ◦ Percent upfront (commonly half, either as a deposit or delivered at kickoff) ◦ Milestone based Quotes 5 Vendor scoping

@ramimacisabird 5 Vendor scoping Engagement Economics and Security Assessments The
Guerilla CISO, ryblov

@ramimacisabird 6 Contracting

@ramimacisabird • Rate • Scope • Overall Level of Effort
• Trade depth for breadth • Reporting • Relationship (volume) Negotiation axis 6 Contracting

@ramimacisabird 6 Contracting

@ramimacisabird What is the greatest challenge in buying security services?
• Balancing quality/price/availability ◦ and differentiating quality ◦ and justifying spend to management ▪ comparing oranges and apples ◦ or even affording it at all 6 Contracting

@ramimacisabird • Explicit proposals • Like-for-like • Reference checks •
Long-term needs • Vet the consultants Vetting 6 Contracting

@ramimacisabird The paperwork • (Mutual) Non-Disclosure Agreement - (m)NDA •
Master Service Agreement - MSA • Statement of Work - SOW • Rules of Engagement • Common clauses • Service Fees; Taxes; Invoicing and Payment • Termination • Proprietary Rights • Confidentiality • Warranties; Limitation of Liability; Insurance • Indemnification 6 Contracting

@ramimacisabird The paperwork What to Look For in a Penetration
Testing Statement of Work? CUSTOMER MASTER SERVICES AGREEMENT Checklist: Starting a Security Consulting Firm 6 Contracting

@ramimacisabird 7 Preparation & Delivery

@ramimacisabird Logistics Tips • “Need about double the resource to
manage than you think!” • “Always have technical staff work with your procurement team -- and that's on both sides (vendor and client).” • “Have one primary person for all contact with vendor” • “Leverage business initiatives (e.g. new product launches) to fund pentest procurement as a capital expenditure, leverage ROI to bring it to routine operational budget for the rest of the landscape.” 7 Preparation & Delivery

@ramimacisabird Internal alignment • Authorization • Buy-in • Blue team
collaboration Logistics Communication channels • Track progress • Respond to questions • Dispatch • Escalation policy Known risks • Risk assessments • Threat models • Previous reports 7 Preparation & Delivery

@ramimacisabird Technical Preparation • Resolve outstanding issues • Test environment
• Integration • Configuration • Feature flags • Roles • Seed data • Change freeze • Out-of-scope controls • APPSEC Cali 2018 - Hunter – Optimize your Pentesters Time • NCC Group - Jerome Smith - The Why Behind Web Application Penetration Test Prerequisites 7 Preparation & Delivery

@ramimacisabird Onboarding • Hardware • Software • Remote access •
Legal, HR, IT • Demos, documentation, code 7 Preparation & Delivery

@ramimacisabird THE ASSESSMENT

@ramimacisabird 8 The Readout

@ramimacisabird 8 The Readout The Report • Assessment details: Scope,
level of effort, tools and methodology, vendor and consultant information • Executive summary: overall outcome, including risk posture, findings of note, and executive or meta recommendations • Findings: details, impact and risk, reproduction information, and remediation guidance • Appendix: additional information on bug classes, detailed remediation steps, custom tools or scripts developed, or raw data from testing

@ramimacisabird •Question bank: • If you had more time, where
would you dig further or look next? • What would you recommend we do differently for our next engagement? • Were there any trends you observed? Are there any systemic mitigations you’d recommend? • Were there any areas that were particularly well hardened? • How does our posture compare to the (industry/benchmark/average engagement)? 8 The Readout

@ramimacisabird > Security consulting firms are the only way you
have to know how you compare to others in your field as only a consulting firm can combine trust-based data acquisition with identity-protecting pooling of that otherwise unobtainable comparability data. Penetration testing: a duet Dan Geer & John Harthorne 8 The Readout

@ramimacisabird Offboarding 8 The Readout

@ramimacisabird No findings? No problem! •“Manage client expectations” • Note
limitations •Detailed test plan and test coverage • “assurance that we have looked diligently” • “internal investigation / quality control” • “follow up with client for a sanity check” •“A bit more conversation on other security related observations and best practices that can be deployed given the lack of findings” • Highlight true negatives 8 The Readout

@ramimacisabird 9 Ingestion

@ramimacisabird 9 Ingestion • Use your standard processes • Triage

◦ Root cause analysis ◦ Variant analysis

◦ Root cause analysis ◦ Variant analysis • Remediation planning ◦ Level of effort ◦ Fix, mitigate, or accept • Parsable reporting • 2020 Global Appsec SF - Clint Gibler & Isaac Evans - Eradicating Vulnerability Classes

@ramimacisabird 10 After the assessment

@ramimacisabird Remediation 1 0 After the assessment

@ramimacisabird Retrospective 1 0 After the assessment •Question bank: •
How were their answers to the questions in the readout? • How was the report quality? • Were there false positives or false negatives? • Are there canary bugs? • How do you feel about value for price given the category and quality of the vendor chosen? • How was the vendor’s communication?

@ramimacisabird 1 1 Now what?

@ramimacisabird 1 1 Now what? Testing Cadence • Annual •
Quarterly • Development cycle aligned • Compliance aligned > “I do them as ongoing events because I dont feel like punching someone in the face once a year qualifies as a means for improving ones ability to dodge or block a punch.”

@ramimacisabird Scope and Vendor • Retro breadth and depth ◦
Listed limitations ◦ Retargeting can enhance one or the other • Vendor rotation 1 1 Now what?

@ramimacisabird Scope and Vendor • “switch vendors and sometimes overlap
for A/B testing” … • “go back to the same person for at least 3 years running.” 1 1 Now what?

@ramimacisabird Scope and Vendor • Vendor rotation - Pros ▪
Cross-vendor comparison ▪ “vendors work hard on new clients” ▪ Firms are fungible quality ▪ Recommended or required by policy or auditors ▪ Different firms have specialties and methodologies • Penetration Testing in the Financial Services Industry (2010) 1 1 Now what?

@ramimacisabird Scope and Vendor • Vendor repetition - Pros ▪
Decreased ramp up time ▪ Improved project management and comms ▪ Improved understanding of business risk ▪ Cost-savings possible on volume or relationship ▪ Expectation of consistency on performance • Penetration Testing in the Financial Services Industry (2010) 1 1 Now what?

@ramimacisabird Scaling your program • Maximize advantage of leverage ◦
price, scheduling, and consultant selection • Optimize for project management • Standardize ◦ Procurement ◦ Internal customer experience ◦ Ingestion • Decide when to staff in-house • Define ROI 1 1 Now what?

@ramimacisabird Scaling your program “Select a handful of companies and
work on a contract framework with them. This lowers the amount of work for procuring individual pentests (everything is pre-approved internally & less paperwork).” “Make friends with your supplier management people. They can make your life easy or difficult.” 1 1 Now what?

@ramimacisabird Return on Investment How are companies calculating… 1 1
Now what?

@ramimacisabird Return on Investment • “We don't” - verbatim… x3
• Quality: ◦ “Look at the overall quality from the pentest provider over time (can't do it for an individual assessment)” ◦ “Depth of analysis and quality of analysis that goes beyond scanning tools.” ◦ Quality of findings, specifically those that are scalable across our company. ◦ “quality of the assessment, quality of the findings” 1 1 Now what?

@ramimacisabird Return on Investment • “identify *and* close critical or
high bugs … a general sentiment from those who hear about it. “ • “Risk reduction” / “Aggregate organizational risk identified” • “Value in contributing to sales success” / “$ business lost from potential risks” • “1. grading the visibility to areas needing improvement, 2. grading the efficacy of monitoring and our response capabilities"” 1 1 Now what?

@ramimacisabird If I can only offer three pieces of advice…
1. Be aware of the market for lemons, and think critically about how you’ll know whether a proposal is good, and how you’ll tell if the assessment delivers value 2. Structure your assessment carefully based on key motivations, to deliver measurable business value 3. Use assessments not to kill bugs, but to kill bug classes

@ramimacisabird http://tldrsec.com/guides/buying-security https://speakerdeck.com/ramimac/buying-security

@ramimacisabird with thanks to • Adrian ("Time to kill the
pen test") Sanabria, Tenchi Security • Edward Farrell, Mercury Information Security Services • Robert Postill, Privay • Elliot Murphy, KindlyOps.com • John Cannady, Palo Alto Networks • Robert Shala, Sentry Cybersecurity • Javier Hijas, Efficience • Damien Wilson, Mindglob.com • Dan Guido, Trail of Bits • Mick Douglas (@BetterSafetyNet), InfoSec Innovations • lvh, Latacora • Travis McPeak • Cristiano Maruti • Joel St. John • Emil Vaagland And anonymous friends from: • MTX • Prompt Security • Carve Systems, an iVision company

@ramimacisabird Topics that didn’t fit into the main slides

@ramimacisabird “It's fucking difficult and nothing is consistent across the
industry.”

@ramimacisabird 5 Vendor scoping

@ramimacisabird Buyer Statistics

@ramimacisabird Quality

@ramimacisabird On Red Teaming •External Pressures: • “Regulatory requirements, insurance
requirements” • “management said so” • “Company board couldn’t be convinced otherwise” • “It's just something we do periodically” • “Regulatory drivers” • “Internal rules demand one Red team per year”

@ramimacisabird On Red Teaming •Maturity curve: • “known risks were
mitigated with preventative or detective controls” • “Self assessments indicated we were mature enough” • “once defensive controls are in a strong position and are ready to be put to the test” • “Results from prior tests/assessments showing maturity”

@ramimacisabird On Red Teaming •Chained threats and external validation: •
“We were concerned about abuse cases of an internal CI/CD service based on our own experience, but the company continued heavy investments in this internal tool. We wanted external validation of the criticality of the issues, so we could use it for making our case internally. It helped.” • “We had a large call center and wanted to clue in leadership that our controls around it were insufficient.” • “individual pentest failed to show the big picture about how vulnerable we were as a whole”

@ramimacisabird Scaling your program > I ran a trial at
Facebook where 10 security consulting companies audited the same code. Code my team had already carefully audited. All 10 found the same pool of shallow bugs (about half) but the remaining issues were all over the map, including one we ourselves had missed. Each person brings their own long tail of security knowledge to bear. Contrast this with something like performance (another attribute of “quality” in software) where it is trivial to measure progress. Why Product Security is Hard Collin Greene 1 1 Now what?

@ramimacisabird Challenges • Collaborating with engineering to prepare ◦ including
a reasonable test environment with reasonable level of effort • Managing logistics across the overall program • Lack of standardization • Scoping • Remediation ◦ And infosec team bandwidth

💰 Buying Security A Client’s Guide Keep up with security
research tldrsec.com By Rami McCarthy @ramimacisabird

Buying Security

Buying Security

More Decks by Rami McCarthy

Other Decks in Technology

Featured

Transcript