Buying Security - Speaker Deck

Slide 1

Slide 1 text

@ramimacisabird BSidesSF 2022 Buying Security A Client’s Guide Rami McCarthy

Slide 2

Slide 2 text

@ramimacisabird 👋 hello • Security @ series d health-tech • Reformed security consultant • with thanks to I’m Rami McCarthy

Slide 3

Slide 3 text

@ramimacisabird

Slide 4

Slide 4 text

@ramimacisabird Why are we here? Security services are important and buying them is hard

Slide 5

Slide 5 text

@ramimacisabird Challenges “There is too little reliable information about the type and quality of services offered, and too little knowledge about how to sell and buy them.”

Slide 6

Slide 6 text

@ramimacisabird

Slide 7

Slide 7 text

@ramimacisabird Security Assessment > a consulting engagement focused on evaluating security design, architecture, and/or implemented controls to identify whether they are operating as intended and helping the organization to meet its security requirements NIST

Slide 8

Slide 8 text

@ramimacisabird What will you get? ● ~200 resources synthesized ● ~100 security professionals surveyed ● A comprehensive guide to buying and getting value from security services

Slide 9

Slide 9 text

@ramimacisabird

Slide 10

Slide 10 text

@ramimacisabird

Slide 11

Slide 11 text

@ramimacisabird

Slide 12

Slide 12 text

@ramimacisabird

Slide 13

Slide 13 text

@ramimacisabird

Slide 14

Slide 14 text

@ramimacisabird Why are we here?

Slide 15

Slide 15 text

@ramimacisabird Penetration Testing Considered Harmful Today (2012) Thinkst’s Haroon Meer, 44CON 2011 [video]

Slide 16

Slide 16 text

@ramimacisabird Point/Counterpoint: Penetration Testing Counterpoint: Marcus Ranum, 2007 Application Security Tools: Good or Bad? Gary McGraw, Freedom to Tinker, 2006

Slide 17

Slide 17 text

@ramimacisabird The average security services vendor delivers a quality assessment

Slide 18

Slide 18 text

@ramimacisabird The average security services vendor delivers a quality assessment Buyers Sellers 2.64 3.14

Slide 19

Slide 19 text

@ramimacisabird Quality

Slide 20

Slide 20 text

@ramimacisabird Security Assessments

Slide 21

Slide 21 text

@ramimacisabird Patrick Thomas Types of Security Assessments

Slide 22

Slide 22 text

@ramimacisabird Types of Security Assessments • Vulnerability assessment • Penetration testing • White-box (“full knowledge”) • Grey-box (“partial knowledge”) • Black-box (“no knowledge”) • Code review • Threat Model • Red Team (Adversary Simulation) • Social Engineering (Phishing, Vishing, Smishing) • Technical Specialty (hardware, cryptography, cryptocurrency) ● The Difference Between a Vulnerability Assessment and a Penetration Test, Daniel Miessler ● What Are The Different Types Of Penetration Testing?, Jason Firch, PurpleSec.us

Slide 23

Slide 23 text

@ramimacisabird On newer models…

Slide 24

Slide 24 text

@ramimacisabird “Getting tough to find something that's not just a dressed up Nessus scan.”

Slide 25

Slide 25 text

@ramimacisabird GOTO 1 Security Assessment Process 8 The Readout 6 Contracting Gathering Proposals 4 2 Finding Vendors 9 Ingestion 7 Preparation & Delivery 5 Vendor scoping Client scoping and requirements 3 1 Defining Motivation 1 0 After the assessment 1 1 Now what?

Slide 26

Slide 26 text

@ramimacisabird 1 Defining Motivation

Slide 27

Slide 27 text

@ramimacisabird • Risk Reduction • Compliance • Internal attestation • Investment or M&A • Sales • Post-breach 1 Defining Motivation

Slide 28

Slide 28 text

@ramimacisabird • Risk Reduction • Compliance • Internal attestation • Investment or M&A • Sales • Post-breach 1 Defining Motivation

Slide 29

Slide 29 text

@ramimacisabird 2 Finding Vendors https://twitter.com/cure53berlin/status/976463307942088706

Slide 30

Slide 30 text

@ramimacisabird Taxonomy of Vendors

Slide 31

Slide 31 text

@ramimacisabird Taxonomy Types of Vendors •Global/enterprise consulting •Cybersecurity services •Boutique •Specialty •Sole practitioner •Researcher/bug hunter •Low cost •Managed Security Service Provider •Value Added Reseller

Slide 32

Slide 32 text

@ramimacisabird What is the greatest challenge in buying security services? ● Finding good vendors, who provide significant value ○ who are available when you need them ○ who can support specific systems or architecture ○ who provide consistent quality staff 2 Finding Vendors

Slide 33

Slide 33 text

@ramimacisabird •Network recommendations •Follow-the-leader •Research •Conference speakers •Published research •Prominent staff •Public reports •Compliance approved •Assessment standards work •Certifications •Analyst recommendations ● https://github.com/juliocesarfort/public-pentesting-reports ● http://www.pentest-standard.org/index.php/FAQ#Q:_Who_is_involved_with_this_standard.3F 2 Finding Vendors

Slide 34

Slide 34 text

@ramimacisabird ● https://github.com/juliocesarfort/public-pentesting-reports ● http://www.pentest-standard.org/index.php/FAQ#Q:_Who_is_involved_with_this_standard.3F 2 Finding Vendors

Slide 35

Slide 35 text

@ramimacisabird Client scoping and requirements 3

Slide 36

Slide 36 text

@ramimacisabird Client scoping and requirements 3 > Scope management is the process of defining what work is required, and then making sure that all of that work, and only that work, is done. SANS Scoping Security Assessments - A Project Management Approach

Slide 37

Slide 37 text

@ramimacisabird Client scoping and requirements 3

Slide 38

Slide 38 text

@ramimacisabird > Strike a balance between performing a comprehensive set of tests and evaluating functionality and features that present the greatest risk. GSA IT Security Procedural Guide: Conducting Penetration Test Exercises Client scoping and requirements 3

Slide 39

Slide 39 text

@ramimacisabird Client scoping and requirements 3

Slide 40

Slide 40 text

@ramimacisabird • Budget ● https://web.archive.org/web/20071207150024/http://securitybuddha.com/2007/08/22/the-art-of-scoping-application-security-reviews-part-1-the-business/ Client scoping and requirements 3

Slide 41

Slide 41 text

@ramimacisabird • Budget • Motivations • Documentation needs • Measurement goals • Breadth vs. Depth • Review your: • Risk assessment • Threat model • Data classification Client scoping and requirements 3 • Mark Curphey, The Art of Scoping Application Security Reviews • 4Armed, Scoping a penetration test • PTES, Pre-engagement Interactions • Trustwave, Missing Critical Vulnerabilities Through Narrow Scoping

Slide 42

Slide 42 text

@ramimacisabird • Follow-on requirements ● https://web.archive.org/web/20071207150024/http://securitybuddha.com/2007/08/22/the-art-of-scoping-application-security-reviews-part-1-the-business/ • Remediation Assistance • Assessor requirements • Onsite • Citizenship • Clearance • Specific methodologies • Certification Client scoping and requirements 3

Slide 43

Slide 43 text

@ramimacisabird Gathering Proposals 4

Slide 44

Slide 44 text

@ramimacisabird 1. Request for proposals 2. Shortlisting (3-5) 3. Initial call Gathering Proposals 4

Slide 45

Slide 45 text

@ramimacisabird • Risk Reduction -> Flexibility. Focus on collaboration and business risk • Compliance -> Certification, balance of substance, auditor relationship • Internal attestation -> Audience, executive summary quality • Investment or M&A -> Speed to engage, experience with M&A, not your call • Sales -> Client relationship, brand name, deliverables • Post-breach -> Incident experience, legal counsel, advisory work 1 Defining Motivation Gathering Proposals 4

Slide 46

Slide 46 text

@ramimacisabird ● Question bank: ○ How soon would you be able to staff this engagement? ○ What experience do you have with organizations like ours? ○ What is your engagement model? ■ Collaboration ■ Staffing ■ Project management ■ Methodology and tools ● https://owasp.org/www-project-application-security-verification-standard/ Gathering Proposals 4

Slide 47

Slide 47 text

@ramimacisabird 5 Vendor scoping

Slide 48

Slide 48 text

@ramimacisabird 5 Vendor scoping > The penetration test team should identify what testing they believe will give a full picture of the vulnerability status of the estate. Advice on how to get the most from penetration testing National Cybersecurity Centre

Slide 49

Slide 49 text

@ramimacisabird Scoping by naive metrics

Slide 50

Slide 50 text

@ramimacisabird How Vendors Scope 1. Questionnaire a. or Scan b. or Code 2. Conversation 3. Demonstration

Slide 51

Slide 51 text

@ramimacisabird ● Fixed price or time and materials ● Detailed pricing ● Earmarked discounts ● Payment terms ○ Net 30 (or 60, 180, 365, with penalties for late payment) ○ Percent upfront (commonly half, either as a deposit or delivered at kickoff) ○ Milestone based Quotes 5 Vendor scoping

Slide 52

Slide 52 text

@ramimacisabird 5 Vendor scoping Engagement Economics and Security Assessments The Guerilla CISO, ryblov

Slide 53

Slide 53 text

@ramimacisabird 6 Contracting

Slide 54

Slide 54 text

@ramimacisabird • Rate • Scope • Overall Level of Effort • Trade depth for breadth • Reporting • Relationship (volume) Negotiation axis 6 Contracting

Slide 55

Slide 55 text

@ramimacisabird 6 Contracting

Slide 56

Slide 56 text

@ramimacisabird What is the greatest challenge in buying security services? ● Balancing quality/price/availability ○ and differentiating quality ○ and justifying spend to management ■ comparing oranges and apples ○ or even affording it at all 6 Contracting

Slide 57

Slide 57 text

@ramimacisabird • Explicit proposals • Like-for-like • Reference checks • Long-term needs • Vet the consultants Vetting 6 Contracting

Slide 58

Slide 58 text

@ramimacisabird The paperwork • (Mutual) Non-Disclosure Agreement - (m)NDA • Master Service Agreement - MSA • Statement of Work - SOW • Rules of Engagement • Common clauses • Service Fees; Taxes; Invoicing and Payment • Termination • Proprietary Rights • Confidentiality • Warranties; Limitation of Liability; Insurance • Indemnification 6 Contracting

Slide 59

Slide 59 text

@ramimacisabird The paperwork What to Look For in a Penetration Testing Statement of Work? CUSTOMER MASTER SERVICES AGREEMENT Checklist: Starting a Security Consulting Firm 6 Contracting

Slide 60

Slide 60 text

@ramimacisabird 7 Preparation & Delivery

Slide 61

Slide 61 text

@ramimacisabird Logistics Tips • “Need about double the resource to manage than you think!” • “Always have technical staff work with your procurement team -- and that's on both sides (vendor and client).” • “Have one primary person for all contact with vendor” • “Leverage business initiatives (e.g. new product launches) to fund pentest procurement as a capital expenditure, leverage ROI to bring it to routine operational budget for the rest of the landscape.” 7 Preparation & Delivery

Slide 62

Slide 62 text

@ramimacisabird Internal alignment • Authorization • Buy-in • Blue team collaboration Logistics Communication channels • Track progress • Respond to questions • Dispatch • Escalation policy Known risks • Risk assessments • Threat models • Previous reports 7 Preparation & Delivery

Slide 63

Slide 63 text

@ramimacisabird Technical Preparation • Resolve outstanding issues • Test environment • Integration • Configuration • Feature flags • Roles • Seed data • Change freeze • Out-of-scope controls ● APPSEC Cali 2018 - Hunter – Optimize your Pentesters Time ● NCC Group - Jerome Smith - The Why Behind Web Application Penetration Test Prerequisites 7 Preparation & Delivery

Slide 64

Slide 64 text

@ramimacisabird Onboarding • Hardware • Software • Remote access • Legal, HR, IT • Demos, documentation, code 7 Preparation & Delivery

Slide 65

Slide 65 text

@ramimacisabird THE ASSESSMENT

Slide 66

Slide 66 text

@ramimacisabird 8 The Readout

Slide 67

Slide 67 text

@ramimacisabird 8 The Readout The Report • Assessment details: Scope, level of effort, tools and methodology, vendor and consultant information • Executive summary: overall outcome, including risk posture, findings of note, and executive or meta recommendations • Findings: details, impact and risk, reproduction information, and remediation guidance • Appendix: additional information on bug classes, detailed remediation steps, custom tools or scripts developed, or raw data from testing

Slide 68

Slide 68 text

@ramimacisabird •Question bank: • If you had more time, where would you dig further or look next? • What would you recommend we do differently for our next engagement? • Were there any trends you observed? Are there any systemic mitigations you’d recommend? • Were there any areas that were particularly well hardened? • How does our posture compare to the (industry/benchmark/average engagement)? 8 The Readout

Slide 69

Slide 69 text

@ramimacisabird > Security consulting firms are the only way you have to know how you compare to others in your field as only a consulting firm can combine trust-based data acquisition with identity-protecting pooling of that otherwise unobtainable comparability data. Penetration testing: a duet Dan Geer & John Harthorne 8 The Readout

Slide 70

Slide 70 text

@ramimacisabird Offboarding 8 The Readout

Slide 71

Slide 71 text

@ramimacisabird Offboarding 8 The Readout

Slide 72

Slide 72 text

@ramimacisabird No findings? No problem! •“Manage client expectations” • Note limitations •Detailed test plan and test coverage • “assurance that we have looked diligently” • “internal investigation / quality control” • “follow up with client for a sanity check” •“A bit more conversation on other security related observations and best practices that can be deployed given the lack of findings” • Highlight true negatives 8 The Readout

Slide 73

Slide 73 text

@ramimacisabird 9 Ingestion

Slide 74

Slide 74 text

@ramimacisabird 9 Ingestion ● Use your standard processes ● Triage

Slide 75

Slide 75 text

@ramimacisabird 9 Ingestion ● Use your standard processes ● Triage ○ Root cause analysis ○ Variant analysis

Slide 76

Slide 76 text

@ramimacisabird 9 Ingestion ● Use your standard processes ● Triage ○ Root cause analysis ○ Variant analysis ● Remediation planning ○ Level of effort ○ Fix, mitigate, or accept ● Parsable reporting ● 2020 Global Appsec SF - Clint Gibler & Isaac Evans - Eradicating Vulnerability Classes

Slide 77

Slide 77 text

@ramimacisabird 10 After the assessment

Slide 78

Slide 78 text

@ramimacisabird Remediation 1 0 After the assessment

Slide 79

Slide 79 text

@ramimacisabird Retrospective 1 0 After the assessment •Question bank: • How were their answers to the questions in the readout? • How was the report quality? • Were there false positives or false negatives? • Are there canary bugs? • How do you feel about value for price given the category and quality of the vendor chosen? • How was the vendor’s communication?

Slide 80

Slide 80 text

@ramimacisabird 1 1 Now what?

Slide 81

Slide 81 text

@ramimacisabird 1 1 Now what? Testing Cadence ● Annual ● Quarterly ● Development cycle aligned ● Compliance aligned > “I do them as ongoing events because I dont feel like punching someone in the face once a year qualifies as a means for improving ones ability to dodge or block a punch.”

Slide 82

Slide 82 text

@ramimacisabird Scope and Vendor ● Retro breadth and depth ○ Listed limitations ○ Retargeting can enhance one or the other ● Vendor rotation 1 1 Now what?

Slide 83

Slide 83 text

@ramimacisabird Scope and Vendor ● “switch vendors and sometimes overlap for A/B testing” … ● “go back to the same person for at least 3 years running.” 1 1 Now what?

Slide 84

Slide 84 text

@ramimacisabird Scope and Vendor ● Vendor rotation - Pros ■ Cross-vendor comparison ■ “vendors work hard on new clients” ■ Firms are fungible quality ■ Recommended or required by policy or auditors ■ Different firms have specialties and methodologies ● Penetration Testing in the Financial Services Industry (2010) 1 1 Now what?

Slide 85

Slide 85 text

@ramimacisabird Scope and Vendor ● Vendor repetition - Pros ■ Decreased ramp up time ■ Improved project management and comms ■ Improved understanding of business risk ■ Cost-savings possible on volume or relationship ■ Expectation of consistency on performance ● Penetration Testing in the Financial Services Industry (2010) 1 1 Now what?

Slide 86

Slide 86 text

@ramimacisabird Scaling your program ● Maximize advantage of leverage ○ price, scheduling, and consultant selection ● Optimize for project management ● Standardize ○ Procurement ○ Internal customer experience ○ Ingestion ● Decide when to staff in-house ● Define ROI 1 1 Now what?

Slide 87

Slide 87 text

@ramimacisabird Scaling your program “Select a handful of companies and work on a contract framework with them. This lowers the amount of work for procuring individual pentests (everything is pre-approved internally & less paperwork).” “Make friends with your supplier management people. They can make your life easy or difficult.” 1 1 Now what?

Slide 88

Slide 88 text

@ramimacisabird Return on Investment How are companies calculating… 1 1 Now what?

Slide 89

Slide 89 text

@ramimacisabird Return on Investment ● “We don't” - verbatim… x3 ● Quality: ○ “Look at the overall quality from the pentest provider over time (can't do it for an individual assessment)” ○ “Depth of analysis and quality of analysis that goes beyond scanning tools.” ○ Quality of findings, specifically those that are scalable across our company. ○ “quality of the assessment, quality of the findings” 1 1 Now what?

Slide 90

Slide 90 text

@ramimacisabird Return on Investment ● “identify *and* close critical or high bugs … a general sentiment from those who hear about it. “ ● “Risk reduction” / “Aggregate organizational risk identified” ● “Value in contributing to sales success” / “$ business lost from potential risks” ● “1. grading the visibility to areas needing improvement, 2. grading the efficacy of monitoring and our response capabilities"” 1 1 Now what?

Slide 91

Slide 91 text

@ramimacisabird If I can only offer three pieces of advice… 1. Be aware of the market for lemons, and think critically about how you’ll know whether a proposal is good, and how you’ll tell if the assessment delivers value 2. Structure your assessment carefully based on key motivations, to deliver measurable business value 3. Use assessments not to kill bugs, but to kill bug classes

Slide 92

Slide 92 text

@ramimacisabird http://tldrsec.com/guides/buying-security https://speakerdeck.com/ramimac/buying-security

Slide 93

Slide 93 text

@ramimacisabird with thanks to • Adrian ("Time to kill the pen test") Sanabria, Tenchi Security • Edward Farrell, Mercury Information Security Services • Robert Postill, Privay • Elliot Murphy, KindlyOps.com • John Cannady, Palo Alto Networks • Robert Shala, Sentry Cybersecurity • Javier Hijas, Efficience • Damien Wilson, Mindglob.com • Dan Guido, Trail of Bits • Mick Douglas (@BetterSafetyNet), InfoSec Innovations • lvh, Latacora • Travis McPeak • Cristiano Maruti • Joel St. John • Emil Vaagland And anonymous friends from: • MTX • Prompt Security • Carve Systems, an iVision company

Slide 94

Slide 94 text

@ramimacisabird Topics that didn’t fit into the main slides

Slide 95

Slide 95 text

@ramimacisabird “It's fucking difficult and nothing is consistent across the industry.”

Slide 96

Slide 96 text

@ramimacisabird 5 Vendor scoping

Slide 97

Slide 97 text

@ramimacisabird Buyer Statistics

Slide 98

Slide 98 text

@ramimacisabird Quality

Slide 99

Slide 99 text

@ramimacisabird On Red Teaming •External Pressures: • “Regulatory requirements, insurance requirements” • “management said so” • “Company board couldn’t be convinced otherwise” • “It's just something we do periodically” • “Regulatory drivers” • “Internal rules demand one Red team per year”

Slide 100

Slide 100 text

@ramimacisabird On Red Teaming •Maturity curve: • “known risks were mitigated with preventative or detective controls” • “Self assessments indicated we were mature enough” • “once defensive controls are in a strong position and are ready to be put to the test” • “Results from prior tests/assessments showing maturity”

Slide 101

Slide 101 text

@ramimacisabird On Red Teaming •Chained threats and external validation: • “We were concerned about abuse cases of an internal CI/CD service based on our own experience, but the company continued heavy investments in this internal tool. We wanted external validation of the criticality of the issues, so we could use it for making our case internally. It helped.” • “We had a large call center and wanted to clue in leadership that our controls around it were insufficient.” • “individual pentest failed to show the big picture about how vulnerable we were as a whole”

Slide 102

Slide 102 text

@ramimacisabird Scaling your program > I ran a trial at Facebook where 10 security consulting companies audited the same code. Code my team had already carefully audited. All 10 found the same pool of shallow bugs (about half) but the remaining issues were all over the map, including one we ourselves had missed. Each person brings their own long tail of security knowledge to bear. Contrast this with something like performance (another attribute of “quality” in software) where it is trivial to measure progress. Why Product Security is Hard Collin Greene 1 1 Now what?

Slide 103

Slide 103 text

@ramimacisabird Challenges ● Collaborating with engineering to prepare ○ including a reasonable test environment with reasonable level of effort ● Managing logistics across the overall program ● Lack of standardization ● Scoping ● Remediation ○ And infosec team bandwidth

Slide 104

Slide 104 text

💰 Buying Security A Client’s Guide Keep up with security research tldrsec.com By Rami McCarthy @ramimacisabird