Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling Detection and Response at Sentry

Avatar for Misha Kuenstner Misha Kuenstner
April 18, 2025
98

Scaling Detection and Response at Sentry

Avatar for Misha Kuenstner

Misha Kuenstner

April 18, 2025

Transcript

  1. 1 Pillars of D&R 2 Categories and Stages 5 Scaling

    Technically SOAR Platform Agenda 6 In Depth Example Architecture, Data Flow, Components 3 Scaling People Security culture and team profile 4 Scaling Processes Introducing vs Removing
  2. Introduction Security Engineering Manager Building out D&R (Detection and Response),

    supporting all Security programs Senior Security Engineer Helped build D&R and Security, Infrastructure Security Little bit of everything D&R, Vuln Management, InfraSec, Network Configuration, Incident Response, Developer Education
  3. Detection and Response Pillars Detection Response Infrastructure KTLO (Keep The

    Lights On) Operational Work Threat Modeling Detection Engineering DaC Threat Intelligence Threat Hunting Incident Response IR Playbooks IR Processes Log Visibility Alert Fidelity SIEM SOAR Log Ingestion Pipelines Enrichment Alert Automation
  4. Security Programs and Maturity People Process Technology Team Leadership Cultural

    Indicators Policies Compliance Metrics Visibility SIEM Application Security Enterprise Security Detection and Response IT Security
  5. People Growth Late Stage Team Leadership Culture - Introduction of

    specialized roles - Increased investment in tools and staff to meet demand - Integration with Engineering and other functions - Psychological safety - Awareness around how to leverage Security to drive revenue and sales - Specialized roles - Team at capacity, tight workload - Team is used to reaching out to Security (SIRT@) - Phishing reports, vuln management - Positive reinforcement around security reporting - Cultural norm
  6. People Challenges Challenges - Often the hardest to change -

    New norms, behavioral changes or modifications take time - Broader cultural influences may be out of our control, but directly impact how effective Security can be - Ex. Security incidents and how the following play a role in our IR: - Psychological safety - High accountability, but low blame - Lack of strong ownership
  7. Processes Growth Late Stage Metrics Compliance - Ownership of controls

    is in flux - Internal Auditing processes are getting established - No automation in place - Metrics exist, but monitoring is just starting - Data is not consistently aggregated - Security policies are inconsistent - Clear ownership of controls - Risk Assessment and Register exist with likelihood, impact, classification - Organization has a more structured approach to audits - Metrics are defined. Programs have KPIs/OKRs - Leading and lagging indicators are identified Policies
  8. Process Challenges Challenges - Finding the balance between moving fast

    and adding friction in the right places at the right time - Metrics are hard - Data can be manipulated to tell a story - Deciding what to track - Compliance is not a replacement for Security, but it does help to shape more secure practices - Lack of ownership at start ups - Vuln management or a bug bounty - Incident Response
  9. Scaling Technology Challenges • Build vs Buy ◦ Engineering capacity

    to build, maintain, and iterate ◦ Budgets • At startups, you generally may have more flexibility in adopting vendors or tools that are still in their own growth phase ◦ You can accept some shortcomings in order to adopt bleeding edge tech, get better customer support, cheaper prices ◦ Better to buy some technology (maybe don’t build your own EDR)
  10. How we built our SOAR Problem Statement • We have

    a SIEM (security incident and event management) actively running detections on all of our log sources • Lack of a SOAR (security orchestration and automation response) poses a significant risk in our ability to do one of our main responsibilities, respond to Security issues • We are a small team • We need to respond to the alerts in a timely way • We want to spend less time responding, without sacrificing quality
  11. Basic Requirements of a SOAR SOAR - Security Orchestration and

    Automation Response • Automation • Orchestration • Incident Response • Threat Intelligence • Reporting and Analytics
  12. Existing Commercial Solutions • Out of our current budget ◦

    Tines, Crowdstrike, Torq • Do not have strong features ◦ Qradar • Will be complex and costly to maintain ◦ The Hive, Catalyst
  13. What is S3F? S3F is Sentry’s Serverless SOAR Framework •

    Custom built using GCP Functions & Workflows to cheaply scale ◦ When we say cheap, we mean it (~$40/month) vs 250k (Tines) • v1 - Main Functionality Goals ◦ Simple deployment pipeline ◦ Processes alerts from Panther ▪ Enrich alerts with additional data ▪ Suppress edge-case rules ▪ Additional deduplication layer ◦ Runs automation from Jira ▪ Execute response workflows in comments (ended up being vNext) ▪ Keep findings in one place to consolidate information and for compliance
  14. Technology Terraform - All infra is code - Modules for

    speed & simplicity - Atlantis: apply in PR Cloud Functions - Run code w/o infra - Expose webhooks - Granular SAs Cloud Workflows - Tie together APIs - Handle control flow, retries, errors - Basically free
  15. Technology Firestore - Key -> Document storage - Stores alert

    data - Cheap, fast, scales Cloud Scheduler (Cron) - Run cloud functions on a schedule - No resource cost while waiting
  16. Ingestion • Dynamically create a SA when the cloud function

    is created ◦ SA will create a Panther ingestion bucket ◦ Request to the webhook creates an alert file in Firestore ◦ Writes alert metadata from Panther ▪ Alert ID - created by Panther (rep. as p_alert_id) ▪ Rule ID - created by Panther (rep. as p_rule_id) Panther API Ingestion Jira Alerts Firestore • First step after Panther creates a ticket from a triggered detection • Public webhook Grab Alert events and write to Firestore and Jira
  17. Interesting Challenges with Ingestion • Panther API gives you an

    alert ID and rule ID upon creating an alert • We attach relevant events to the JIRA ticket as events.json using alert ID for easy response • The challenge is with the Panther API and pulling the relevant events
  18. Potential Long Term Solutions • We considered building our own

    table to map rule IDs to log types ◦ We had other priorities and we accepted this as tech debt • Panther implements another mechanism to grab relevant events and stores them more intuitively • Addition of log source list in alerts
  19. Enrichment Create the Request query Populate workflow vars with the

    list of results Write map to Firestore Dump JSON to JIRA as a comment We run these concurrently where we can Set of queries to pull usernames, IPs, hashes, domains for a given alert
  20. Response Workflows Manually Trigger commands in JIRA Jira Alerts Firestore

    Response Triggers Response Workflows • Second public webhook • Trigger-JIRA-command cloud function ◦ Parse the command ◦ Trigger pre-registered cloud functions that make the request to the API (such as github, permiso)
  21. Monitoring Infrastructure and Alerts Turns out GCP Dashboards were not

    enough • Spun up a Grafana instance ◦ Total spend is $40/month for our SOAR, $36 go to Grafana ◦ Consistently 80 - 90% of our monthly costs • Started collecting metrics for alerts ◦ Jira ▪ Created vs Resolved, Ticket duration by state, Avg ticket age, severity, etc ◦ Grafana ▪ Workflow Failures, Alerts Published, Unique Rule IDs, Severity ▪ Infrastructure Monitoring ◦ GCP Dashboards ▪ Total Function Executions, Crashes, etc
  22. Cost Breakdown Total Cost: $35.95 Grafana: $29.40 (84%) Rest: $5.60

    (16%) What Cost % of Total Networking LB (Grafana only) 14.69 40.8% Cloud SQL (Grafana only) 6.17 17.1% Cloud Armor 3.97 11.0% Cloud Run (CPU Time) 2.30 6.4% Cloud Functions (CPU Time) 2.08 5.8% Secrets Manager 1.41 3.9% * Everything else is < $1 and covers the remaining ~15% Total Data: ~20GiB/day (21.47 gb/day) Monthly data: 665 gbs/month Cost / GB: $ 0.00841 / GB Aug 1st - Aug 31st
  23. Summary - Building Security from the ground up is hard

    - Jump on the train, not in front of it - 3 Main takeaways: - Make the secure path the easy path when you can - Listen to your customers, internal stakeholders, and make working with security a collaborative process. May need to put the business needs first more that you’d like - Be patient, Security is a long term investment
  24. Usage Breakdown • 352 Function Executions • 6 Function Crashes

    • 24 workflow executions • Alerts ◦ Nov 1 - Nov 31: 138 alerts ◦ Aug 1 - Aug 31: 270 alerts • Firestore ◦ Free tier Function Daily mean (Aug 1-Aug 31) ingest-panther-webhook 87 MiB find-alert-events 87 MiB populate-alert-firestore 112 MiB publish-jira 112 MiB Workflow Daily mean (Aug 1-Aug 31) Ingest (Bytes/sec) 30 B/s - 5 B/s Enrichment (Mebibyte) 87 MiB/day Panther Alerts GCS bucket (Gibibytes/sec) 23 GiB/s
  25. Challenge #1 Problem: Scanning every single DB for each log

    source for matching alert events • Panther doesn’t let us use the rule ID to pull events from their GraphQL API using the log source ◦ Detections can be generic and run across multiple sources • Each log source has its own table of events that triggered an alert Workaround: Query every single log source table for matching p_alert_id • We have to rerun this every time we scan a table and come up empty • Panther cache may be taking a while to populate tables with events so we rerun it • We run this component within seconds of an alert getting generated so our guess is they are caching somewhere and we’re querying the DB too soon Why it works: Very little extra cost to us, we want to make response easier, no other way to query the table
  26. Challenge #2 Problem: GraphQL Request to run SQL query to

    pull events doesn’t return confirmation • Panther’s API doesn’t confirm after our SQL query is done and we’ve successfully grabbed the events • We submit the query, then it tells us “it’s running” Workaround: We send multiple requests to ask status of initial SQL query • We have to run this 2*n times, wait 30 seconds • They have to pay for the compute time for the GraphQL request and the SQL request • ~150 alerts/week, 30 seconds/run, ~20 seconds/compute time Why it works: Very little extra cost to us
  27. The Team • Engineers: ◦ Early in career ( ~2

    full time YOE) ◦ First time at a SaaS company ◦ Needed guidance on how to write RFCs, trade-offs, decisions ◦ An opportunity to demonstrate technical aptitude and impact • My role ◦ Review PRs ◦ Design guidance ◦ Feedback at every stage ◦ Identify opportunities for cross functional adoption to increase impact ◦ Coaching ◦ Plan for open source ◦ Guidance during or tie-break for decisions ◦ Identify partners for enrichments
  28. Suppression After enrichment, we can suppress based on enrichment or

    private data Suppression Panther Ingestion Jira Alerts Firestore
  29. Publish Publish data to JIRA • Enrichment data is not

    written to the ticket, only to Firestore • Builds a JSON representation of JSON body and updates the body to our format and adds it to the Jira ticket