Learning from AWS Customer Security Incidents [2022]

This show will discuss the public catalog of AWS Customer Security Incidents (https://github.com/ramimac/aws-customer-security-incidents), covering over twenty different public breaches. We’ll walk through the technical details of these attacks, establish the common root causes, look at lessons learned, and establish how you can proactively secure your environment against these real world risks.

Transcript

1. @ramimacisabird
   Learning from
   AWS Customer Security Incidents
   OWASP DevSlop - May 14, 2022
   Rami McCarthy
   

   View Slide

2. @ramimacisabird
   Hello!
   I’m Rami McCarthy 👋
   • Security @ series d health-tech
   • Reformed security consultant
   • AWS Certified Security, Specialty & CCSKv4
   • -creator & -contributor
   

   View Slide

3. @ramimacisabird
   Today
   This is what you’ll get!
   • A strong understanding of how AWS accounts are
   breached in the real world
   • Common initial access and escalation vectors in AWS
   • Actionable practices to prevent joining this esteemed
   cohort
   

   View Slide

4. @ramimacisabird
   

   View Slide

5. @ramimacisabird
   Disclaimers!
   

   View Slide

6. @ramimacisabird
   The Shared Responsibility Model
   

   View Slide

7. @ramimacisabird
   

   View Slide

8. @ramimacisabird
   "Postmortem Culture: Learning from Failure”
   by John Lunney and Sue Lueder
   ”Avoid Blame and Keep It Constructive”
   https://landing.google.com/sre/sre-book/chapters/postmortem-culture
   ”Blameless postmortems can be challenging to write, because the
   postmortem format clearly identifies the actions that led to the incident.
   Removing blame from a postmortem gives people the confidence to
   escalate issues without fear. It is also important not to stigmatize
   frequent production of postmortems by a person or team. An
   atmosphere of blame risks creating a culture in which incidents and
   issues are swept under the rug, leading to greater risk for the
   organization [Boy13]."
   Best Practice:
   

   View Slide

9. @ramimacisabird
   Survivorship Bias Infrequent disclosure
   

   View Slide

10. @ramimacisabird
    DisruptOps’ Top 10
    Cloud Attack Killchains
    • Static API Credential Exposure to Account
    Hijack
    • Compromised Server via Exposed Remote
    Access Ports
    • Compromised Database via Inadvertent
    Exposure
    • Object Storage Public Data Exposure
    • Server Side Request Forgery
    ● Cryptomining
    ● Network Attack
    ● Compromised Secrets
    ● Novel Cloud Data Exposure and Exfiltration
    ● Subdomain Takeover
    

    View Slide

11. @ramimacisabird
    

    View Slide

12. @ramimacisabird
    The Common Cases
    

    View Slide

13. @ramimacisabird
    

    View Slide

14. @ramimacisabird
    Open S3 Buckets and Other Exposed Data Stores
    

    View Slide

15. @ramimacisabird
    Secure Defaults
    • 2017: prominent indicator next to each S3 bucket that is
    publicly accessible
    • 2017: Clarified UX (“Authenticated - Anyone with an AWS
    account”)
    • 2018: Trusted Advisor S3 Public Access rule
    • 2018: Block public access
    • 2019: Access Analyzer for S3
    • 2020: Amazon GuardDuty to Protect Your S3 Buckets
    

    View Slide

16. @ramimacisabird
    Open S3 Buckets and Other Exposed Data Stores
    

    View Slide

17. @ramimacisabird
    Database Ransomware
    • AWS services or user managed
    • Generally, internet exposed with a weak password
    • BTC ransom
    • Examples:
    • https:/
    /mangolassi.it/topic/19664/database-h
    eld-for-ransom-anyone-experience-this-befor
    e/16
    • https:/
    /forums.aws.amazon.com/thread.jspa?t
    hreadID=249445
    

    View Slide

18. @ramimacisabird
    “Through 2025, more than 99% of cloud breaches will have a root cause of
    preventable misconfigurations or mistakes by end users.”
    - Gartner. (H/T Anton Chuvakin)
    

    View Slide

19. @ramimacisabird
    Case Study Speed Run
    

    View Slide

20. @ramimacisabird
    

    View Slide

21. @ramimacisabird
    S3 Global Write: Magecart
    https://www.riskiq.com/blog/labs/magecart-amazon-s3-buckets/
    

    View Slide

22. @ramimacisabird
    S3 Global Write
    Politifact
    2017
    Initial Access: “Misconfigured cloud computing server”
    Impact: Coinhive cryptojacking
    LA Times
    2018
    Initial Access: S3 global write access
    Impact: Coinhive cryptojacking added to homicide.latimes.com
    Twilio
    2020
    Initial Access: S3 global write access
    Impact: Magecart
    

    View Slide

23. @ramimacisabird
    S3 Global Write
    AWS IAM Access Analyzer
    for S3
    Cloud Security Posture
    Management
    Prevention:
    Infrastructure as code +
    SAST
    

    View Slide

24. @ramimacisabird
    Malicious AMI
    Cryptomining AMI
    2018
    Initial Access: Unknown AMI
    Impact: "mines cryptocurrencies, asks for ransom money,
    and tries to exploit things to spread”
    Cryptomining AMI
    2020
    Initial Access: Windows 2008 Server Community AMI
    Impact: Cryptojacking for Monero
    Subscription Scam
    2020
    Initial Access: CentOS AMI squatting
    Impact: $$$ subscription price
    

    View Slide

25. @ramimacisabird
    Malicious AMI
    Using random community
    AMIs
    Prevention:
    

    View Slide

26. @ramimacisabird
    Application Vulnerability
    Tesla
    2018
    Initial Access: Globally exposed Kubernetes console, pod
    with AWS credentials
    Impact: Cryptojacking
    Imperva
    2018
    Initial Access: “Internal compute instance” globally accessible,
    “contained” AWS API key
    Impact: RDS snapshot stolen
    JW Player
    2019
    Initial Access: Weave Scope (publicly exposed), RCE by design
    Impact: Cryptojacking
    1/3
    

    View Slide

27. @ramimacisabird
    Capital One
    2019
    Initial Access: Misconfigured “firewall” (WAF), SSRF access
    to IMDS (metadata service)
    Impact: 100M+ credit card applications stored in S3
    TeamTNT Worm
    2020
    Initial Access: Misconfigured Docker & k8s platforms
    Impact: Cryptojacking for Monero
    Uran Company
    2021
    Initial Access: Compromised Drupal with API keys
    Impact: Cryptomining
    Application Vulnerability
    2/3
    

    View Slide

28. @ramimacisabird
    Application Vulnerability
    Onus
    2021
    Initial Access: Log4Shell vulnerability in Cyclos server
    Impact: 2 million ONUS users’ information including EKYC
    data, personal information, and password hash was leaked
    “Cloud Metadata Abuse
    by UNC2903”
    2022
    Initial Access: Adminer CVE
    Impact: Unknown
    Escalation/
    Persistence
    1. AmazonS3FullAccess creds (and DB creds) in Cyclos config
    2. Steals AWS credentials from ~/.aws/*
    3/3
    

    View Slide

29. @ramimacisabird
    Application Vulnerability:SSRF
    

    View Slide

30. @ramimacisabird
    Application Vulnerability
    Using IMDSv2 - check out
    SSDLC: Threat Model -> Design
    Review -> Code Review -> SAST
    -> Assessments
    Asset Inventory
    Prevention:
    Patch Management
    Putting internal applications on the internet
    

    View Slide

31. @ramimacisabird
    Abuse of Valid Credentials
    Malindo Air
    2019
    Initial Access: Former employees for a third party
    e-commerce provider abused their access
    Impact: 35 million customer records
    1/2
    Voova
    2019
    Initial Access: Stolen credentials by former employee
    Impact: Deleted 23 servers
    Cisco
    2018
    Initial Access: Former employee with AWS access 5
    months post-resignation
    Impact: Deleted ~450 EC2 instances
    

    View Slide

32. @ramimacisabird
    Abuse of Valid Credentials
    Ubiquiti
    2021
    Initial Access: Compromised credentials from IT employee
    Lastpass (alleged former employee insider threat)
    Impact: root administrator access to all AWS accounts, extortion
    “Insider Threat Scenario”
    2020
    Initial Access: Fired employee uses credentials
    Impact: Deleted production databases
    2/2
    Escalation/
    Persistence 1. Access CI/CD server, create a new user, steal credentials
    

    View Slide

33. @ramimacisabird
    Abuse of Valid Credentials
    Standardize and automate
    offboarding
    Manage third party risk
    Least privilege applications
    and services
    Prevention:
    Improve logging, monitoring,
    and detection:
    Time/Location/Activity
    heuristics
    

    View Slide

34. @ramimacisabird
    Abuse of Stolen Credentials
    Code Spaces
    2014
    Initial Access: AWS Console Credentials (Phishing?)
    Impact: Wiped S3 buckets, EC2 instances, AMIs, EBS snapshots
    Datadog
    2016
    Initial Access: CI/CD AWS access key and SSH private key leaked
    Impact: 3 EC2 instances and subset of S3 buckets
    Uber
    2016
    Initial Access: Private Github Repo with AWS credentials
    Impact: Names and driver’s license numbers of 600k drivers; PII
    of 57 million users
    1/5
    

    View Slide

35. @ramimacisabird
    Abuse of Stolen Credentials
    OneLogin
    2017
    Initial Access: AWS keys
    Impact: Accessed database tables (with encrypted data)
    DXC Technologies
    2017
    Initial Access: Private AWS key exposed via Github
    Impact: 244 EC2 instances started
    Cameo
    2020
    Initial Access: Credentials in mobile app package
    Impact: Access to backend infrastructure, including user data
    2/5
    Open Exchange Rates
    2020
    Initial Access: Third-party compromise exposing access key
    Impact: User database
    

    View Slide

36. @ramimacisabird
    Abuse of Stolen Credentials
    Natures Basket
    2020
    Initial Access: Hard-coded root keys in source code
    exposed via public S3 bucket
    Impact: Responsible disclosure
    Animal Jam
    2020
    Initial Access: Slack compromise exposes AWS credentials
    Impact: User database
    Juspay
    2021
    Initial Access: Compromised old, unrecycled Amazon Web
    Services (AWS) access key
    Impact: Masked card data, email IDs and phone numbers
    3/5
    

    View Slide

37. @ramimacisabird
    Abuse of Stolen Credentials
    20/20 Network
    2021
    Initial Access: Compromised credential
    Impact: S3 buckets accessed then deleted
    LogicGate
    2021
    Initial Access: Compromised credentials
    Impact: Backup files in S3 stolen
    Kaspersky
    2021
    Initial Access: Compromised SES token from third party
    Impact: Phishing attacks
    4/5
    

    View Slide

38. @ramimacisabird
    Abuse of Stolen Credentials
    “Alert-to-fix in AWS”
    2020
    Initial Access: Root IAM user access key compromised
    Impact: Cryptojacking
    “A key pair to remember”
    2021
    Initial Access: 8 IAM access keys compromised
    Impact: Command line access to EC2 instances
    “From CLI to console,
    chasing an attacker in
    AWS”
    2021
    Initial Access: Credentials in publicly available code repository
    Impact: Cryptomining (prevented)
    5/5
    

    View Slide

39. @ramimacisabird
    Abuse of Stolen Credentials
    Escalation/
    Persistence
    1. Attacker created additional accounts/access keys
    2. Attacker attempted to pivot with customer credentials
    3. Attacker created EC2 instances
    4. Attacker generated SSH keys for EC2 instances
    5. Attacker backdoored security groups
    6. Attacker used AttachUserPolicy for privilege escalation
    

    View Slide

40. @ramimacisabird
    Abuse of Stolen Credentials
    Follow IAM Best Practices:
    MFA, key rotation
    Audit and monitor privileging
    Prevention:
    Using IAM users
    Storing credentials in code
    

    View Slide

41. @ramimacisabird
    Unknown
    DNC Hack by the GRU
    2016
    Initial Access: Unknown, test clusters breached
    Impact: Tableau and Vertica Queries
    Flexbooker
    2021
    Initial Access: ???
    Impact: 3.7M first and last names, email addresses, phone
    numbers, "encrypted" passwords
    Escalation/
    Persistence 1. EC2 Snapshots copied to attacker AWS accounts
    

    View Slide

42. @ramimacisabird
    Trends
    

    View Slide

43. @ramimacisabird
    

    View Slide

44. @ramimacisabird
    Threat Actors
    1. Monero mining is primary monitization
    a. RCE & Brute force passwords
    b. 8220 Mining Group (chinese speaking)
    • Docker and k8s targeting
    c. Rocke (chinese speaking)
    • JS backdoors
    d. Pacha Group (chinese speaking)
    • lot of evasion, advanced anti-analysis
    2. Dark web market exists for public cloud access
    3. Docker-focused malware (XoRDDOS, Groundhog and Tsunami)
    4. Denonia (lambda targeting malware)
    The Usual Suspects: A Look at Threat Actors Targeting the Cloud and their Battle for Superiority
    2021 IBM Security X-Force Cloud Threat Landscape Report
    https://www.cadosecurity.com/cado-discovers-denonia-the-first-malware-specifically-targeting-lambda/
    

    View Slide

45. @ramimacisabird
    Thank you! 👋
    Key references:
    https:/
    /blog.christophetd.fr/cloud-security-breaches-and-vulnerabilities-2021-in-review/
    https:/
    /tldrsec.com/blog/cloud-security-orienteering/
    https:/
    /summitroute.com/downloads/aws_security_maturity_roadmap-Summit_Route.pdf
    https:/
    /www.marcolancini.it/2021/blog-cloud-security-roadmap/
    Stop by Adrien Coquet from NounProject.com
    Starting by Luis Prado from NounProject.com
    Rami McCarthy, 2022
    Slides:
    https:/
    /speakerdeck.com/ramimac/learning-from-aws-custome
    r-security-incidents-2022
    

    View Slide

46. @ramimacisabird
    Subdomain Takeovers
    https:/
    /0xpatrik.com/subdomain-takeover-basics/
    

    View Slide