$30 off During Our Annual Pro Sale. View Details »

Bringing a Machete to the Amazon - Blackhat EU 2014

Erik Peterson
October 17, 2014

Bringing a Machete to the Amazon - Blackhat EU 2014

Erik Peterson

October 17, 2014
Tweet

More Decks by Erik Peterson

Other Decks in Research

Transcript

  1. Bringing a Machete to
    the Amazon
    Erik Peterson - @silvexis - [email protected]

    View Slide

  2. Hi, I’m Erik
    • AppSec guy for 17 years
    • Director of Technology Strategy at
    Veracode
    • Have been researching the Cloud
    and building products on top of
    AWS since 2009
    @silvexis

    View Slide

  3. AGENDA
    I II III IV
    INTRO GETTING
    ACCESS
    V VI IV
    PROBLEMS
    MACHETE
    FULL
    STACK

    HACKS
    THE API
    IS KING
    EMERGENT
    INSECURITY

    View Slide

  4. CLOUD

    IS AN
    OPERATING
    SYSTEM

    View Slide

  5. CLOUD
    INFRASTRUCTURE 

    IS CODE

    View Slide

  6. A Fully Realized AWS Application

    View Slide

  7. Traditional App represents small % of system
    • Java, .Net, etc… including 3rd party code in red

    View Slide

  8. Majority of the system provided by AWS
    Using it all securely however is your responsibility

    View Slide

  9. What you get from AWS
    • The digital equivalent of infinite empty rack
    space
    • A friendly looking web interface
    • A mile long list of compliance certifications
    • The rest is up to you

    View Slide

  10. …and yet after a few months this might be you

    View Slide

  11. Forklifting also Dangerous
    Forklifting is the process of taking legacy data center applications
    and loading them into a cloud provider with little or no changes
    Often this is the most expensive way to use AWS
    It is also dangerous
    IDS and network access controls are unaware of AWS API activity
    Previously ignored or de-prioritized vulnerabilities can become critical
    The meaning of availability changes

    View Slide

  12. EMERGENT INSECURITY
    The individual components of your
    system may be secure but when
    deployed into the Cloud, the system
    becomes insecure
    AND IT GETS WORSE AT SCALE

    View Slide

  13. EMERGENT INSECURITY FACTORS
    “Internet Weather” - AWS systems and API calls are subject to
    unpredictable, non-persistent, network latency, system performance and
    connection interruption
    Guaranteed Failure - System availability is a factor of redundancy and
    automation, not the stability and performance of monolithic systems
    Software defined everything (SDe) - Virtualized networks, network
    interfaces, file systems, computing power and more can change independent
    of the underlying system
    Out-of-band management - AWS API’s operate outside of traditional
    security controls, can make all existing controls irrelevant

    View Slide

  14. EMBRACE AN EVENTUALLY CONSISTENT

    SECURITY MODEL
    Instead of trying to enforce change control which creates
    brittle systems that are insecure and not survivable, design
    your systems to be eventually consistent with your
    security and operational goals
    If your system requires strict change control to maintain
    order, in the Cloud, you will eventually have chaos.

    View Slide

  15. ALL HAIL THE KING
    In the cloud,
    the king of
    the jungle is
    the API

    View Slide

  16. What’s your Real Attack Surface?
    “If your security sucks now, you’ll be pleasantly surprised by the
    lack of change when you move to Cloud.”*
    In reality is it gets worse
    AWS API endpoints are open by design, trumps all existing
    controls
    Private IP’s might be public, data leaks through 3rd party services
    Only thing standing between total compromise of your _entire_
    datacenter is the secrecy of your API keys
    *Chris Hoff, @beaker

    View Slide

  17. API ACCESS PHYSICAL ACCESS
    IN THE CLOUD

    View Slide

  18. API Credential Exposure Impact
    The attacker can:
    Sets up bitcoin mining operations within your cloud environment
    Alter your applications to spread malware
    Uses your environment as a means to launch additional attacks
    Downloads all your customers data

    View Slide

  19. Destroy your entire virtual datacenter

    View Slide

  20. Bypassing traditional controls via the API
    Start with an EC2 system in AWS-VPC, block all network traffic, throw away SSH keys/
    passwords, place IDS on VPC Gateway to detect intrusion attempts, log all network
    traffic to/from system
    In a traditional data center, system is now inaccessible, any attempt to access would
    be detected
    In AWS, use API to snapshot AMI/EBS vol’s, mount snapshots on different system,
    extract whatever I need/want.
    Zero indication from traditional controls that any access has taken place
    Same is true of RDS, I don’t need your passwords, I just need to snapshot your DB

    View Slide

  21. Getting Access to the API
    Checking your API keys into Github
    Exploiting vulnerabilities to access Metadata

    View Slide

  22. Oops

    View Slide

  23. API Honeypot?
    Curious how fast an API key can be stolen from GitHub?
    Create a key, lock it to only one IP address and watch your logs
    You should see failed access attempts in 60 minutes or less

    View Slide

  24. View Slide

  25. What is Cloud Metadata?
    Based on RFC 3927 - Dynamic Configuration of IPv4 Link-Local Addresses
    Metadata contains all kinds of awesome things, like startup scripts and your
    AWS access credentials
    It’s not just AWS, all cloud providers have it (except for MS Azure!)
    AWS: http://169.254.169.254/latest/user-data
    Google: http://169.254.169.254/computeMetadata/v1/
    OpenStack/RackSpace: http://169.254.169.254/openstack
    HP Helion: http://169.254.169.254/2009-04-04/meta-data/
    On Azure, metadata is not dynamic but is copied to /var/lib/waagent (linux) or
    %SYSTEMDRIVE%\AzureData\CustomData.bin (Windows)
    There is nothing wrong with any of this, as long as you are aware and protect it

    View Slide

  26. $ wget -q -O - http://169.254.169.254/latest/meta-data/
    ami-id
    ami-launch-index
    ami-manifest-path
    block-device-mapping/
    hostname
    iam/
    instance-action
    instance-id
    instance-type
    kernel-id
    local-hostname
    local-ipv4
    mac
    metrics/
    network/
    placement/
    profile
    public-hostname
    public-ipv4
    public-keys/
    reservation-id
    security-groups
    Accessing AWS Metadata
    AWS command line tool ec2-metadata will extract some (but not all!)
    of the metadata
    To get everything use wget or curl
    Each one of these represents
    meta data you can access
    $ wget -q -O - http://169.254.169.254/latest/meta-data/instance-id
    i-0496132e

    View Slide

  27. METADATA CREDENTIAL EXTRACTION
    [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/security-
    credentials/
    {
    "Code" : "Success",
    "LastUpdated" : "2014-08-13T16:43:54Z",
    "Type" : "AWS-HMAC",
    "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA",
    "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV",
    "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD
    +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/iF94PTi8+kO5sxgboy4XPB
    +Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g+iyZLw8mT5ADz9zGUQTf
    +P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf
    +GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS+sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/
    Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia
    +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr
    +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt
    +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc
    +iFQaFBvCD3q66fBQ==",
    "Expiration" : "2014-08-13T23:05:19Z"
    }

    View Slide

  28. METADATA CREDENTIAL EXTRACTION
    [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/security-
    credentials/
    {
    "Code" : "Success",
    "LastUpdated" : "2014-08-13T16:43:54Z",
    "Type" : "AWS-HMAC",
    "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA",
    "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV",
    "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD
    +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/iF94PTi8+kO5sxgboy4XPB
    +Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g+iyZLw8mT5ADz9zGUQTf
    +P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf
    +GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS+sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/
    Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia
    +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr
    +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt
    +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc
    +iFQaFBvCD3q66fBQ==",
    "Expiration" : "2014-08-13T23:05:19Z"
    }

    View Slide

  29. OLD VULNS NEW LIFE
    These vulnerabilities can result in a

    total data center compromise

    in AWS
    CWE-918: SSRF
    CWE-611: XXE
    CWE-441: Unintended Proxy or Intermediary
    CWE-77: Command Injection
    CWE-200: Information Exposure
    Why?
    All of these can lead to unintended exposure of metadata

    View Slide

  30. Remote Access to Metadata
    Real world Examples
    Prezi experienced this first hand
    http://engineering.prezi.com/blog/2014/03/24/prezi-got-
    pwned-a-tale-of-responsible-disclosure/
    Andres Riancho demonstrated this in his BlackHat talk
    “Pivoting in Amazon Clouds”
    Exploited SSRF vuln to inject malicious AWS SQS messages
    to then exploit a celery/python vuln and take control

    View Slide

  31. Controlling API Access
    IP Restrictions, they are not just for security groups!
    Keys locked down to specific IP’s (like your VPC gateway) have less chance of damage
    Doesn’t fully eliminate the problem however
    {
    "Version": "2012-10-17",
    "Statement":[{
    "Effect":"Deny",
    "Action":"*",
    "Resource":"*",
    "Condition":{
    "NotIpAddress":{
    “aws:SourceIp”:[“VPC GATEWAY IP/32”]
    }
    }
    }]
    }

    View Slide

  32. Your bill is not an IDS
    Many AWS customers
    today use their bill as
    their IDS
    So far, most of these people
    have been pretty lucky

    View Slide

  33. Implement API Logging…please!
    Logging is off by default in AWS
    In AWS, turn on CloudTrail for all
    regions
    Use LogStash, develop your own
    or buy a commercial solution to
    make API activity logs accessible

    View Slide

  34. EXCESSIVE LACK OF

    ACCESS CONTROL
    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*"
    }
    ]
    }

    View Slide

  35. DevOps Culture tends to “Fail Open”
    I love DevOps but….
    Be wary of Developers building systems who are new to the Cloud because
    they might now be the ones configuring the firewall rules and API Access
    Policies as well
    Developers and Operations are often pre-disposed to “Just make it work”
    This is one of the reasons DevOps is such a powerful movement
    It’s on us as security people to adapt and keep DevOps powerful

    View Slide

  36. Leaking Tags
    Tags in AWS are one of the primary means of keeping track of things
    Tags however often can contain private data that does not have strong access protections
    I’ve seen API keys and passwords in tags!!!
    If you use any 3rd party AWS management system like StackDriver, CloudHealth or
    others, your tag data is replicated to those systems

    View Slide

  37. PRIVATE DNS IS PUBLIC DNS
    AWS DNS from within AWS will resolve EC2 Classic public DNS entries to RFC
    1918 private address
    Good chance of Information Exposure (CWE-200) from web servers listening
    on private IP’s
    A casual poke around the EC2 10.0.0.0/8 address space reveals hundreds of
    web servers
    Remediation: Migrate EC2 Classic environments to VPC
    [ec2-user@heyzeus ~]$ host example.veracodelabs.com
    example.veracodelabs.com is an alias for
    ec2-107-22-197-187.compute-1.amazonaws.com.
    ec2-107-22-197-187.compute-1.amazonaws.com has address 10.72.187.67

    View Slide

  38. Data and File Leakage
    AWS S3
    1 in 7 buckets left open to anyone attempting to access them
    This might very well be intentional but security through obscurity is a real problem
    AWS EBS
    If I have API access, I can clone your running systems and EBS volumes into an
    external account
    You can prevent this by encrypting your EBS volumes (you can’t encrypt root
    volumes however)

    View Slide

  39. The Full Stack Hack
    3. SSRF or XXE
    vulnerability exposes
    EC2 Metadata, revealing
    AWS API Keys
    1. AWS DNS
    Reveals Private
    IP of web server
    2. Private IP of web
    server reveals
    detailed errors or
    admin interface
    4. AWS API Key lets
    you create new
    instances where your
    clone and mount any
    existing EBS vol.
    5. Cloned system
    gets you SSH keys
    to app servers and
    API key with IAM:*
    giving you access to
    everything
    6. With new root
    credentials create trust
    relationship with external
    account and clone DB for
    quiet extraction

    View Slide

  40. CUTTING THROUGH THE JUNGLE
    MACHETE

    View Slide

  41. Capabilities
    Analyzes IAM users and permissions
    Requires only readonly API access
    Records all services and objects in use
    Track changes, know what is in your account quickly
    Image Cracker
    Crack open AMI and EBS snapshots and extract access keys, user accounts, data
    Coming Soon
    Not quite ready yet, sorry!

    View Slide

  42. IAM Permissions Analysis
    epeterson@epeterson-mac:~/work/machete$ bin/machete -e 67d54268-dab1-4c73-a424-dbe75d466bb5 refresh
    Machete - v0.1.5
    - Connecting...
    ...connected
    Starting scan of environment 67d54268-dab1-4c73-a424-dbe75d466bb5
    ================================================================================
    id: 67d54268-dab1-4c73-a424-dbe75d466bb5|AROAIEMNXXXX3FXXXXXHM6
    cust uuid: f4cec92f-9184-4394-a4d3-13b7d8708a13
    env uuid: 67d54268-dab1-4c73-a424-dbe75d466bb5
    First seen: 2014-10-11 13:38:20.335481+00:00
    Last seen: 2014-10-17 10:04:00.754355+00:00
    Parent: None
    Type: IAM_ROLE
    Tags: {}
    Metadata:
    --------------------------------------------------------------------------------
    {u'arn': u'arn:aws:iam::034817543207:role/cloudatlas_node',
    u'assume_role_policy_document': {u'Statement': [{u'Action': u'sts:AssumeRole',
    u'Effect': u'Allow',
    u'Principal': {u'Service': u'ec2.amazonaws.com'},
    u'Sid': u''}],
    u'Version': u'2012-10-17'},
    u'create_date': u'2014-06-11T20:25:43Z',
    u'id': u'AXXXXXXNKYAIXXXXXXXX',
    u'name': u'cloudatlas_node',
    u'path': u'/',
    u'role_id': u'AXXXXXXNKYAIXXXXXXXX',
    u'role_name': u'cloudatlas_node'}
    --------------------------------------------------------------------------------
    Versions: 1
    --------------------------------------------------------------------------------
    Change detected on: 2014-10-17 12:03:48
    {u'added': {},
    u'changed': {u'assume_role_policy_document': {u'Statement': [{u'Action': u'sts:AssumeRole',
    u'Effect': u'Allow',
    u'Principal': {u'Service': u'ec2.amazonaws.com'},
    u'Sid': u''}],
    u'Version': u'2012-10-17'}},
    u'removed': {}}
    --------------------------------------------------------------------------------

    View Slide

  43. Resource tracking
    (machete)epeterson@epeterson-mac:~/work/machete/bin$ ./machete list -e 67d54268-dab1-4c73-a424-dbe75d466bb5
    # : ID Type State First Seen Last Seen Versions
    --------------------------------------------------------------------------------------------------------------------------------------------
    1 : 54.84.xxx.xxx ADDRESS None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.617682+00:00 No changes
    2 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.619777+00:00 No changes
    3 : i-xxxxxxxx EC2 stopped 2014-08-21 18:09:49.462541+00:00 2014-09-02 18:23:59.688451+00:00
    {u'1408649760.41495': {u'added': {},
    u'changed': {u'dns_name': u'',
    u'ip_address': None,
    u'public_dns_name': u'',
    u'reason': u'User initiated (2014-08-21 19:34:02 GMT)',
    u'state': [u'stopped', 80],
    u'state_reason': {u'code': u'Client.UserInitiatedShutdown',
    u'message': u'Client.UserInitiatedShutdown: User initiated shutdown'}},
    u'removed': {}}}
    4 : i-xxxxxxxx EC2 running 2014-08-29 19:41:35.097611+00:00 2014-09-02 18:23:59.682205+00:00 No changes
    5 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.693405+00:00 No changes
    6 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.678996+00:00 No changes
    7 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.219698+00:00 No changes
    8 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.936409+00:00 No changes
    9 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.893741+00:00 No changes
    10 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.240372+00:00 No changes
    11 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.098728+00:00 No changes
    12 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:27.484144+00:00 No changes
    13 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.956196+00:00 No changes
    14 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.691644+00:00 No changes
    15 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.354823+00:00 No changes
    16 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.590336+00:00 No changes
    17 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.966074+00:00 No changes
    18 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.913403+00:00 No changes
    19 : sg-xxxxxxxx EC2_SECURITY_GROUP None 2014-09-02 18:12:16.600848+00:00 2014-09-02 22:39:30.547159+00:00 No changes

    View Slide

  44. Other Tools
    Netflix Security Monkey
    http://techblog.netflix.com/2014/06/announcing-security-monkey-aws-security.html
    …and a whole ton of other OSS stuff you should check out
    AWS Trusted Advisor
    https://aws.amazon.com/premiumsupport/trustedadvisor/
    NimboStratus
    https://andresriancho.github.io/nimbostratus/

    View Slide

  45. @silvexis [email protected]
    THANK YOU
    May all your Clouds come with a Luck Dragon

    View Slide