Bringing a Machete to the Amazon - Blackhat EU 2014

Transcript

1. Bringing a Machete to
   the Amazon
   Erik Peterson - @silvexis - epeterson@veracode.com
   

   View Slide

2. Hi, I’m Erik
   • AppSec guy for 17 years
   • Director of Technology Strategy at
   Veracode
   • Have been researching the Cloud
   and building products on top of
   AWS since 2009
   @silvexis
   

   View Slide

3. AGENDA
   I II III IV
   INTRO GETTING
   ACCESS
   V VI IV
   PROBLEMS
   MACHETE
   FULL
   STACK

   HACKS
   THE API
   IS KING
   EMERGENT
   INSECURITY
   

   View Slide

4. CLOUD

   IS AN
   OPERATING
   SYSTEM
   

   View Slide

5. CLOUD
   INFRASTRUCTURE 

   IS CODE
   

   View Slide

6. A Fully Realized AWS Application
   

   View Slide

7. Traditional App represents small % of system
   • Java, .Net, etc… including 3rd party code in red
   

   View Slide

8. Majority of the system provided by AWS
   Using it all securely however is your responsibility
   

   View Slide

9. What you get from AWS
   • The digital equivalent of infinite empty rack
   space
   • A friendly looking web interface
   • A mile long list of compliance certifications
   • The rest is up to you
   

   View Slide

10. …and yet after a few months this might be you
    

    View Slide

11. Forklifting also Dangerous
    Forklifting is the process of taking legacy data center applications
    and loading them into a cloud provider with little or no changes
    Often this is the most expensive way to use AWS
    It is also dangerous
    IDS and network access controls are unaware of AWS API activity
    Previously ignored or de-prioritized vulnerabilities can become critical
    The meaning of availability changes
    

    View Slide

12. EMERGENT INSECURITY
    The individual components of your
    system may be secure but when
    deployed into the Cloud, the system
    becomes insecure
    AND IT GETS WORSE AT SCALE
    

    View Slide

13. EMERGENT INSECURITY FACTORS
    “Internet Weather” - AWS systems and API calls are subject to
    unpredictable, non-persistent, network latency, system performance and
    connection interruption
    Guaranteed Failure - System availability is a factor of redundancy and
    automation, not the stability and performance of monolithic systems
    Software defined everything (SDe) - Virtualized networks, network
    interfaces, file systems, computing power and more can change independent
    of the underlying system
    Out-of-band management - AWS API’s operate outside of traditional
    security controls, can make all existing controls irrelevant
    

    View Slide

14. EMBRACE AN EVENTUALLY CONSISTENT

    SECURITY MODEL
    Instead of trying to enforce change control which creates
    brittle systems that are insecure and not survivable, design
    your systems to be eventually consistent with your
    security and operational goals
    If your system requires strict change control to maintain
    order, in the Cloud, you will eventually have chaos.
    

    View Slide

15. ALL HAIL THE KING
    In the cloud,
    the king of
    the jungle is
    the API
    

    View Slide

16. What’s your Real Attack Surface?
    “If your security sucks now, you’ll be pleasantly surprised by the
    lack of change when you move to Cloud.”*
    In reality is it gets worse
    AWS API endpoints are open by design, trumps all existing
    controls
    Private IP’s might be public, data leaks through 3rd party services
    Only thing standing between total compromise of your _entire_
    datacenter is the secrecy of your API keys
    *Chris Hoff, @beaker
    

    View Slide

17. API ACCESS PHYSICAL ACCESS
    IN THE CLOUD
    

    View Slide

18. API Credential Exposure Impact
    The attacker can:
    Sets up bitcoin mining operations within your cloud environment
    Alter your applications to spread malware
    Uses your environment as a means to launch additional attacks
    Downloads all your customers data
    

    View Slide

19. Destroy your entire virtual datacenter
    

    View Slide

20. Bypassing traditional controls via the API
    Start with an EC2 system in AWS-VPC, block all network traffic, throw away SSH keys/
    passwords, place IDS on VPC Gateway to detect intrusion attempts, log all network
    traffic to/from system
    In a traditional data center, system is now inaccessible, any attempt to access would
    be detected
    In AWS, use API to snapshot AMI/EBS vol’s, mount snapshots on different system,
    extract whatever I need/want.
    Zero indication from traditional controls that any access has taken place
    Same is true of RDS, I don’t need your passwords, I just need to snapshot your DB
    

    View Slide

21. Getting Access to the API
    Checking your API keys into Github
    Exploiting vulnerabilities to access Metadata
    

    View Slide

22. Oops
    

    View Slide

23. API Honeypot?
    Curious how fast an API key can be stolen from GitHub?
    Create a key, lock it to only one IP address and watch your logs
    You should see failed access attempts in 60 minutes or less
    

    View Slide

24. View Slide

25. What is Cloud Metadata?
    Based on RFC 3927 - Dynamic Configuration of IPv4 Link-Local Addresses
    Metadata contains all kinds of awesome things, like startup scripts and your
    AWS access credentials
    It’s not just AWS, all cloud providers have it (except for MS Azure!)
    AWS: http://169.254.169.254/latest/user-data
    Google: http://169.254.169.254/computeMetadata/v1/
    OpenStack/RackSpace: http://169.254.169.254/openstack
    HP Helion: http://169.254.169.254/2009-04-04/meta-data/
    On Azure, metadata is not dynamic but is copied to /var/lib/waagent (linux) or
    %SYSTEMDRIVE%\AzureData\CustomData.bin (Windows)
    There is nothing wrong with any of this, as long as you are aware and protect it
    

    View Slide

26. $ wget -q -O - http://169.254.169.254/latest/meta-data/
    ami-id
    ami-launch-index
    ami-manifest-path
    block-device-mapping/
    hostname
    iam/
    instance-action
    instance-id
    instance-type
    kernel-id
    local-hostname
    local-ipv4
    mac
    metrics/
    network/
    placement/
    profile
    public-hostname
    public-ipv4
    public-keys/
    reservation-id
    security-groups
    Accessing AWS Metadata
    AWS command line tool ec2-metadata will extract some (but not all!)
    of the metadata
    To get everything use wget or curl
    Each one of these represents
    meta data you can access
    $ wget -q -O - http://169.254.169.254/latest/meta-data/instance-id
    i-0496132e
    

    View Slide

27. METADATA CREDENTIAL EXTRACTION
    [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/security-
    credentials/
    {
    "Code" : "Success",
    "LastUpdated" : "2014-08-13T16:43:54Z",
    "Type" : "AWS-HMAC",
    "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA",
    "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV",
    "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD
    +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/iF94PTi8+kO5sxgboy4XPB
    +Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g+iyZLw8mT5ADz9zGUQTf
    +P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf
    +GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS+sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/
    Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia
    +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr
    +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt
    +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc
    +iFQaFBvCD3q66fBQ==",
    "Expiration" : "2014-08-13T23:05:19Z"
    }
    

    View Slide

28. METADATA CREDENTIAL EXTRACTION
    [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/security-
    credentials/
    {
    "Code" : "Success",
    "LastUpdated" : "2014-08-13T16:43:54Z",
    "Type" : "AWS-HMAC",
    "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA",
    "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV",
    "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD
    +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/iF94PTi8+kO5sxgboy4XPB
    +Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g+iyZLw8mT5ADz9zGUQTf
    +P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf
    +GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS+sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/
    Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia
    +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr
    +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt
    +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc
    +iFQaFBvCD3q66fBQ==",
    "Expiration" : "2014-08-13T23:05:19Z"
    }
    

    View Slide

29. OLD VULNS NEW LIFE
    These vulnerabilities can result in a

    total data center compromise

    in AWS
    CWE-918: SSRF
    CWE-611: XXE
    CWE-441: Unintended Proxy or Intermediary
    CWE-77: Command Injection
    CWE-200: Information Exposure
    Why?
    All of these can lead to unintended exposure of metadata
    

    View Slide

30. Remote Access to Metadata
    Real world Examples
    Prezi experienced this first hand
    http://engineering.prezi.com/blog/2014/03/24/prezi-got-
    pwned-a-tale-of-responsible-disclosure/
    Andres Riancho demonstrated this in his BlackHat talk
    “Pivoting in Amazon Clouds”
    Exploited SSRF vuln to inject malicious AWS SQS messages
    to then exploit a celery/python vuln and take control
    

    View Slide

31. Controlling API Access
    IP Restrictions, they are not just for security groups!
    Keys locked down to specific IP’s (like your VPC gateway) have less chance of damage
    Doesn’t fully eliminate the problem however
    {
    "Version": "2012-10-17",
    "Statement":[{
    "Effect":"Deny",
    "Action":"*",
    "Resource":"*",
    "Condition":{
    "NotIpAddress":{
    “aws:SourceIp”:[“VPC GATEWAY IP/32”]
    }
    }
    }]
    }
    

    View Slide

32. Your bill is not an IDS
    Many AWS customers
    today use their bill as
    their IDS
    So far, most of these people
    have been pretty lucky
    

    View Slide

33. Implement API Logging…please!
    Logging is off by default in AWS
    In AWS, turn on CloudTrail for all
    regions
    Use LogStash, develop your own
    or buy a commercial solution to
    make API activity logs accessible
    

    View Slide

34. EXCESSIVE LACK OF

    ACCESS CONTROL
    {
    "Version": "2012-10-17",
    "Statement": [
    {
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*"
    }
    ]
    }
    

    View Slide

35. DevOps Culture tends to “Fail Open”
    I love DevOps but….
    Be wary of Developers building systems who are new to the Cloud because
    they might now be the ones configuring the firewall rules and API Access
    Policies as well
    Developers and Operations are often pre-disposed to “Just make it work”
    This is one of the reasons DevOps is such a powerful movement
    It’s on us as security people to adapt and keep DevOps powerful
    

    View Slide

36. Leaking Tags
    Tags in AWS are one of the primary means of keeping track of things
    Tags however often can contain private data that does not have strong access protections
    I’ve seen API keys and passwords in tags!!!
    If you use any 3rd party AWS management system like StackDriver, CloudHealth or
    others, your tag data is replicated to those systems
    

    View Slide

37. PRIVATE DNS IS PUBLIC DNS
    AWS DNS from within AWS will resolve EC2 Classic public DNS entries to RFC
    1918 private address
    Good chance of Information Exposure (CWE-200) from web servers listening
    on private IP’s
    A casual poke around the EC2 10.0.0.0/8 address space reveals hundreds of
    web servers
    Remediation: Migrate EC2 Classic environments to VPC
    [ec2-user@heyzeus ~]$ host example.veracodelabs.com
    example.veracodelabs.com is an alias for
    ec2-107-22-197-187.compute-1.amazonaws.com.
    ec2-107-22-197-187.compute-1.amazonaws.com has address 10.72.187.67
    

    View Slide

38. Data and File Leakage
    AWS S3
    1 in 7 buckets left open to anyone attempting to access them
    This might very well be intentional but security through obscurity is a real problem
    AWS EBS
    If I have API access, I can clone your running systems and EBS volumes into an
    external account
    You can prevent this by encrypting your EBS volumes (you can’t encrypt root
    volumes however)
    

    View Slide

39. The Full Stack Hack
    3. SSRF or XXE
    vulnerability exposes
    EC2 Metadata, revealing
    AWS API Keys
    1. AWS DNS
    Reveals Private
    IP of web server
    2. Private IP of web
    server reveals
    detailed errors or
    admin interface
    4. AWS API Key lets
    you create new
    instances where your
    clone and mount any
    existing EBS vol.
    5. Cloned system
    gets you SSH keys
    to app servers and
    API key with IAM:*
    giving you access to
    everything
    6. With new root
    credentials create trust
    relationship with external
    account and clone DB for
    quiet extraction
    

    View Slide

40. CUTTING THROUGH THE JUNGLE
    MACHETE
    

    View Slide

41. Capabilities
    Analyzes IAM users and permissions
    Requires only readonly API access
    Records all services and objects in use
    Track changes, know what is in your account quickly
    Image Cracker
    Crack open AMI and EBS snapshots and extract access keys, user accounts, data
    Coming Soon
    Not quite ready yet, sorry!
    

    View Slide

42. IAM Permissions Analysis
    epeterson@epeterson-mac:~/work/machete$ bin/machete -e 67d54268-dab1-4c73-a424-dbe75d466bb5 refresh
    Machete - v0.1.5
    - Connecting...
    ...connected
    Starting scan of environment 67d54268-dab1-4c73-a424-dbe75d466bb5
    ================================================================================
    id: 67d54268-dab1-4c73-a424-dbe75d466bb5|AROAIEMNXXXX3FXXXXXHM6
    cust uuid: f4cec92f-9184-4394-a4d3-13b7d8708a13
    env uuid: 67d54268-dab1-4c73-a424-dbe75d466bb5
    First seen: 2014-10-11 13:38:20.335481+00:00
    Last seen: 2014-10-17 10:04:00.754355+00:00
    Parent: None
    Type: IAM_ROLE
    Tags: {}
    Metadata:
    --------------------------------------------------------------------------------
    {u'arn': u'arn:aws:iam::034817543207:role/cloudatlas_node',
    u'assume_role_policy_document': {u'Statement': [{u'Action': u'sts:AssumeRole',
    u'Effect': u'Allow',
    u'Principal': {u'Service': u'ec2.amazonaws.com'},
    u'Sid': u''}],
    u'Version': u'2012-10-17'},
    u'create_date': u'2014-06-11T20:25:43Z',
    u'id': u'AXXXXXXNKYAIXXXXXXXX',
    u'name': u'cloudatlas_node',
    u'path': u'/',
    u'role_id': u'AXXXXXXNKYAIXXXXXXXX',
    u'role_name': u'cloudatlas_node'}
    --------------------------------------------------------------------------------
    Versions: 1
    --------------------------------------------------------------------------------
    Change detected on: 2014-10-17 12:03:48
    {u'added': {},
    u'changed': {u'assume_role_policy_document': {u'Statement': [{u'Action': u'sts:AssumeRole',
    u'Effect': u'Allow',
    u'Principal': {u'Service': u'ec2.amazonaws.com'},
    u'Sid': u''}],
    u'Version': u'2012-10-17'}},
    u'removed': {}}
    --------------------------------------------------------------------------------
    

    View Slide

43. Resource tracking
    (machete)epeterson@epeterson-mac:~/work/machete/bin$ ./machete list -e 67d54268-dab1-4c73-a424-dbe75d466bb5
    # : ID Type State First Seen Last Seen Versions
    --------------------------------------------------------------------------------------------------------------------------------------------
    1 : 54.84.xxx.xxx ADDRESS None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.617682+00:00 No changes
    2 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.619777+00:00 No changes
    3 : i-xxxxxxxx EC2 stopped 2014-08-21 18:09:49.462541+00:00 2014-09-02 18:23:59.688451+00:00
    {u'1408649760.41495': {u'added': {},
    u'changed': {u'dns_name': u'',
    u'ip_address': None,
    u'public_dns_name': u'',
    u'reason': u'User initiated (2014-08-21 19:34:02 GMT)',
    u'state': [u'stopped', 80],
    u'state_reason': {u'code': u'Client.UserInitiatedShutdown',
    u'message': u'Client.UserInitiatedShutdown: User initiated shutdown'}},
    u'removed': {}}}
    4 : i-xxxxxxxx EC2 running 2014-08-29 19:41:35.097611+00:00 2014-09-02 18:23:59.682205+00:00 No changes
    5 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.693405+00:00 No changes
    6 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.678996+00:00 No changes
    7 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.219698+00:00 No changes
    8 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.936409+00:00 No changes
    9 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.893741+00:00 No changes
    10 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.240372+00:00 No changes
    11 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.098728+00:00 No changes
    12 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:27.484144+00:00 No changes
    13 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.956196+00:00 No changes
    14 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.691644+00:00 No changes
    15 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.354823+00:00 No changes
    16 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.590336+00:00 No changes
    17 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.966074+00:00 No changes
    18 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.913403+00:00 No changes
    19 : sg-xxxxxxxx EC2_SECURITY_GROUP None 2014-09-02 18:12:16.600848+00:00 2014-09-02 22:39:30.547159+00:00 No changes
    

    View Slide

44. Other Tools
    Netflix Security Monkey
    http://techblog.netflix.com/2014/06/announcing-security-monkey-aws-security.html
    …and a whole ton of other OSS stuff you should check out
    AWS Trusted Advisor
    https://aws.amazon.com/premiumsupport/trustedadvisor/
    NimboStratus
    https://andresriancho.github.io/nimbostratus/
    

    View Slide

45. @silvexis epeterson@veracode.com
    THANK YOU
    May all your Clouds come with a Luck Dragon
    

    View Slide