Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bringing a Machete to the Amazon - Blackhat EU 2014

Bringing a Machete to the Amazon - Blackhat EU 2014

Erik Peterson

October 17, 2014

More Decks by Erik Peterson

Other Decks in Research


  1. Hi, I’m Erik • AppSec guy for 17 years •

    Director of Technology Strategy at Veracode • Have been researching the Cloud and building products on top of AWS since 2009 @silvexis

  3. Traditional App represents small % of system • Java, .Net,

    etc… including 3rd party code in red
  4. Majority of the system provided by AWS Using it all

    securely however is your responsibility
  5. What you get from AWS • The digital equivalent of

    infinite empty rack space • A friendly looking web interface • A mile long list of compliance certifications • The rest is up to you
  6. Forklifting also Dangerous Forklifting is the process of taking legacy

    data center applications and loading them into a cloud provider with little or no changes Often this is the most expensive way to use AWS It is also dangerous IDS and network access controls are unaware of AWS API activity Previously ignored or de-prioritized vulnerabilities can become critical The meaning of availability changes
  7. EMERGENT INSECURITY The individual components of your system may be

    secure but when deployed into the Cloud, the system becomes insecure AND IT GETS WORSE AT SCALE
  8. EMERGENT INSECURITY FACTORS “Internet Weather” - AWS systems and API

    calls are subject to unpredictable, non-persistent, network latency, system performance and connection interruption Guaranteed Failure - System availability is a factor of redundancy and automation, not the stability and performance of monolithic systems Software defined everything (SDe) - Virtualized networks, network interfaces, file systems, computing power and more can change independent of the underlying system Out-of-band management - AWS API’s operate outside of traditional security controls, can make all existing controls irrelevant
 SECURITY MODEL Instead of trying to

    enforce change control which creates brittle systems that are insecure and not survivable, design your systems to be eventually consistent with your security and operational goals If your system requires strict change control to maintain order, in the Cloud, you will eventually have chaos.
  10. What’s your Real Attack Surface? “If your security sucks now,

    you’ll be pleasantly surprised by the lack of change when you move to Cloud.”* In reality is it gets worse AWS API endpoints are open by design, trumps all existing controls Private IP’s might be public, data leaks through 3rd party services Only thing standing between total compromise of your _entire_ datacenter is the secrecy of your API keys *Chris Hoff, @beaker
  11. API Credential Exposure Impact The attacker can: Sets up bitcoin

    mining operations within your cloud environment Alter your applications to spread malware Uses your environment as a means to launch additional attacks Downloads all your customers data
  12. Bypassing traditional controls via the API Start with an EC2

    system in AWS-VPC, block all network traffic, throw away SSH keys/ passwords, place IDS on VPC Gateway to detect intrusion attempts, log all network traffic to/from system In a traditional data center, system is now inaccessible, any attempt to access would be detected In AWS, use API to snapshot AMI/EBS vol’s, mount snapshots on different system, extract whatever I need/want. Zero indication from traditional controls that any access has taken place Same is true of RDS, I don’t need your passwords, I just need to snapshot your DB
  13. Getting Access to the API Checking your API keys into

    Github Exploiting vulnerabilities to access Metadata
  14. API Honeypot? Curious how fast an API key can be

    stolen from GitHub? Create a key, lock it to only one IP address and watch your logs You should see failed access attempts in 60 minutes or less
  15. What is Cloud Metadata? Based on RFC 3927 - Dynamic

    Configuration of IPv4 Link-Local Addresses Metadata contains all kinds of awesome things, like startup scripts and your AWS access credentials It’s not just AWS, all cloud providers have it (except for MS Azure!) AWS: Google: OpenStack/RackSpace: HP Helion: On Azure, metadata is not dynamic but is copied to /var/lib/waagent (linux) or %SYSTEMDRIVE%\AzureData\CustomData.bin (Windows) There is nothing wrong with any of this, as long as you are aware and protect it
  16. $ wget -q -O - ami-id ami-launch-index ami-manifest-path block-device-mapping/

    hostname iam/ instance-action instance-id instance-type kernel-id local-hostname local-ipv4 mac metrics/ network/ placement/ profile public-hostname public-ipv4 public-keys/ reservation-id security-groups Accessing AWS Metadata AWS command line tool ec2-metadata will extract some (but not all!) of the metadata To get everything use wget or curl Each one of these represents meta data you can access $ wget -q -O - i-0496132e
  17. METADATA CREDENTIAL EXTRACTION [ec2-user@ip-10-0-1-125 ~]$ wget -q -O -

    credentials/<YOUR PROFILE HERE> { "Code" : "Success", "LastUpdated" : "2014-08-13T16:43:54Z", "Type" : "AWS-HMAC", "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA", "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV", "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/iF94PTi8+kO5sxgboy4XPB +Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g+iyZLw8mT5ADz9zGUQTf +P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf +GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS+sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/ Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc +iFQaFBvCD3q66fBQ==", "Expiration" : "2014-08-13T23:05:19Z" }
  18. METADATA CREDENTIAL EXTRACTION [ec2-user@ip-10-0-1-125 ~]$ wget -q -O -

    credentials/<YOUR PROFILE HERE> { "Code" : "Success", "LastUpdated" : "2014-08-13T16:43:54Z", "Type" : "AWS-HMAC", "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA", "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV", "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/iF94PTi8+kO5sxgboy4XPB +Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g+iyZLw8mT5ADz9zGUQTf +P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf +GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS+sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/ Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc +iFQaFBvCD3q66fBQ==", "Expiration" : "2014-08-13T23:05:19Z" }
  19. OLD VULNS NEW LIFE These vulnerabilities can result in a

    total data center compromise
 in AWS CWE-918: SSRF CWE-611: XXE CWE-441: Unintended Proxy or Intermediary CWE-77: Command Injection CWE-200: Information Exposure Why? All of these can lead to unintended exposure of metadata
  20. Remote Access to Metadata Real world Examples Prezi experienced this

    first hand http://engineering.prezi.com/blog/2014/03/24/prezi-got- pwned-a-tale-of-responsible-disclosure/ Andres Riancho demonstrated this in his BlackHat talk “Pivoting in Amazon Clouds” Exploited SSRF vuln to inject malicious AWS SQS messages to then exploit a celery/python vuln and take control
  21. Controlling API Access IP Restrictions, they are not just for

    security groups! Keys locked down to specific IP’s (like your VPC gateway) have less chance of damage Doesn’t fully eliminate the problem however { "Version": "2012-10-17", "Statement":[{ "Effect":"Deny", "Action":"*", "Resource":"*", "Condition":{ "NotIpAddress":{ “aws:SourceIp”:[“VPC GATEWAY IP/32”] } } }] }
  22. Your bill is not an IDS Many AWS customers today

    use their bill as their IDS So far, most of these people have been pretty lucky
  23. Implement API Logging…please! Logging is off by default in AWS

    In AWS, turn on CloudTrail for all regions Use LogStash, develop your own or buy a commercial solution to make API activity logs accessible
 ACCESS CONTROL { "Version": "2012-10-17", "Statement": [

    { "Effect": "Allow", "Action": "*", "Resource": "*" } ] }
  25. DevOps Culture tends to “Fail Open” I love DevOps but….

    Be wary of Developers building systems who are new to the Cloud because they might now be the ones configuring the firewall rules and API Access Policies as well Developers and Operations are often pre-disposed to “Just make it work” This is one of the reasons DevOps is such a powerful movement It’s on us as security people to adapt and keep DevOps powerful
  26. Leaking Tags Tags in AWS are one of the primary

    means of keeping track of things Tags however often can contain private data that does not have strong access protections I’ve seen API keys and passwords in tags!!! If you use any 3rd party AWS management system like StackDriver, CloudHealth or others, your tag data is replicated to those systems

    will resolve EC2 Classic public DNS entries to RFC 1918 private address Good chance of Information Exposure (CWE-200) from web servers listening on private IP’s A casual poke around the EC2 address space reveals hundreds of web servers Remediation: Migrate EC2 Classic environments to VPC [ec2-user@heyzeus ~]$ host example.veracodelabs.com example.veracodelabs.com is an alias for ec2-107-22-197-187.compute-1.amazonaws.com. ec2-107-22-197-187.compute-1.amazonaws.com has address
  28. Data and File Leakage AWS S3 1 in 7 buckets

    left open to anyone attempting to access them This might very well be intentional but security through obscurity is a real problem AWS EBS If I have API access, I can clone your running systems and EBS volumes into an external account You can prevent this by encrypting your EBS volumes (you can’t encrypt root volumes however)
  29. The Full Stack Hack 3. SSRF or XXE vulnerability exposes

    EC2 Metadata, revealing AWS API Keys 1. AWS DNS Reveals Private IP of web server 2. Private IP of web server reveals detailed errors or admin interface 4. AWS API Key lets you create new instances where your clone and mount any existing EBS vol. 5. Cloned system gets you SSH keys to app servers and API key with IAM:* giving you access to everything 6. With new root credentials create trust relationship with external account and clone DB for quiet extraction
  30. Capabilities Analyzes IAM users and permissions Requires only readonly API

    access Records all services and objects in use Track changes, know what is in your account quickly Image Cracker Crack open AMI and EBS snapshots and extract access keys, user accounts, data Coming Soon Not quite ready yet, sorry!
  31. IAM Permissions Analysis epeterson@epeterson-mac:~/work/machete$ bin/machete -e 67d54268-dab1-4c73-a424-dbe75d466bb5 refresh Machete -

    v0.1.5 - Connecting... ...connected Starting scan of environment 67d54268-dab1-4c73-a424-dbe75d466bb5 ================================================================================ id: 67d54268-dab1-4c73-a424-dbe75d466bb5|AROAIEMNXXXX3FXXXXXHM6 cust uuid: f4cec92f-9184-4394-a4d3-13b7d8708a13 env uuid: 67d54268-dab1-4c73-a424-dbe75d466bb5 First seen: 2014-10-11 13:38:20.335481+00:00 Last seen: 2014-10-17 10:04:00.754355+00:00 Parent: None Type: IAM_ROLE Tags: {} Metadata: -------------------------------------------------------------------------------- {u'arn': u'arn:aws:iam::034817543207:role/cloudatlas_node', u'assume_role_policy_document': {u'Statement': [{u'Action': u'sts:AssumeRole', u'Effect': u'Allow', u'Principal': {u'Service': u'ec2.amazonaws.com'}, u'Sid': u''}], u'Version': u'2012-10-17'}, u'create_date': u'2014-06-11T20:25:43Z', u'id': u'AXXXXXXNKYAIXXXXXXXX', u'name': u'cloudatlas_node', u'path': u'/', u'role_id': u'AXXXXXXNKYAIXXXXXXXX', u'role_name': u'cloudatlas_node'} -------------------------------------------------------------------------------- Versions: 1 -------------------------------------------------------------------------------- Change detected on: 2014-10-17 12:03:48 {u'added': {}, u'changed': {u'assume_role_policy_document': {u'Statement': [{u'Action': u'sts:AssumeRole', u'Effect': u'Allow', u'Principal': {u'Service': u'ec2.amazonaws.com'}, u'Sid': u''}], u'Version': u'2012-10-17'}}, u'removed': {}} --------------------------------------------------------------------------------
  32. Resource tracking (machete)epeterson@epeterson-mac:~/work/machete/bin$ ./machete list -e 67d54268-dab1-4c73-a424-dbe75d466bb5 # : ID

    Type State First Seen Last Seen Versions -------------------------------------------------------------------------------------------------------------------------------------------- 1 : 54.84.xxx.xxx ADDRESS None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.617682+00:00 No changes 2 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.619777+00:00 No changes 3 : i-xxxxxxxx EC2 stopped 2014-08-21 18:09:49.462541+00:00 2014-09-02 18:23:59.688451+00:00 {u'1408649760.41495': {u'added': {}, u'changed': {u'dns_name': u'', u'ip_address': None, u'public_dns_name': u'', u'reason': u'User initiated (2014-08-21 19:34:02 GMT)', u'state': [u'stopped', 80], u'state_reason': {u'code': u'Client.UserInitiatedShutdown', u'message': u'Client.UserInitiatedShutdown: User initiated shutdown'}}, u'removed': {}}} 4 : i-xxxxxxxx EC2 running 2014-08-29 19:41:35.097611+00:00 2014-09-02 18:23:59.682205+00:00 No changes 5 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.693405+00:00 No changes 6 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.678996+00:00 No changes 7 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.219698+00:00 No changes 8 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.936409+00:00 No changes 9 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.893741+00:00 No changes 10 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.240372+00:00 No changes 11 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.098728+00:00 No changes 12 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:27.484144+00:00 No changes 13 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.956196+00:00 No changes 14 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.691644+00:00 No changes 15 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.354823+00:00 No changes 16 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.590336+00:00 No changes 17 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.966074+00:00 No changes 18 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.913403+00:00 No changes 19 : sg-xxxxxxxx EC2_SECURITY_GROUP None 2014-09-02 18:12:16.600848+00:00 2014-09-02 22:39:30.547159+00:00 No changes
  33. Other Tools Netflix Security Monkey http://techblog.netflix.com/2014/06/announcing-security-monkey-aws-security.html …and a whole ton

    of other OSS stuff you should check out AWS Trusted Advisor https://aws.amazon.com/premiumsupport/trustedadvisor/ NimboStratus https://andresriancho.github.io/nimbostratus/