Bringing a Machete to the Amazon - AppSec USA 2014

So you are on AWS, now what? How do you cut through the jungle and secure your environment for the Cloud?

Transcript

1. AppSec USA 2014! Denver, Colorado Erik Peterson - @silvexis - epeterson@veracode.com Bringing a Machete to the Amazon
2. ! • AppSec guy for 17 years ! • Director of Technology Strategy at Veracode ! • Have been researching the Cloud and building products on top of AWS since 2009 Hi, I’m Erik @silvexis
3. AGENDA I II III IV INTRO OLD  VULNS,   NEW  LIFE V VI IV METADATA CLOUDATLAS   PROJECT FULL   STACK  HACKS THE  API   IS  KING EMERGENT   INSECURITY
4. CLOUD
 IS AN OPERATING SYSTEM
5. CLOUD INFRASTRUCTURE IS CODE
6. A Fully Realized AWS Application
7. • Java, .Net, etc… including 3rd party code in red Traditional App represents small % of system
8. • Using it all securely however is your responsibility Majority of the system provided by AWS
9. What you get from AWS • The digital equivalent of infinite empty rack space • A friendly looking web interface • A mile long list of compliance certifications ! ! ! ! ! • The rest is up to you
10. …and after a few months this might be you • You have 39 security groups named “Launch-wizard-x”, half of which allow all from 0.0.0.0 • Your bill is 5x what you were expecting • 2 teams are asking you why their systems disappeared and 8 more are un-reachable • You have 176 instances running and you only know what 7 of them do
11. • Treating AWS as just another hosting company is dangerous • For starters, it’s likely the most expensive way to use AWS • IDS and network access controls are unaware of AWS API activity • Certain low priority vulnerabilities become critical Forklifting isn’t safe either
12. • “If your security sucks now, you’ll be pleasantly surprised by the lack of change when you move to Cloud.”* • In reality is it gets worse because of API access and unexpected information leakage – AWS API endpoints are open by default, trumps all existing controls – Private IP’s might be public – Only thing standing between total compromise of your _entire_ datacenter is the secrecy of your API keys What’s your Attack Surface? *Chris  Hoff,  @beaker
13. The individual components of your system may be secure but when deployed into the Cloud, the system becomes insecure EMERGENT INSECURITY AND IT GETS WORSE AT SCALE
14. Instead of trying to enforce change control which creates brittle systems that are insecure and not survivable focus on creating environments that are eventually consistent with your security and operational goals ! If your system requires strict change control to maintain order, in the cloud, you will eventually have chaos. EMERGENT INSECURITY
15. In the cloud, the king of the jungle is the API THE API IS KING
16. • The following vulnerabilities can mean total data center compromise in AWS • CWE-918: SSRF • CWE-611: XXE • CWE-441: Unintended Proxy or Intermediary • CWE-77: Command Injection • Why? • Metadata! OLD VULNS NEW LIFE
18. • Based on RFC 3927 - Dynamic Configuration of IPv4 Link-Local Addresses • Metadata contains all kinds of awesome things, like startup scripts and your AWS access credentials • It’s not just AWS, all cloud providers have it – AWS: http://169.254.169.254/latest/user-data – Google: http://169.254.169.254/computeMetadata/v1/ – OpenStack/RackSpace: http://169.254.169.254/openstack – HP Helion: http://169.254.169.254/2009-04-04/meta-data/ – etc… • There is nothing wrong with any of this, as long as you are aware and protect it What is Cloud Metadata?
19. • AWS command line tool ec2-metadata will extract some (but not all!) of the metadata • To get everything use wget or curl Accessing AWS Metadata $ wget -q -O - http://169.254.169.254/latest/meta-data/! ami-id! ami-launch-index! ami-manifest-path! block-device-mapping/! hostname! iam/! instance-action! instance-id! instance-type! kernel-id! local-hostname! local-ipv4! mac! metrics/! network/! placement/! profile! public-hostname! public-ipv4! public-keys/! reservation-id! security-groups Each  one  of  these   represents  meta   data  you  can  access $ wget -q -O - http://169.254.169.254/latest/meta-data/instance-id! i-0496132e
20. METADATA CREDENTIAL EXTRACTION [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/ security-credentials/<YOUR PROFILE HERE>! {! "Code" : "Success",! "LastUpdated" : "2014-08-13T16:43:54Z",! "Type" : "AWS-HMAC",! "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA",! "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV",! "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/ iF94PTi8+kO5sxgboy4XPB+Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g +iyZLw8mT5ADz9zGUQTf+P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/ L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf+GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS +sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc +iFQaFBvCD3q66fBQ==",! "Expiration" : "2014-08-13T23:05:19Z"! }
21. METADATA CREDENTIAL EXTRACTION [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/ security-credentials/<YOUR PROFILE HERE>! {! "Code" : "Success",! "LastUpdated" : "2014-08-13T16:43:54Z",! "Type" : "AWS-HMAC",! "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA",! "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV",! "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/ iF94PTi8+kO5sxgboy4XPB+Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g +iyZLw8mT5ADz9zGUQTf+P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/ L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf+GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS +sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc +iFQaFBvCD3q66fBQ==",! "Expiration" : "2014-08-13T23:05:19Z"! }
22. • Prezi experienced this first hand • http://engineering.prezi.com/blog/ 2014/03/24/prezi-got-pwned-a-tale-of- responsible-disclosure/ • Andres Riancho demonstrated this in his BlackHat talk “Pivoting in Amazon Clouds” Remote Access to Metadata
23. • IP Restrictions, they are not just for security groups! • Keys locked down to specific IP’s could accidentally find their way in GitHub with less chance of damage • If you don’t need it or use it, you could also block it • route add -host 169.254.169.254 reject Controlling API Access {! "Version": "2012-10-17", ! "Statement":[{! "Effect":"Deny",! "Action":"*",! "Resource":"*",! "Condition":{! "NotIpAddress":{! “aws:SourceIp”:[“YOUR VPC NAT GATEWAY IP/32”]! }! }! }]! }
24. • Curious how fast an API key can be stolen from GitHub? • Create a key, lock it to only one IP address and watch your CloudTrail logs • You should see failed access attempts in 60 minutes or less API Honeypot Anyone?
25. The Full Stack Hack 3.  SSRF  or  XXE   vulnerability  exposes   EC2  Metadata,   revealing  AWS  API   Keys 1.  AWS  DNS   Reveals   Private  IP  of   web  server 2.  Private  IP  of   web  server  reveals   detailed  errors  or   admin  interface 4.  AWS  API  Key   lets  you  create  new   instances  where   your  clone  and   mount  any  existing   EBS  vol. 5.  Cloned  system   gets  you  SSH  keys  to   app  servers  and  API   key  with  IAM:*   giving  you  access  to   everything 6.  With  new  root   credentials  create  trust   relationship  with  external   account  and  clone  DB  for   quiet  extraction
26. RECOVERY DISASTER After a full compromise, the attacker can do more than just kill a few systems to cover their tracks, they can nuke the entire datacenter Nuke  the  site  from  orbit,  it’s   the  only  way  to  be  sure
27. CLONE  BYPASS
28. • Lets say I have read only to EBS but no access to your servers – Actually, I have access to your servers – Clone the EBS vol’s, mount, extract whatever I need/want • Same is true of RDS, I don’t need your passwords, I just need to snapshot your DB Bypassing Auth with Clones
29. EXCESSIVE  LACK  of  ACCESS  CONTROL
30. • How many of you have accounts or API keys with a policy that looks like this? EXCESSIVE LACK of ACCESS CONTROL {! "Version": "2012-10-17",! "Statement": [! {! "Effect": "Allow",! "Action": "*",! "Resource": "*"! }! ]! }
31. • Tags • IP Addresses YOU’RE LEAKING
32. • Tags in AWS are one of the primary means of keeping track of things • Tags however often can contain private data that does not have strong access protections • If you use any 3rd party AWS management system like StackDriver, CloudHealth or others, your tag data is replicated to those systems TAGS
33. • AWS DNS from within AWS will resolve EC2 Classic public DNS entries to RFC 1918 private address • Good chance of Information Exposure (CWE-200) from web servers listening on private IP’s • A casual poke around the EC2 10.0.0.0/8 address space reveals hundreds of web servers • Remediation: Migrate EC2 Classic environments to VPC PRIVATE IS PUBLIC DNS [ec2-user@heyzeus ~]$ host example.veracodelabs.com! example.veracodelabs.com is an alias for ec2-107-22-197-187.compute-1.amazonaws.com.! ec2-107-22-197-187.compute-1.amazonaws.com has address 10.72.187.67
34. • Many AWS users use their bill as their IDS • So far, most of these people have been pretty damn lucky ! • Don’t be one of these people, turn on CloudTrail • Use LogStash, develop your own or buy a commercial solution to keep track of API activity LACK OF AWARENESS
35. CLOUDATLAS LOOKING BEHIND THE CLOUDS
36. • CloudAtlas analyzes the full stack of a cloud based application • Requires only readonly API access • Free SaaS service for exploring your AWS environment • www.cloudatlas.veracodelabs.com ! • Coming Soon • Web Interface not ready yet, sorry! CloudAtlas
37. • Things it does: • Enumerate all your services, user accounts and permissions • Track and alert on changes in near real time • Map out your applications and services • Perform analysis on services in use CloudAtlas
38. Service Discovery and Change Tracking (cloudatlas)epeterson@epeterson-mac:~/work/cloudatlas/bin$ ./cloudatlas list -e 67d54268-dab1-4c73-a424-dbe75d466bb5! # : ID Type State First Seen Last Seen Versions! --------------------------------------------------------------------------------------------------------------------------------------------! 1 : 54.84.xxx.xxx ADDRESS None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.617682+00:00 No changes! 2 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.619777+00:00 No changes! 3 : i-xxxxxxxx EC2 stopped 2014-08-21 18:09:49.462541+00:00 2014-09-02 18:23:59.688451+00:00! {u'1408649760.41495': {u'added': {},! u'changed': {u'dns_name': u'',! u'ip_address': None,! u'public_dns_name': u'',! u'reason': u'User initiated (2014-08-21 19:34:02 GMT)',! u'state': [u'stopped', 80],! u'state_reason': {u'code': u'Client.UserInitiatedShutdown',! u'message': u'Client.UserInitiatedShutdown: User initiated shutdown'}},! u'removed': {}}}! ! 4 : i-xxxxxxxx EC2 running 2014-08-29 19:41:35.097611+00:00 2014-09-02 18:23:59.682205+00:00 No changes! 5 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.693405+00:00 No changes! 6 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.678996+00:00 No changes! 7 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.219698+00:00 No changes! 8 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.936409+00:00 No changes! 9 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.893741+00:00 No changes! 10 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.240372+00:00 No changes! 11 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.098728+00:00 No changes! 12 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:27.484144+00:00 No changes! 13 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.956196+00:00 No changes! 14 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.691644+00:00 No changes! 15 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.354823+00:00 No changes! 16 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.590336+00:00 No changes! 17 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.966074+00:00 No changes! 18 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.913403+00:00 No changes! 19 : sg-xxxxxxxx EC2_SECURITY_GROUP None 2014-09-02 18:12:16.600848+00:00 2014-09-02 22:39:30.547159+00:00 No changes
39. • Image cracking: • Capable of cracking open AMI’s and EBS volumes to enumerate applications and systems • IAM Permission Analysis • Complete analysis and reporting of who can do what to what, when and where from Adv. Capabilities
40. • Netflix  Security  Monkey   – http://techblog.netflix.com/2014/06/announcing-­‐ security-­‐monkey-­‐aws-­‐security.html   – …and  a  whole  ton  of  other  OSS   • AWS  Trusted  Advisor   – https://aws.amazon.com/premiumsupport/ trustedadvisor/   • NimboStratus   – https://andresriancho.github.io/nimbostratus/ Other Tools
41. THANK YOU May all your Clouds come with a Luck Dragon AppSec  USA  2014 @silvexis epeterson@veracode.com