$30 off During Our Annual Pro Sale. View Details »

Bringing a Machete to the Amazon - AppSec USA 2014

Erik Peterson
September 19, 2014

Bringing a Machete to the Amazon - AppSec USA 2014

So you are on AWS, now what? How do you cut through the jungle and secure your environment for the Cloud?

Erik Peterson

September 19, 2014
Tweet

More Decks by Erik Peterson

Other Decks in Research

Transcript

  1. ! • AppSec guy for 17 years ! • Director

    of Technology Strategy at Veracode ! • Have been researching the Cloud and building products on top of AWS since 2009 Hi, I’m Erik @silvexis
  2. AGENDA I II III IV INTRO OLD  VULNS,   NEW

     LIFE V VI IV METADATA CLOUDATLAS   PROJECT FULL   STACK  HACKS THE  API   IS  KING EMERGENT   INSECURITY
  3. • Java, .Net, etc… including 3rd party code in red

    Traditional App represents small % of system
  4. What you get from AWS • The digital equivalent of

    infinite empty rack space • A friendly looking web interface • A mile long list of compliance certifications ! ! ! ! ! • The rest is up to you
  5. …and after a few months this might be you •

    You have 39 security groups named “Launch-wizard-x”, half of which allow all from 0.0.0.0 • Your bill is 5x what you were expecting • 2 teams are asking you why their systems disappeared and 8 more are un-reachable • You have 176 instances running and you only know what 7 of them do
  6. • Treating AWS as just another hosting company is dangerous

    • For starters, it’s likely the most expensive way to use AWS • IDS and network access controls are unaware of AWS API activity • Certain low priority vulnerabilities become critical Forklifting isn’t safe either
  7. • “If your security sucks now, you’ll be pleasantly surprised

    by the lack of change when you move to Cloud.”* • In reality is it gets worse because of API access and unexpected information leakage – AWS API endpoints are open by default, trumps all existing controls – Private IP’s might be public – Only thing standing between total compromise of your _entire_ datacenter is the secrecy of your API keys What’s your Attack Surface? *Chris  Hoff,  @beaker
  8. The individual components of your system may be secure but

    when deployed into the Cloud, the system becomes insecure EMERGENT INSECURITY AND IT GETS WORSE AT SCALE
  9. Instead of trying to enforce change control which creates brittle

    systems that are insecure and not survivable focus on creating environments that are eventually consistent with your security and operational goals ! If your system requires strict change control to maintain order, in the cloud, you will eventually have chaos. EMERGENT INSECURITY
  10. • The following vulnerabilities can mean total data center compromise

    in AWS • CWE-918: SSRF • CWE-611: XXE • CWE-441: Unintended Proxy or Intermediary • CWE-77: Command Injection • Why? • Metadata! OLD VULNS NEW LIFE
  11. • Based on RFC 3927 - Dynamic Configuration of IPv4

    Link-Local Addresses • Metadata contains all kinds of awesome things, like startup scripts and your AWS access credentials • It’s not just AWS, all cloud providers have it – AWS: http://169.254.169.254/latest/user-data – Google: http://169.254.169.254/computeMetadata/v1/ – OpenStack/RackSpace: http://169.254.169.254/openstack – HP Helion: http://169.254.169.254/2009-04-04/meta-data/ – etc… • There is nothing wrong with any of this, as long as you are aware and protect it What is Cloud Metadata?
  12. • AWS command line tool ec2-metadata will extract some (but

    not all!) of the metadata • To get everything use wget or curl Accessing AWS Metadata $ wget -q -O - http://169.254.169.254/latest/meta-data/! ami-id! ami-launch-index! ami-manifest-path! block-device-mapping/! hostname! iam/! instance-action! instance-id! instance-type! kernel-id! local-hostname! local-ipv4! mac! metrics/! network/! placement/! profile! public-hostname! public-ipv4! public-keys/! reservation-id! security-groups Each  one  of  these   represents  meta   data  you  can  access $ wget -q -O - http://169.254.169.254/latest/meta-data/instance-id! i-0496132e
  13. METADATA CREDENTIAL EXTRACTION [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/

    security-credentials/<YOUR PROFILE HERE>! {! "Code" : "Success",! "LastUpdated" : "2014-08-13T16:43:54Z",! "Type" : "AWS-HMAC",! "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA",! "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV",! "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/ iF94PTi8+kO5sxgboy4XPB+Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g +iyZLw8mT5ADz9zGUQTf+P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/ L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf+GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS +sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc +iFQaFBvCD3q66fBQ==",! "Expiration" : "2014-08-13T23:05:19Z"! }
  14. METADATA CREDENTIAL EXTRACTION [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/

    security-credentials/<YOUR PROFILE HERE>! {! "Code" : "Success",! "LastUpdated" : "2014-08-13T16:43:54Z",! "Type" : "AWS-HMAC",! "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA",! "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV",! "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/ iF94PTi8+kO5sxgboy4XPB+Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g +iyZLw8mT5ADz9zGUQTf+P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/ L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf+GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS +sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc +iFQaFBvCD3q66fBQ==",! "Expiration" : "2014-08-13T23:05:19Z"! }
  15. • Prezi experienced this first hand • http://engineering.prezi.com/blog/ 2014/03/24/prezi-got-pwned-a-tale-of- responsible-disclosure/

    • Andres Riancho demonstrated this in his BlackHat talk “Pivoting in Amazon Clouds” Remote Access to Metadata
  16. • IP Restrictions, they are not just for security groups!

    • Keys locked down to specific IP’s could accidentally find their way in GitHub with less chance of damage • If you don’t need it or use it, you could also block it • route add -host 169.254.169.254 reject Controlling API Access {! "Version": "2012-10-17", ! "Statement":[{! "Effect":"Deny",! "Action":"*",! "Resource":"*",! "Condition":{! "NotIpAddress":{! “aws:SourceIp”:[“YOUR VPC NAT GATEWAY IP/32”]! }! }! }]! }
  17. • Curious how fast an API key can be stolen

    from GitHub? • Create a key, lock it to only one IP address and watch your CloudTrail logs • You should see failed access attempts in 60 minutes or less API Honeypot Anyone?
  18. The Full Stack Hack 3.  SSRF  or  XXE   vulnerability

     exposes   EC2  Metadata,   revealing  AWS  API   Keys 1.  AWS  DNS   Reveals   Private  IP  of   web  server 2.  Private  IP  of   web  server  reveals   detailed  errors  or   admin  interface 4.  AWS  API  Key   lets  you  create  new   instances  where   your  clone  and   mount  any  existing   EBS  vol. 5.  Cloned  system   gets  you  SSH  keys  to   app  servers  and  API   key  with  IAM:*   giving  you  access  to   everything 6.  With  new  root   credentials  create  trust   relationship  with  external   account  and  clone  DB  for   quiet  extraction
  19. RECOVERY DISASTER After a full compromise, the attacker can do

    more than just kill a few systems to cover their tracks, they can nuke the entire datacenter Nuke  the  site  from  orbit,  it’s   the  only  way  to  be  sure
  20. • Lets say I have read only to EBS but

    no access to your servers – Actually, I have access to your servers – Clone the EBS vol’s, mount, extract whatever I need/want • Same is true of RDS, I don’t need your passwords, I just need to snapshot your DB Bypassing Auth with Clones
  21. • How many of you have accounts or API keys

    with a policy that looks like this? EXCESSIVE LACK of ACCESS CONTROL {! "Version": "2012-10-17",! "Statement": [! {! "Effect": "Allow",! "Action": "*",! "Resource": "*"! }! ]! }
  22. • Tags in AWS are one of the primary means

    of keeping track of things • Tags however often can contain private data that does not have strong access protections • If you use any 3rd party AWS management system like StackDriver, CloudHealth or others, your tag data is replicated to those systems TAGS
  23. • AWS DNS from within AWS will resolve EC2 Classic

    public DNS entries to RFC 1918 private address • Good chance of Information Exposure (CWE-200) from web servers listening on private IP’s • A casual poke around the EC2 10.0.0.0/8 address space reveals hundreds of web servers • Remediation: Migrate EC2 Classic environments to VPC PRIVATE IS PUBLIC DNS [ec2-user@heyzeus ~]$ host example.veracodelabs.com! example.veracodelabs.com is an alias for ec2-107-22-197-187.compute-1.amazonaws.com.! ec2-107-22-197-187.compute-1.amazonaws.com has address 10.72.187.67
  24. • Many AWS users use their bill as their IDS

    • So far, most of these people have been pretty damn lucky ! • Don’t be one of these people, turn on CloudTrail • Use LogStash, develop your own or buy a commercial solution to keep track of API activity LACK OF AWARENESS
  25. • CloudAtlas analyzes the full stack of a cloud based

    application • Requires only readonly API access • Free SaaS service for exploring your AWS environment • www.cloudatlas.veracodelabs.com ! • Coming Soon • Web Interface not ready yet, sorry! CloudAtlas
  26. • Things it does: • Enumerate all your services, user

    accounts and permissions • Track and alert on changes in near real time • Map out your applications and services • Perform analysis on services in use CloudAtlas
  27. Service Discovery and Change Tracking (cloudatlas)epeterson@epeterson-mac:~/work/cloudatlas/bin$ ./cloudatlas list -e 67d54268-dab1-4c73-a424-dbe75d466bb5!

    # : ID Type State First Seen Last Seen Versions! --------------------------------------------------------------------------------------------------------------------------------------------! 1 : 54.84.xxx.xxx ADDRESS None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.617682+00:00 No changes! 2 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.619777+00:00 No changes! 3 : i-xxxxxxxx EC2 stopped 2014-08-21 18:09:49.462541+00:00 2014-09-02 18:23:59.688451+00:00! {u'1408649760.41495': {u'added': {},! u'changed': {u'dns_name': u'',! u'ip_address': None,! u'public_dns_name': u'',! u'reason': u'User initiated (2014-08-21 19:34:02 GMT)',! u'state': [u'stopped', 80],! u'state_reason': {u'code': u'Client.UserInitiatedShutdown',! u'message': u'Client.UserInitiatedShutdown: User initiated shutdown'}},! u'removed': {}}}! ! 4 : i-xxxxxxxx EC2 running 2014-08-29 19:41:35.097611+00:00 2014-09-02 18:23:59.682205+00:00 No changes! 5 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.693405+00:00 No changes! 6 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.678996+00:00 No changes! 7 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.219698+00:00 No changes! 8 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.936409+00:00 No changes! 9 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.893741+00:00 No changes! 10 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.240372+00:00 No changes! 11 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.098728+00:00 No changes! 12 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:27.484144+00:00 No changes! 13 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.956196+00:00 No changes! 14 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.691644+00:00 No changes! 15 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.354823+00:00 No changes! 16 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.590336+00:00 No changes! 17 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.966074+00:00 No changes! 18 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.913403+00:00 No changes! 19 : sg-xxxxxxxx EC2_SECURITY_GROUP None 2014-09-02 18:12:16.600848+00:00 2014-09-02 22:39:30.547159+00:00 No changes
  28. • Image cracking: • Capable of cracking open AMI’s and

    EBS volumes to enumerate applications and systems • IAM Permission Analysis • Complete analysis and reporting of who can do what to what, when and where from Adv. Capabilities
  29. • Netflix  Security  Monkey   – http://techblog.netflix.com/2014/06/announcing-­‐ security-­‐monkey-­‐aws-­‐security.html   –

    …and  a  whole  ton  of  other  OSS   • AWS  Trusted  Advisor   – https://aws.amazon.com/premiumsupport/ trustedadvisor/   • NimboStratus   – https://andresriancho.github.io/nimbostratus/ Other Tools