$30 off During Our Annual Pro Sale. View Details »

Bringing a Machete to the Amazon - AppSec USA 2014

Erik Peterson
September 19, 2014

Bringing a Machete to the Amazon - AppSec USA 2014

So you are on AWS, now what? How do you cut through the jungle and secure your environment for the Cloud?

Erik Peterson

September 19, 2014
Tweet

More Decks by Erik Peterson

Other Decks in Research

Transcript

  1. AppSec USA 2014!
    Denver, Colorado
    Erik Peterson - @silvexis - [email protected]
    Bringing a Machete to the Amazon

    View Slide

  2. !
    • AppSec guy for 17 years
    !
    • Director of Technology
    Strategy at Veracode
    !
    • Have been researching the
    Cloud and building products
    on top of AWS since 2009
    Hi, I’m Erik
    @silvexis

    View Slide

  3. AGENDA
    I II III IV
    INTRO OLD  VULNS,  
    NEW  LIFE
    V VI IV
    METADATA CLOUDATLAS  
    PROJECT
    FULL  
    STACK  HACKS
    THE  API  
    IS  KING
    EMERGENT  
    INSECURITY

    View Slide

  4. CLOUD

    IS AN
    OPERATING
    SYSTEM

    View Slide

  5. CLOUD
    INFRASTRUCTURE
    IS
    CODE

    View Slide

  6. A Fully Realized AWS Application

    View Slide

  7. • Java, .Net, etc… including 3rd party code in red
    Traditional App represents small % of system

    View Slide

  8. • Using it all securely however is your responsibility
    Majority of the system provided by AWS

    View Slide

  9. What you get from AWS
    • The digital equivalent of
    infinite empty rack space
    • A friendly looking web
    interface
    • A mile long list of
    compliance certifications
    !
    !
    !
    !
    !
    • The rest is up to you

    View Slide

  10. …and after a few months this might be you
    • You have 39 security groups
    named “Launch-wizard-x”,
    half of which allow all from
    0.0.0.0
    • Your bill is 5x what you were
    expecting
    • 2 teams are asking you why
    their systems disappeared
    and 8 more are un-reachable
    • You have 176 instances
    running and you only know
    what 7 of them do

    View Slide

  11. • Treating AWS as just another hosting company is
    dangerous
    • For starters, it’s likely the most expensive way to
    use AWS
    • IDS and network access controls are unaware of
    AWS API activity
    • Certain low priority vulnerabilities become
    critical
    Forklifting isn’t safe either

    View Slide

  12. • “If your security sucks now, you’ll be pleasantly
    surprised by the lack of change when you move to
    Cloud.”*
    • In reality is it gets worse because of API access and
    unexpected information leakage
    – AWS API endpoints are open by default, trumps
    all existing controls
    – Private IP’s might be public
    – Only thing standing between total compromise
    of your _entire_ datacenter is the secrecy of
    your API keys
    What’s your Attack Surface?
    *Chris  Hoff,  @beaker

    View Slide

  13. The individual components of
    your system may be secure but
    when deployed into the Cloud,
    the system becomes insecure
    EMERGENT INSECURITY
    AND IT GETS WORSE AT SCALE

    View Slide

  14. Instead of trying to enforce change control
    which creates brittle systems that are insecure
    and not survivable focus on creating
    environments that are eventually consistent
    with your security and operational goals
    !
    If your system requires strict change control to
    maintain order, in the cloud, you will eventually
    have chaos.
    EMERGENT INSECURITY

    View Slide

  15. In the cloud,
    the king of
    the jungle is
    the API
    THE API IS KING

    View Slide

  16. • The following vulnerabilities can mean
    total data center compromise in AWS
    • CWE-918: SSRF
    • CWE-611: XXE
    • CWE-441: Unintended Proxy or
    Intermediary
    • CWE-77: Command Injection
    • Why?
    • Metadata!
    OLD VULNS NEW LIFE

    View Slide

  17. View Slide

  18. • Based on RFC 3927 - Dynamic Configuration of IPv4 Link-Local
    Addresses
    • Metadata contains all kinds of awesome things, like startup scripts
    and your AWS access credentials
    • It’s not just AWS, all cloud providers have it
    – AWS: http://169.254.169.254/latest/user-data
    – Google: http://169.254.169.254/computeMetadata/v1/
    – OpenStack/RackSpace: http://169.254.169.254/openstack
    – HP Helion: http://169.254.169.254/2009-04-04/meta-data/
    – etc…
    • There is nothing wrong with any of this, as long as you are aware
    and protect it
    What is Cloud Metadata?

    View Slide

  19. • AWS command line tool ec2-metadata will extract some
    (but not all!) of the metadata
    • To get everything use wget or curl
    Accessing AWS Metadata
    $ wget -q -O - http://169.254.169.254/latest/meta-data/!
    ami-id!
    ami-launch-index!
    ami-manifest-path!
    block-device-mapping/!
    hostname!
    iam/!
    instance-action!
    instance-id!
    instance-type!
    kernel-id!
    local-hostname!
    local-ipv4!
    mac!
    metrics/!
    network/!
    placement/!
    profile!
    public-hostname!
    public-ipv4!
    public-keys/!
    reservation-id!
    security-groups
    Each  one  of  these  
    represents  meta  
    data  you  can  access
    $ wget -q -O - http://169.254.169.254/latest/meta-data/instance-id!
    i-0496132e

    View Slide

  20. METADATA CREDENTIAL EXTRACTION
    [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/
    security-credentials/!
    {!
    "Code" : "Success",!
    "LastUpdated" : "2014-08-13T16:43:54Z",!
    "Type" : "AWS-HMAC",!
    "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA",!
    "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV",!
    "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD
    +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/
    iF94PTi8+kO5sxgboy4XPB+Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g
    +iyZLw8mT5ADz9zGUQTf+P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/
    L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf+GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS
    +sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia
    +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr
    +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt
    +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc
    +iFQaFBvCD3q66fBQ==",!
    "Expiration" : "2014-08-13T23:05:19Z"!
    }

    View Slide

  21. METADATA CREDENTIAL EXTRACTION
    [ec2-user@ip-10-0-1-125 ~]$ wget -q -O - http://169.254.169.254/latest/meta-data/iam/
    security-credentials/!
    {!
    "Code" : "Success",!
    "LastUpdated" : "2014-08-13T16:43:54Z",!
    "Type" : "AWS-HMAC",!
    "AccessKeyId" : "ASIAJ4YSTDWONSUFPHIA",!
    "SecretAccessKey" : "iqFuJWcj9AUaBXe0tbuc6MC70oQW2wehWufZ9cQV",!
    "Token" : "AQoDYXdzEGIa0AMskSUj1Ing9DHLT1QmD
    +vDimxTCnAbrNGcGPbV9jEPYO5LDMMLBAjVdklFo7vS8HnEDrH3ea0T7f8aXW9BGMSdc/
    iF94PTi8+kO5sxgboy4XPB+Bh44xHSKFV4WIrMKfMUwAftcieER7z6CakegOoe6Q/H0PsK9GpS1pO6g
    +iyZLw8mT5ADz9zGUQTf+P3anQ3dAl32SWYEiJR0fTQCuKqE8/dpLbnmdhOn3WyW8eF3TJFPd8/
    L0MQak3EMgo1pAxm+eWAMj1B5Crewy4sbvBzf+GcemFYiMClsY9gFxZCxOexV09j9nPos/d9VRpFakm1tWAS
    +sqHKz1zxLidWJewUfuhyLSxcR5xOeZYJ6/Pt6bQitf21ep6FJExEGE3Ho0A10z4tv9Yo5c2tPafEhWsACBOia
    +kpQExftmuIulmkRK9NugNuKcd0OzDkoftkpIFAj09oP2tgsDuImc0R3LScijbmhgLZsG1UuEpNpTnr
    +7DmbvmrJWihfTD2hJzFAzhvsvN8ytt
    +mWkSGAeBhA6UosCgj2VjBIkHG6kkTcL9FnEU7XPoKqVsiusRqwZYOOeITsL4dE38pcfHhmBPaCI6H5TbjO4FuRQc
    +iFQaFBvCD3q66fBQ==",!
    "Expiration" : "2014-08-13T23:05:19Z"!
    }

    View Slide

  22. • Prezi experienced this first hand
    • http://engineering.prezi.com/blog/
    2014/03/24/prezi-got-pwned-a-tale-of-
    responsible-disclosure/
    • Andres Riancho demonstrated this in his
    BlackHat talk “Pivoting in Amazon Clouds”
    Remote Access to Metadata

    View Slide

  23. • IP Restrictions, they are not just for security
    groups!
    • Keys locked down to specific IP’s could accidentally find
    their way in GitHub with less chance of damage
    • If you don’t need it or use it, you could also block it
    • route add -host 169.254.169.254 reject
    Controlling API Access
    {!
    "Version": "2012-10-17", !
    "Statement":[{!
    "Effect":"Deny",!
    "Action":"*",!
    "Resource":"*",!
    "Condition":{!
    "NotIpAddress":{!
    “aws:SourceIp”:[“YOUR VPC NAT GATEWAY IP/32”]!
    }!
    }!
    }]!
    }

    View Slide

  24. • Curious how fast an API key can be
    stolen from GitHub?
    • Create a key, lock it to only one IP
    address and watch your CloudTrail
    logs
    • You should see failed access
    attempts in 60 minutes or less
    API Honeypot Anyone?

    View Slide

  25. The Full Stack Hack
    3.  SSRF  or  XXE  
    vulnerability  exposes  
    EC2  Metadata,  
    revealing  AWS  API  
    Keys
    1.  AWS  DNS  
    Reveals  
    Private  IP  of  
    web  server
    2.  Private  IP  of  
    web  server  reveals  
    detailed  errors  or  
    admin  interface
    4.  AWS  API  Key  
    lets  you  create  new  
    instances  where  
    your  clone  and  
    mount  any  existing  
    EBS  vol.
    5.  Cloned  system  
    gets  you  SSH  keys  to  
    app  servers  and  API  
    key  with  IAM:*  
    giving  you  access  to  
    everything
    6.  With  new  root  
    credentials  create  trust  
    relationship  with  external  
    account  and  clone  DB  for  
    quiet  extraction

    View Slide

  26. RECOVERY DISASTER
    After a full compromise, the attacker can do more
    than just kill a few systems to cover their tracks, they
    can nuke the entire datacenter
    Nuke  the  site  from  orbit,  it’s  
    the  only  way  to  be  sure

    View Slide

  27. CLONE  BYPASS

    View Slide

  28. • Lets say I have read only to EBS but no
    access to your servers
    – Actually, I have access to your servers
    – Clone the EBS vol’s, mount, extract
    whatever I need/want
    • Same is true of RDS, I don’t need your
    passwords, I just need to snapshot your
    DB
    Bypassing Auth with Clones

    View Slide

  29. EXCESSIVE  LACK  of  ACCESS  CONTROL

    View Slide

  30. • How many of you have accounts or
    API keys with a policy that looks like
    this?
    EXCESSIVE LACK of ACCESS CONTROL
    {!
    "Version": "2012-10-17",!
    "Statement": [!
    {!
    "Effect": "Allow",!
    "Action": "*",!
    "Resource": "*"!
    }!
    ]!
    }

    View Slide

  31. • Tags
    • IP Addresses
    YOU’RE LEAKING

    View Slide

  32. • Tags in AWS are one of the primary means of
    keeping track of things
    • Tags however often can contain private data that
    does not have strong access protections
    • If you use any 3rd party AWS management system
    like StackDriver, CloudHealth or others, your tag
    data is replicated to those systems
    TAGS

    View Slide

  33. • AWS DNS from within AWS will resolve EC2 Classic public
    DNS entries to RFC 1918 private address
    • Good chance of Information Exposure (CWE-200) from
    web servers listening on private IP’s
    • A casual poke around the EC2 10.0.0.0/8 address space
    reveals hundreds of web servers
    • Remediation: Migrate EC2 Classic environments to VPC
    PRIVATE IS PUBLIC DNS
    [ec2-user@heyzeus ~]$ host example.veracodelabs.com!
    example.veracodelabs.com is an alias for
    ec2-107-22-197-187.compute-1.amazonaws.com.!
    ec2-107-22-197-187.compute-1.amazonaws.com has address 10.72.187.67

    View Slide

  34. • Many AWS users use their bill as their IDS
    • So far, most of these people have been
    pretty damn lucky
    !
    • Don’t be one of these people, turn on
    CloudTrail
    • Use LogStash, develop your own or buy
    a commercial solution to keep track of
    API activity
    LACK OF AWARENESS

    View Slide

  35. CLOUDATLAS
    LOOKING BEHIND THE CLOUDS

    View Slide

  36. • CloudAtlas analyzes the full stack of a
    cloud based application
    • Requires only readonly API access
    • Free SaaS service for exploring your
    AWS environment
    • www.cloudatlas.veracodelabs.com
    !
    • Coming Soon
    • Web Interface not ready yet, sorry!
    CloudAtlas

    View Slide

  37. • Things it does:
    • Enumerate all your services, user
    accounts and permissions
    • Track and alert on changes in near
    real time
    • Map out your applications and
    services
    • Perform analysis on services in use
    CloudAtlas

    View Slide

  38. Service Discovery and Change Tracking
    (cloudatlas)epeterson@epeterson-mac:~/work/cloudatlas/bin$ ./cloudatlas list -e 67d54268-dab1-4c73-a424-dbe75d466bb5!
    # : ID Type State First Seen Last Seen Versions!
    --------------------------------------------------------------------------------------------------------------------------------------------!
    1 : 54.84.xxx.xxx ADDRESS None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.617682+00:00 No changes!
    2 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.619777+00:00 No changes!
    3 : i-xxxxxxxx EC2 stopped 2014-08-21 18:09:49.462541+00:00 2014-09-02 18:23:59.688451+00:00!
    {u'1408649760.41495': {u'added': {},!
    u'changed': {u'dns_name': u'',!
    u'ip_address': None,!
    u'public_dns_name': u'',!
    u'reason': u'User initiated (2014-08-21 19:34:02 GMT)',!
    u'state': [u'stopped', 80],!
    u'state_reason': {u'code': u'Client.UserInitiatedShutdown',!
    u'message': u'Client.UserInitiatedShutdown: User initiated shutdown'}},!
    u'removed': {}}}!
    !
    4 : i-xxxxxxxx EC2 running 2014-08-29 19:41:35.097611+00:00 2014-09-02 18:23:59.682205+00:00 No changes!
    5 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.693405+00:00 No changes!
    6 : i-xxxxxxxx EC2 running 2014-08-21 15:34:16.964497+00:00 2014-09-02 18:23:59.678996+00:00 No changes!
    7 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.219698+00:00 No changes!
    8 : igw-xxxxxxxx VPC_INTERNET_GATEWAY None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.936409+00:00 No changes!
    9 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.893741+00:00 No changes!
    10 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.240372+00:00 No changes!
    11 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.098728+00:00 No changes!
    12 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:27.484144+00:00 No changes!
    13 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.956196+00:00 No changes!
    14 : r-xxxxxxxx EC2_RESERVATION None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:28.691644+00:00 No changes!
    15 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.354823+00:00 No changes!
    16 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:24.590336+00:00 No changes!
    17 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:26.966074+00:00 No changes!
    18 : rtb-xxxxxxxx VPC_ROUTE_TABLE None 2014-09-02 18:23:45.874644+00:00 2014-09-02 22:39:25.913403+00:00 No changes!
    19 : sg-xxxxxxxx EC2_SECURITY_GROUP None 2014-09-02 18:12:16.600848+00:00 2014-09-02 22:39:30.547159+00:00 No changes

    View Slide

  39. • Image cracking:
    • Capable of cracking open AMI’s and
    EBS volumes to enumerate
    applications and systems
    • IAM Permission Analysis
    • Complete analysis and reporting of
    who can do what to what, when and
    where from
    Adv. Capabilities

    View Slide

  40. • Netflix  Security  Monkey  
    – http://techblog.netflix.com/2014/06/announcing-­‐
    security-­‐monkey-­‐aws-­‐security.html  
    – …and  a  whole  ton  of  other  OSS  
    • AWS  Trusted  Advisor  
    – https://aws.amazon.com/premiumsupport/
    trustedadvisor/  
    • NimboStratus  
    – https://andresriancho.github.io/nimbostratus/
    Other Tools

    View Slide

  41. THANK YOU
    May all your Clouds come with a Luck Dragon
    AppSec  USA  2014
    @silvexis [email protected]

    View Slide