QCon NY 2015: AWS Security Incident Tutorial

https://qconnewyork.com/ny2015/tutorial/security-incident-response-aws-best-practices-and-prerequisites

Tutorial: Security Incident Response in AWS: Best Practices and Prerequisites

What do you do when you are woken up from sleep for a security incident in your AWS environment?

In this detailed session, we’ll go over how to survive security incidents in AWS:
• Anatomy of an attack or compromise
• Engaging AWS Support for fast response
• Log and data gathering
• Mitigation techniques
• Top security best practices in AWS

Transcript

1. Security Incident Response in AWS: Best Practices and Prerequisites John Martinez Principal Solutions Architect 1 About Me ▪Been doing DevOps and Cloud stuff for ~5 years ▪Helped architect and build Creative Cloud @ Adobe ▪Cut my teeth on “the cloud” at Netflix ▪UNIX and Linux throat beard for >20 years ▪Have been involved in countless security and production incidents, most at 2:00 AM ▪I now talk to people about security for a living ▪Live in Silicon Valley ▪I’m completely addicted to building Raspberry Pi 2’s for random things around the house 2 2 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
2. About You ▪Who are you? Name, Company, Title/Job, Location, AWS Experience, Random Fact 3 3 Agenda - 1st Hour ▪Intros ▪AWS Shared Security Responsibility Model ▪Incident Response in AWS ▪Anatomy of an Attack ▪~10 minute break 4 AGENDA IS FLEXIBLE ADJUSTMENTS WILL BE MADE 4 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
3. Agenda - 2nd Hour ▪Engaging AWS Support ▪Log and Data on your side ▪Mitigation Techniques ▪In-class Exercise (time permitting) ▪~10 minute break 5 5 Agenda - 3rd Hour ▪Top Security Best Practices and Exploitability ▪A Practitioner’s Tool Chest ▪Real-world experiences from you ▪Review, Q&A, Lunch time! 6 6 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
4. Expectations ▪You have created at least one AWS account ▪You have started at least once EC2 instance ▪You have basic familiarity with AWS IAM ▪You have a browser open to the AWS Console and/or have a Terminal open with aws-­‐cli installed, configured and ready to go ▪Hacking in class is not only allowed, it’s encouraged! ▪And above all else, PLEASE INTERRUPT ME 7 7 Please respect each other’s privacy Sensitive security topics will be discussed and shared by all of us. Please keep that in mind when you leave here. If you must use anecdote please make sure to anonymize. 8 8 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
5. 1st Hour 9 9 AWS Shared Security Responsibility 10 10 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
6. Shared Security Responsibility Model 11 http://aws.amazon.com/compliance/shared-responsibility-model/ 11 Shared Security Responsibility - AWS ▪AWS owns responsibility for all things physical ▪They build the services and APIs you need to deploy and run your applications on AWS ▪They deliver to their customers the ability to create identity credentials to access AWS services and resources ▪They secure the hypervisor and the network below layer 7 12 12 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
7. Shared Security Responsibility - You ▪Creating and managing IAM entities and their authentication tokens ▪Creating and managing IAM access policies ▪Creating and managing network access to EC2 instances and VPC networks ▪Deploying applications and creating virtual infrastructure ▪OS Patching and Maintenance ▪Monitoring activity and collecting data ▪Designing for failure 13 13 AWS Security Services - Identity ▪Identity and Access Management (IAM) ▪Secure Token Service (STS) ▪Directory Service 14 14 For IAM, it’s important to learn as much as you can about managing IAM policies. IAM policies are very powerful and provide a great way for you to limit access to specific resources via the API (and AWS Console). Also, use IAM managed policies instead of in-line policies. STS is a great way for you to provide access to 3rd parties like Evident.io into your AWS accounts. Temporary authentication tokens are brokered by AWS. No key management by you or by your 3rd party vendor. STS is also useful for federating identities to an identity provider external to AWS, such as SAML. Manage your API Access Keys. More will be covered later, but a fair warning that in the customer responsibility list, API access keys and EC2 Security Groups are high on the list of AWS resources to secure. QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
8. AWS Security Services - Encryption ▪Key Management Service ▪CloudHSM 15 15 KMS is a great way to encrypt AWS services with your keys AWS Security Services - EC2 ▪EC2 Security Groups ▪EC2 Keypairs (SSH) ▪VPC Subnet ACLs 16 16 EC2 security groups are the #1 touch-point for customer responsibility. Along with API Access Keys, the top, as discussed. Security Groups are stateful firewalls VPC Subnet ACLs are stateless SSH keypairs can be managed outside of AWS and uploaded. Commands to use: ssh-­‐keygen  -­‐t  rsa  -­‐C  <key-­‐name>   aws  -­‐-­‐profile  <profile>  ec2  import-­‐key-­‐pair  -­‐-­‐key-­‐name  <key-­‐name>  —public-­‐ key-­‐material  <file>   QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
9. AWS Security Services - Data/Log Analysis ▪CloudTrail ▪Config ▪CloudWatch Logs 17 17 CloudTrail is priority #0 when it comes to customer responsibility. As seen later, data gathering is important in incident response, and CloudTrail data is a big part of that. AWS Security Services - Storage ▪S3 ACLs and Bucket Policies ▪S3 Lifecycle Rules ▪EBS Volume Encryption 18 18 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
10. Incident Response in AWS 19 19 Incident Response: Not very different ▪Have a plan ▪NOC and SOC roles ▪Communication is key ▪Your AWS/Cloud guru may be your best ally if you don’t have an explicit “Cloud Security” Architect/Engineer 20 20 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
11. Incident Response: somewhat different ▪Your AWS/Cloud guru may be your best ally if you don’t have an explicit “Cloud Security” Architect/Engineer ▪DevOps can play a major role ▪Engage AWS Support often and early (more on that later) ▪Come armed with data (also more on that later) ▪Download latest CSA Guidance, Domain 9 “Incident Response” https://cloudsecurityalliance.org/research/security-guidance/ 21 21 Triage the Incident Evaluate the situation ▪What’s going on right now? ▪No time to freak out 22 22 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
12. Triage the Incident Stop the bleeding ▪Secure the site/isolate the damage ▪May need to bring down the site temporarily 23 23 Triage the Incident Start the breathing ▪Get the business running again ▪Restart services when you’re ready 24 24 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
13. Triage the Incident Protect the wound ▪Investigate root cause ▪Data is king 25 25 Triage the Incident Treat for Shock ▪Make it better (so it does not happen again) ▪Learn from your mistakes ▪Monitor continuously 26 26 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
14. Security Incident Checklist ✓Have both a plan and tools (and have tested them) ✓Define what and when it happened and/or ✓Define what and when is was not happening ✓Initial triage of the problem ✓The more detail you can collect, the better ✓Scale up - Scale out - Isolate the issue ✓Review CloudTrail Logs ✓Snapshot/dd effected resources ✓Open a support case with AWS early ✓Update Often 27 27 Anatomy of an Attack 28 28 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
15. What does an attack in AWS look like? ▪More than likely, you don’t even know it’s going on ▪Sometimes, an email from the AWS Security will come in that an attack is originating from one of your EC2 instances ▪Preventative tools like Evident.io can help reduce the amount of surprises ▪Dev accounts are just as important to secure as production 29 29 Many times, a compromised instance will be used to attack other AWS customers. This is pretty common, especially when an attacker is trying to deploy a botnet. This is why it’s so important to safeguard your credentials and other critical data in AWS. We will discuss prevention in more detail when we cover the “Top 10 Security Best Practices Section” What does an attack in AWS look like? for EC2 ▪Most common type of attack ▪EC2 IP address space is public https://ip-ranges.amazonaws.com/ip-ranges.json ▪Security Group rules are too open ▪Vulnerable applications ▪Vulnerable operating systems ▪Instances in unused regions (look for micros) ▪AutoScaling gone wild 30 30 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
16. What does an attack in AWS look like? for IAM ▪Exposed and compromised API Access Keys ▪New users/roles ▪Deleted users/roles ▪IAM policy changes ▪Root user locked out ▪AWS resources being created or deleted ▪Costs going up trough the roof 31 31 What does an attack in AWS look like? for S3 ▪Wide open ACLs and/or Policies ▪Watch AuthenticatedUsers grantee in ACLs ▪Missing buckets and objects ▪Created buckets and objects 32 32 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
17. 10 Minute Break 33 33 2nd Hour 34 34 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
18. Engaging AWS Support 35 35 What Will AWS do WITH you? ▪Communicate via Support Ticket ▪Can provide you guidance on ▪Scaling up ▪Scaling out ▪Isolating the issue ▪May shift traffic to help ▪Can provide you forensic image for analysis in AWS 36 36 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
19. AWS Support is Part of YOUR Team ▪Always include AWS Support in your incident response program ▪If you are an Enterprise Support customer, include your TAM ▪On occasion, an AWS Support person can be on your call ▪Always report abuse and security issues to AWS Support, their support center is bigger than yours! 37 37 Penetration Testing ▪ALWAYS use the official AWS Pen Test Request form ▪Is not immediate, but you can request immediate via AWS Support http://aws.amazon.com/security/penetration-testing/ 38 38 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
20. Logging and Data on Your Side 39 39 Come Prepared ▪The more data you have, the better ▪Make sure you have data for ▪AWS infrastructure and resource state (Evident.io can help) ▪AWS Activity and events (CloudTrail) ▪OS and Application state (FIM) ▪S3 buckets (S3 Logs) ▪ELBs (ELB Logs) 40 40 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
21. CloudTrail FTW ▪CloudTrail provides a pretty comprehensive amount of data for events on AWS resources ▪CloudTrail produces tons of data, be prepared to sift through it ▪Send CloudTrail data to a centralized bucket for easy consumption into log aggregation systems ▪AWS may request CloudTrail data during an incident ▪Protect your CloudTrail buckets as if they had customer sensitive PII 41 41 Forensic Images ▪A great way to determine root cause ▪Law enforcement may want later ▪Create EBS snapshot or AMI from affected instance ▪Use an isolated VPC with no route to the internet to bring up and test 42 42 Forensic imaging in AWS is very easy with EC2 snapshotting capabilities. The instance does not need to go down. This is especially important if you are early in the discovery phase of the incident. Also, AWS Support could create a snapshot in your behalf in special circumstances. QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
22. Mitigation Techniques 43 43 Preventative Medicine ▪Follow the Top 10 Security Best Practices ▪Read and be familiar with the AWS Security Best Practices White Paper ▪Automate data gathering 44 44 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
23. Security + DevOps ▪Configuration Management can help get you to the last known good state ▪Immutable AMIs are also helpful in restoring operating systems and applications ▪CloudFormation templates for infrastructure management ▪Test your CI/CD pipeline 45 45 In-class Exercise: Kali Linux or Evident.io Your choice 46 46 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
24. 10 Minute Break 47 47 3rd Hour 48 48 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
25. Top 10 AWS Security Best Practices 49 49 50 Top 10 AWS Security Best Practices 1. Disable root API access key and secret key 2. Enable MFA tokens everywhere 3. Reduce number of IAM users with Admin rights 4. Use Roles for EC2 5. Least privilege: limit what IAM entities can do with strong/ explicit policies 6. Rotate all the keys regularly 7. Use IAM roles with STS AssumeRole where possible 8. Use AutoScaling to dampen DDoS effects 9. Do not allow 0.0.0.0/0 in any EC2/ELB security group unless you mean it 10. Watch world-readable/listable S3 bucket policies 50 If there is a #0, it would be to enable CloudTrail and log to a central bucket for monitoring of all AWS activity My employer’s product, Evident Security Platform, is designed to help AWS customers monitor and remediate security issues and alerts when the above best practices are not being implemented. QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
26. #1 - Disable Root Account API Access Key ▪“Root” account has no restrictions ▪Create administrative IAM users ▪Use Roles for EC2 (#4) ▪Make sure billing and contact questions are filled out ▪Bonus: Set up MFA on root and throw away the key! 51 51 A root API access key is special in that the root user does not have any restrictions on it, nor can there be any restrictions on it. Unlike an IAM user, role, or group, there are no policies to limit access. If the root API access key is compromised, it is equivalent to removing your home’s front door. #2 - 1 Enable MFA Tokens Everywhere ▪Provide an additional factor to the authentication step ▪MFA is assigned to root account and IAM users ▪Can be assigned to roles ▪Physical or virtual ▪Virtual has choices (Google Authenticator, Authy, etc.) 52 52 MFA tokens are a great way to challenge users with a secondary authentication token. They’re mostly used when logging in via AWS Console, but they can also be used to challenge aws-cli usage via the AWS STS service. I recommend using the Authy MFA token as it makes your MFA token available on multiple devices (phone, tablet, computer) and supports multiple services like GitHub, AWS and others. QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
27. #3 - Reduce Number of IAM users with Admin ▪How many people have the keys to your kingdom? ▪Not just people - apps ▪Review IAM policies on Users, Groups and Roles ▪Remember #1 ▪Consider Identity Federation 53 53 Be careful who you give your house key to! #4 - Use Roles for EC2 ▪Do your EC2 instances need to contact other AWS Services? ▪AWS SDKs and aws-cli support EC2 Roles ▪Reduced attack surface area ▪Secure DevOps on EC2 ▪Create an EC2 specific role ▪Assign a specific policy to that role ▪Launch an EC2 instance with that role ▪Easy to test with aws-cli on EC2 54 54 Roles for EC2 are a great way to remove the burden of deploying an API access key pair onto EC2 instances. However, care must be taken in multi- user environments. Since the temporary key is stored in EC2 instance metadata, any authenticated user on your EC2 instances will be able to access it. In certain use cases (SSH bastion, for instance), I recommend managing API access keys for individual users. QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
28. #5 - Least Privilege ▪Programs should operate using the least amount of privilege to get the job done ▪IAM can get very granular ▪Works in tandem with #4 on EC2 ▪Should be applied to all automated workflows, too ▪Very specific IAM policies - only allow what you mean ▪IAM managed policies make this easier ▪Use the IAM policy generator and policy simulator to help 55 55 “Least privilege: Every program and every user of the system should operate using the least set of privileges necessary to complete the job. Primarily, this principle limits the damage that can result from an accident or error. It also reduces the number of potential interactions among privileged programs to the minimum for correct operation, so that unintentional, unwanted, or improper uses of privilege are less likely to occur. Thus, if a question arises related to misuse of a privilege, the number of programs that must be audited is minimized. Put another way, if a mechanism can provide "firewalls," the principle of least privilege provides a rationale for where to install the firewalls. The military security rule of "need-to-know" is an example of this principle.” http://web.mit.edu/saltzer/www/publications/protection/Basic.html #6 - Rotate all the Keys Regularly ▪Compromised access keys are very annoying and can cost your business dearly ▪IAM users should have keys rotated every 90 days minimum ▪Mostly useful for when Roles for EC2 won’t work in automated workflows Sample process: ▪Track age of Access Keys ▪Create new key ▪Supply key to automation process ▪Test ▪Deactivate old key 56 56 Key management is hard. If you must use API access keys, and we all do, make sure you rotate and manage well. I recommend using a credentials management system like LastPass to store and share keys amongst your team. QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
29. #7 - Use IAM Roles with STS AssumeRole ▪Similar to EC2 Roles ▪Can be used in place of privileged IAM user Access Keys ▪Temporary credentials ▪Allows for 3rd parties such as Evident.io to access your AWS accounts more securely ▪Extended version of AssumeRole allows for Identity Federation 57 57 Friends don’t let their friends provide API access keys to their 3rd party vendors. Use IAM Roles and the STS AssumeRole call with 3rd party trust. Make sure your vendor also provides a randomized external ID. Identity Federation allows you to use your internal directory service with AWS. The mechanism by which this is done is STS AssumeRoleWithSAML. Can be used to authenticate both the AWS Console and the aws-cli, however, the latter is a little trick to set up. Authorization is set up by mapping internal Groups with Roles. You can then assign specific access permissions to AWS resources with IAM policies. #8 - Use AutoScaling to Dampen DDoS ▪AutoScaling allows you to increase number of EC2 instances automatically ▪More instances means site stays up ▪Small price to pay for site reliability ▪You may need a temporary increase in EC2 limits ▪You may need to temporarily increase desired number of instances in ASG ▪Work with AWS, they may be able to help you on the network edge 58 58 While AutoScaling is not in itself a security service, you can certainly use it to provide better quality of service to your users and customers. In the case of DDoS attacks, AutoScaling can be quite effective in keeping your site/ service up while AWS reacts to a DDoS attack at the network layer. Caveat: it will cost money to do this, since your service will create new EC2 instances in response to an increase in traffic or load. Make sure you understand the characteristics of your application and service well, and have the CloudWatch triggers for AutoScaling well tuned. QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
30. #9 - Do not allow ALL in Security Groups ▪Unless you really mean it ▪Like leaving the door wide open ▪EC2 IP address range is a favorite for scanners ▪Monitor Security Groups regularly (HINT: Evident.io can help) ▪Affects not just EC2 instances, but: ▪ELBs ▪RDS Database Servers ▪ElastiCache Clusters ▪EMR Nodes ▪and others… 59 59 #10 - Watch Readable and Listable S3 Buckets ▪Open S3 buckets a favorite for trolling for API Access Keys ▪Check your Bucket ACLs regularly ▪Watch for all grantees, including AuthenticatedUsers ▪Check your Bucket Policies regularly 60 60 S3 buckets can be tricky to manage. Make sure you are explicit with objects that need to be public and those that do not. In most cases, they do not, which is also the default ACL for buckets. Even still, it’s very easy to flip the public bit on an object ACL or open a bucket wide open with the wrong S3 policy. QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
31. A Practitioner’s Tool Chest 61 61 Some tools that can help you ▪Evident.io (infrastructure) ▪AWS Trusted Advisor (infrastructure) ▪Cloudability (cost) ▪Splunk (monitoring and logging) ▪AWS CloudWatch and CW Logs (monitoring and logging) ▪Qualys (app and vulnerability scanning) ▪nmap (network scanning) ▪Autopsy and SluethKit (forensics) ▪Kali Linux (pen testing) ▪…and many others… 62 62 https://www.evident.io/ https://www.cloudability.com/ https://www.splunk.com/ https://www.qualys.com/ http://nmap.org http://www.sleuthkit.org http://systeminterrupt.me/blog/2014/08/25/setting-up-sleuthkit-and- autopsy-on-an-aws-ec2-instance/ https://www.kali.org QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015
32. Real-World Experiences from You with Review and Q&A 63 63 Thank You QCon NY! john@evident.io • john@johnmartinez.com • @johnmartinez • LinkedIn 64 64 QConNY-AWS-SecurityIncident-Tutorial.key - June 9, 2015