QCon NY 2015: AWS Security Incident Tutorial (Full Size)

Transcript

1. Security Incident Response in AWS: Best Practices and Prerequisites John Martinez Principal Solutions Architect
2. Download slides here: https://goo.gl/UG0yX7 2
3. About Me ▪Been doing DevOps and Cloud stuff for ~5 years ▪Helped architect and build Creative Cloud @ Adobe ▪Cut my teeth on “the cloud” at Netflix ▪UNIX and Linux throat beard for >20 years ▪Have been involved in countless security and production incidents, most at 2:00 AM ▪I now talk to people about security for a living ▪Live in Silicon Valley ▪I’m completely addicted to building Raspberry Pi 2’s for random things around the house 3
4. About You ▪Who are you? Name, Company, Title/Job, Location, AWS Experience, Random Fact 4
5. Agenda - 1st Hour ▪Intros ▪AWS Shared Security Responsibility Model ▪Incident Response in AWS ▪Anatomy of an Attack ▪~10 minute break 5 AGENDA IS FLEXIBLE ADJUSTMENTS WILL BE MADE
6. Agenda - 2nd Hour ▪Engaging AWS Support ▪Log and Data on your side ▪Mitigation Techniques ▪In-class Exercise (time permitting) ▪~10 minute break 6
7. Agenda - 3rd Hour ▪Top Security Best Practices and Exploitability ▪A Practitioner’s Tool Chest ▪Real-world experiences from you ▪Review, Q&A, Lunch time! 7
8. Expectations ▪You have created at least one AWS account ▪You have started at least once EC2 instance ▪You have basic familiarity with AWS IAM ▪You have a browser open to the AWS Console and/or have a Terminal open with aws-­‐cli installed, configured and ready to go ▪Hacking in class is not only allowed, it’s encouraged! ▪And above all else, PLEASE INTERRUPT ME 8
9. Please respect each other’s privacy Sensitive security topics will be discussed and shared by all of us. Please keep that in mind when you leave here. If you must use anecdote please make sure to anonymize. 9
10. 1st Hour 10
11. AWS Shared Security Responsibility 11
12. Shared Security Responsibility Model 12 http://aws.amazon.com/compliance/shared-responsibility-model/
13. Shared Security Responsibility - AWS ▪AWS owns responsibility for all things physical ▪They build the services and APIs you need to deploy and run your applications on AWS ▪They deliver to their customers the ability to create identity credentials to access AWS services and resources ▪They secure the hypervisor and the network below layer 7 13
14. Shared Security Responsibility - You ▪Creating and managing IAM entities and their authentication tokens ▪Creating and managing IAM access policies ▪Creating and managing network access to EC2 instances and VPC networks ▪Deploying applications and creating virtual infrastructure ▪OS Patching and Maintenance ▪Monitoring activity and collecting data ▪Designing for failure 14
15. AWS Security Services - Identity ▪Identity and Access Management (IAM) ▪Secure Token Service (STS) ▪Directory Service 15
16. AWS Security Services - Encryption ▪Key Management Service ▪CloudHSM 16
17. AWS Security Services - EC2 ▪EC2 Security Groups ▪EC2 Keypairs (SSH) ▪VPC Subnet ACLs 17
18. AWS Security Services - Data/Log Analysis ▪CloudTrail ▪Config ▪CloudWatch Logs 18
19. AWS Security Services - Storage ▪S3 ACLs and Bucket Policies ▪S3 Lifecycle Rules ▪EBS Volume Encryption 19
20. Incident Response in AWS 20
21. Incident Response: Not very different ▪Have a plan ▪NOC and SOC roles ▪Communication is key ▪Your AWS/Cloud guru may be your best ally if you don’t have an explicit “Cloud Security” Architect/Engineer 21
22. Incident Response: somewhat different ▪Your AWS/Cloud guru may be your best ally if you don’t have an explicit “Cloud Security” Architect/Engineer ▪DevOps can play a major role ▪Engage AWS Support often and early (more on that later) ▪Come armed with data (also more on that later) ▪Download latest CSA Guidance, Domain 9 “Incident Response” https://cloudsecurityalliance.org/research/security-guidance/ 22
23. Triage the Incident Evaluate the situation ▪What’s going on right now? ▪No time to freak out 23
24. Triage the Incident Stop the bleeding ▪Secure the site/isolate the damage ▪May need to bring down the site temporarily 24
25. Triage the Incident Start the breathing ▪Get the business running again ▪Restart services when you’re ready 25
26. Triage the Incident Protect the wound ▪Investigate root cause ▪Data is king 26
27. Triage the Incident Treat for Shock ▪Make it better (so it does not happen again) ▪Learn from your mistakes ▪Monitor continuously 27
28. Security Incident Checklist ✓Have both a plan and tools (and have tested them) ✓Define what and when it happened and/or ✓Define what and when is was not happening ✓Initial triage of the problem ✓The more detail you can collect, the better ✓Scale up - Scale out - Isolate the issue ✓Review CloudTrail Logs ✓Snapshot/dd effected resources ✓Open a support case with AWS early ✓Update Often 28
29. Anatomy of an Attack 29
30. What does an attack in AWS look like? ▪More than likely, you don’t even know it’s going on ▪Sometimes, an email from the AWS Security will come in that an attack is originating from one of your EC2 instances ▪Preventative tools like Evident.io can help reduce the amount of surprises ▪Dev accounts are just as important to secure as production 30
31. What does an attack in AWS look like? for EC2 ▪Most common type of attack ▪EC2 IP address space is public https://ip-ranges.amazonaws.com/ip-ranges.json ▪Security Group rules are too open ▪Vulnerable applications ▪Vulnerable operating systems ▪Instances in unused regions (look for micros) ▪AutoScaling gone wild 31
32. What does an attack in AWS look like? for IAM ▪Exposed and compromised API Access Keys ▪New users/roles ▪Deleted users/roles ▪IAM policy changes ▪Root user locked out ▪AWS resources being created or deleted ▪Costs going up trough the roof 32
33. What does an attack in AWS look like? for S3 ▪Wide open ACLs and/or Policies ▪Watch AuthenticatedUsers grantee in ACLs ▪Missing buckets and objects ▪Created buckets and objects 33
34. 10 Minute Break 34
35. 2nd Hour 35
36. Engaging AWS Support 36
37. What Will AWS do WITH you? ▪Communicate via Support Ticket ▪Can provide you guidance on ▪Scaling up ▪Scaling out ▪Isolating the issue ▪May shift traffic to help ▪Can provide you forensic image for analysis in AWS 37
38. AWS Support is Part of YOUR Team ▪Always include AWS Support in your incident response program ▪If you are an Enterprise Support customer, include your TAM ▪On occasion, an AWS Support person can be on your call ▪Always report abuse and security issues to AWS Support, their support center is bigger than yours! 38
39. Penetration Testing ▪ALWAYS use the official AWS Pen Test Request form ▪Is not immediate, but you can request immediate via AWS Support http://aws.amazon.com/security/penetration-testing/ 39
40. Logging and Data on Your Side 40
41. Come Prepared ▪The more data you have, the better ▪Make sure you have data for ▪AWS infrastructure and resource state (Evident.io can help) ▪AWS Activity and events (CloudTrail) ▪OS and Application state (FIM) ▪S3 buckets (S3 Logs) ▪ELBs (ELB Logs) 41
42. CloudTrail FTW ▪CloudTrail provides a pretty comprehensive amount of data for events on AWS resources ▪CloudTrail produces tons of data, be prepared to sift through it ▪Send CloudTrail data to a centralized bucket for easy consumption into log aggregation systems ▪AWS may request CloudTrail data during an incident ▪Protect your CloudTrail buckets as if they had customer sensitive PII 42
43. Forensic Images ▪A great way to determine root cause ▪Law enforcement may want later ▪Create EBS snapshot or AMI from affected instance ▪Use an isolated VPC with no route to the internet to bring up and test 43
44. Mitigation Techniques 44
45. Preventative Medicine ▪Follow the Top 10 Security Best Practices ▪Read and be familiar with the AWS Security Best Practices White Paper ▪Automate data gathering 45
46. Security + DevOps ▪Configuration Management can help get you to the last known good state ▪Immutable AMIs are also helpful in restoring operating systems and applications ▪CloudFormation templates for infrastructure management ▪Test your CI/CD pipeline 46
47. In-class Exercise: Kali Linux or Evident.io Your choice 47
48. 10 Minute Break 48
49. 3rd Hour 49
50. Top 10 AWS Security Best Practices 50
51. 51 Top 10 AWS Security Best Practices 1. Disable root API access key and secret key 2. Enable MFA tokens everywhere 3. Reduce number of IAM users with Admin rights 4. Use Roles for EC2 5. Least privilege: limit what IAM entities can do with strong/ explicit policies 6. Rotate all the keys regularly 7. Use IAM roles with STS AssumeRole where possible 8. Use AutoScaling to dampen DDoS effects 9. Do not allow 0.0.0.0/0 in any EC2/ELB security group unless you mean it 10. Watch world-readable/listable S3 bucket policies
52. #1 - Disable Root Account API Access Key ▪“Root” account has no restrictions ▪Create administrative IAM users ▪Use Roles for EC2 (#4) ▪Make sure billing and contact questions are filled out ▪Bonus: Set up MFA on root and throw away the key! 52
53. #2 - 1 Enable MFA Tokens Everywhere ▪Provide an additional factor to the authentication step ▪MFA is assigned to root account and IAM users ▪Can be assigned to roles ▪Physical or virtual ▪Virtual has choices (Google Authenticator, Authy, etc.) 53
54. #3 - Reduce Number of IAM users with Admin ▪How many people have the keys to your kingdom? ▪Not just people - apps ▪Review IAM policies on Users, Groups and Roles ▪Remember #1 ▪Consider Identity Federation 54
55. #4 - Use Roles for EC2 ▪Do your EC2 instances need to contact other AWS Services? ▪AWS SDKs and aws-cli support EC2 Roles ▪Reduced attack surface area ▪Secure DevOps on EC2 ▪Create an EC2 specific role ▪Assign a specific policy to that role ▪Launch an EC2 instance with that role ▪Easy to test with aws-cli on EC2 55
56. #5 - Least Privilege ▪Programs should operate using the least amount of privilege to get the job done ▪IAM can get very granular ▪Works in tandem with #4 on EC2 ▪Should be applied to all automated workflows, too ▪Very specific IAM policies - only allow what you mean ▪IAM managed policies make this easier ▪Use the IAM policy generator and policy simulator to help 56
57. #6 - Rotate all the Keys Regularly ▪Compromised access keys are very annoying and can cost your business dearly ▪IAM users should have keys rotated every 90 days minimum ▪Mostly useful for when Roles for EC2 won’t work in automated workflows Sample process: ▪Track age of Access Keys ▪Create new key ▪Supply key to automation process ▪Test ▪Deactivate old key 57
58. #7 - Use IAM Roles with STS AssumeRole ▪Similar to EC2 Roles ▪Can be used in place of privileged IAM user Access Keys ▪Temporary credentials ▪Allows for 3rd parties such as Evident.io to access your AWS accounts more securely ▪Extended version of AssumeRole allows for Identity Federation 58
59. #8 - Use AutoScaling to Dampen DDoS ▪AutoScaling allows you to increase number of EC2 instances automatically ▪More instances means site stays up ▪Small price to pay for site reliability ▪You may need a temporary increase in EC2 limits ▪You may need to temporarily increase desired number of instances in ASG ▪Work with AWS, they may be able to help you on the network edge 59
60. #9 - Do not allow ALL in Security Groups ▪Unless you really mean it ▪Like leaving the door wide open ▪EC2 IP address range is a favorite for scanners ▪Monitor Security Groups regularly (HINT: Evident.io can help) ▪Affects not just EC2 instances, but: ▪ELBs ▪RDS Database Servers ▪ElastiCache Clusters ▪EMR Nodes ▪and others… 60
61. #10 - Watch Readable and Listable S3 Buckets ▪Open S3 buckets a favorite for trolling for API Access Keys ▪Check your Bucket ACLs regularly ▪Watch for all grantees, including AuthenticatedUsers ▪Check your Bucket Policies regularly 61
62. A Practitioner’s Tool Chest 62
63. Some tools that can help you ▪Evident.io (infrastructure) ▪AWS Trusted Advisor (infrastructure) ▪Cloudability (cost) ▪Splunk (monitoring and logging) ▪AWS CloudWatch and CW Logs (monitoring and logging) ▪Qualys (app and vulnerability scanning) ▪nmap (network scanning) ▪Autopsy and SluethKit (forensics) ▪Kali Linux (pen testing) ▪…and many others… 63
64. Real-World Experiences from You with Review and Q&A 64
65. Thank You QCon NY! john@evident.io • john@johnmartinez.com • @johnmartinez • LinkedIn 65