Share
Embed
QCon NY 2015: AWS Security Incident Tutorial (Full Size)
John Martinez
June 09, 2015
Transcript
65 slides
-
Security Incident Response in AWS: Best Practices and Prerequisites John
Martinez Principal Solutions Architect -
About Me ▪Been doing DevOps and Cloud stuff for ~5
years ▪Helped architect and build Creative Cloud @ Adobe ▪Cut my teeth on “the cloud” at Netflix ▪UNIX and Linux throat beard for >20 years ▪Have been involved in countless security and production incidents, most at 2:00 AM ▪I now talk to people about security for a living ▪Live in Silicon Valley ▪I’m completely addicted to building Raspberry Pi 2’s for random things around the house 3 -
About You ▪Who are you? Name, Company, Title/Job, Location, AWS
Experience, Random Fact 4 -
Agenda - 1st Hour ▪Intros ▪AWS Shared Security Responsibility Model
▪Incident Response in AWS ▪Anatomy of an Attack ▪~10 minute break 5 AGENDA IS FLEXIBLE ADJUSTMENTS WILL BE MADE -
Agenda - 2nd Hour ▪Engaging AWS Support ▪Log and Data
on your side ▪Mitigation Techniques ▪In-class Exercise (time permitting) ▪~10 minute break 6 -
Agenda - 3rd Hour ▪Top Security Best Practices and Exploitability
▪A Practitioner’s Tool Chest ▪Real-world experiences from you ▪Review, Q&A, Lunch time! 7 -
Expectations ▪You have created at least one AWS account ▪You
have started at least once EC2 instance ▪You have basic familiarity with AWS IAM ▪You have a browser open to the AWS Console and/or have a Terminal open with aws-‐cli installed, configured and ready to go ▪Hacking in class is not only allowed, it’s encouraged! ▪And above all else, PLEASE INTERRUPT ME 8 -
Please respect each other’s privacy Sensitive security topics will be
discussed and shared by all of us. Please keep that in mind when you leave here. If you must use anecdote please make sure to anonymize. 9 -
Shared Security Responsibility - AWS ▪AWS owns responsibility for all
things physical ▪They build the services and APIs you need to deploy and run your applications on AWS ▪They deliver to their customers the ability to create identity credentials to access AWS services and resources ▪They secure the hypervisor and the network below layer 7 13 -
Shared Security Responsibility - You ▪Creating and managing IAM entities
and their authentication tokens ▪Creating and managing IAM access policies ▪Creating and managing network access to EC2 instances and VPC networks ▪Deploying applications and creating virtual infrastructure ▪OS Patching and Maintenance ▪Monitoring activity and collecting data ▪Designing for failure 14 -
AWS Security Services - Identity ▪Identity and Access Management (IAM)
▪Secure Token Service (STS) ▪Directory Service 15 -
AWS Security Services - EC2 ▪EC2 Security Groups ▪EC2 Keypairs
(SSH) ▪VPC Subnet ACLs 17 -
AWS Security Services - Storage ▪S3 ACLs and Bucket Policies
▪S3 Lifecycle Rules ▪EBS Volume Encryption 19 -
Incident Response: Not very different ▪Have a plan ▪NOC and
SOC roles ▪Communication is key ▪Your AWS/Cloud guru may be your best ally if you don’t have an explicit “Cloud Security” Architect/Engineer 21 -
Incident Response: somewhat different ▪Your AWS/Cloud guru may be your
best ally if you don’t have an explicit “Cloud Security” Architect/Engineer ▪DevOps can play a major role ▪Engage AWS Support often and early (more on that later) ▪Come armed with data (also more on that later) ▪Download latest CSA Guidance, Domain 9 “Incident Response” https://cloudsecurityalliance.org/research/security-guidance/ 22 -
Triage the Incident Evaluate the situation ▪What’s going on right
now? ▪No time to freak out 23 -
Triage the Incident Stop the bleeding ▪Secure the site/isolate the
damage ▪May need to bring down the site temporarily 24 -
Triage the Incident Start the breathing ▪Get the business running
again ▪Restart services when you’re ready 25 -
Triage the Incident Treat for Shock ▪Make it better (so
it does not happen again) ▪Learn from your mistakes ▪Monitor continuously 27 -
Security Incident Checklist ✓Have both a plan and tools (and
have tested them) ✓Define what and when it happened and/or ✓Define what and when is was not happening ✓Initial triage of the problem ✓The more detail you can collect, the better ✓Scale up - Scale out - Isolate the issue ✓Review CloudTrail Logs ✓Snapshot/dd effected resources ✓Open a support case with AWS early ✓Update Often 28 -
What does an attack in AWS look like? ▪More than
likely, you don’t even know it’s going on ▪Sometimes, an email from the AWS Security will come in that an attack is originating from one of your EC2 instances ▪Preventative tools like Evident.io can help reduce the amount of surprises ▪Dev accounts are just as important to secure as production 30 -
What does an attack in AWS look like? for EC2
▪Most common type of attack ▪EC2 IP address space is public https://ip-ranges.amazonaws.com/ip-ranges.json ▪Security Group rules are too open ▪Vulnerable applications ▪Vulnerable operating systems ▪Instances in unused regions (look for micros) ▪AutoScaling gone wild 31 -
What does an attack in AWS look like? for IAM
▪Exposed and compromised API Access Keys ▪New users/roles ▪Deleted users/roles ▪IAM policy changes ▪Root user locked out ▪AWS resources being created or deleted ▪Costs going up trough the roof 32 -
What does an attack in AWS look like? for S3
▪Wide open ACLs and/or Policies ▪Watch AuthenticatedUsers grantee in ACLs ▪Missing buckets and objects ▪Created buckets and objects 33 -
What Will AWS do WITH you? ▪Communicate via Support Ticket
▪Can provide you guidance on ▪Scaling up ▪Scaling out ▪Isolating the issue ▪May shift traffic to help ▪Can provide you forensic image for analysis in AWS 37 -
AWS Support is Part of YOUR Team ▪Always include AWS
Support in your incident response program ▪If you are an Enterprise Support customer, include your TAM ▪On occasion, an AWS Support person can be on your call ▪Always report abuse and security issues to AWS Support, their support center is bigger than yours! 38 -
Penetration Testing ▪ALWAYS use the official AWS Pen Test Request
form ▪Is not immediate, but you can request immediate via AWS Support http://aws.amazon.com/security/penetration-testing/ 39 -
Come Prepared ▪The more data you have, the better ▪Make
sure you have data for ▪AWS infrastructure and resource state (Evident.io can help) ▪AWS Activity and events (CloudTrail) ▪OS and Application state (FIM) ▪S3 buckets (S3 Logs) ▪ELBs (ELB Logs) 41 -
CloudTrail FTW ▪CloudTrail provides a pretty comprehensive amount of data
for events on AWS resources ▪CloudTrail produces tons of data, be prepared to sift through it ▪Send CloudTrail data to a centralized bucket for easy consumption into log aggregation systems ▪AWS may request CloudTrail data during an incident ▪Protect your CloudTrail buckets as if they had customer sensitive PII 42 -
Forensic Images ▪A great way to determine root cause ▪Law
enforcement may want later ▪Create EBS snapshot or AMI from affected instance ▪Use an isolated VPC with no route to the internet to bring up and test 43 -
Preventative Medicine ▪Follow the Top 10 Security Best Practices ▪Read
and be familiar with the AWS Security Best Practices White Paper ▪Automate data gathering 45 -
Security + DevOps ▪Configuration Management can help get you to
the last known good state ▪Immutable AMIs are also helpful in restoring operating systems and applications ▪CloudFormation templates for infrastructure management ▪Test your CI/CD pipeline 46 -
51 Top 10 AWS Security Best Practices 1. Disable root
API access key and secret key 2. Enable MFA tokens everywhere 3. Reduce number of IAM users with Admin rights 4. Use Roles for EC2 5. Least privilege: limit what IAM entities can do with strong/ explicit policies 6. Rotate all the keys regularly 7. Use IAM roles with STS AssumeRole where possible 8. Use AutoScaling to dampen DDoS effects 9. Do not allow 0.0.0.0/0 in any EC2/ELB security group unless you mean it 10. Watch world-readable/listable S3 bucket policies -
#1 - Disable Root Account API Access Key ▪“Root” account
has no restrictions ▪Create administrative IAM users ▪Use Roles for EC2 (#4) ▪Make sure billing and contact questions are filled out ▪Bonus: Set up MFA on root and throw away the key! 52 -
#2 - 1 Enable MFA Tokens Everywhere ▪Provide an additional
factor to the authentication step ▪MFA is assigned to root account and IAM users ▪Can be assigned to roles ▪Physical or virtual ▪Virtual has choices (Google Authenticator, Authy, etc.) 53 -
#3 - Reduce Number of IAM users with Admin ▪How
many people have the keys to your kingdom? ▪Not just people - apps ▪Review IAM policies on Users, Groups and Roles ▪Remember #1 ▪Consider Identity Federation 54 -
#4 - Use Roles for EC2 ▪Do your EC2 instances
need to contact other AWS Services? ▪AWS SDKs and aws-cli support EC2 Roles ▪Reduced attack surface area ▪Secure DevOps on EC2 ▪Create an EC2 specific role ▪Assign a specific policy to that role ▪Launch an EC2 instance with that role ▪Easy to test with aws-cli on EC2 55 -
#5 - Least Privilege ▪Programs should operate using the least
amount of privilege to get the job done ▪IAM can get very granular ▪Works in tandem with #4 on EC2 ▪Should be applied to all automated workflows, too ▪Very specific IAM policies - only allow what you mean ▪IAM managed policies make this easier ▪Use the IAM policy generator and policy simulator to help 56 -
#6 - Rotate all the Keys Regularly ▪Compromised access keys
are very annoying and can cost your business dearly ▪IAM users should have keys rotated every 90 days minimum ▪Mostly useful for when Roles for EC2 won’t work in automated workflows Sample process: ▪Track age of Access Keys ▪Create new key ▪Supply key to automation process ▪Test ▪Deactivate old key 57 -
#7 - Use IAM Roles with STS AssumeRole ▪Similar to
EC2 Roles ▪Can be used in place of privileged IAM user Access Keys ▪Temporary credentials ▪Allows for 3rd parties such as Evident.io to access your AWS accounts more securely ▪Extended version of AssumeRole allows for Identity Federation 58 -
#8 - Use AutoScaling to Dampen DDoS ▪AutoScaling allows you
to increase number of EC2 instances automatically ▪More instances means site stays up ▪Small price to pay for site reliability ▪You may need a temporary increase in EC2 limits ▪You may need to temporarily increase desired number of instances in ASG ▪Work with AWS, they may be able to help you on the network edge 59 -
#9 - Do not allow ALL in Security Groups ▪Unless
you really mean it ▪Like leaving the door wide open ▪EC2 IP address range is a favorite for scanners ▪Monitor Security Groups regularly (HINT: Evident.io can help) ▪Affects not just EC2 instances, but: ▪ELBs ▪RDS Database Servers ▪ElastiCache Clusters ▪EMR Nodes ▪and others… 60 -
#10 - Watch Readable and Listable S3 Buckets ▪Open S3
buckets a favorite for trolling for API Access Keys ▪Check your Bucket ACLs regularly ▪Watch for all grantees, including AuthenticatedUsers ▪Check your Bucket Policies regularly 61 -
Some tools that can help you ▪Evident.io (infrastructure) ▪AWS Trusted
Advisor (infrastructure) ▪Cloudability (cost) ▪Splunk (monitoring and logging) ▪AWS CloudWatch and CW Logs (monitoring and logging) ▪Qualys (app and vulnerability scanning) ▪nmap (network scanning) ▪Autopsy and SluethKit (forensics) ▪Kali Linux (pen testing) ▪…and many others… 63