Ben Whaley @iAmTheWhaley Cloudy & Not-So-Cloudy Security Considerations The Other Hat Infracoders August 20, 2015

How Many Hats Do You Wear? Design a network that supports global redundancy with low latency Write CM recipes for managing apps & services Implement a monitoring & alerting system Manage a continuous integration & deployment pipeline … also “security”

Ben Whaley @iAmTheWhaley “Security is Everybody’s Responsibility”

Defense In Depth The Principle of Least Privilege Reduce the Attack Surface CIA Triad Asset: the thing to protect Vulnerability: A weakness that exposes an asset Threat: What we’re protecting against (bad actor, virus…) Risk = Threat + Asset + Vulnerability Shit Security People Say

BitcoinTalk forums compromised by social engineering 1/5 of Android Apps in Google Play store are Malware John McAfee plants key loggers in laptops, gifts them to Belizean government workers Stuxnet targets centrifuges in Iranian nuclear enrichment plants IETF standards working groups undermined by NSA What is a Sys Admin to do?

Defense in Depth is the ONLY way

Authentication & Authorization Issues tl;dr Use MFA

Passwords aren’t good enough Trust passwords? GPUs/FPGAs can make hundreds of billions of guesses/sec Brainflayer, DefCon 2015: “a single day should be more than enough time for a botnet to check every possible eight character ASCII password and XKCD- style passphrase against every Bitcoin address that has ever received funds”

Administration Enable MFA wherever it’s available • AWS console, AWS API, Google account, GitHub, etc • SSH via Duo Sec Do not use shared and/or default accounts Use sudo for auditability SSH private keys should always have a passphrase Beware of SSH agent forwarding hijack - Demo

Web Apps HTTP Basic Auth - Only viable over SSL, and only for simple cases. credentials sent with every request. HTTP Digest Auth - Safer to use without SSL, but server must store pw with a weak hash.. but without SSL the server cannot be trusted Mutual/2-way SSL - Underappreciated today, esp. for server-to- server comms. Not scalable for large # of clients. Username/Pw + token/Oauth - Most widely deployed & understood. Often implemented poorly

APIs HTTP Basic auth + SSL reasonable but not great, or mutual SSL, or API key & secret Secure all API endpoints, even those used only internally Possibly use path-based routing to limit access to sensitive interfaces Custom/NIH solutions should be vehemently mistrusted

HAProxy path-based routing with mutual SSL bind *:443 ssl crt myapp.example.com.pem verify optional ca-file client_ca.crt crl-file client_crl.pem no-sslv3 ciphers ECDHE-RSA-AES128- GCM-SHA256:!RC4:!DES:!MD5:!aNULL:!EDH:!AESGCM acl pri_https path_beg -i /admin /private /secret acl valid_cert ssl_c_verify 0 use_backend app_https if pri_https valid_cert

Databases Obvious: Apps should not use DB users with superuser Apply principle of least privilege for accessing the DB from apps App to DB connection must be encrypted Replication between cluster nodes must also be encrypted & authenticated (usually via mutual SSL) Some data stores have NO option for encryption and only weak auth (redis)

AWS IAM Use roles for EC2 instances (duh) Beware of services that ask for IAM access & secret keys Instead, use the IAM AssumeRole feature IAM administrator access considered harmful Enable MFA for root account, disable API keys, do not use root account On instances, replace or disable the default username and associated key CloudTrail

Replace Default User #!/bin/bash -x # Via Eric Hammond, https://alestic.com # Swap the default user user=myuser usermod -l $user ubuntu groupmod -n $user ubuntu usermod -d /home/$user -m $user if [ -f /etc/sudoers.d/90-cloudimg-ubuntu ]; then mv /etc/sudoers.d/90-cloudimg-ubuntu /etc/sudoers.d/90-cloud- init-users fi sed -i "s/ubuntu/${user}/g" /etc/sudoers.d/90-cloud-init-users

A Few Thoughts On Network Security

AWS Security Groups Do Don’t Use security group ID (sg-abcd1234) for sources in SG rules Allow all traffic (0.0.0.0/0) or large network ranges (10.0.0.0/8, 192.168.0.0/16) Create one SG per system “role” Assign many security groups to an instance Programmatically manage SGs (via CloudFormation, custom scripts, etc) Permit any ports that are unneeded Name & Tag security groups Include granular outbound rules

Host-based firewalls (IPTables) A second tier beyond security groups (defense in depth) Enable with only SSH allowed, save as part of a base image Open ports as needed based on the system’s services & functionality Best managed via CM “But troubleshooting is so much harder!”

Other Considerations • “Trusted internal network” fallacy • All access to the network through a bastion or VPN • NACLs complicate things but can be useful to segment subnets • Complete VPC isolation is better • Subnets do (almost) nothing for security • Port knocking is a fun toy • 3rd party services (Dome9) can enable/disable group rules via browser extension • VPC Flow Logs • Peered VPCs & VPN Gateway • Beware of connected network proliferation • Note SG limitations for communication between VPCs

SSL/TLS “Well, I ain't passed the bar but I know a little bit Enough that you won't illegally search my shit” - JayZ

Efficacy Trustworthiness damaged by multiple vulnerabilities, both implementation and protocol (Heartbleed, POODLE, FREAK, LogJam) Selection of ciphersuite of paramount importance ECDHE defeats all known crypto attacks Ultra paranoid? Replace Diffie Hellman group parameters, and/or only use >= 2048 bit primes awk '$5 > 2000' /etc/ssh/moduli > new_moduli && mv new_moduli /etc/ssh/moduli See weakdh.org

Other Challenges Certificate & key management, expiration, revocation (Un)Trustworthiness of Certificate Authorities 300+ trusted CAs by default Interference with the operations of middleboxes “Cryptographic Overhead” Like everything else, thwarted by weak end user security

TLS Usage “HTTPS everywhere” Not just for confidentiality! Recommended for internal networks too Wildcard certificates: risks & benefits OCSP (Online Certificate Status Protocol) stapling openssl s_client -connect monitor.mozillalabs.com:443 -status OCSP Response Data: OCSP Response Status: successful (0x0) Response Type: Basic OCSP Response AWS quickly updates available ciphersuites when vulnerabilities are announced

TLS Usage https://istlsfastyet.com

VPC Peering Admins Management 10.100.0.0/16 Control Systems Production 10.120.0.0/16 Security Group Autoscale Group Security Group Users Security Group Security Group Persistence Services Public Private https https MySQL TLS token auth Redis - Auth only SSH or VPN

Some System Security Stuff

The Basics Bake a secure base image (Packer) with smart defaults CIS Benchmarks Standard Disable default services Create separate partitions & set mount options Harden the SSH config (https://goo.gl/PBqEWr) Configure log rotation & centralized system logs Remove unused SUID binaries AWS Share the base image between accounts Avoid community AMIs

Patching Exploits for new vulnerabilities found in the wild in < 1 week (Defense in depth is only defense) Must be able to orchestrate updates across fleets of servers Using immutable systems? Now go replace them all Which is worse - downtime because a patch broke something, or exposure of customer data due to vulnerable software? But don’t be premature about it Can software repositories be trusted?

Security Tools Choose security tools wisely (overhead) Level 1 (must have): Centralized logging MFA System monitoring (CPU/Memory/Disk/IO) Level 2 (nice to have): host-based intrusion detection file integrity monitoring centralized authentication Antivirus SELinux grsecurity

Configuration Management Useful for preventing drift from the secure base image Can be a source of vulnerability Secrets in plain text in repositories (db passwords, API keys, private keys, etc) Runs as root or root-ish Can be triggered remotely, often without further auth Authenticated using public key crypto Mitigation Use vault/encrypted attributes rather than plain text Keep config mgmt software patched Follow vendor security guidelines

Provisioning & Deployment Credentials/secrets in EC2 User Data (Pocket vulnerability) Attack vectors via CI/CD Integrations with Slack/Hipchat Does your bot have root? Dependencies on external services When containers don’t contain Docker is the new wheel Colocated containers can subvert the principle of least privilege Insecure configurations --privileged, --net=host

References & Resources List of AWS Security Resources https://gist.github.com/chanj/6c48c059ad4b72a60bf3 AWS Security Center https://aws.amazon.com/security/ AWS security best practices http://media.amazonwebservices.com/AWS_Security_Best_Practices.pdf Secure OpenSSH Config https://goo.gl/PBqEWr CIS Benchmarks https://benchmarks.cisecurity.org/downloads/benchmarks/ Pocket Vulnerability https://www.gnu.gl/blog/Posts/multiple-vulnerabilities-in-pocket/ VPC Security http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Security.html How to securely hash passwords http://security.stackexchange.com/questions/211/how-to-securely-hash-passwords Countdown to Zero Day (Stuxnet book) http://www.amazon.com/Countdown-Zero-Day-Stuxnet-Digital/dp/077043617X

No Hay Más www.whaletech.co @iAmTheWhaley