Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Infrastructure Security Considerations

Ben Whaley
August 20, 2015

Infrastructure Security Considerations

Broad strokes and thoughts on security with a focus on the cloud and in particular AWS.

Ben Whaley

August 20, 2015

More Decks by Ben Whaley

Other Decks in Technology


  1. How Many Hats Do You Wear? Design a network that

    supports global redundancy with low latency Write CM recipes for managing apps & services Implement a monitoring & alerting system Manage a continuous integration & deployment pipeline … also “security”
  2. Defense In Depth The Principle of Least Privilege Reduce the

    Attack Surface CIA Triad Asset: the thing to protect Vulnerability: A weakness that exposes an asset Threat: What we’re protecting against (bad actor, virus…) Risk = Threat + Asset + Vulnerability Shit Security People Say
  3. BitcoinTalk forums compromised by social engineering 1/5 of Android Apps

    in Google Play store are Malware John McAfee plants key loggers in laptops, gifts them to Belizean government workers Stuxnet targets centrifuges in Iranian nuclear enrichment plants IETF standards working groups undermined by NSA What is a Sys Admin to do?
  4. Passwords aren’t good enough Trust passwords? GPUs/FPGAs can make hundreds

    of billions of guesses/sec Brainflayer, DefCon 2015: “a single day should be more than enough time for a botnet to check every possible eight character ASCII password and XKCD- style passphrase against every Bitcoin address that has ever received funds”
  5. Administration Enable MFA wherever it’s available • AWS console, AWS

    API, Google account, GitHub, etc • SSH via Duo Sec Do not use shared and/or default accounts Use sudo for auditability SSH private keys should always have a passphrase Beware of SSH agent forwarding hijack - Demo
  6. Web Apps HTTP Basic Auth - Only viable over SSL,

    and only for simple cases. credentials sent with every request. HTTP Digest Auth - Safer to use without SSL, but server must store pw with a weak hash.. but without SSL the server cannot be trusted Mutual/2-way SSL - Underappreciated today, esp. for server-to- server comms. Not scalable for large # of clients. Username/Pw + token/Oauth - Most widely deployed & understood. Often implemented poorly
  7. APIs HTTP Basic auth + SSL reasonable but not great,

    or mutual SSL, or API key & secret Secure all API endpoints, even those used only internally Possibly use path-based routing to limit access to sensitive interfaces Custom/NIH solutions should be vehemently mistrusted
  8. HAProxy path-based routing with mutual SSL bind *:443 ssl crt

    myapp.example.com.pem verify optional ca-file client_ca.crt crl-file client_crl.pem no-sslv3 ciphers ECDHE-RSA-AES128- GCM-SHA256:!RC4:!DES:!MD5:!aNULL:!EDH:!AESGCM acl pri_https path_beg -i /admin /private /secret acl valid_cert ssl_c_verify 0 use_backend app_https if pri_https valid_cert
  9. Databases Obvious: Apps should not use DB users with superuser

    Apply principle of least privilege for accessing the DB from apps App to DB connection must be encrypted Replication between cluster nodes must also be encrypted & authenticated (usually via mutual SSL) Some data stores have NO option for encryption and only weak auth (redis)
  10. AWS IAM Use roles for EC2 instances (duh) Beware of

    services that ask for IAM access & secret keys Instead, use the IAM AssumeRole feature IAM administrator access considered harmful Enable MFA for root account, disable API keys, do not use root account On instances, replace or disable the default username and associated key CloudTrail
  11. Replace Default User #!/bin/bash -x # Via Eric Hammond, https://alestic.com

    # Swap the default user user=myuser usermod -l $user ubuntu groupmod -n $user ubuntu usermod -d /home/$user -m $user if [ -f /etc/sudoers.d/90-cloudimg-ubuntu ]; then mv /etc/sudoers.d/90-cloudimg-ubuntu /etc/sudoers.d/90-cloud- init-users fi sed -i "s/ubuntu/${user}/g" /etc/sudoers.d/90-cloud-init-users
  12. AWS Security Groups Do Don’t Use security group ID (sg-abcd1234)

    for sources in SG rules Allow all traffic ( or large network ranges (, Create one SG per system “role” Assign many security groups to an instance Programmatically manage SGs (via CloudFormation, custom scripts, etc) Permit any ports that are unneeded Name & Tag security groups Include granular outbound rules
  13. Host-based firewalls (IPTables) A second tier beyond security groups (defense

    in depth) Enable with only SSH allowed, save as part of a base image Open ports as needed based on the system’s services & functionality Best managed via CM “But troubleshooting is so much harder!”
  14. Other Considerations • “Trusted internal network” fallacy • All access

    to the network through a bastion or VPN • NACLs complicate things but can be useful to segment subnets • Complete VPC isolation is better • Subnets do (almost) nothing for security • Port knocking is a fun toy • 3rd party services (Dome9) can enable/disable group rules via browser extension • VPC Flow Logs • Peered VPCs & VPN Gateway • Beware of connected network proliferation • Note SG limitations for communication between VPCs
  15. SSL/TLS “Well, I ain't passed the bar but I know

    a little bit Enough that you won't illegally search my shit” - JayZ
  16. Efficacy Trustworthiness damaged by multiple vulnerabilities, both implementation and protocol

    (Heartbleed, POODLE, FREAK, LogJam) Selection of ciphersuite of paramount importance ECDHE defeats all known crypto attacks Ultra paranoid? Replace Diffie Hellman group parameters, and/or only use >= 2048 bit primes awk '$5 > 2000' /etc/ssh/moduli > new_moduli && mv new_moduli /etc/ssh/moduli See weakdh.org
  17. Other Challenges Certificate & key management, expiration, revocation (Un)Trustworthiness of

    Certificate Authorities 300+ trusted CAs by default Interference with the operations of middleboxes “Cryptographic Overhead” Like everything else, thwarted by weak end user security
  18. TLS Usage “HTTPS everywhere” Not just for confidentiality! Recommended for

    internal networks too Wildcard certificates: risks & benefits OCSP (Online Certificate Status Protocol) stapling openssl s_client -connect monitor.mozillalabs.com:443 -status OCSP Response Data: OCSP Response Status: successful (0x0) Response Type: Basic OCSP Response AWS quickly updates available ciphersuites when vulnerabilities are announced
  19. VPC Peering Admins Management Control Systems Production Security

    Group Autoscale Group Security Group Users Security Group Security Group Persistence Services Public Private https https MySQL TLS token auth Redis - Auth only SSH or VPN
  20. The Basics Bake a secure base image (Packer) with smart

    defaults CIS Benchmarks Standard Disable default services Create separate partitions & set mount options Harden the SSH config (https://goo.gl/PBqEWr) Configure log rotation & centralized system logs Remove unused SUID binaries AWS Share the base image between accounts Avoid community AMIs
  21. Patching Exploits for new vulnerabilities found in the wild in

    < 1 week (Defense in depth is only defense) Must be able to orchestrate updates across fleets of servers Using immutable systems? Now go replace them all Which is worse - downtime because a patch broke something, or exposure of customer data due to vulnerable software? But don’t be premature about it Can software repositories be trusted?
  22. Security Tools Choose security tools wisely (overhead) Level 1 (must

    have): Centralized logging MFA System monitoring (CPU/Memory/Disk/IO) Level 2 (nice to have): host-based intrusion detection file integrity monitoring centralized authentication Antivirus SELinux grsecurity
  23. Configuration Management Useful for preventing drift from the secure base

    image Can be a source of vulnerability Secrets in plain text in repositories (db passwords, API keys, private keys, etc) Runs as root or root-ish Can be triggered remotely, often without further auth Authenticated using public key crypto Mitigation Use vault/encrypted attributes rather than plain text Keep config mgmt software patched Follow vendor security guidelines
  24. Provisioning & Deployment Credentials/secrets in EC2 User Data (Pocket vulnerability)

    Attack vectors via CI/CD Integrations with Slack/Hipchat Does your bot have root? Dependencies on external services When containers don’t contain Docker is the new wheel Colocated containers can subvert the principle of least privilege Insecure configurations --privileged, --net=host
  25. References & Resources List of AWS Security Resources https://gist.github.com/chanj/6c48c059ad4b72a60bf3 AWS

    Security Center https://aws.amazon.com/security/ AWS security best practices http://media.amazonwebservices.com/AWS_Security_Best_Practices.pdf Secure OpenSSH Config https://goo.gl/PBqEWr CIS Benchmarks https://benchmarks.cisecurity.org/downloads/benchmarks/ Pocket Vulnerability https://www.gnu.gl/blog/Posts/multiple-vulnerabilities-in-pocket/ VPC Security http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Security.html How to securely hash passwords http://security.stackexchange.com/questions/211/how-to-securely-hash-passwords Countdown to Zero Day (Stuxnet book) http://www.amazon.com/Countdown-Zero-Day-Stuxnet-Digital/dp/077043617X