SECRETS IN OUR CLOUDS Akash Mahajan (Founder Appsecco)

ABOUT ME 2

A PERSONAL JOURNEY 3

RESEARCH ➤ At Appsecco we are a small specialist application security company - As part of consulting, assessments etc. ➤ being aware of various ways to manage secrets ➤ if the software keeping them safe are actually secure etc.
 IT/OPS ➤ We manage IT/Ops for the company - We have deployed production workloads which require secrets to be ➤ safely created ➤ provided to applications/processes ➤ refresh them and expire them 4 WHY ARE WE TALKING ABOUT SECRETS?

OUR PROD AND STAGING REQUIREMENTS ARE TYPICAL FOR A SMALL COMPANY 1. We run multiple web applications with integrated authentication 2. We run multiple websites with authentication and authorization 3. In most cases we terminate SSL/TLS at Nginx and reverse proxy to internal applications 4. TLS certificates are generated using Lets Encrypt 5. Whenever we add authentication as a middleware we rely on our Single Sign-On provider 6. The applications which are running need access to certain secrets such as passwords, authentication tokens 5 WHAT ARE WE TRYING TO PROTECT?

➤ Install Software and Configure Software ➤ Vault and Goldfish ➤ Easy to follow install instructions ➤ Vault Server ➤ Goldfish Front end application ➤ Most of the complexity completely hidden by the brilliant instructions ➤ Both for a local test install ➤ A vagrant based install ➤ This can be expanded to setup a Digital Ocean droplet with the commands given 6 SO I STARTED WITH WHAT I KNOW BEST

HASHICORP VAULT IS A DECENT SOLUTION FOR THE FOLLOWING REASONS (FOR ME) ➤ I didn’t need large scale, its just a binary file that gets setup ➤ Has a simple enough configuration to start with ➤ Bare minimum Role Based Access Control to start with ➤ But based on the documentation, seems easy enough to expand later ➤ Vault unsealing is a manual effort for now ➤ This could be automated, but then we will need to store the unseal keys “securely” 7 GREAT THINGS ABOUT HASHICORP VAULT (IMHO) Save this to a server.hcl file and type the following command $ vault server -config server.hcl Step by step http://bit.ly/simple-vault-install

LETS GO FOR A SIMPLE ENOUGH TO SETUP SOLUTION WHICH ISN’T FULLY AUTOMATED 8 REFERENCE ARCHITECTURE WE CAN USE Client Internet Server Filesystem SSO

➤ How do I bootstrap secrets? ➤ Where do we store the keys that are generated when we “unseal” the vault? 9 OPERATIONAL QUESTIONS - WHERE TO STORE

WHAT ARE THESE SECRETS THAT WE NEED TO WORRY ABOUT? ➤ Certificates ➤ SSH Keys ➤ API Tokens ➤ Passwords (database, system, user)
 IS THERE ANYTHING ELSE WE CAN STORE SAFELY? ➤ Store the original QR code of 2FA enrolment, if more than 1 person requires login access ➤ Audit logs from other critical systems which may contain sensitive details ➤ TOTP Generation Keys 10 OPERATIONAL QUESTIONS - WHAT TO STORE

THINGS THAT KEEP ME UP AT NIGHT ➤ Is it the best way to manage secrets? ➤ Should I be using something which is cloud specific? ➤ How resilient is this? ➤ How will I automate to be able to make it reproducible? GNAWING FEARS ➤ What if I have made a mistake in configuration and this will leak data ➤ What if my backups fail 11 THE NAGGING DOUBT AT THE BACK OF MY MIND

12 SWITCHING HATS FROM OPS TO SECURITY

➤ Step back ➤ Think of abstractions which are simple enough for me to understand
 CONCEPTS THAT HELPED ME ➤ PlainText Key(s) ➤ Secure Storage ➤ Secure Access ➤ Ability to chose which user sees which secret ➤ Some secrets can be temporary 13 DON'T PANIC

SOME ATTRIBUTES THAT HELPED ME REMEMBERING 1. If a digital secret is stolen, we still have our copy 2. Therefore, the best way to store something that is important to us, is in a form which is encrypted and not in plain text 3. If we encrypt something, there exists a key which is in plain text 4. This ‘PlainText Key’ needs to be protected 5. This ‘PlainText Key’ should have secure storage and we should be able to prove that we are authorised to access it when we want it. 6. Our ‘PlainText Key’ can be stored inside the Vault behind keys which are with multiple people 1. Notice that we have shifted the burden of security from technology to a people process 14 A BIT ABOUT DIGITAL SECRETS Original Digital Copy Turtles

BIT MORE ABOUT ‘PLAINTEXT KEYS’ 1. Secrets as ‘PlainText Keys’ will be stored in file system and if they have been read, in memory 2. If we are orchestrating servers and provisioning applications, the ‘PlainText key’ may be copied somewhere closer (cached) 1. For example we have a linux server with a disk volume which is encrypted using EncFS and when the server starts, we mount it. This requires providing a ‘PlainText key’ 2. Another example is, if we are automating Jenkins to be installed and configured the admin user has a password. For us to run our commands, we need to be able to read this password. Usually this gets stored either in a predictable disk path or in an environment variable 15 A FEW MORE ABSTRACTIONS

CPU 16 CLOUD IS SOMEONE ELSE’S COMPUTER DISK MEMORY STILL WITH OUR ABSTRACTIONS ➤ If the CPU, Disk and Memory belong to someone else’s computer there is nothing you can keep as a secret which is in ‘PlainText’ ➤ But the question is, can you trust the security procedures, processes of a company like AWS, GCP , Azure

AWS ➤ Cloud HSM ➤ Key Management Service (KMS) ➤ Parameter Store 17 WHAT EXACTLY CAN I FIND ON AWS/AZURE/GCP AZURE ➤ Key Vault ➤ It is a combination of AWS CloudHSM & KMS GCP ➤ Google Cloud Key Management Service ➤ Generate, Rotate & destroy encryption keys COMMON FEATURES ➤ All of them provide a straightforward way to store the ‘PlainText key’ ➤ API driven services, integrated with their Identity and Access Management and Logs for audits ➤ Allow keys managed by them or Bring Your Own Key to the setup

➤ A secret that should be kept safe in the world of servers are SSH private keys ➤ Usual wisdom suggests that the private keys should be protected with pass-phrases and guarded ➤ But what if generate a new SSH key pair whenever access is required to a server and trash them once our work is done? GCLOUD CONNECTING TO LINUX INSTANCES ➤ By using the gcloud command, SSH keys are generated by the Google IAM service ➤ They are generated afresh because authentication and authorisation is being done out of band 18 APPROACHES (ALREADY EXISTING IN THE WORLD)

➤ Most of these should be considered as like *nix utilities ➤ You may chose to store the root credentials of AWS account in Keepass shared with your team over Dropbox ➤ That allows you to revoke the credentials of the admin users for AWS ➤ You may use, the AWS KMS to decrypt the secrets required to decrypt the provisioning software to unseal the Vault ➤ If you use this kind of an approach, then your KMS is as good as a Cloud HSM ➤ If you are running a SAAS, you may want to use tenant admin’s password as a salt to encrypt with a key provided by a cloud provider and completely offload the process to the provider 19 THESE ARE TOOLS AVAILABLE TO YOU

20 ADDRESSING FEARS AND DOUBTS FEAR WAY FORWARD Is it the best way to manage secrets? Enough smart people have done the thinking. Choose one of the patterns and go with it Should I be doing something cloud specific? If you already have stuff in multiple clouds use Vault to begin with. If you are on a single cloud, go with cloud provider and review later How resilient is this? If the worry is, something catastrophic may cause everything to be gone, store admin level secrets in a physical safe by printing them How will I be able to automate all of this? The concepts are same, abstractions are present, if you really need something custom, you will figure out the approach. You may not be able to figure out the approach on day 1, but in iterations. What if my backups fail? The focus should always be to test your restore, quickly and regularly.

THINK ABOUT SOME OF THESE BEFORE YOU GET GOING 1.Storage Security - Can you expect to have better security than the cloud provider? 2.Access Security - Can your internal processes do better security than the cloud provider? 3.Automation possibilities - If you do go ahead, what are the parts that will not be automated 4.Cooking required - If you plan to have a multi-cloud strategy do you need to write your applications, automation scripts to have wrapper functions to ensure that you can plug-in/plug- out cloud provider specific code 5.Compliance - Do you have a specific compliance requirement, which requires you to think differently about all of this? If yes, then none of what we have discussed may be relevant and you may need to look at the compliance documentation for guidance 21 IF YOU PLAN TO STORE SECRETS IN YOUR CLOUD

➤ The Right Way to Manage Secrets with AWS - https://segment.com/blog/the-right-way-to- manage-secrets/ ➤ Secrets management in the cloud https://www.slideshare.net/EvanJJohnson/secrets- management-in-the-cloud ➤ Secrets at Scale Automated Bootstrapping of Secrets and Identity in the Cloud - https:// www.usenix.org/sites/default/files/conference/protected-files/enigma_haken_slides.pdf ➤ Storing Secrets in the Cloud and in the Data Center - http://schd.ws/hosted_files/ appsecusa2015/a5/Turtles.pdf ➤ Turtles All The Way Down - https://danielsomerfield.github.io/turtles/reviews/ 22 TALKS AND PRESENTATIONS

START SIMPLE ➤ If you are new to this, consider two approaches for managing secrets. ➤ One for humans ➤ Another one for systems ➤ Going from easy to relatively harder 23 5 STEPS FOR HUMANS AND SYSTEMS HUMANS Setup Shared Keepass databases synced with each other Setup a Vault server behind Nginx and create entries for SSH keys and database passwords Use a provider which can match company directory, this will make it easy to revoke access to an employee if required SYSTEMS Use the same cloud provider where your servers are hosted If possible deploy server less app which exposes secrets both from the Cloud KMS and the Vault server Easy Hard

No content

No content

QUESTIONS @makash | https://linkd.in/webappsecguy | akash@appsecco.com