Seatec Astronomy

Where are your credentials and secrets stored?
In .env files or in environment variables, or even worse in config files?
Are your primary AWS keys shared amongst developers?
Do you still have SSH keys from former employees on your servers?

If your answer is "Yes" to one or more of these questions you probably haven't heard the term "secrets management"

In this talk we will look into managing secrets in development and operations, and expose the problems related to them.
I will give you an overview of the current state of techniques to mitigate these problems and we'll take a brief look at how an open source tool like Hashicorp Vault can provide a solution to managing secrets in the years to come.

Transcript

1. SEATEC ASTRONOMY 1 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
2. WHOAMI 2 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
3. TOO MANY SECRETS 3 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
4. 4 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
5. INTERACTION TIME 5 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
6. C SUITE, VP ENGINEERING 6 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
7. TEAM LEADS 7 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
8. SENIOR DEVELOPERS/ENGINEERS 8 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
9. UNCLE BOB 9 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
10. CLEAN CODE 10 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
11. BUSINESS PROBLEM 11 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
12. BUSINESS PROBLEM IS YOU! 12 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
13. Take off your business hat Focus on the business goals less on academics — David Cramer (Sentry CEO) from "Mastering Duct Tape" PyCon balkan 2018 13 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
14. management needs to recognize that its engineers are a critical component of any major product decision. Too often, a company’s engineers are seen as a service organization that can be steamrolled when important business decisions arise (ensuring a healthy product- engineering relationship is one more reason it’s critical to have hired the right engineers) — Nemil Dalal 14 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
15. BUSINESS IS NOT INFALLIBLE 15 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
16. DEVELOPERS DESPISE BUSINESS? 16 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
17. BLUE COLLAR ON A MYSTICAL JOURNEY 17 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
18. CARDINAL RULE OF BUSINESS 18 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
19. PRIMARY GOAL OF A SOFTWARE DEVELOPER 19 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
20. SALES MAKE MONEY, ENGINEERS COST MONEY 20 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
21. WASTE AND HUBRIS 21 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
22. "I am not a salesperson, I am just a developer" — well known developer 22 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
23. CAPITAL ALLOCATION 23 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
24. Technical debt can have massive interest when not addressed. People don't even realize how much interest they're paying until they stop paying it. Imagine paying back a rather reasonable loan and suddenly you have 50x more money at the end of each month. The problem is that most people don't bo ther estimating their tech debt. — Anna Filina 24 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
25. Later equals never — Dave LeBlanc 25 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
26. WHAT IS THE MANDATE OF A SOFTWARE ENGINEER? 26 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
27. Your mandate as a software engineer is to find solutions for the problems presented, with acceptable compromises between time, cost and quality, with buy-in from the management/leadership. 27 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
28. 28 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
29. 29 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
30. ENGINEERING OBSESSION 30 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
31. I might not have to work with you, you will have to work with someone like me, so you better be prepared 31 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
32. OK.... SECURITY? 32 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
33. BUSINESS EXPERIENCE AND SECURITY ARE AFTERTHOUGHT 33 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
34. BY THINKING IN TERMS OF BUSINESS EXPERIENCE AND COST AND EXPENSES SECURITY STARTS TO CLIMB MORE AND MORE ON THE PRIORITY LIST 34 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
35. LOW HANGING FRUIT 35 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
36. 36 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
37. COMMON WAYS OF REVEALING YOUR SECRETS? > making your information public, for example by committing them to the repository (and making the repo public) > malicious party gaining access to your infrastructure, ie server 37 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
38. MALICIOUS PARTY GAINING ACCESS TO YOUR INFRASTRUCTURE, IE SERVER 38 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
39. # location: /etc/pam_scripts/login-email-notification.sh #!/bin/sh EMAIL_TO="sysadmin@example.com" EMAIL_FROM="ssh-alert@server1.example.com" SUBJECT="SSH Login Notification" MESSAGE=" A user signed into your server through SSH. ------------------------------------------- Username: ${PAM_USER} IP Address: ${PAM_RHOST}" if [ ${PAM_TYPE} = "open_session" ]; then echo "${MESSAGE}" | mail -n -r "${EMAIL_FROM}" -s "${SUBJECT}" "${EMAIL_TO}" fi exit 0 # location: /etc/pam.d/sshd # Login Email Notification session required pam_exec.so /etc/pam_scripts/login-email-notification.sh 39 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
40. You don't store your users passwords in your database, yet the access-credentials to said database are written down in cleartext in a file on your server. Sounds familiar? — Andreas Heigl 40 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
41. STORY TIME! 41 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
42. ANOTHER ONE! IN CASE YOU ARE STILL NOT CONVINCED 42 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
43. ENVIRONMENT VARIABLES 43 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
44. 44 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
45. tr '\0' '\n' < /proc/<pid>/environ 45 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
46. CONTAINERS!!!111 docker inspect -f` or `docker inspect -f \ '{{range $index, $value := .Config.Env}}{{println $value}}{{end}}' \ container_name 46 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
47. web_app: image: code4hire/dev-images:php-7.2-cli hostname: "web_app" working_dir: ${WEB_DESTINATION_PATH} volumes: - ${WEB_APP_PATH}:${WEB_DESTINATION_PATH} - ${WEB_REPORTS_PATH}:${WEB_REPORTS_DESTINATION_PATH} - ./auth.json:/root/.composer/auth.json - "${DATA_PATH}/datadog:/var/run/datadog:ro" environment: - APPLICATION_ENV=${APPLICATION_ENV} - WEB_LOGGER_NAME=${WEB_LOGGER_NAME} - WEB_LOG_PATH=${WEB_LOG_PATH} - WEB_LOG_LEVEL=${LOG_LEVEL} - WEB_LOG_TO_CONSOLE=${LOG_TO_CONSOLE} - SENTRY_DSN=${SENTRY_DSN} - RMQ_HOST=${RMQ_HOST} - RMQ_PORT=${RMQ_PORT} - RMQ_USERNAME=${RMQ_USERNAME} - RMQ_PASSWORD=${RMQ_PASSWORD} - RMQ_VHOST=${RMQ_VHOST} - RMQ_PREFETCH_COUNT=${RMQ_PREFETCH_COUNT} - RMQ_DEFAULT_EXCHANGE_NAME=${RMQ_DEFAULT_EXCHANGE_NAME} - RMQ_DEFAULT_EXCHANGE_TYPE=${RMQ_DEFAULT_EXCHANGE_TYPE} ... 47 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
48. DOCKER SECRETS!!! /run/secrets/NAME 48 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
49. OPS TEAM ALLOWS THIS! 49 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
50. LIABILITY 50 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
51. INSURANCE 51 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
52. CERTIFICATION 52 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
53. BUSINESS EXPERIENCE SECURITY COST/DAMAGE CONTROL 53 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
54. HOW CAN THIS SITUATION BE IMPROVED? SECRETS MANAGEMENT APPLICATIONS 54 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
55. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS SHOULD BE: > Ease of setup and operation 55 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
56. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS SHOULD BE: > Ease of setup and operation > Secret rotation 56 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
57. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS SHOULD BE: > Ease of setup and operation > Secret rotation > Dynamic secrets 57 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
58. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS SHOULD BE: > Ease of setup and operation > Secret rotation > Dynamic secrets > Encryption in transport and at rest, and choices of backends 58 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
59. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS SHOULD BE: > Ease of setup and operation > Secret rotation > Dynamic secrets > Encryption in transport and at rest, and choices of backends > Cost 59 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
60. Ansible Vault, Barbican, Chef Data Bags, Chef Vault, Citadel, Confidant, Configuration Storage Systems (Consul, etcd, Zookeeper), Conjur, Crypt, EJSON, Keywhiz, Knox, Red October, Trousseau, Vault (Hashicorp) 60 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
61. AWS SECRETS MANAGER 61 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
62. EXAMPLE 1: PRODUCTION-SCALE WEB APPLICATION Cost Dimensions - 2 SSH keys per server and 5 database credentials per database. - 2 API calls per SSH key per day. 24 API calls per database credential per day. - 7 API calls per database credential per week to rotate credentials safely. 15 secrets (2 SSH keys * 1 load balancer + 2 SSH keys * 2 web servers + 2 SSH keys * 2 app servers + 5 database credentials * 1 database) @ $0.40 / secret / month 4,040 API calls (2 SSH keys/server * 5 servers * 1 API call/day * 30 days + 5 database credentials * 1 database * 24 API calls/day * 30 days + 5 database credentials * 1 database * 7 API calls/week * 4 weeks) @ $0.05/10,000 calls $6.02 TOTAL (PER MONTH) 62 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
63. EXAMPLE 2: USING EPHEMERAL SECRETS TO AUTHENTICATE MICRO SERVICES Cost Dimensions 5M secrets (each valid for 1 hour). 2 API calls per secret per month. Note: Since these secrets are stored in Secrets Manager for an hour, the price per secret is calculated as $0.40 * 1 hour / (30 days * 24 hours) = $0.00056 / secret/ hour $2,800.00 5M secrets @ $0.00056 / secret/ hour $50.00 10M API calls (5M secret * 2 API calls) @ $0.05/10,000 calls $2,850.00 TOTAL (PER MONTH) 63 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
64. HASHICORP VAULT 64 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
65. GOOD FIT? 65 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
66. SECURITY MODEL > confidentiality > integrity > availability > accountability > authentication. 66 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
67. 67 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
68. SHAMIR'S SECRET SHARING ALGORITHM 68 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
69. 69 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
70. UNSEALED 70 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
71. FIRST CONTACT 71 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
72. POLICY 72 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
73. UNSEALED AND READY 73 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
74. VAULT STATUS $ vault status Key Value --- ----- Seal Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version 1.0.2 Cluster Name vault-cluster-3cdf26fe Cluster ID 08082f3a-b58d-1abf-a770-fbb8d87359ee HA Enabled false 74 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
75. WRITE A SECRET $ vault kv put secret/hello foo=world excited=yes Key Value --- ----- created_time 2019-02-04T19:54:03.250328Z deletion_time n/a destroyed false version 2 $ curl \ -H "X-Vault-Token: $VAULT_TOKEN" \ -H "Content-Type: application/json" \ -X POST \ -d '{"foo":"world"}' \ -d '{"excited":"yes"}' \ http://127.0.0.1:8200/v1/secret/hello 75 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
76. READ A SECRET $ vault kv get secret/hello ====== Metadata ====== Key Value --- ----- created_time 2019-02-04T19:54:03.250328Z deletion_time n/a destroyed false version 2 ===== Data ===== Key Value --- ----- excited yes foo world $ curl \ -H "X-Vault-Token: $VAULT_TOKEN" \ -X LIST \ http://127.0.0.1:8200/v1/secret/hello 76 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
77. READ ONLY THE VALUE FOR A SECRET $ vault kv get -field=excited secret/hello yes $ curl \ -H "X-Vault-Token: $TOKEN" \ -X GET \ http://127.0.0.1:8200/v1/secret/hello/excited 77 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
78. JSON? SURE $ vault kv get -format=json secret/hello | jq -r .data.data.excited yes 78 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
79. DELETE A SECRET $ vault kv delete secret/hello Success! Data deleted (if it existed) at: secret/hello $ curl \ --header "X-Vault-Token: $VAULT_TOKEN \ --request DELETE \ https://127.0.0.1:8200/v1/secret/hello 79 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
80. PHP $client = new \GuzzleHttp\Client([ 'base_uri' => $baseUrl, 'timeout' => 2.0, 'headers' => [ 'X-Vault-Token' => $accessToken, 'Accept' => 'application/json', ] ]); $response = $client->request('GET', '/v1/secret/hello/excited'); $response->getBody()->seek(0); $output = json_decode(trim($responseBody->getContents())); print_r($output->data->excited) yes 80 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
81. EVERYTHING IS AWESOME? RIGHT? WELL... 81 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
82. SEALING/UNSEALING 82 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
83. HTTP CALLS 83 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
84. FINAL WORDS 84 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
85. average cost of a large data breach (in which more than one million records are lost) in 2018 was $3.9 MILLION DOLLARS 85 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
86. THE END 86 — Srdjan Vranac, Code4Hire, PHPSerbia 2019