Seatec Astronomy

Seatec Astronomy

Where are your credentials and secrets stored?
In .env files or in environment variables, or even worse in config files?
Are your primary AWS keys shared amongst developers?
Do you still have SSH keys from former employees on your servers?

If your answer is "Yes" to one or more of these questions you probably haven't heard the term "secrets management"

In this talk we will look into managing secrets in development and operations, and expose the problems related to them.
I will give you an overview of the current state of techniques to mitigate these problems and we'll take a brief look at how an open source tool like Hashicorp Vault can provide a solution to managing secrets in the years to come.

F2d82b268a7cbccc9809c939428df64f?s=128

Vranac Srdjan

May 25, 2019
Tweet

Transcript

  1. SEATEC ASTRONOMY 1 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  2. WHOAMI 2 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  3. TOO MANY SECRETS 3 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  4. 4 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  5. INTERACTION TIME 5 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  6. C SUITE, VP ENGINEERING 6 — Srdjan Vranac, Code4Hire, PHPSerbia

    2019
  7. TEAM LEADS 7 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  8. SENIOR DEVELOPERS/ENGINEERS 8 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  9. UNCLE BOB 9 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  10. CLEAN CODE 10 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  11. BUSINESS PROBLEM 11 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  12. BUSINESS PROBLEM IS YOU! 12 — Srdjan Vranac, Code4Hire, PHPSerbia

    2019
  13. Take off your business hat Focus on the business goals

    less on academics — David Cramer (Sentry CEO) from "Mastering Duct Tape" PyCon balkan 2018 13 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  14. management needs to recognize that its engineers are a critical

    component of any major product decision. Too often, a company’s engineers are seen as a service organization that can be steamrolled when important business decisions arise (ensuring a healthy product- engineering relationship is one more reason it’s critical to have hired the right engineers) — Nemil Dalal 14 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  15. BUSINESS IS NOT INFALLIBLE 15 — Srdjan Vranac, Code4Hire, PHPSerbia

    2019
  16. DEVELOPERS DESPISE BUSINESS? 16 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  17. BLUE COLLAR ON A MYSTICAL JOURNEY 17 — Srdjan Vranac,

    Code4Hire, PHPSerbia 2019
  18. CARDINAL RULE OF BUSINESS 18 — Srdjan Vranac, Code4Hire, PHPSerbia

    2019
  19. PRIMARY GOAL OF A SOFTWARE DEVELOPER 19 — Srdjan Vranac,

    Code4Hire, PHPSerbia 2019
  20. SALES MAKE MONEY, ENGINEERS COST MONEY 20 — Srdjan Vranac,

    Code4Hire, PHPSerbia 2019
  21. WASTE AND HUBRIS 21 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  22. "I am not a salesperson, I am just a developer"

    — well known developer 22 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  23. CAPITAL ALLOCATION 23 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  24. Technical debt can have massive interest when not addressed. People

    don't even realize how much interest they're paying until they stop paying it. Imagine paying back a rather reasonable loan and suddenly you have 50x more money at the end of each month. The problem is that most people don't bo ther estimating their tech debt. — Anna Filina 24 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  25. Later equals never — Dave LeBlanc 25 — Srdjan Vranac,

    Code4Hire, PHPSerbia 2019
  26. WHAT IS THE MANDATE OF A SOFTWARE ENGINEER? 26 —

    Srdjan Vranac, Code4Hire, PHPSerbia 2019
  27. Your mandate as a software engineer is to find solutions

    for the problems presented, with acceptable compromises between time, cost and quality, with buy-in from the management/leadership. 27 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  28. 28 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  29. 29 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  30. ENGINEERING OBSESSION 30 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  31. I might not have to work with you, you will

    have to work with someone like me, so you better be prepared 31 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  32. OK.... SECURITY? 32 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  33. BUSINESS EXPERIENCE AND SECURITY ARE AFTERTHOUGHT 33 — Srdjan Vranac,

    Code4Hire, PHPSerbia 2019
  34. BY THINKING IN TERMS OF BUSINESS EXPERIENCE AND COST AND

    EXPENSES SECURITY STARTS TO CLIMB MORE AND MORE ON THE PRIORITY LIST 34 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  35. LOW HANGING FRUIT 35 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  36. 36 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  37. COMMON WAYS OF REVEALING YOUR SECRETS? > making your information

    public, for example by committing them to the repository (and making the repo public) > malicious party gaining access to your infrastructure, ie server 37 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  38. MALICIOUS PARTY GAINING ACCESS TO YOUR INFRASTRUCTURE, IE SERVER 38

    — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  39. # location: /etc/pam_scripts/login-email-notification.sh #!/bin/sh EMAIL_TO="sysadmin@example.com" EMAIL_FROM="ssh-alert@server1.example.com" SUBJECT="SSH Login Notification" MESSAGE="

    A user signed into your server through SSH. ------------------------------------------- Username: ${PAM_USER} IP Address: ${PAM_RHOST}" if [ ${PAM_TYPE} = "open_session" ]; then echo "${MESSAGE}" | mail -n -r "${EMAIL_FROM}" -s "${SUBJECT}" "${EMAIL_TO}" fi exit 0 # location: /etc/pam.d/sshd # Login Email Notification session required pam_exec.so /etc/pam_scripts/login-email-notification.sh 39 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  40. You don't store your users passwords in your database, yet

    the access-credentials to said database are written down in cleartext in a file on your server. Sounds familiar? — Andreas Heigl 40 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  41. STORY TIME! 41 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  42. ANOTHER ONE! IN CASE YOU ARE STILL NOT CONVINCED 42

    — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  43. ENVIRONMENT VARIABLES 43 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  44. 44 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  45. tr '\0' '\n' < /proc/<pid>/environ 45 — Srdjan Vranac, Code4Hire,

    PHPSerbia 2019
  46. CONTAINERS!!!111 docker inspect -f` or `docker inspect -f \ '{{range

    $index, $value := .Config.Env}}{{println $value}}{{end}}' \ container_name 46 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  47. web_app: image: code4hire/dev-images:php-7.2-cli hostname: "web_app" working_dir: ${WEB_DESTINATION_PATH} volumes: - ${WEB_APP_PATH}:${WEB_DESTINATION_PATH}

    - ${WEB_REPORTS_PATH}:${WEB_REPORTS_DESTINATION_PATH} - ./auth.json:/root/.composer/auth.json - "${DATA_PATH}/datadog:/var/run/datadog:ro" environment: - APPLICATION_ENV=${APPLICATION_ENV} - WEB_LOGGER_NAME=${WEB_LOGGER_NAME} - WEB_LOG_PATH=${WEB_LOG_PATH} - WEB_LOG_LEVEL=${LOG_LEVEL} - WEB_LOG_TO_CONSOLE=${LOG_TO_CONSOLE} - SENTRY_DSN=${SENTRY_DSN} - RMQ_HOST=${RMQ_HOST} - RMQ_PORT=${RMQ_PORT} - RMQ_USERNAME=${RMQ_USERNAME} - RMQ_PASSWORD=${RMQ_PASSWORD} - RMQ_VHOST=${RMQ_VHOST} - RMQ_PREFETCH_COUNT=${RMQ_PREFETCH_COUNT} - RMQ_DEFAULT_EXCHANGE_NAME=${RMQ_DEFAULT_EXCHANGE_NAME} - RMQ_DEFAULT_EXCHANGE_TYPE=${RMQ_DEFAULT_EXCHANGE_TYPE} ... 47 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  48. DOCKER SECRETS!!! /run/secrets/NAME 48 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  49. OPS TEAM ALLOWS THIS! 49 — Srdjan Vranac, Code4Hire, PHPSerbia

    2019
  50. LIABILITY 50 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  51. INSURANCE 51 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  52. CERTIFICATION 52 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  53. BUSINESS EXPERIENCE SECURITY COST/DAMAGE CONTROL 53 — Srdjan Vranac, Code4Hire,

    PHPSerbia 2019
  54. HOW CAN THIS SITUATION BE IMPROVED? SECRETS MANAGEMENT APPLICATIONS 54

    — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  55. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS SHOULD BE: >

    Ease of setup and operation 55 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  56. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS SHOULD BE: >

    Ease of setup and operation > Secret rotation 56 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  57. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS SHOULD BE: >

    Ease of setup and operation > Secret rotation > Dynamic secrets 57 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  58. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS SHOULD BE: >

    Ease of setup and operation > Secret rotation > Dynamic secrets > Encryption in transport and at rest, and choices of backends 58 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  59. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS SHOULD BE: >

    Ease of setup and operation > Secret rotation > Dynamic secrets > Encryption in transport and at rest, and choices of backends > Cost 59 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  60. Ansible Vault, Barbican, Chef Data Bags, Chef Vault, Citadel, Confidant,

    Configuration Storage Systems (Consul, etcd, Zookeeper), Conjur, Crypt, EJSON, Keywhiz, Knox, Red October, Trousseau, Vault (Hashicorp) 60 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  61. AWS SECRETS MANAGER 61 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  62. EXAMPLE 1: PRODUCTION-SCALE WEB APPLICATION Cost Dimensions - 2 SSH

    keys per server and 5 database credentials per database. - 2 API calls per SSH key per day. 24 API calls per database credential per day. - 7 API calls per database credential per week to rotate credentials safely. 15 secrets (2 SSH keys * 1 load balancer + 2 SSH keys * 2 web servers + 2 SSH keys * 2 app servers + 5 database credentials * 1 database) @ $0.40 / secret / month 4,040 API calls (2 SSH keys/server * 5 servers * 1 API call/day * 30 days + 5 database credentials * 1 database * 24 API calls/day * 30 days + 5 database credentials * 1 database * 7 API calls/week * 4 weeks) @ $0.05/10,000 calls $6.02 TOTAL (PER MONTH) 62 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  63. EXAMPLE 2: USING EPHEMERAL SECRETS TO AUTHENTICATE MICRO SERVICES Cost

    Dimensions 5M secrets (each valid for 1 hour). 2 API calls per secret per month. Note: Since these secrets are stored in Secrets Manager for an hour, the price per secret is calculated as $0.40 * 1 hour / (30 days * 24 hours) = $0.00056 / secret/ hour $2,800.00 5M secrets @ $0.00056 / secret/ hour $50.00 10M API calls (5M secret * 2 API calls) @ $0.05/10,000 calls $2,850.00 TOTAL (PER MONTH) 63 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  64. HASHICORP VAULT 64 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  65. GOOD FIT? 65 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  66. SECURITY MODEL > confidentiality > integrity > availability > accountability

    > authentication. 66 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  67. 67 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  68. SHAMIR'S SECRET SHARING ALGORITHM 68 — Srdjan Vranac, Code4Hire, PHPSerbia

    2019
  69. 69 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  70. UNSEALED 70 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  71. FIRST CONTACT 71 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  72. POLICY 72 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  73. UNSEALED AND READY 73 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  74. VAULT STATUS $ vault status Key Value --- ----- Seal

    Type shamir Initialized true Sealed false Total Shares 1 Threshold 1 Version 1.0.2 Cluster Name vault-cluster-3cdf26fe Cluster ID 08082f3a-b58d-1abf-a770-fbb8d87359ee HA Enabled false 74 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  75. WRITE A SECRET $ vault kv put secret/hello foo=world excited=yes

    Key Value --- ----- created_time 2019-02-04T19:54:03.250328Z deletion_time n/a destroyed false version 2 $ curl \ -H "X-Vault-Token: $VAULT_TOKEN" \ -H "Content-Type: application/json" \ -X POST \ -d '{"foo":"world"}' \ -d '{"excited":"yes"}' \ http://127.0.0.1:8200/v1/secret/hello 75 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  76. READ A SECRET $ vault kv get secret/hello ====== Metadata

    ====== Key Value --- ----- created_time 2019-02-04T19:54:03.250328Z deletion_time n/a destroyed false version 2 ===== Data ===== Key Value --- ----- excited yes foo world $ curl \ -H "X-Vault-Token: $VAULT_TOKEN" \ -X LIST \ http://127.0.0.1:8200/v1/secret/hello 76 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  77. READ ONLY THE VALUE FOR A SECRET $ vault kv

    get -field=excited secret/hello yes $ curl \ -H "X-Vault-Token: $TOKEN" \ -X GET \ http://127.0.0.1:8200/v1/secret/hello/excited 77 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  78. JSON? SURE $ vault kv get -format=json secret/hello | jq

    -r .data.data.excited yes 78 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  79. DELETE A SECRET $ vault kv delete secret/hello Success! Data

    deleted (if it existed) at: secret/hello $ curl \ --header "X-Vault-Token: $VAULT_TOKEN \ --request DELETE \ https://127.0.0.1:8200/v1/secret/hello 79 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  80. PHP $client = new \GuzzleHttp\Client([ 'base_uri' => $baseUrl, 'timeout' =>

    2.0, 'headers' => [ 'X-Vault-Token' => $accessToken, 'Accept' => 'application/json', ] ]); $response = $client->request('GET', '/v1/secret/hello/excited'); $response->getBody()->seek(0); $output = json_decode(trim($responseBody->getContents())); print_r($output->data->excited) yes 80 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  81. EVERYTHING IS AWESOME? RIGHT? WELL... 81 — Srdjan Vranac, Code4Hire,

    PHPSerbia 2019
  82. SEALING/UNSEALING 82 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  83. HTTP CALLS 83 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  84. FINAL WORDS 84 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

  85. average cost of a large data breach (in which more

    than one million records are lost) in 2018 was $3.9 MILLION DOLLARS 85 — Srdjan Vranac, Code4Hire, PHPSerbia 2019
  86. THE END 86 — Srdjan Vranac, Code4Hire, PHPSerbia 2019