Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Seatec Astronomy

Seatec Astronomy

Where are your credentials and secrets stored?
In .env files or in environment variables, or even worse in config files?
Are your primary AWS keys shared amongst developers?
Do you still have SSH keys from former employees on your servers?

If your answer is "Yes" to one or more of these questions you probably haven't heard the term "secrets management"

In this talk we will look into managing secrets in development and operations, and expose the problems related to them.
I will give you an overview of the current state of techniques to mitigate these problems and we'll take a brief look at how an open source tool like Hashicorp Vault can provide a solution to managing secrets in the years to come.

Vranac Srdjan

May 25, 2019
Tweet

More Decks by Vranac Srdjan

Other Decks in Technology

Transcript

  1. SEATEC
    ASTRONOMY
    1 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  2. WHOAMI
    2 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  3. TOO MANY SECRETS
    3 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  4. 4 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  5. INTERACTION TIME
    5 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  6. C SUITE, VP ENGINEERING
    6 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  7. TEAM LEADS
    7 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  8. SENIOR
    DEVELOPERS/ENGINEERS
    8 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  9. UNCLE BOB
    9 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  10. CLEAN CODE
    10 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  11. BUSINESS PROBLEM
    11 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  12. BUSINESS PROBLEM
    IS
    YOU!
    12 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  13. Take off your business hat
    Focus on the business goals
    less on academics
    — David Cramer (Sentry CEO)
    from "Mastering Duct Tape" PyCon balkan 2018
    13 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  14. management needs to recognize that its engineers are a
    critical component of any major product decision. Too
    often, a company’s engineers are seen as a service
    organization that can be steamrolled when important
    business decisions arise (ensuring a healthy product-
    engineering relationship is one more reason it’s critical
    to have hired the right engineers)
    — Nemil Dalal
    14 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  15. BUSINESS IS NOT
    INFALLIBLE
    15 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  16. DEVELOPERS DESPISE
    BUSINESS?
    16 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  17. BLUE COLLAR ON A
    MYSTICAL JOURNEY
    17 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  18. CARDINAL RULE OF
    BUSINESS
    18 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  19. PRIMARY GOAL OF A
    SOFTWARE DEVELOPER
    19 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  20. SALES MAKE MONEY,
    ENGINEERS COST MONEY
    20 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  21. WASTE AND HUBRIS
    21 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  22. "I am not a salesperson,
    I am just a developer"
    — well known developer
    22 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  23. CAPITAL ALLOCATION
    23 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  24. Technical debt can have massive interest when not
    addressed. People don't even realize how much interest
    they're paying until they stop paying it. Imagine paying back
    a rather reasonable loan and suddenly you have 50x more
    money at the end of each month. The problem is that most
    people don't bo ther estimating their tech debt.
    — Anna Filina
    24 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  25. Later equals never
    — Dave LeBlanc
    25 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  26. WHAT IS THE MANDATE OF
    A SOFTWARE ENGINEER?
    26 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  27. Your mandate as a software engineer is
    to find solutions for the problems presented,
    with acceptable compromises between time, cost and
    quality, with buy-in from the management/leadership.
    27 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  28. 28 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  29. 29 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  30. ENGINEERING OBSESSION
    30 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  31. I might not have to work with you,
    you will have to work with someone like me,
    so you better be prepared
    31 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  32. OK.... SECURITY?
    32 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  33. BUSINESS EXPERIENCE AND SECURITY
    ARE AFTERTHOUGHT
    33 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  34. BY THINKING IN TERMS OF BUSINESS
    EXPERIENCE AND COST AND EXPENSES
    SECURITY STARTS TO CLIMB MORE AND
    MORE ON THE PRIORITY LIST
    34 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  35. LOW HANGING FRUIT
    35 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  36. 36 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  37. COMMON WAYS OF REVEALING YOUR SECRETS?
    > making your information public, for example by
    committing them to the repository (and making the repo
    public)
    > malicious party gaining access to your infrastructure,
    ie server
    37 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  38. MALICIOUS PARTY GAINING ACCESS TO
    YOUR INFRASTRUCTURE, IE SERVER
    38 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  39. # location: /etc/pam_scripts/login-email-notification.sh
    #!/bin/sh
    EMAIL_TO="[email protected]"
    EMAIL_FROM="[email protected]"
    SUBJECT="SSH Login Notification"
    MESSAGE="
    A user signed into your server through SSH.
    -------------------------------------------
    Username: ${PAM_USER}
    IP Address: ${PAM_RHOST}"
    if [ ${PAM_TYPE} = "open_session" ]; then
    echo "${MESSAGE}" | mail -n -r "${EMAIL_FROM}" -s "${SUBJECT}" "${EMAIL_TO}"
    fi
    exit 0
    # location: /etc/pam.d/sshd
    # Login Email Notification
    session required pam_exec.so /etc/pam_scripts/login-email-notification.sh
    39 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  40. You don't store your users passwords in your database,
    yet the access-credentials to said database are written
    down in cleartext in a file on your server. Sounds
    familiar?
    — Andreas Heigl
    40 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  41. STORY TIME!
    41 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  42. ANOTHER ONE!
    IN CASE YOU ARE STILL NOT CONVINCED
    42 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  43. ENVIRONMENT VARIABLES
    43 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  44. 44 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  45. tr '\0' '\n' < /proc//environ
    45 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  46. CONTAINERS!!!111
    docker inspect -f` or `docker inspect -f \
    '{{range $index, $value := .Config.Env}}{{println $value}}{{end}}' \
    container_name
    46 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  47. web_app:
    image: code4hire/dev-images:php-7.2-cli
    hostname: "web_app"
    working_dir: ${WEB_DESTINATION_PATH}
    volumes:
    - ${WEB_APP_PATH}:${WEB_DESTINATION_PATH}
    - ${WEB_REPORTS_PATH}:${WEB_REPORTS_DESTINATION_PATH}
    - ./auth.json:/root/.composer/auth.json
    - "${DATA_PATH}/datadog:/var/run/datadog:ro"
    environment:
    - APPLICATION_ENV=${APPLICATION_ENV}
    - WEB_LOGGER_NAME=${WEB_LOGGER_NAME}
    - WEB_LOG_PATH=${WEB_LOG_PATH}
    - WEB_LOG_LEVEL=${LOG_LEVEL}
    - WEB_LOG_TO_CONSOLE=${LOG_TO_CONSOLE}
    - SENTRY_DSN=${SENTRY_DSN}
    - RMQ_HOST=${RMQ_HOST}
    - RMQ_PORT=${RMQ_PORT}
    - RMQ_USERNAME=${RMQ_USERNAME}
    - RMQ_PASSWORD=${RMQ_PASSWORD}
    - RMQ_VHOST=${RMQ_VHOST}
    - RMQ_PREFETCH_COUNT=${RMQ_PREFETCH_COUNT}
    - RMQ_DEFAULT_EXCHANGE_NAME=${RMQ_DEFAULT_EXCHANGE_NAME}
    - RMQ_DEFAULT_EXCHANGE_TYPE=${RMQ_DEFAULT_EXCHANGE_TYPE}
    ...
    47 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  48. DOCKER SECRETS!!!
    /run/secrets/NAME
    48 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  49. OPS TEAM ALLOWS THIS!
    49 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  50. LIABILITY
    50 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  51. INSURANCE
    51 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  52. CERTIFICATION
    52 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  53. BUSINESS EXPERIENCE
    SECURITY
    COST/DAMAGE CONTROL
    53 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  54. HOW CAN THIS SITUATION BE
    IMPROVED?
    SECRETS MANAGEMENT
    APPLICATIONS
    54 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  55. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS
    SHOULD BE:
    > Ease of setup and operation
    55 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  56. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS
    SHOULD BE:
    > Ease of setup and operation
    > Secret rotation
    56 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  57. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS
    SHOULD BE:
    > Ease of setup and operation
    > Secret rotation
    > Dynamic secrets
    57 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  58. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS
    SHOULD BE:
    > Ease of setup and operation
    > Secret rotation
    > Dynamic secrets
    > Encryption in transport and at rest, and choices of
    backends
    58 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  59. IMPORTANT FEATURES FOR ANY OF THESE SYSTEMS
    SHOULD BE:
    > Ease of setup and operation
    > Secret rotation
    > Dynamic secrets
    > Encryption in transport and at rest, and choices of
    backends
    > Cost
    59 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  60. Ansible Vault, Barbican, Chef Data Bags, Chef Vault,
    Citadel, Confidant, Configuration Storage Systems
    (Consul, etcd, Zookeeper), Conjur, Crypt, EJSON, Keywhiz,
    Knox, Red October, Trousseau, Vault (Hashicorp)
    60 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  61. AWS SECRETS MANAGER
    61 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  62. EXAMPLE 1: PRODUCTION-SCALE WEB APPLICATION
    Cost Dimensions
    - 2 SSH keys per server and 5 database credentials per database.
    - 2 API calls per SSH key per day. 24 API calls per database credential per day.
    - 7 API calls per database credential per week to rotate credentials safely.
    15 secrets (2 SSH keys * 1 load balancer
    + 2 SSH keys * 2 web servers
    + 2 SSH keys * 2 app servers
    + 5 database credentials * 1 database)
    @ $0.40 / secret / month
    4,040 API calls (2 SSH keys/server * 5 servers * 1 API call/day * 30 days
    + 5 database credentials * 1 database * 24 API calls/day * 30 days
    + 5 database credentials * 1 database * 7 API calls/week * 4 weeks) @ $0.05/10,000 calls
    $6.02 TOTAL (PER MONTH)
    62 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  63. EXAMPLE 2: USING EPHEMERAL SECRETS TO AUTHENTICATE MICRO
    SERVICES
    Cost Dimensions
    5M secrets (each valid for 1 hour).
    2 API calls per secret per month.
    Note: Since these secrets are stored in Secrets Manager for an hour,
    the price per secret is calculated as
    $0.40 * 1 hour / (30 days * 24 hours) = $0.00056 / secret/ hour
    $2,800.00 5M secrets @ $0.00056 / secret/ hour
    $50.00 10M API calls (5M secret * 2 API calls) @ $0.05/10,000 calls
    $2,850.00 TOTAL (PER MONTH)
    63 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  64. HASHICORP VAULT
    64 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  65. GOOD FIT?
    65 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  66. SECURITY MODEL
    > confidentiality
    > integrity
    > availability
    > accountability
    > authentication.
    66 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  67. 67 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  68. SHAMIR'S SECRET SHARING ALGORITHM
    68 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  69. 69 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  70. UNSEALED
    70 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  71. FIRST CONTACT
    71 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  72. POLICY
    72 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  73. UNSEALED AND READY
    73 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  74. VAULT STATUS
    $ vault status
    Key Value
    --- -----
    Seal Type shamir
    Initialized true
    Sealed false
    Total Shares 1
    Threshold 1
    Version 1.0.2
    Cluster Name vault-cluster-3cdf26fe
    Cluster ID 08082f3a-b58d-1abf-a770-fbb8d87359ee
    HA Enabled false
    74 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  75. WRITE A SECRET
    $ vault kv put secret/hello foo=world excited=yes
    Key Value
    --- -----
    created_time 2019-02-04T19:54:03.250328Z
    deletion_time n/a
    destroyed false
    version 2
    $ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -H "Content-Type: application/json" \
    -X POST \
    -d '{"foo":"world"}' \
    -d '{"excited":"yes"}' \
    http://127.0.0.1:8200/v1/secret/hello
    75 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  76. READ A SECRET
    $ vault kv get secret/hello
    ====== Metadata ======
    Key Value
    --- -----
    created_time 2019-02-04T19:54:03.250328Z
    deletion_time n/a
    destroyed false
    version 2
    ===== Data =====
    Key Value
    --- -----
    excited yes
    foo world
    $ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X LIST \
    http://127.0.0.1:8200/v1/secret/hello
    76 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  77. READ ONLY THE VALUE FOR A SECRET
    $ vault kv get -field=excited secret/hello
    yes
    $ curl \
    -H "X-Vault-Token: $TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/secret/hello/excited
    77 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  78. JSON? SURE
    $ vault kv get -format=json secret/hello | jq -r .data.data.excited
    yes
    78 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  79. DELETE A SECRET
    $ vault kv delete secret/hello
    Success! Data deleted (if it existed) at: secret/hello
    $ curl \
    --header "X-Vault-Token: $VAULT_TOKEN \
    --request DELETE \
    https://127.0.0.1:8200/v1/secret/hello
    79 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  80. PHP
    $client = new \GuzzleHttp\Client([
    'base_uri' => $baseUrl,
    'timeout' => 2.0,
    'headers' => [
    'X-Vault-Token' => $accessToken,
    'Accept' => 'application/json',
    ]
    ]);
    $response = $client->request('GET', '/v1/secret/hello/excited');
    $response->getBody()->seek(0);
    $output = json_decode(trim($responseBody->getContents()));
    print_r($output->data->excited)
    yes
    80 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  81. EVERYTHING IS AWESOME? RIGHT?
    WELL...
    81 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  82. SEALING/UNSEALING
    82 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  83. HTTP CALLS
    83 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  84. FINAL WORDS
    84 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  85. average cost of a large data breach (in which more than
    one million records are lost) in 2018 was
    $3.9 MILLION DOLLARS
    85 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide

  86. THE END
    86 — Srdjan Vranac, Code4Hire, PHPSerbia 2019

    View Slide