Secrets Management for Development & Operations

Originally presented at DevOpsDays Istanbul, 2021.

Your company tells you they want to secure your application and infrastructure with a secrets manager. How should you use it, and how will it affect your code? In this session, you’ll learn how to refactor your applications, delivery pipelines, local development, and infrastructure to accommodate rotating secrets. You'll take away the best practices and patterns for integrating with a secrets manager while minimizing interruptions to your development workflow.

Transcript

1. DevOpsDays Istanbul
   
   
   September 2021
   

   

   @joatmon08
   Secrets Management
   for Development &
   Operations
   1
   

   View Slide

2. Secrets
   
   
   Passwords, API tokens, SSL Certificates, or any other sensitive information
   used to access something else.
   2
   

   View Slide

3. We have a secrets
   manager!
   3
   

   View Slide

4. It will help us securely
   access, store, revoke,
   and rotate secrets.
   4
   

   View Slide

5. Immutability & ephemerality
   
   
   in security changes development and operations workflows.
   5
   

   View Slide

6. How does it affect my
   development or
   operations workflows?
   6
   

   View Slide

7. What do I need to
   know to adopt a
   secrets manager?
   7
   

   View Slide

8. Developer Advocate at HashiCorp
   

   she/her
   
   
   

   joatmon08.github.io
   

   @joatmon08
   Rosemary Wang
   8
   

   View Slide

9. What you
   need to
   know
   ▪ Organizing your secrets manager
   
   
   ▪ Comparing patterns for secrets injection
   
   
   ▪ Adding a secrets manager to your:
   
   
   – Delivery pipelines
   
   
   – Infrastructure
   
   
   – Local development
   
   
   – Applications
   

   View Slide

10. Organize your
    secrets manager
    10
    

    View Slide

11. Authentication
    
    
    Try using protocols or
    built-in methods
    TARGET
    SECRETS MANAGER
    /BUSINESS/SECRET
    AUTHENTICATE
    OIDC/JWT
    
    
    KUBERNETES
    
    
    CLOUD PROVIDERS
    
    
    TOKENS
    
    
    (AVOID USERNAME/PASSWORD!)
    11
    

    View Slide

12. Access
    Control
    
    
    Implement least-
    privilege access by:
    business domain,
    environment, and
    usage
    TARGET (DEV)
    SECRETS MANAGER
    /BUSINESS/SECRET
    ALLOW READ FROM
    
    
    /BUSINESS/DEV/SECRET
    ADMINISTRATORS
    ALLOW WRITE TO
    
    
    /BUSINESS/*
    TARGET (PROD)
    12
    ALLOW READ FROM
    
    
    /BUSINESS/PROD/SECRET
    

    View Slide

13. Reasonable
    TTLs
    
    
    You don’t need a
    database password
    rotated every five
    seconds.
    TARGET
    SECRETS MANAGER
    /BUSINESS/SECRET
    AUTHENTICATE
    GET
    SET RENEWAL INTERVAL
    
    
    TO BE AT LEAST HALF
    
    
    THE TTL OF THE SECRET.
    13
    

    View Slide

14. 1. Use built-in authentication protocols
    
    
    2. Set up access control for secrets
    
    
    3. Configure a reasonable time-to-live
    14
    

    View Slide

15. Compare secrets
    injection patterns
    15
    

    View Slide

16. Direct
    
    
    Use script or client
    library.
    TARGET
    SECRETS MANAGER
    /BUSINESS/SECRET
    AUTHENTICATE
    GET
    16
    

    View Slide

17. Sidecar
    
    
    Separate async
    process.
    TARGET
    READ
    FILE
    SECRETS MANAGER
    /BUSINESS/SECRET
    SIDECAR
    WRITE
    AUTHENTICATE
    GET
    17
    

    View Slide

18. Add a secrets
    manager
    18
    

    View Slide

19. Delivery Pipelines
    
    
    Use secrets for deployment
    19
    

    View Slide

20. UNIT TESTS INTEGRATION
    
    
    TESTS
    DEPLOY
    
    
    TO DEV
    GET SECRETS
    
    
    FOR DEV
    SECRETS MANAGER
    /BUSINESS/SECRET
    INTEGRATION
    
    
    TESTS
    DEPLOY
    
    
    TO PROD
    GET SECRETS
    
    
    FOR PROD
    SECRETS MANAGER
    /BUSINESS/SECRET
    Direct Consider…
    
    
    • Separate by manager versus path
    
    
    • Only allow certain CI runners to authenticate
    20
    

    View Slide

21. Infrastructure
    
    
    Defining secrets “as code”
    21
    

    View Slide

22. CODE EDITOR
    resource "aws_db_instance" "products"
    {
    
    allocated_storage = 1
    0
    
    engine = "postgres
    "
    
    engine_version = "11.6
    "
    
    instance_class = "db.t3.micro
    "
    
    name = "products
    "
    
    identifier = "${var.name}-products
    "
    
    username = var.database_usernam
    e
    
    password = var.database_passwor
    d
    
    }
    
    22
    

    View Slide

23. PASS TO
    
    
    INFRASTRUCTURE
    
    
    AS CODE
    GET
    
    
    “BOOTSTRAP”
    
    
    DATABASE
    
    
    PASSWORD
    SECRETS MANAGER
    /BUSINESS/SECRET
    CONFIGURE
    
    
    DATABASE
    ROTATE
    
    
    “BOOTSTRAP”
    
    
    DATABASE
    
    
    PASSWORD
    Consider…
    
    
    • Configuration drift versus security
    
    
    • Secret storage in infrastructure as code
    23
    

    View Slide

24. Local development
    
    
    Retrieving secrets for testing
    24
    

    View Slide

25. SECRETS MANAGER
    /BUSINESS/SECRET
    SECURE ACCESS
    
    
    MANAGEMENT
    GET CREDENTIALS
    
    
    FROM SECRETS MANAGER
    LOG INTO
    
    
    DATABASE
    Consider…
    
    
    • Tracking human versus service access
    
    
    • Evaluate TTL for development testing
    25
    

    View Slide

26. Applications
    
    
    “Inject” secrets while running
    26
    

    View Slide

27. APPLICATION
    
    
    USES CLIENT
    
    
    LIBRARY TO
    
    
    GET SECRET
    SECRETS MANAGER
    /BUSINESS/SECRET
    Direct
    START
    
    
    APPLICATION
    NEW SECRET?
    
    
    RESTART
    
    
    APPLICATION.
    You must…
    
    
    • Implement application reload or separate thread to
    get secrets
    
    
    • Account for connection failure to secrets manager
    
    
    (in your application code)
    27
    

    View Slide

28. Spring
    Cloud
    
    
    Add to bootstrap.yml
    spring.cloud.vault
    :
    
    authentication: APPROL
    E
    
    app-role
    :
    
    role-id: REDACTE
    D
    
    secret-id: REDACTE
    D
    
    mysql
    :
    
    enabled: tru
    e
    
    role: readonl
    y
    
    backend: mysq
    l
    
    username-property: spring.datasource.usernam
    e
    
    password-property: spring.datasource.password
    CODE EDITOR
    

    View Slide

29. .NET
    Example
    
    
    Add configuration provider
    public VaultConfigurationProvider(VaultOptions config
    )
    
    {
    
    _config = config
    ;
    
    var vaultClientSettings = new VaultClientSettings
    (
    
    _config.Address
    ,
    
    new AppRoleAuthMethodInfo(_config.Role
    ,
    
    _config.Secret
    )
    
    )
    ;
    
    _client = new VaultClient(vaultClientSettings)
    ;
    
    }
    CODE EDITOR
    

    View Slide

30. .NET
    Example
    
    
    Add extension method
    public static class VaultExtension
    s
    
    {
    
    public static IConfigurationBuilder AddVault
    (
    
    this IConfigurationBuilder configuration
    ,
    
    Action options
    )
    
    {
    
    var vaultOptions =
    
    
    new VaultConfigurationSource(options)
    ;
    
    configuration.Add(vaultOptions)
    ;
    
    return configuration
    ;
    
    }
    
    }
    CODE EDITOR
    

    View Slide

31. CONFIGURATION
    
    
    FILE
    SECRETS MANAGER
    /BUSINESS/SECRET
    SIDECAR PROCESS
    Sidecar
    APPLICATION
    
    
    READS
    
    
    CONFIGURATION
    
    
    FROM FILE
    CHANGE TO FILE?
    
    
    RESTART
    
    
    APPLICATION.
    START
    
    
    APPLICATION
    You must…
    
    
    • Configure separate process
    
    
    • Access control to file
    
    
    • Configure application reload if file changes (Spring,
    .NET)
    31
    

    View Slide

32. What you
    need to
    know
    ▪ Organizing a secrets manager
    
    
    ▪ Comparing patterns for secrets injection
    
    
    ▪ Adding a secrets manager to your:
    
    
    – Delivery pipelines
    
    
    – Infrastructure
    
    
    – Local development
    
    
    – Applications
    

    View Slide

33. Learn more…
    
    
    ▪ Delivery pipeline: youtu.be/qgF7XquqVSA
    
    
    ▪ Application + secrets management deep dive: youtu.be/gO4i_s0h1uo
    
    
    ▪ Application Client Libraries (Tutorials & Getting Started)
    
    
    – learn.hashicorp.com/collections/vault/app-integration
    
    
    – cloud.spring.io/spring-cloud-vault/reference/html/
    
    
    ▪ Kubernetes + Vault
    
    
    – learn.hashicorp.com/tutorials/vault/kubernetes-sidecar
    
    
    – learn.hashicorp.com/tutorials/vault/kubernetes-secret-store-driver
    33
    

    View Slide

34. joatmon08.github.io
    

    Rosemary Wang
    

    @joatmon08
    Thank you!
    34
    

    View Slide