Think Like a Vault Developer - Secure Introduction at Scale

These slides correspond to the talk given at https://www.youtube.com/watch?v=R-jJXm3QGLQ&list=PL81sUbsFNc5bDS1lH0HPJFyBnAP8Iv4t0&index=22

Transcript

1. Think Like a Vault Developer: Secure Introduction at Scale Jeff Mitchell HashiCorp @jefferai
2. • Understand basic tenets of security ◦ The definitions we use and the way we think about them • Explore properties necessary for secret management and (initial) secret distribution Goals
3. • Applied security in action: secure introduction, response wrapping and orchestrators ◦ Not a Nomad/Vault talk Goals
4. Non-Goals • Discussion of crypto theory ◦ Crypto properties (algorithm selection, etc.) are important for applied security ◦ Can stay high-level • Comparison of container/hypervisor/OS security ◦ Let’s talk Linux/Docker
5. Non-Goals • Deep dive into security rabbit holes ◦ Let’s keep paranoia in check
6. Tenets / Defining Security
7. “Security” • Security is the practice of risk management ◦ Deciding which risks can be accepted and which cannot ◦ Guarding against violations of norms • Risk increases with system complexity ◦ More points of failure, confusion, ingress and egress
8. “Threat” • Anything that can elevate risk is a threat • Modeling threats informs security policy
9. “Secret” • A secret is something that will elevate your risk if exposed to unauthorized entities • Undesired consequences are harm ◦ Unauthorized data access ◦ Identity spoofing ◦ Private data egress ◦ Regulatory fines
10. Exposure • An exposed secret ◦ Elevates risk ◦ May cause harm • An exposed secret is a threat
11. Secrets vs. Identifiers • Some things that can be disclosed are identifiers ◦ Username (identifier) vs. Password (secret) ◦ TLS certificate (identifier) vs. TLS key (secret) ◦ GitHub user name (identifier) vs. API key (secret) • Identifiers aren’t completely risk-free ◦ Have chosen to ignore that risk
12. What is “Trust”? • A trusted entity is one that will not divulge the secrets it has access to • Modeling trusted entities is companion to modeling threats • Two concepts
13. Circle of Trust Entities we trust with any secret. For this talk: RAM root Secret Management Tool Employees? CPU Cloud?
14. Circle of Trust Things not in circle of trust: Persistent Storage Random Users General apps/services NSA Random Wi-Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud? RAM root Secret Management Tool CPU Cloud? Employees?
15. Circle of Trust Only allowed long-term storage is in circle of trust Persistent Storage Random Users General apps/services NSA Random Wi-Fi Hotspot Your Mother’s Notepad.txt Employees? CPU Cloud? RAM root Secret Management Tool CPU Cloud? Employees?
16. Chain of Trust • The set of links (e.g. network hops) that any particular secret travels through from source entity A to destination entity B • Source/destination must be in circle of trust
17. Links • Any link in a chain of trust is a potential access point/interception point ◦ Accidental logging ◦ Exploitation by attacker ◦ Lookup by operator ◦ Employee Post-It™ ◦ Conspiracy of One/Compromised employee
18. Problem Space • We can now establish a problem space: ◦ Managing secrets in a given environment means establishing trust chains in the environment • Links in the chains have associated risk ◦ Minimize hops and minimize risk-per-link
19. Problem Space • In other words: figuring out how to securely get a secret from producer to consumer ◦ Duh, right? • You have to take the roundabout approach ◦ Understanding trust and trust levels along the way is incredibly important ◦ Only after understanding trust can you figure out where and how to minimize risk
20. Problem Space • Risk cannot be fully mitigated; must assume any given secret will eventually be divulged • Ultimate goal: zero trust ◦ Don’t give the opportunity for risks to occur in the first place
21. Scheduler Secret Management Tool Scheduler Agent Container Goal: Securely move secret from originator to new container’s RAM
22. Scheduler Secret Management Tool Scheduler Agent Container ?
23. Scheduler Secret Management Tool Scheduler Agent Container ?
24. Scheduler Secret Management Tool Scheduler Agent Container ?
25. Security Properties / Success Criteria
26. Secret Protection • Establishing a chain of trust requires defining the requirements it must fulfill to keep secrets protected • Good news: we (often) only need to do this for one secret! ◦ First secret authenticates us to allow direct access to more
27. Secure Introduction • If you can protect this secret, you can protect any secret ◦ ...generally • This concept is secure introduction (SI)
28. Secure Introduction • How do you protect secrets (perform SI)? • First: establish success criteria based on your organization’s acceptable risk ◦ Any given organization needs a security policy and associated success criteria that meets their individual needs
29. Secure Introduction • Reasons your organization doesn’t need to establish a security policy: ◦ None
30. Success Criteria • The security properties that must be implemented/met to implement the accepted security policy • For secure introduction for this talk: ◦ Don’t let them live forever (rotate/expire) ◦ Distribute them securely ◦ Limit exposure if disclosed ◦ Have a break-glass procedure ◦ Detect unauthorized access
31. Rotation/Expiration • As lifetime increases, chance for exposure → ∞ ◦ Caches/logs ◦ Cracked over time/enough packets ◦ Debugging
32. Rotation/Expiration • Secrets should be rotated “frequently” and have lifetimes as short as possible • User passwords vs. machine secrets ◦ xkcd’s ______ ______ ______ ______ → bad policies + frequent rotation = written down user passwords ◦ Less frequently/more used = more likely overseen ◦ Machines: it’s just data - plan/build for rotation and rotate often
33. Distribution • The literal movement along the chain of trust ◦ To/From ▪ People ▪ Machines • Base level: never plaintext, always covered ◦ Encryption ◦ Wrapping ◦ Etc.
34. Limit Exposure • Principle of least privilege ◦ DB credentials: only specific tables/operations ◦ Login credentials: not root ◦ API credentials: minimal function set
35. Access Detection • Things have a way of being leaky • Env vars are a common way to pass in container secrets. Also: ◦ Often logged, sometimes multiple places ◦ Easily discoverable by operator ▪ Both Docker commands and non-Docker commands can spill the beans
36. Access Detection • Equally as important as protecting a secret is knowing if an unintended party has intercepted it • Audit logs are great… ◦ ...but do you look at them? • Active detection (when possible) is even better
37. Break-Glass • Compromised? ◦ Stop all further access to protected resources ◦ Perform forensics ◦ Rotate all secrets after re-establishing identities of legitimate secret-holders ◦ … • Figure this out during the planning process -- not after!
38. • You do not have to face this alone
39. • Recent explosion of FOSS Secret Management (SM) tools ◦ KeyWhiz ◦ Knox ◦ Conjur (Docker only) ◦ Many more Secret Management Tools
40. • Recent explosion of FOSS Secret Management (SM) tools ◦ KeyWhiz ◦ Knox ◦ Conjur (Docker only) ◦ Many more ◦ Vault Secret Management Tools
41. • If you only take away two things from this talk, make sure they are the following:
42. • If you only take away two things from this talk, make sure they are the following: • Write your own crypto
43. • If you only take away two things from this talk, make sure they are the following: • Write your own crypto • Use it in a Secret Management tool
44. • If you only take away two things from this talk, make sure they are the following: • Write your own crypto • Use it in a Secret Management tool NO! NO!
45. • If you only take away two things from this talk, make sure they are the following: • Use a Secret Management tool • Don’t roll your own
46. Secret Management Tools • Discussion of SM tools true for Vault and its capabilities • Vault has a focus on providing security primitives ◦ Enables creation of complex workflows ◦ We’ll see how this provides necessary flexibility
47. Secret Management Tools • Central: ◦ Secure storage (avoid sprawl) ◦ Management ◦ Auditing (who has seen/accessed what, when) • Codified, secure access mechanisms • Secret rotation/revocation/expiration
48. SM + SI • Explicit focus on the secure introduction problem ◦ Core competency from ground up ◦ Necessary tools/capabilities/primitives • 100k containers can support existing SI paradigms ◦ But imagine… ▪ Managing 100k LDAP/AD users churning at high rate ▪ Generating/dropping 100k Kerberos keytabs ◦ ...and you’ve solved SI anyways
49. SMs <3 Schedulers • Anyone working with tasks (containers) at scale uses a scheduler ◦ Nomad ◦ Mesos ◦ Fleet ◦ Swarm ◦ Kubernetes
50. SMs <3 Schedulers • Schedulers are sources of truth...and provide hooks • SM tools and schedulers can be a magical combination
51. Scheduler Secret Management Tool Scheduler Agent Container Vault
52. Scheduler Secret Management Tool Scheduler Agent Container SI #1 (Traditional) Vault
53. Scheduler Vault Scheduler Agent Container SI #1 (Traditional): AWS-EC2 with Vault Secure Intro Client Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens
54. Scheduler Secret Management Tool Scheduler Agent Container SI #2 (New) Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Vault
55. Preconditions • A Vault token has: ◦ Unlimited or limited use-count ◦ Limited time-to-live (TTL), possibly renewable ◦ Set of authorization policies ▪ e.g. use first secret (auth token) to get application secrets ◦ Consistent ID in audit logs ◦ Token-scoped secure storage tied to token lifetime ▪ ‘cubbyhole’ • Claim: These primitives allow excellent secret coverage
56. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
57. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
58. Scheduler App: db_writer Security Policy: app_db_rw_pol Secret Management Tool Create Token: Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
59. Scheduler Secret Management Tool Create Token: Policy: app_db_rw_pol Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens App: db_writer Security Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Vault
60. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Image: db_writer TOKEN: (outer token) Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Vault
61. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: 1 TTL: 30s (non-renewable) Policy: read_cubby Private storage: Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Unwrap Vault
62. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Vault
63. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get DB Creds Vault
64. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens User: db_user Password: db_pass Expiration: 24h Container Vault
65. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get S3 Creds Vault
66. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens ACCESS_KEY: ... SECRET_KEY: ... Expiration: 24h Container Vault
67. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Container Uses: ∞ TTL: 1h (renewable) Policy: app_db_rw_pol Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Get TLS Cert Vault
68. Scheduler Secret Management Tool Scheduler Agent Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Uses: ∞ TTL: 8h (renewable) Policy: create_app_tokens Certificate: ... Private Key: ... Issuing CA: ... Container Vault
69. How’d we do? • Look at previous list: ◦ Don’t let them live forever (rotate/expire) ✓ Outer token expires (use limit) with only copy of inner token value ◦ Distribute them securely ✓ Inner token covered the entire way ◦ Limit exposure if disclosed ✓ Only specific policies granted, can only access specific secrets ◦ Have a break-glass procedure ✓ Can lock down access at SM tool ✓ Audit logs ensure we know area of exposure ◦ Detect unauthorized access
70. Access Detection • Can detect unauthorized access of “real” authentication token due to time and use limit • Application: ◦ Reads inner token: success ◦ Reads storage but no inner token found: log error and recover (e.g. fail job) ◦ Invalid token? Check current time vs. issue time + TTL: ▪ Expired? YELLOW FLAG: investigate, but probably just bad timing ▪ Not expired? RED FLAG: Unauthorized use
71. Response Wrapping • This paradigm is so useful that it’s available for nearly any Vault call and known as response wrapping • Mechanism not container- or authentication-specific ◦ CM tool drops wrapped secret on EC2 instance ◦ File with wrapped value injected into chroot ◦ Etc.
72. Job-Finish Revocation • Wrapped responses containing tokens also return the token’s accessor (reference) • Scheduling agent/system can correlate accessor with job ◦ Revoke token / child leases immediately on job finish
73. Integration • Nomad integration uses a variation of this scheme but basic concept is the same ◦ User-supplied token at job submission time checked for access ◦ Agent can unwrap secrets and mount into ramdisk ◦ Etc.
74. Integration • HashiStack tools follow the Unix model • We set out to tackle this problem in a generic way • Any scheduler can implement what Nomad has (or another variation on this model) ◦ Primitives to do so are there in Vault
75. Wrap-Up • Don’t ignore risk, minimize it ◦ Understand your circles and chains of trust • Use Vault, or some other SM tool • Orchestration tools are great facilitators of secure introduction
76. Thanks!