End to End Message Authenticity in Cloud Native Systems Micah Hausler, AWS & Lucas Käldström, Upbound @micahhausler.com & @luxas.dev 

Who are we? Staff Engineer @ Upbound Formerly at Weaveworks Former SIG Cluster Lifecycle co-lead CNCF Ambassador 2017-2024 Principal Engineer @ AWS Working on AWS EKS Kubernetes Security Response Committee SIG-Auth Chair @micahhausler.com & @luxas.dev 

Outline 1. Why does end to end authenticity matter? a. Attacks against common setups with intermediate proxies b. OpenID Connect (OIDC) token stealing 2. How is it possible to mitigate? a. HTTP Message Signatures (RFC 9421) 3. How to perform key distribution? a. OAuth 2.0 Demonstrating Proof of Possession (RFC 9449) b. OpenPubKey key distribution protocol c. Key database (e.g. out-of-band or GitHub) 4. Demo - https://github.com/micahhausler/httpsig-rejekts-demo 5. What’s next? a. WIMSE Service to Service Authentication & SPIFFE Demo Link @micahhausler.com & @luxas.dev 

Why does end to end authenticity matter? @micahhausler.com & @luxas.dev 

(m)TLS in a microservices world TLS Server: www.acmeco.com HTTPS POST /users Authorization: Bearer {“name”: “lucas”} A @micahhausler.com & @luxas.dev 

(m)TLS in a microservices world TLS mTLS Server: www.acmeco.com HTTPS POST /users Authorization: Bearer {“name”: “lucas”} A Server: user-service Client: gateway @micahhausler.com & @luxas.dev 

(m)TLS in a microservices world TLS mTLS Server: www.acmeco.com ✅ Authenticity: server cert & id token valid ✅ Integrity: data is signed ✅ Confidentiality: data is encrypted HTTPS POST /users Authorization: Bearer {“name”: “lucas”} A Server: user-service Client: gateway @micahhausler.com & @luxas.dev 

(m)TLS in a microservices world TLS mTLS ✅ Authenticity: server cert & id token valid ✅ Integrity: data is signed ✅ Confidentiality: data is encrypted ✅ Authenticity: server & client cert valid ✅ Integrity: data is signed ✅ Confidentiality: data is encrypted HTTPS POST /users Authorization: Bearer {“name”: “lucas”} A Server: user-service Client: gateway @micahhausler.com & @luxas.dev Server: www.acmeco.com 

TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) HTTPS POST /users Authorization: Bearer {“name”: “lucas”} Client: gateway A Hijacked (or misconfigured) gateway: Token replay @micahhausler.com & @luxas.dev Server: www.acmeco.com 

TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) HTTPS POST /users Authorization: Bearer {“name”: “lucas”} Client: gateway Seen tokens: id_token_A id_token_B id_token_C A Hijacked (or misconfigured) gateway: Token replay @micahhausler.com & @luxas.dev Server: www.acmeco.com 

Hijacked (or misconfigured) gateway: Token replay TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) HTTPS POST /users Authorization: Bearer {“name”: “lucas”} Client: gateway Seen tokens: id_token_A id_token_B id_token_C HTTPS POST /users Authorization: Bearer {“name”: “lucas”} A @micahhausler.com & @luxas.dev Server: www.acmeco.com 

Hijacked (or misconfigured) gateway: Payload change TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) ❌ Integrity: The gateway could change the payload without user-service noticing HTTPS POST /users Authorization: Bearer {“name”: “lucas”} Client: gateway A HTTPS POST /users Authorization: Bearer {“name”: “micah”} @micahhausler.com & @luxas.dev Server: www.acmeco.com 

TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) ❌ Integrity: The gateway could change the payload without user-service noticing ⇒ Every component in the request chain have to be fully trusted HTTPS POST /users Authorization: Bearer {“name”: “lucas”} Client: gateway A HTTPS POST /users Authorization: Bearer {“name”: “micah”} Hijacked (or misconfigured) gateway: Payload change @micahhausler.com & @luxas.dev Server: www.acmeco.com 

Hijacked (or misconfigured) gateway: Summary TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) ❌ Integrity: The gateway could change the payload without user-service noticing ✅ Confidentiality: data is encrypted HTTPS POST /users Authorization: Bearer {“name”: “lucas”} Client: gateway A @micahhausler.com & @luxas.dev Server: www.acmeco.com 

OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify. 

OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps 1. Web login Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify. 

OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps 1. Web login 2. Signed ID token Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify. 

OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps 1. Web login 2. Signed ID token Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification 3. Request with ID token Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify. 

OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token @micahhausler.com & @luxas.dev 

OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token ✅ Easy to propagate extra information about the user, e.g. group membership @micahhausler.com & @luxas.dev 

OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token ✅ Easy to propagate extra information about the user, e.g. group membership ❌ Symmetric credential => can be forwarded if leaked @micahhausler.com & @luxas.dev 

OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token ✅ Easy to propagate extra information about the user, e.g. group membership ❌ Symmetric credential => can be forwarded if leaked ❌ Not bound to request contents => data can be modified by intermediaries @micahhausler.com & @luxas.dev 

How can you mitigate these issues? @micahhausler.com & @luxas.dev 

HTTP Message Signatures: IETF RFC 9421 Idea: A client can sign the security-sensitive content and headers with either a symmetric or asymmetric key, and the server can verify that the observed content actually matches what the client signed (and vice versa). TLS HTTPS POST /users Authorization: Bearer Signature-Input: “authorization” @body Signature: dMT/A/76ehrdBTD/2Xx8… {“name”: “lucas”} Server: www.acmeco.com A sign choose what data to sign (what is important?) 

HTTP Message Signatures: IETF RFC 9421 Idea: A client can sign the security-sensitive content and headers with either a symmetric or asymmetric key, and the server can verify that the observed content actually matches what the client signed (and vice versa). TLS mTLS Server: user-service HTTPS POST /users Authorization: Bearer Signature-Input: “authorization” @body Signature: dMT/A/76ehrdBTD/2Xx8… {“name”: “lucas”} Server: www.acmeco.com Client: gateway A HTTPS POST /users Authorization: Bearer Signature-Input: “authorization” @body Signature: dMT/A/76ehrdBTD/2Xx8… {“name”: “lucas”} sign verify Content or header tampering => ❌ Unaltered => ✅ 

How to distribute the keys? @micahhausler.com & @luxas.dev 

AWS sigv4: symmetric secret key + HMAC HTTP Message Signatures RFC ≈ generalization of the AWS sigv4 protocol Client gets a access key ID + secret somehow (out of band) Both access key ID + secret are random strings generated by AWS IAM AWS client library uses the access key secret to sign critical HTTP request info AWS ARS verifies the request is correctly signed => user authenticated @micahhausler.com & @luxas.dev 

High-level: transitive trust 1. application trusts identity provider (configuration) 2. identity provider attests the client (login) 3. identity provider binds a client identity to its client key => application can trust client identity and client key through identity provider @micahhausler.com & @luxas.dev 

Binding a key to an OIDC flow 1. Web login page 2. Redirect with code Identity Provider issuer: https://identity-provider.com 0. Download public keys from https://identity-provider.com/.well-known/jwks.json for verification This is the “normal” OIDC flow 3. POST code private sign public verify Authenticate ID token using 5. Request with ID token 4. Get signed ID token 

Demonstrating Proof of Possession (DPoP): RFC 9449 1. Web login page 2. Redirect with code Identity Provider issuer: https://identity-provider.com 0. Download public keys from https://identity-provider.com /.well-known/jwks.json for verification 5. Request with ID token with & Signature-Input: @body “authorization” Signature: dMT/A/76ehrdBTD/ 3. POST code and PoP 4. Get signed ID token, bound to private sign public verify Authenticate ID token using Can verify signature using private sign ✅ Ephemeral user key ❌ Changes needed in OIDC provider { “alg”: “ES256” }.{ "sub":"[email protected]", "iss":"https://server.example.com", "cnf": { "jkt":"" } }. 

OpenPubKey: Make nonce meaningful 1. Web login page, set nonce 2. Redirect with code Identity Provider issuer: https://identity-provider.com 0. Download public keys from https://identity-provider.com/ .well-known/jwks.json for verification 5. Request with ID token with & Signature-Input: @body “authorization” Signature: dMT/A/76ehrdBTD/ 3. POST code 4. Get signed ID token, bound to thru nonce private sign public verify Authenticate ID token using Can verify signature using private sign ✅ Ephemeral user key ✅ Works with any OIDC provider { “alg”: “ES256” }.{ "sub":"[email protected]", "iss":"https://server.example.com", "nonce”: SHA-3({ “alg”: “ES256”, “upk”: , “rz”: crypto.random() }) }. 

GitHub is actually a public key database (!) In order to clone private GitHub repositories, users can: - generate a private key locally - log into GitHub and register their public key - … and GitHub will accept usage of the private key, - AND expose public keys at https://github.com/.keys (!) @micahhausler.com & @luxas.dev 

Demo: Signing Requests luxas 0. Log into Github and register public key 

Demo: Signing Requests 1. TLS HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… Server: rejekts.dev.micahhausler.com luxas private sign 0. Log into Github and register public key 

Demo: Signing Requests 1. TLS 2. mTLS Server: demo-app HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… Server: rejekts.dev.micahhausler.com Client: gateway luxas private sign 0. Log into Github and register public key public verify HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… 

Demo: Signing Requests 1. TLS 2. mTLS Server: demo-app HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… Server: rejekts.dev.micahhausler.com Client: gateway luxas Content or header tampering => ❌ Unaltered => ✅ private sign 0. Log into Github and register public key 3. Download public key public verify HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… 

Live demo, you should participate and do this! 1. Generate a new SSH key using e.g. ssh-keygen -t ecdsa -f test-key 2. Log into your GitHub, and register the public key (e.g. test-key.pub) 3. go install github.com/micahhausler/httpsig-rejekts-demo@latest 4. Run httpsig-rejekts-demo -key test-key -username The program signs the HTTP content with your private key. 5. Execute the curl command the program outputs. Server downloads your public key from Github, and authenticates you! 6. If you change (like an intermediary) e.g. the signed content-type header, the server will notice the mismatch and reject the request 

What’s next? @micahhausler.com & @luxas.dev 

Workload Identity with SPIFFE “Clients” can also be workloads, not just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. @micahhausler.com & @luxas.dev 

Workload Identity with SPIFFE “Clients” can also be workloads, not just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. SPIFFE is meant to solve the “bottom turtle” problem of trust: 1. When a workload starts up, it asks “who am I?” through a gRPC API @micahhausler.com & @luxas.dev 

Workload Identity with SPIFFE “Clients” can also be workloads, not just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. SPIFFE is meant to solve the “bottom turtle” problem of trust: 1. When a workload starts up, it asks “who am I?” through a gRPC API 2. The gRPC API server, the agent, attests the workload @micahhausler.com & @luxas.dev 

Workload Identity with SPIFFE “Clients” can also be workloads, not just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. SPIFFE is meant to solve the “bottom turtle” problem of trust: 1. When a workload starts up, it asks “who am I?” through a gRPC API 2. The gRPC API server, the agent, attests the workload 3. The agent gives the workload the OIDC tokens/x509 certs it’s bound to @micahhausler.com & @luxas.dev 

WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. @micahhausler.com & @luxas.dev 

WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs @micahhausler.com & @luxas.dev 

WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs 2. The issuer (SPIFFE) binds the workload identity to the public key @micahhausler.com & @luxas.dev 

WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs 2. The issuer (SPIFFE) binds the workload identity to the public key 3. By the authenticator trusting SPIFFE, it can also trust the workload’s key @micahhausler.com & @luxas.dev 

WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs 2. The issuer (SPIFFE) binds the workload identity to the public key 3. By the authenticator trusting SPIFFE, it can also trust the workload’s key Thanks to Evan and Pieter from SPIRL for answering questions about WIMSE! Get involved today! https://github.com/spiffe/spiffe/pull/327 @micahhausler.com & @luxas.dev 

Thanks! Bluesky: @luxas.dev LinkedIn: luxas CNCF Slack(s): luxas Bluesky: @micahhausler.com LinkedIn: micahhausler CNCF Slack(s): micahhausler Credits: Icons by Flaticon