Handling GDPR with Apache Kafka: How to Comply Without Freaking Out? (Kafka Summit London 2019)

Do you wonder how to cope with the right to be forgotten? Do you wonder how to only process the events of individuals who have given their consent for processing their data? Do you wonder how to protect PII data of your users? Or do you wonder how to implement these across all your heterogeneous languages, clients and processing frameworks without having to re-implement all your streaming services? This talk is for you!

In this talk, we will answer these questions and show you
1) how transparent end-to-end encryption can be implemented on top of Apache Kafka;
2) how crypto-shredding can be used to forget individuals; and
3) how record based access control can be implemented on top of Apache Kafka.

Above all, we will show how this can be done without touching any applications by using an out-of-process architecture (à la service-mesh).

Transcript

1. Handling GDPR with Apache Kafka: How to comply without freaking out? David Jacot (@davidjacot) Kafka Summit London May 14, 2019
2. @davidjacot GDPR: What are the challenges? Encryption Right to be Forgotten Consent ! 2
3. @davidjacot Kafka: What are the challenges? 3 Encryption Right to be Forgotten Consent Kafka does not have an real encryption story Kafka only provides Topics authorization Kafka only knows how to expire or compact events ! ?
4. @davidjacot Fortunately, solutions exist! 4 Encryption Right to be Forgotten Consent Kafka does not have an real encryption story Kafka only provides Topics authorization Kafka only knows how to expire or compact events E2E Encryption Crypto-Shredding Record based ACLs ! ? ✓
5. @davidjacot End-to-End (Symmetric) Encryption 1 2 3 4 5 6 7 8 9 10 Producer Consumer Encrypt Produce Decrypt Consume 5
6. @davidjacot How do they exchange keys? Key Management Service (KMS) Master Key or Key Encryption Key (KEK) Producer Consumer 6
7. @davidjacot Envelope Encryption Key: clearkey Value: clearvalue Headers: Key: clearkey Value: DEK(clearvalue) Headers: Key: clearkey Value: DEK(clearvalue) Headers: - KEK(DEK) - Ref. to the KEK Generate a DEK locally Use a KEK to wrap the DEK DEK: Data Encryption Key KEK: Key Encryption Key 7
8. @davidjacot Envelope Decryption Key: clearkey Value: clearvalue Headers: Use the DEK to decrypt the Value Use the KEK to unwrap the DEK DEK: Data Encryption Key KEK: Key Encryption Key Key: clearkey Value: DEK(clearvalue) Headers: - KEK(DEK) - Ref. to the KEK 8
9. @davidjacot A Master Key per Cluster ... KMS Topic A Topic B Topic C 9 Cluster A
10. @davidjacot … or a Master Key per Topic ... KMS Topic A Topic B Topic C 10 Topic A Topic B Topic C
11. @davidjacot … or a Master Key per User KMS Topic A Topic B Topic C User 1 User 2 User 3 User 4 11
12. @davidjacot Need to forget a User? Crypto-shred him! 12 KMS Topic A Topic B Topic C User 1 User 2 User 3 User 4
13. @davidjacot Record (or User) based ACLs KMS Topic A Topic B Topic C 13 User 1 User 2 User 3 User 4
14. @davidjacot Record (or User) based ACLs KMS Topic A Topic B Topic C 14 User 1 User 2 User 3 User 4
15. @davidjacot Record (or User) based ACLs KMS Topic A Topic B Topic C 15 User 1 User 2 User 3 User 4
16. @davidjacot Checkpoint 16 Encryption Right to be Forgotten Consent Kafka does not have an real encryption story Kafka only provides Topics authorization Kafka only knows how to expire or compact events E2E Encryption Crypto-Shredding Record based ACLs ! ? ✓
17. @davidjacot How do I implement it in practice? 17 KMS Vendor
18. @davidjacot Does this guy really believe that we can integrate this in all our apps? 18
19. @davidjacot Can’t we solve this at the infrastructure level? 19
20. @davidjacot What about using sidecars? 20 KMS Vendor Control Plane Pod Pod Pod Server VM Pod
21. @davidjacot Let’s see how it works ... 21
22. @davidjacot client-marketing Setup on Kubernetes 22 Kafka client-eventsgen ksql-gen L7 Proxy client-analytics console -consumer L7 Proxy console -consumer L7 Proxy Control Plane Data Plane control-plane KMS Policies ACL Clear mTLS clickstream
23. @davidjacot click stream { "ip": "122.152.45.245", "userid": 38, "time": 1556469798309, "request": "GET / HTTP/1.1", "status": "302", "bytes": "4096", "agent": "Mozilla/5.0 ..." } 23
24. @davidjacot Demo 24
25. @davidjacot Kafka Transparent L7 Proxy for Apache Kafka 25 Pod / VM App Proxy Broker 1 Broker 2 Broker 3 Kafka Client Interceptors 1 2 3 1 2 3 TCP connections going to Kafka are redirected to the proxy Each connection is proxied to its real destination Requests & Responses are intercepted and possibly altered Clear mTLS
26. @davidjacot What is intercepted by the proxy? ApisRequest / ApisResponse ProduceRequest / ProduceResponse FetchRequest / FetchResponse 26
27. @davidjacot RecordBatch (v2): well thought! baseOffset: int64 batchLength: int32 partitionLeaderEpoch: int32 magic: int8 (current magic value is 2) crc: int32 attributes: int16 lastOffsetDelta: int32 firstTimestamp: int64 maxTimestamp: int64 producerId: int64 producerEpoch: int16 baseSequence: int32 records: [Record] 27 https://kafka.apache.org/documentation/#recordbatch
28. @davidjacot + Privacy & Compliance Transparent Integration Solved once for all 29 - Encryption comes with a cost Latency is increased (KMS) Message Format >v2
29. @davidjacot What’s next? 30 External KMS Field level encryption Field level ACL Avro support Schema validation Lineage ...
30. @davidjacot Summary 31 Encryption Right to be Forgotten Consent Kafka does not have an real encryption story Kafka only provides Topics authorization Kafka only knows how to expire or compact events E2E Encryption Crypto-Shredding Record based ACLs ! ? ✓
31. Thank you! David Jacot (@davidjacot) Kafka Summit London May 14, 2019 32