Client-Side Field-Level Encryption for Apache Kafka Connect @ VoxxedDays Luxembourg 2022

Abstract:
Apache Kafka offers several security features ranging from authentication and authorisation mechanisms to over-the-wire encryption. This notwithstanding, data encryption performed at the client-side, which leads to explicit data-at-rest protection in topics at the broker's side, can still be considered a blind spot.

After highlighting the main benefits for data-at-rest protection, this session discusses in-depth how to selectively encrypt and decrypt sensitive payload fields in the context of Apache Kafka Connect pipelines. In particular, Kryptonite is introduced which is an ecosystem community project written and open-sourced by the speaker in 2021.

During this talk, you will learn how to benefit from a configurable single message transformation that lets you perform encryption and decryption operations in Kafka Connect worker nodes without any custom code. Client-side cryptography makes your Kafka-based data integration scenarios more secure by safeguarding the most sensitive and precious data against any form of uncontrolled or illegal access once it hits the Apache Kafka brokers.

Recording:

https://www.youtube.com/watch?v=4FmbWir_abM

Code:

- Kryptonite for Kafka Project: https://github.com/hpgrahsl/kryptonite-for-kafka
- Demo Scenarios: https://github.com/hpgrahsl/vdlux22-k4k-demo

Transcript

1. Client-Side Field-Level Encryption for Apache Kafka Connect @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
2. Why should we care? @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 2
3. 61% of breaches involved credential data1 1 Verzion DBIR 2021 - https://www.verizon.com/dbir @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 3
4. 85% of breaches involved the human element1 1 Verzion DBIR 2021 - https://www.verizon.com/dbir @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 4
5. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 5
6. compromised external cloud assets more common than on-premises assets1 1 Verzion DBIR 2021 - https://www.verizon.com/dbir @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 6
7. Don't forget about the price tag of data breaches. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 7
8. Don't forget about the price tag of data breaches. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 8
9. $4.24M average cost of data breach2 2 IBM Cost of Data Breach Report - https://www.ibm.com/security/data-breach @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 9
10. $180 per record cost of customer pii2 2 IBM Cost of Data Breach Report - https://www.ibm.com/security/data-breach @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 10
11. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 11
12. It's me ... ! • technical trainer at NETCONOMY • independent engineer & consultant • Confluent Community Catalyst • MongoDB Champion @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 12
13. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 13
14. ! But Kafka related? Yes! 3 3 https://spectralops.io/blog/misconfigured-kafdrop-puts-companies-apache-kafka-completely-exposed/ @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 14
15. ! They found it "all" ... 3 3 https://spectralops.io/blog/misconfigured-kafdrop-puts-companies-apache-kafka-completely-exposed/ @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 15
16. unhappy @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 16
17. Core Kafka Security Mechanisms @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 17
18. Table Stakes ? @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 18
19. over-the-wire encryption @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 19
20. authentication @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 20
21. authorization @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 21
22. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 22
23. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 23
24. disturbing @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 24
25. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 25
26. Core Security Necessary ! @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 26
27. Core Security Sufficient ? @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 27
28. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 28
29. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 29
30. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 30
31. ? in use by brokers @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 31
32. brokers see everything ... and so does any legitimate Kafka client @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 32
33. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 33
34. human promise is NOT technical promise @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 34
35. end-to-end encryption ? ? ? @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 35
36. Community Project Kryptonite for Kafka @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 36
37. client-side field level cryptography @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 37
38. Client-Side Cryptography @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 38
39. Client-Side Cryptography @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 39
40. Field Level Encryption @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 40
41. Field Level Encryption @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 41
42. Field Level Decryption @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 42
43. Field Level Decryption @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 43
44. Kafka Connect Single Message Transform @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 44
45. CSFLC with Source Connectors @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 45
46. CSFLC with Source Connectors @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 46
47. CSFLC with Source Connectors @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 47
48. CSFLC with Source Connectors @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 48
49. CSFLC with Sink Connectors @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 49
50. CSFLC with Sink Connectors @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 50
51. CSFLC with Sink Connectors @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 51
52. CSFLC with Sink Connectors @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 52
53. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 53
54. Demo Scenario 1 @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 54
55. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 55
56. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 56
57. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 57
58. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 58
59. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 59
60. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 60
61. Demo Scenario 2 @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 61
62. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 62
63. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 63
64. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 64
65. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 65
66. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 66
67. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 67
68. Behind the Curtain ? @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 68
69. Cryptography • Tink by Google • AEAD based on AES GCM • DAEAD based on AES SIV • key rotation support @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 69
70. Keyset Management • within SMT config (not recommended) • externalized to separate file (okayish) • remote / cloud KMS (recommended) • currently Azure Key Vault @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 70
71. ! Little Ideas ! • wildcard / regex matching for field names • dynamic keyset selection based on payload • additional KMS providers (GCP, AWS, ...) @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 71
72. ! Bigger Ideas ! • add further cryptography options (e.g. FPE) • language / runtime agnostic data serialization • extend scope beyond Kafka Connect @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 72
73. data should continue to be a valuable asset not become a costly liability @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 73
74. twitter @hpgrahsl @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 74
75. Go check it out ! • Project Code https://bit.ly/vdlux22-k4k • Demo Scenarios https://bit.ly/vdlux22-demo @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 75
76. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
77. Photo Credits in order of appearance (c) Chunli Ju - https://unsplash.com/photos/8fs1X0JFgFE (c) Wolf Zimmermann - https://unsplash.com/photos/6sf5rf8QYFE (c) Jason Leung - https://unsplash.com/photos/SAYzxuS1O3M (c) Dev Asangbam - https://unsplash.com/photos/sh9vkVbVgo (c) Keenan Constance - https://unsplash.com/photos/VTLcvV6UVaI (c) Steve Johnson - https://unsplash.com/photos/hokONTrHIAQ (c) Pete Linforth - https://pixabay.com/illustrations/biometrics-access-identification-4503187/ (c) Miguel Á. Padriñán - https://www.pexels.com/photo/close-up-shot-of-keys-on-a-red-surface-2882687/ (c) Camila Quintero Franco - https://unsplash.com/photos/mC852jACK1g (c) Gerd Altmann - https://pixabay.com/illustrations/board-excuse-me-excuse-discharge-1848736/ (c) Vijaya narasimha - https://pixabay.com/photos/crevasse-sand-stone-hills-rock-399957/ (c) Gerd Altmann - https://pixabay.com/photos/trust-man-hood-map-prompt-4321822/ (c) Matheo JBT - https://unsplash.com/photos/HLhvZ9HRAwo (c) Rob Laughter - https://unsplash.com/photos/WW1jsInXgwM (c) Markus Spiske - https://unsplash.com/photos/iar-afB0QQw (c) Nerene Grobler - https://unsplash.com/photos/sLxcfdsqLQ (c) Wilhelm Gunkel - https://unsplash.com/photos/L04Kczg_Jvs (c) Matt Walsh - https://unsplash.com/photos/tVkdGtEe2C4