Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Client-Side Field-Level Encryption for Apache Kafka Connect @ VoxxedDays Luxembourg 2022

Client-Side Field-Level Encryption for Apache Kafka Connect @ VoxxedDays Luxembourg 2022

Abstract:
Apache Kafka offers several security features ranging from authentication and authorisation mechanisms to over-the-wire encryption. This notwithstanding, data encryption performed at the client-side, which leads to explicit data-at-rest protection in topics at the broker's side, can still be considered a blind spot.

After highlighting the main benefits for data-at-rest protection, this session discusses in-depth how to selectively encrypt and decrypt sensitive payload fields in the context of Apache Kafka Connect pipelines. In particular, Kryptonite is introduced which is an ecosystem community project written and open-sourced by the speaker in 2021.

During this talk, you will learn how to benefit from a configurable single message transformation that lets you perform encryption and decryption operations in Kafka Connect worker nodes without any custom code. Client-side cryptography makes your Kafka-based data integration scenarios more secure by safeguarding the most sensitive and precious data against any form of uncontrolled or illegal access once it hits the Apache Kafka brokers.

Recording:

https://www.youtube.com/watch?v=4FmbWir_abM

Code:

- Kryptonite for Kafka Project: https://github.com/hpgrahsl/kryptonite-for-kafka
- Demo Scenarios: https://github.com/hpgrahsl/vdlux22-k4k-demo

744f1c2c6cbea2ff5104b0ac512936bd?s=128

Hans-Peter Grahsl

June 21, 2022
Tweet

More Decks by Hans-Peter Grahsl

Other Decks in Programming

Transcript

  1. Client-Side Field-Level Encryption for Apache Kafka Connect @hpgrahsl | #VoxxedDays

    Luxembourg | June 21-22, 2022
  2. Why should we care? @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 2
  3. 61% of breaches involved credential data1 1 Verzion DBIR 2021

    - https://www.verizon.com/dbir @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 3
  4. 85% of breaches involved the human element1 1 Verzion DBIR

    2021 - https://www.verizon.com/dbir @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 4
  5. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 5

  6. compromised external cloud assets more common than on-premises assets1 1

    Verzion DBIR 2021 - https://www.verizon.com/dbir @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 6
  7. Don't forget about the price tag of data breaches. @hpgrahsl

    | #VoxxedDays Luxembourg | June 21-22, 2022 7
  8. Don't forget about the price tag of data breaches. @hpgrahsl

    | #VoxxedDays Luxembourg | June 21-22, 2022 8
  9. $4.24M average cost of data breach2 2 IBM Cost of

    Data Breach Report - https://www.ibm.com/security/data-breach @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 9
  10. $180 per record cost of customer pii2 2 IBM Cost

    of Data Breach Report - https://www.ibm.com/security/data-breach @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 10
  11. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 11

  12. It's me ... ! • technical trainer at NETCONOMY •

    independent engineer & consultant • Confluent Community Catalyst • MongoDB Champion @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 12
  13. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 13

  14. ! But Kafka related? Yes! 3 3 https://spectralops.io/blog/misconfigured-kafdrop-puts-companies-apache-kafka-completely-exposed/ @hpgrahsl |

    #VoxxedDays Luxembourg | June 21-22, 2022 14
  15. ! They found it "all" ... 3 3 https://spectralops.io/blog/misconfigured-kafdrop-puts-companies-apache-kafka-completely-exposed/ @hpgrahsl

    | #VoxxedDays Luxembourg | June 21-22, 2022 15
  16. unhappy @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 16

  17. Core Kafka Security Mechanisms @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 17
  18. Table Stakes ? @hpgrahsl | #VoxxedDays Luxembourg | June 21-22,

    2022 18
  19. over-the-wire encryption @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022

    19
  20. authentication @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 20

  21. authorization @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 21

  22. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 22

  23. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 23

  24. disturbing @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 24

  25. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 25

  26. Core Security Necessary ! @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 26
  27. Core Security Sufficient ? @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 27
  28. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 28

  29. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 29

  30. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 30

  31. ? in use by brokers @hpgrahsl | #VoxxedDays Luxembourg |

    June 21-22, 2022 31
  32. brokers see everything ... and so does any legitimate Kafka

    client @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 32
  33. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 33

  34. human promise is NOT technical promise @hpgrahsl | #VoxxedDays Luxembourg

    | June 21-22, 2022 34
  35. end-to-end encryption ? ? ? @hpgrahsl | #VoxxedDays Luxembourg |

    June 21-22, 2022 35
  36. Community Project Kryptonite for Kafka @hpgrahsl | #VoxxedDays Luxembourg |

    June 21-22, 2022 36
  37. client-side field level cryptography @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 37
  38. Client-Side Cryptography @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022

    38
  39. Client-Side Cryptography @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022

    39
  40. Field Level Encryption @hpgrahsl | #VoxxedDays Luxembourg | June 21-22,

    2022 40
  41. Field Level Encryption @hpgrahsl | #VoxxedDays Luxembourg | June 21-22,

    2022 41
  42. Field Level Decryption @hpgrahsl | #VoxxedDays Luxembourg | June 21-22,

    2022 42
  43. Field Level Decryption @hpgrahsl | #VoxxedDays Luxembourg | June 21-22,

    2022 43
  44. Kafka Connect Single Message Transform @hpgrahsl | #VoxxedDays Luxembourg |

    June 21-22, 2022 44
  45. CSFLC with Source Connectors @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 45
  46. CSFLC with Source Connectors @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 46
  47. CSFLC with Source Connectors @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 47
  48. CSFLC with Source Connectors @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 48
  49. CSFLC with Sink Connectors @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 49
  50. CSFLC with Sink Connectors @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 50
  51. CSFLC with Sink Connectors @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 51
  52. CSFLC with Sink Connectors @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 52
  53. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 53

  54. Demo Scenario 1 @hpgrahsl | #VoxxedDays Luxembourg | June 21-22,

    2022 54
  55. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 55

  56. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 56

  57. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 57

  58. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 58

  59. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 59

  60. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 60

  61. Demo Scenario 2 @hpgrahsl | #VoxxedDays Luxembourg | June 21-22,

    2022 61
  62. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 62

  63. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 63

  64. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 64

  65. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 65

  66. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 66

  67. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 67

  68. Behind the Curtain ? @hpgrahsl | #VoxxedDays Luxembourg | June

    21-22, 2022 68
  69. Cryptography • Tink by Google • AEAD based on AES

    GCM • DAEAD based on AES SIV • key rotation support @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 69
  70. Keyset Management • within SMT config (not recommended) • externalized

    to separate file (okayish) • remote / cloud KMS (recommended) • currently Azure Key Vault @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 70
  71. ! Little Ideas ! • wildcard / regex matching for

    field names • dynamic keyset selection based on payload • additional KMS providers (GCP, AWS, ...) @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 71
  72. ! Bigger Ideas ! • add further cryptography options (e.g.

    FPE) • language / runtime agnostic data serialization • extend scope beyond Kafka Connect @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 72
  73. data should continue to be a valuable asset not become

    a costly liability @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 73
  74. twitter @hpgrahsl @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022

    74
  75. Go check it out ! • Project Code https://bit.ly/vdlux22-k4k •

    Demo Scenarios https://bit.ly/vdlux22-demo @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022 75
  76. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022

  77. Photo Credits in order of appearance (c) Chunli Ju -

    https://unsplash.com/photos/8fs1X0JFgFE (c) Wolf Zimmermann - https://unsplash.com/photos/6sf5rf8QYFE (c) Jason Leung - https://unsplash.com/photos/SAYzxuS1O3M (c) Dev Asangbam - https://unsplash.com/photos/sh9vkVbVgo (c) Keenan Constance - https://unsplash.com/photos/VTLcvV6UVaI (c) Steve Johnson - https://unsplash.com/photos/hokONTrHIAQ (c) Pete Linforth - https://pixabay.com/illustrations/biometrics-access-identification-4503187/ (c) Miguel Á. Padriñán - https://www.pexels.com/photo/close-up-shot-of-keys-on-a-red-surface-2882687/ (c) Camila Quintero Franco - https://unsplash.com/photos/mC852jACK1g (c) Gerd Altmann - https://pixabay.com/illustrations/board-excuse-me-excuse-discharge-1848736/ (c) Vijaya narasimha - https://pixabay.com/photos/crevasse-sand-stone-hills-rock-399957/ (c) Gerd Altmann - https://pixabay.com/photos/trust-man-hood-map-prompt-4321822/ (c) Matheo JBT - https://unsplash.com/photos/HLhvZ9HRAwo (c) Rob Laughter - https://unsplash.com/photos/WW1jsInXgwM (c) Markus Spiske - https://unsplash.com/photos/iar-afB0QQw (c) Nerene Grobler - https://unsplash.com/photos/sLxcfdsqLQ (c) Wilhelm Gunkel - https://unsplash.com/photos/L04Kczg_Jvs (c) Matt Walsh - https://unsplash.com/photos/tVkdGtEe2C4