Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Client-Side Field-Level Encryption for Apache Kafka Connect @ VoxxedDays Luxembourg 2022

Client-Side Field-Level Encryption for Apache Kafka Connect @ VoxxedDays Luxembourg 2022

Abstract:
Apache Kafka offers several security features ranging from authentication and authorisation mechanisms to over-the-wire encryption. This notwithstanding, data encryption performed at the client-side, which leads to explicit data-at-rest protection in topics at the broker's side, can still be considered a blind spot.

After highlighting the main benefits for data-at-rest protection, this session discusses in-depth how to selectively encrypt and decrypt sensitive payload fields in the context of Apache Kafka Connect pipelines. In particular, Kryptonite is introduced which is an ecosystem community project written and open-sourced by the speaker in 2021.

During this talk, you will learn how to benefit from a configurable single message transformation that lets you perform encryption and decryption operations in Kafka Connect worker nodes without any custom code. Client-side cryptography makes your Kafka-based data integration scenarios more secure by safeguarding the most sensitive and precious data against any form of uncontrolled or illegal access once it hits the Apache Kafka brokers.

Recording:

https://www.youtube.com/watch?v=4FmbWir_abM

Code:

- Kryptonite for Kafka Project: https://github.com/hpgrahsl/kryptonite-for-kafka
- Demo Scenarios: https://github.com/hpgrahsl/vdlux22-k4k-demo

Hans-Peter Grahsl

June 21, 2022
Tweet

More Decks by Hans-Peter Grahsl

Other Decks in Programming

Transcript

  1. Client-Side
    Field-Level Encryption
    for Apache Kafka
    Connect
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022

    View Slide

  2. Why should we care?
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    2

    View Slide

  3. 61%
    of breaches involved
    credential data1
    1 Verzion DBIR 2021 - https://www.verizon.com/dbir
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    3

    View Slide

  4. 85%
    of breaches involved
    the human element1
    1 Verzion DBIR 2021 - https://www.verizon.com/dbir
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    4

    View Slide

  5. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    5

    View Slide

  6. compromised external
    cloud assets
    more common than
    on-premises assets1
    1 Verzion DBIR 2021 - https://www.verizon.com/dbir
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    6

    View Slide

  7. Don't
    forget about the price tag
    of data breaches.
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    7

    View Slide

  8. Don't
    forget about the price tag
    of data breaches.
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    8

    View Slide

  9. $4.24M
    average cost of data
    breach2
    2 IBM Cost of Data Breach Report - https://www.ibm.com/security/data-breach
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    9

    View Slide

  10. $180
    per record cost of
    customer pii2
    2 IBM Cost of Data Breach Report - https://www.ibm.com/security/data-breach
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    10

    View Slide

  11. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    11

    View Slide

  12. It's me ...
    !
    • technical trainer at NETCONOMY
    • independent engineer & consultant
    • Confluent Community Catalyst
    • MongoDB Champion
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    12

    View Slide

  13. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    13

    View Slide

  14. !
    But Kafka related? Yes!
    3
    3 https://spectralops.io/blog/misconfigured-kafdrop-puts-companies-apache-kafka-completely-exposed/
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    14

    View Slide

  15. !
    They found it "all" ...
    3
    3 https://spectralops.io/blog/misconfigured-kafdrop-puts-companies-apache-kafka-completely-exposed/
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    15

    View Slide

  16. unhappy
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    16

    View Slide

  17. Core Kafka
    Security Mechanisms
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    17

    View Slide

  18. Table Stakes ?
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    18

    View Slide

  19. over-the-wire encryption
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    19

    View Slide

  20. authentication
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    20

    View Slide

  21. authorization
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    21

    View Slide

  22. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    22

    View Slide

  23. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    23

    View Slide

  24. disturbing
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    24

    View Slide

  25. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    25

    View Slide

  26. Core Security
    Necessary !
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    26

    View Slide

  27. Core Security
    Sufficient ?
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    27

    View Slide

  28. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    28

    View Slide

  29. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    29

    View Slide

  30. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    30

    View Slide

  31. ?
    in use by brokers
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    31

    View Slide

  32. brokers
    see everything ...
    and so does
    any legitimate
    Kafka client
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    32

    View Slide

  33. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    33

    View Slide

  34. human promise
    is NOT
    technical promise
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    34

    View Slide

  35. end-to-end
    encryption
    ? ? ?
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    35

    View Slide

  36. Community Project
    Kryptonite for Kafka
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    36

    View Slide

  37. client-side
    field level
    cryptography
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    37

    View Slide

  38. Client-Side Cryptography
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    38

    View Slide

  39. Client-Side Cryptography
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    39

    View Slide

  40. Field Level Encryption
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    40

    View Slide

  41. Field Level Encryption
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    41

    View Slide

  42. Field Level Decryption
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    42

    View Slide

  43. Field Level Decryption
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    43

    View Slide

  44. Kafka Connect
    Single Message
    Transform
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    44

    View Slide

  45. CSFLC with Source Connectors
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    45

    View Slide

  46. CSFLC with Source Connectors
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    46

    View Slide

  47. CSFLC with Source Connectors
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    47

    View Slide

  48. CSFLC with Source Connectors
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    48

    View Slide

  49. CSFLC with Sink Connectors
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    49

    View Slide

  50. CSFLC with Sink Connectors
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    50

    View Slide

  51. CSFLC with Sink Connectors
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    51

    View Slide

  52. CSFLC with Sink Connectors
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    52

    View Slide

  53. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    53

    View Slide

  54. Demo Scenario 1
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    54

    View Slide

  55. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    55

    View Slide

  56. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    56

    View Slide

  57. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    57

    View Slide

  58. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    58

    View Slide

  59. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    59

    View Slide

  60. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    60

    View Slide

  61. Demo Scenario 2
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    61

    View Slide

  62. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    62

    View Slide

  63. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    63

    View Slide

  64. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    64

    View Slide

  65. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    65

    View Slide

  66. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    66

    View Slide

  67. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    67

    View Slide

  68. Behind the Curtain ?
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    68

    View Slide

  69. Cryptography
    • Tink by Google
    • AEAD based on AES GCM
    • DAEAD based on AES SIV
    • key rotation support
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    69

    View Slide

  70. Keyset
    Management
    • within SMT config (not recommended)
    • externalized to separate file (okayish)
    • remote / cloud KMS (recommended)
    • currently Azure Key Vault
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    70

    View Slide

  71. !
    Little Ideas
    !
    • wildcard / regex matching for field names
    • dynamic keyset selection based on payload
    • additional KMS providers (GCP, AWS, ...)
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    71

    View Slide

  72. !
    Bigger Ideas
    !
    • add further cryptography options (e.g. FPE)
    • language / runtime agnostic data serialization
    • extend scope beyond Kafka Connect
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    72

    View Slide

  73. data should continue
    to be a valuable
    asset not become
    a costly liability
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    73

    View Slide

  74. twitter
    @hpgrahsl
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    74

    View Slide

  75. Go check it out !
    • Project Code
    https://bit.ly/vdlux22-k4k
    • Demo Scenarios
    https://bit.ly/vdlux22-demo
    @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022
    75

    View Slide

  76. @hpgrahsl | #VoxxedDays Luxembourg | June 21-22, 2022

    View Slide

  77. Photo Credits
    in order of appearance
    (c) Chunli Ju - https://unsplash.com/photos/8fs1X0JFgFE
    (c) Wolf Zimmermann - https://unsplash.com/photos/6sf5rf8QYFE
    (c) Jason Leung - https://unsplash.com/photos/SAYzxuS1O3M
    (c) Dev Asangbam - https://unsplash.com/photos/sh9vkVbVgo
    (c) Keenan Constance - https://unsplash.com/photos/VTLcvV6UVaI
    (c) Steve Johnson - https://unsplash.com/photos/hokONTrHIAQ
    (c) Pete Linforth - https://pixabay.com/illustrations/biometrics-access-identification-4503187/
    (c) Miguel Á. Padriñán - https://www.pexels.com/photo/close-up-shot-of-keys-on-a-red-surface-2882687/
    (c) Camila Quintero Franco - https://unsplash.com/photos/mC852jACK1g
    (c) Gerd Altmann - https://pixabay.com/illustrations/board-excuse-me-excuse-discharge-1848736/
    (c) Vijaya narasimha - https://pixabay.com/photos/crevasse-sand-stone-hills-rock-399957/
    (c) Gerd Altmann - https://pixabay.com/photos/trust-man-hood-map-prompt-4321822/
    (c) Matheo JBT - https://unsplash.com/photos/HLhvZ9HRAwo
    (c) Rob Laughter - https://unsplash.com/photos/WW1jsInXgwM
    (c) Markus Spiske - https://unsplash.com/photos/iar-afB0QQw
    (c) Nerene Grobler - https://unsplash.com/photos/sLxcfdsqLQ
    (c) Wilhelm Gunkel - https://unsplash.com/photos/L04Kczg_Jvs
    (c) Matt Walsh - https://unsplash.com/photos/tVkdGtEe2C4

    View Slide