Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Towards Client-Side Field-Level Cryptography for Streaming Data Pipelines @ Current 2022, Austin Texas

Towards Client-Side Field-Level Cryptography for Streaming Data Pipelines @ Current 2022, Austin Texas

Abstract:

Apache Kafka offers several security features ranging from authentication and authorization mechanisms to over-the-wire encryption. This notwithstanding, end-to-end encryption between Kafka-based client applications, which fully protects payloads from fraudulent access at the broker's side can still be considered a blind spot. After highlighting the main benefits of explicit data-at-rest protection, this session discusses in-depth how to selectively encrypt and decrypt sensitive payload fields in the context of streaming data pipelines built upon Apache Kafka Connect and ksqlDB apps. In particular, an ecosystem community project named Kryptonite for Kafka - written and open-sourced by the speaker - is introduced. During this demo-driven talk, you will experience how to benefit from:

*a configurable single message transformation (SMT) that lets you perform encryption and decryption operations in Kafka Connect worker nodes without any additional code
*and a custom user-defined function (UDF) for ksqlDB to conveniently encrypt and decrypt specific columns in your SQL-based stream processing apps

Client-side field-level cryptography makes streaming data pipelines more secure by safeguarding your most sensitive and precious data against any form of uncontrolled or illegal access once it hits the Apache Kafka brokers.

Kryptonite for Kafka Project Repository:

https://github.com/hpgrahsl/kryptonite-for-kafka/

Live Demo Scenario Repository:

https://github.com/hpgrahsl/current22-k4k-demo

Recording:

https://www.confluent.io/events/current-2022/towards-client-side-field-level-cryptography/

Hans-Peter Grahsl

October 04, 2022
Tweet

More Decks by Hans-Peter Grahsl

Other Decks in Programming

Transcript

  1. Towards
    Client-Side Field-Level
    Cryptography
    for Streaming Data Pipelines
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022

    View full-size slide

  2. Why should we care?
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    2

    View full-size slide

  3. 61 %
    of breaches involved
    credential data1
    1 Verizon DBIR 2021 - https://www.verizon.com/dbir
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    3

    View full-size slide

  4. 85 %
    of breaches involved
    the human element1
    1 Verizon DBIR 2021 - https://www.verizon.com/dbir
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    4

    View full-size slide

  5. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    5

    View full-size slide

  6. compromised external
    cloud assets
    more common than
    on-premises assets1
    1 Verizon DBIR 2021 - https://www.verizon.com/dbir
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    6

    View full-size slide

  7. Let's don't
    forget about the price tag
    of data breaches.
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    7

    View full-size slide

  8. Let's don't
    forget about the price tag
    of data breaches.
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    8

    View full-size slide

  9. $4.24M
    average cost of data
    breach2
    2 IBM Cost of Data Breach Report - https://www.ibm.com/security/data-breach
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    9

    View full-size slide

  10. $180
    per record cost of
    customer PII2
    2 IBM Cost of Data Breach Report - https://www.ibm.com/security/data-breach
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    10

    View full-size slide

  11. It's me ... Hans-Peter
    • Developer
    !
    Advocate @ Red Hat
    • Open-Source Enthusiast
    • Confluent Community Catalyst since 2019
    • MongoDB Champion since 2020
    • based in Graz, Austria
    "
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    11

    View full-size slide

  12. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    12

    View full-size slide

  13. !
    But Kafka related? Yes!
    3
    3 https://spectralops.io/blog/misconfigured-kafdrop-puts-companies-apache-kafka-completely-exposed/
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    13

    View full-size slide

  14. unhappy
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    14

    View full-size slide

  15. Core Kafka
    Security Mechanisms
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    15

    View full-size slide

  16. over-the-wire encryption
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    16

    View full-size slide

  17. authentication
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    17

    View full-size slide

  18. authorization
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    18

    View full-size slide

  19. table stakes
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    19

    View full-size slide

  20. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    20

    View full-size slide

  21. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    21

    View full-size slide

  22. disturbing
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    22

    View full-size slide

  23. core security
    necessary !
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    23

    View full-size slide

  24. core security
    sufficient ?
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    24

    View full-size slide

  25. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    25

    View full-size slide

  26. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    26

    View full-size slide

  27. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    27

    View full-size slide

  28. ?
    data in use by
    BROKERS
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    28

    View full-size slide

  29. BROKERS
    see everything ...
    and so does
    any legitimate
    Kafka client
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    29

    View full-size slide

  30. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    30

    View full-size slide

  31. human promise
    is NOT
    technical promise
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    31

    View full-size slide

  32. ? ? ?
    end-to-end
    encryption
    ? ? ?
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    32

    View full-size slide

  33. Open-Source
    Community Project
    Kryptonite for Kafka
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    33

    View full-size slide

  34. client-side
    field level
    cryptography
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    34

    View full-size slide

  35. Client-Side Cryptography
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    35

    View full-size slide

  36. Client-Side Cryptography
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    36

    View full-size slide

  37. Field Level Cryptography
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    37

    View full-size slide

  38. Field Level Cryptography
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    38

    View full-size slide

  39. Kafka Connect Sources
    Single Message
    Transform
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    39

    View full-size slide

  40. CSFLC with Source Connectors
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    40

    View full-size slide

  41. Demo Part 1
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    41

    View full-size slide

  42. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    42

    View full-size slide

  43. ksqlDB
    User-Defined Function
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    43

    View full-size slide

  44. CSFLC with Streaming SQL
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    44

    View full-size slide

  45. CSFLC with Streaming SQL
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    45

    View full-size slide

  46. Demo Part 2
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    46

    View full-size slide

  47. Here be
    Dragons
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    47

    View full-size slide

  48. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    48

    View full-size slide

  49. Kafka Connect Sinks
    Single Message
    Transform
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    49

    View full-size slide

  50. CSFLC with Sink Connectors
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    50

    View full-size slide

  51. Demo Part 3
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    51

    View full-size slide

  52. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    52

    View full-size slide

  53. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    53

    View full-size slide

  54. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    54

    View full-size slide

  55. Behind the Scenes?
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    55

    View full-size slide

  56. Cryptography
    • Tink by Google
    • AEAD based on AES GCM
    • DAEAD based on AES SIV
    • key rotation support
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    56

    View full-size slide

  57. Keyset Management
    • within SMT config (not
    recommended)
    • externalized to separate file (okayish)
    • remote / cloud KMS (recommended)
    • preliminary Azure Key Vault support
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    57

    View full-size slide

  58. !
    Little Ideas
    !
    • wildcard / regex matching for field names
    • dynamic keyset selection based on payload
    • additional KMS providers (GCP, AWS, ...)
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    58

    View full-size slide

  59. !
    Bigger Ideas
    !
    • add cryptography options (e.g. FPE)
    • extend scope beyond Kafka Connect and ksqlDB
    • make CSFLC language / runtime agnostic
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    59

    View full-size slide

  60. @hpgrahsl
    Let's stay in touch
    !
    on Twitter
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    60

    View full-size slide

  61. !
    TRY IT
    "
    • Project Code
    https://bit.ly/current22-k4k
    • Demo Scenarios
    https://bit.ly/current22-demo
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    61

    View full-size slide

  62. Data should continue
    to be a valuable
    asset, not become
    a costly liability.
    @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022
    62

    View full-size slide

  63. @hpgrahsl | #Current22 - Austin, Texas | Oct 4-5, 2022

    View full-size slide

  64. Photo Credits
    in order of appearance
    (c) John Salvino - https://unsplash.com/photos/bqGBbLq_yfc
    (c) Wolf Zimmermann - https://unsplash.com/photos/6sf5rf8QYFE
    (c) Jason Leung - https://unsplash.com/photos/SAYzxuS1O3M
    (c) Dev Asangbam - https://unsplash.com/photos/sh9vkVbVgo
    (c) Keenan Constance - https://unsplash.com/photos/VTLcvV6UVaI
    (c) Steve Johnson - https://unsplash.com/photos/hokONTrHIAQ
    (c) Pete Linforth - https://pixabay.com/illustrations/biometrics-access-identification-4503187/
    (c) Miguel Á. Padriñán - https://www.pexels.com/photo/close-up-shot-of-keys-on-a-red-surface-2882687/
    (c) Camila Quintero Franco - https://unsplash.com/photos/mC852jACK1g
    (c) Gerd Altmann - https://pixabay.com/illustrations/board-excuse-me-excuse-discharge-1848736/
    (c) Vijaya narasimha - https://pixabay.com/photos/crevasse-sand-stone-hills-rock-399957/
    (c) Gerd Altmann - https://pixabay.com/photos/trust-man-hood-map-prompt-4321822/
    (c) Matheo JBT - https://unsplash.com/photos/HLhvZ9HRAwo
    (c) Rob Laughter - https://unsplash.com/photos/WW1jsInXgwM
    (c) Markus Spiske - https://unsplash.com/photos/iar-afB0QQw
    (c) Nerene Grobler - https://unsplash.com/photos/sLxcfdsqLQ
    (c) Wilhelm Gunkel - https://unsplash.com/photos/L04Kczg_Jvs
    (c) Matt Walsh - https://unsplash.com/photos/tVkdGtEe2C4

    View full-size slide