Transparent End-to-End security for Apache Kafka_ [email protected] 19.04.2018 

• Kafka supports • Authentication & Authorization • SSL/TLS encryption • but there is no encryption of the message itself Transparent End-to-End security for Apache Kafka_ 2 

• Why data encryption is maybe useful • It protects from reading the message for anyone without the key(s) • It does also protect from altering messages • So no worries about insecure backup places • No worries about disk/hardware thefts or the „evil cloud“ • Broker never sees unencrypted data • Might help to be (GDPR) compliant • SSL/TLS can (under certain conditions) be omitted • leverage sendfile (for linux kernel < 4.1.3) • skip complex and potential dangerous setup • Java SSL (in GCM mode) is slow Transparent End-to-End security for Apache Kafka_ 3 

• Data encryption requirements • fast (but secure) • end-to-end • transparent (to avoid side effects) • Make Kafka totally unaware of encryption • detect if a message in encrypted or not • easy to use/apply Transparent End-to-End security for Apache Kafka_ 4 

• How to achieve these requirements • Producers encrypt • Consumers decrypt • Wrap original serializer Transparent End-to-End security for Apache Kafka_ 5 

• Setup • We need a fast algorithm with hardware support • AES in GCM (Galois/Counter Mode) mode • Authenticated Encryption with Associated Data (AEAD) • But AES is symmetric and we need to distribute the key • Use Diffie-Hellman key exchange • Derive AES key from two entangled EC keypairs (curve25519) • But what about semantically secureness • We use an unencrypted Initialization Vector (IV) and append it to the message • IV is randomly generated for each message Transparent End-to-End security for Apache Kafka_ 6 

• Setup • O: Original plain message (arbitrary bytes) • M: Magic bytes (0xBD 0xDD) (2 bytes) • L: Length information about IV (1 byte) • IV: Initialization Vector (12 bytes normally) • aes(O): AES encrypted message (+ GCM Tag) Transparent End-to-End security for Apache Kafka_ 7 M M L IV aes(O) 

• Producer • Derive AES key • HKDF (HMAC-based Extract-and-Expand Key Derivation Function) • for key stretching and key strengthening • Generate random initialization vector → IV • Encrypt message with AES key and IV -> aes(O) • Replace original message O with M-M-L-IV-aes(O) Transparent End-to-End security for Apache Kafka_ 8 

• Consumer • Derive AES key • Check magic bytes (M). Bypass unencrypted messages • Extract IV by looking at L • Decrypt aes(O) with K and IV • Replace M-M-L-IV-aes(O) with O Transparent End-to-End security for Apache Kafka_ 9 

• Performance • Brokers are totally unaffected • Single producer/consumer on reasonable hardware • encrypt/decrypt up to 1,3 Gb/s (with AES GCM and native OpenSSL) • Message overhead is only 15 byte • Works also well with Kafka Streams Transparent End-to-End security for Apache Kafka_ 10 

• What we have got • Confidentiality • Integrity Transparent End-to-End security for Apache Kafka_ 11 

• Limitations • No authenticity/accountability (yet, signatures needed) • No non-repudiation • Message dropping, replaying or reordering still possible for MITM • No forward secrecy (but we do not want it here) • Java/Scala consumer/producer only - for the moment Transparent End-to-End security for Apache Kafka_ 12 

• Use it • Install OpenSSL • Add dependency • Create EC key pairs Transparent End-to-End security for Apache Kafka_ 13 de.saly kafka-end-2-end-encryption 1.1.0 java -cp kafka-end-2-end-encryption-1.1.0.jar \ de.saly.kafka.crypto.ECKeyGen 

• Use it • Apply producer config • Apply consumer config Transparent End-to-End security for Apache Kafka_ 14 value.serializer: de.saly.kafka.crypto.EncryptingSerializer crypto.wrapped_serializer: org.apache.kafka.common.serialization.StringSerializer crypto.publickey.filepath: /opt/ec_consumer_public.key crypto.privatekey.filepath: /opt/ec_producer_private.key value.deserializer: de.saly.kafka.crypto.DecryptingDeserializer crypto.wrapped_deserializer: org.apache.kafka.common.serialization.StringDeserializer crypto.publickey.filepath: /opt/ec_producer_public.key crypto.privatekey.filepath: /opt/ec_consumer_private.key 

• Further reading • https://github.com/salyh/kafka-end-2-end-encryption/wiki • https://safecurves.cr.yp.to Transparent End-to-End security for Apache Kafka_ 15 

[email protected] Questions? 16