Transparent End-to-End security for Apache Kafka_ [email protected] 22.02.2017 

• Kafka supports • Authentication & Authorization • SSL/TLS encryption • but there is no • Data encryption of the message itself Transparent End-to-End security for Apache Kafka_ 2 

• Why data encryption is maybe useful • It protects from reading the message for anyone without the key • It does also protect from altering messages • So no worries about insecure backup places • No worries about disk/hardware thefts • SSL/TLS can (under certain conditions ) be omitted • leverage sendfile • skip complex setup Transparent End-to-End security for Apache Kafka_ 3 

• Data encryption requirements • fast (but secure) • end-to-end • transparent (to avoid side effects) • Make Kafka totally unaware of encryption • detect if a message in encrypted or not • easy to use/apply Transparent End-to-End security for Apache Kafka_ 4 

• How to achieve these requirements • Producers encrypt • Consumers decrypt • Wrap original serializer Transparent End-to-End security for Apache Kafka_ 5 

• Setup • We need a fast algorithm with hardware support -> AES • But AES is symmetric and we want not encrypt every message with the same key • And it would be hard to get the key from the producer to the consumer • So lets encrypt the AES key with RSA and attach it to every message • But RSA is sooo slooow • We could cache it • But what about semantically secureness • We use an unencrypted Initialization Vector (IV) for that Transparent End-to-End security for Apache Kafka_ 6 

• Setup • O: Original plain message (arbitrary bytes) • K: Plain AES key • M: Magic bytes (0xDF 0xBB) • hash(K): SHA-256 hash of plain AES key • rsa(K): RSA encrypted plain AES key • aes(O): AES encrypted message • IV: Initialization Vector • L: Length information about hash(K), rsa(K) and IV Transparent End-to-End security for Apache Kafka_ 7 

• Producer • If no AES key exists create a random one → (K) • Encrypt AES key with RSA public key → rsa(K) • Calculate SHA-256 hash of AES key → hash(K) • Generate random initialization vector → IV • Encrypt message with AES key and I -> aes(O) • Replace original message O with M-L-hash(K)-rsa(K)-I-aes(O) Transparent End-to-End security for Apache Kafka_ 8 

• Consumer • Check magic bytes (M). Bypass unencrypted messages • Extract hash(K) by looking at L • Extract IV by looking at L • If hash(K) is in cache get plain AES key (K) • If hash(K) is no in cache get decrypt rsa(K) to get plain AES key (and put them into the cache) • Decrypt aes(O) with K and IV • Replace M-L-hash(K)-rsa(K)-IV-aes(O) with O Transparent End-to-End security for Apache Kafka_ 9 

• Performance • Single broker on reasonable hardware • encrypt approx. 300 mb/s in average • decrypt approx. 1,3 Gb/s in average • Message overhead max 324 byte • Depends on original message size • Depends on RSA key length Transparent End-to-End security for Apache Kafka_ 10 

• Limitations • No accountability • No non-repudiation • Message dropping, replaying or reordering still possible • No forward secrecy • Java consumer/producer only for the moment Transparent End-to-End security for Apache Kafka_ 11 

• Use it • Add dependency • Create RSA key pair Transparent End-to-End security for Apache Kafka_ 12 de.saly kafka-end-2-end-encryption 1.0.1 java -cp kafka-end-2-end-encryption-1.0.1.jar \ de.saly.kafka.crypto.RsaKeyGen 2048 

• Use it • Apply producer config • Apply consumer config Transparent End-to-End security for Apache Kafka_ 13 value.serializer: de.saly.kafka.crypto.EncryptingSerializer crypto.wrapped_serializer: org.apache.kafka.common.serialization.StringSerializer crypto.rsa.publickey.filepath: /opt/rsa_publickey.key value.deserializer: de.saly.kafka.crypto.DecryptingDeserializer crypto.wrapped_deserializer: org.apache.kafka.common.serialization.StringDeserializer crypto.rsa.privatekey.filepath: /opt/rsa_privatekey.key 

• Further reading • https://blog.codecentric.de/en/2016/10/transparent-end-end-security-apache-kafka-part-1/ • https://github.com/salyh/kafka-end-2-end-encryption • https://github.com/salyh/kafka-end-2-end-encryption-bench-it Transparent End-to-End security for Apache Kafka_ 14 

[email protected] Questions? 15