Transparent End-to-End security for Apache Kafka (DH)

Transparent End-to-End security for Apache Kafka with Diffie Hellman

Transcript

1. Transparent End-to-End security for Apache Kafka_ hendrik.saly@codecentric.de 19.04.2018
2. • Kafka supports • Authentication & Authorization • SSL/TLS encryption • but there is no encryption of the message itself Transparent End-to-End security for Apache Kafka_ 2
3. • Why data encryption is maybe useful • It protects from reading the message for anyone without the key(s) • It does also protect from altering messages • So no worries about insecure backup places • No worries about disk/hardware thefts or the „evil cloud“ • Broker never sees unencrypted data • Might help to be (GDPR) compliant • SSL/TLS can (under certain conditions) be omitted • leverage sendfile (for linux kernel < 4.1.3) • skip complex and potential dangerous setup • Java SSL (in GCM mode) is slow Transparent End-to-End security for Apache Kafka_ 3
4. • Data encryption requirements • fast (but secure) • end-to-end • transparent (to avoid side effects) • Make Kafka totally unaware of encryption • detect if a message in encrypted or not • easy to use/apply Transparent End-to-End security for Apache Kafka_ 4
5. • How to achieve these requirements • Producers encrypt • Consumers decrypt • Wrap original serializer Transparent End-to-End security for Apache Kafka_ 5
6. • Setup • We need a fast algorithm with hardware support • AES in GCM (Galois/Counter Mode) mode • Authenticated Encryption with Associated Data (AEAD) • But AES is symmetric and we need to distribute the key • Use Diffie-Hellman key exchange • Derive AES key from two entangled EC keypairs (curve25519) • But what about semantically secureness • We use an unencrypted Initialization Vector (IV) and append it to the message • IV is randomly generated for each message Transparent End-to-End security for Apache Kafka_ 6
7. • Setup • O: Original plain message (arbitrary bytes) • M: Magic bytes (0xBD 0xDD) (2 bytes) • L: Length information about IV (1 byte) • IV: Initialization Vector (12 bytes normally) • aes(O): AES encrypted message (+ GCM Tag) Transparent End-to-End security for Apache Kafka_ 7 M M L IV aes(O)
8. • Producer • Derive AES key • HKDF (HMAC-based Extract-and-Expand Key Derivation Function) • for key stretching and key strengthening • Generate random initialization vector → IV • Encrypt message with AES key and IV -> aes(O) • Replace original message O with M-M-L-IV-aes(O) Transparent End-to-End security for Apache Kafka_ 8
9. • Consumer • Derive AES key • Check magic bytes (M). Bypass unencrypted messages • Extract IV by looking at L • Decrypt aes(O) with K and IV • Replace M-M-L-IV-aes(O) with O Transparent End-to-End security for Apache Kafka_ 9
10. • Performance • Brokers are totally unaffected • Single producer/consumer on reasonable hardware • encrypt/decrypt up to 1,3 Gb/s (with AES GCM and native OpenSSL) • Message overhead is only 15 byte • Works also well with Kafka Streams Transparent End-to-End security for Apache Kafka_ 10
11. • What we have got • Confidentiality • Integrity Transparent End-to-End security for Apache Kafka_ 11
12. • Limitations • No authenticity/accountability (yet, signatures needed) • No non-repudiation • Message dropping, replaying or reordering still possible for MITM • No forward secrecy (but we do not want it here) • Java/Scala consumer/producer only - for the moment Transparent End-to-End security for Apache Kafka_ 12
13. • Use it • Install OpenSSL • Add dependency • Create EC key pairs Transparent End-to-End security for Apache Kafka_ 13 <dependency> <groupId>de.saly</groupId> <artifactId>kafka-end-2-end-encryption</artifactId> <version>1.1.0</version> </dependency> java -cp kafka-end-2-end-encryption-1.1.0.jar \ de.saly.kafka.crypto.ECKeyGen
14. • Use it • Apply producer config • Apply consumer config Transparent End-to-End security for Apache Kafka_ 14 value.serializer: de.saly.kafka.crypto.EncryptingSerializer crypto.wrapped_serializer: org.apache.kafka.common.serialization.StringSerializer crypto.publickey.filepath: /opt/ec_consumer_public.key crypto.privatekey.filepath: /opt/ec_producer_private.key value.deserializer: de.saly.kafka.crypto.DecryptingDeserializer crypto.wrapped_deserializer: org.apache.kafka.common.serialization.StringDeserializer crypto.publickey.filepath: /opt/ec_producer_public.key crypto.privatekey.filepath: /opt/ec_consumer_private.key
15. • Further reading • https://github.com/salyh/kafka-end-2-end-encryption/wiki • https://safecurves.cr.yp.to Transparent End-to-End security for Apache Kafka_ 15
16. hendrik.saly@codecentric.de Questions? 16