Share
Embed
Transparent End-to-End security for Apache Kafka
Transparent End-to-End security for Apache Kafka_
Hendrik Saly
February 23, 2017
Transcript
15 slides
-
• Kafka supports • Authentication & Authorization • SSL/TLS encryption
• but there is no • Data encryption of the message itself Transparent End-to-End security for Apache Kafka_ 2 -
• Why data encryption is maybe useful • It protects
from reading the message for anyone without the key • It does also protect from altering messages • So no worries about insecure backup places • No worries about disk/hardware thefts • SSL/TLS can (under certain conditions ) be omitted • leverage sendfile • skip complex setup Transparent End-to-End security for Apache Kafka_ 3 -
• Data encryption requirements • fast (but secure) • end-to-end
• transparent (to avoid side effects) • Make Kafka totally unaware of encryption • detect if a message in encrypted or not • easy to use/apply Transparent End-to-End security for Apache Kafka_ 4 -
• How to achieve these requirements • Producers encrypt •
Consumers decrypt • Wrap original serializer Transparent End-to-End security for Apache Kafka_ 5 -
• Setup • We need a fast algorithm with hardware
support -> AES • But AES is symmetric and we want not encrypt every message with the same key • And it would be hard to get the key from the producer to the consumer • So lets encrypt the AES key with RSA and attach it to every message • But RSA is sooo slooow • We could cache it • But what about semantically secureness • We use an unencrypted Initialization Vector (IV) for that Transparent End-to-End security for Apache Kafka_ 6 -
• Setup • O: Original plain message (arbitrary bytes) •
K: Plain AES key • M: Magic bytes (0xDF 0xBB) • hash(K): SHA-256 hash of plain AES key • rsa(K): RSA encrypted plain AES key • aes(O): AES encrypted message • IV: Initialization Vector • L: Length information about hash(K), rsa(K) and IV Transparent End-to-End security for Apache Kafka_ 7 -
• Producer • If no AES key exists create a
random one → (K) • Encrypt AES key with RSA public key → rsa(K) • Calculate SHA-256 hash of AES key → hash(K) • Generate random initialization vector → IV • Encrypt message with AES key and I -> aes(O) • Replace original message O with M-L-hash(K)-rsa(K)-I-aes(O) Transparent End-to-End security for Apache Kafka_ 8 -
• Consumer • Check magic bytes (M). Bypass unencrypted messages
• Extract hash(K) by looking at L • Extract IV by looking at L • If hash(K) is in cache get plain AES key (K) • If hash(K) is no in cache get decrypt rsa(K) to get plain AES key (and put them into the cache) • Decrypt aes(O) with K and IV • Replace M-L-hash(K)-rsa(K)-IV-aes(O) with O Transparent End-to-End security for Apache Kafka_ 9 -
• Performance • Single broker on reasonable hardware • encrypt
approx. 300 mb/s in average • decrypt approx. 1,3 Gb/s in average • Message overhead max 324 byte • Depends on original message size • Depends on RSA key length Transparent End-to-End security for Apache Kafka_ 10 -
• Limitations • No accountability • No non-repudiation • Message
dropping, replaying or reordering still possible • No forward secrecy • Java consumer/producer only for the moment Transparent End-to-End security for Apache Kafka_ 11 -
• Use it • Add dependency • Create RSA key
pair Transparent End-to-End security for Apache Kafka_ 12 <dependency> <groupId>de.saly</groupId> <artifactId>kafka-end-2-end-encryption</artifactId> <version>1.0.1</version> </dependency> java -cp kafka-end-2-end-encryption-1.0.1.jar \ de.saly.kafka.crypto.RsaKeyGen 2048 -
• Use it • Apply producer config • Apply consumer
config Transparent End-to-End security for Apache Kafka_ 13 value.serializer: de.saly.kafka.crypto.EncryptingSerializer crypto.wrapped_serializer: org.apache.kafka.common.serialization.StringSerializer crypto.rsa.publickey.filepath: /opt/rsa_publickey.key value.deserializer: de.saly.kafka.crypto.DecryptingDeserializer crypto.wrapped_deserializer: org.apache.kafka.common.serialization.StringDeserializer crypto.rsa.privatekey.filepath: /opt/rsa_privatekey.key