Adventures in the Dungeons of OpenSSL

12/15/2023 @ RubyConf Taiwan 2023 https://2023.rubyconf.tw/

----

(original description from timetable)

As part of implementing Hybrid Public Key Encryption (HPKE; RFC 9180) in Ruby, I had a chance to send a patch into Ruby's OpenSSL gem. Missing functionalities? Major version upgrade of the OpenSSL backend? There is a deep dungeon behind one of Ruby's default gem, and I will talk about one adventure through this dungeon.

Transcript

1. Adventures in the Dungeons of OpenSSL Ryo Kajiwara/ 梶原 龍 (sylph01) 2023/12/15 @ RubyConf Taiwan 2023 1
2. Slides are available at: https:/ /speakerdeck.co m/sylph01/adventures -in-the-dungeons-of- openssl 2
3. Hi! 3
4. I do stuff Play rhythm games (especially DanceDanceRevolution) Play the bassoon/contrabassoon Ride a lot of trains (Rails!) (travelled on 99% of JR) Build keyboards if anything catches your interest let's talk! 4
5. And I do stuff that is more relevant to this talk: Freelance web developer focused on Digital Identity and Security Mainly working at codeTakt, developing ID platforms for schools Worked/ing on writing/editing and implementing standards HTTPS in Local Network CG / Web of Things WG @ W3C, OAuth / Messaging Layer Security WG @ IETF Worked as an Officer of Internet Society Japan Chapter (2020-23) 5
6. I'm from the Forgotten Realm of Japan called Shikoku image https:/ /twitter.com/Mitchan_599/status/994221711942971392 6
7. My ticket this time was from 松山 (Matsuyama) to 松山 (Songshan) 7
9. Caution before I start... 9
10. I'm going to talk about Dungeons 10
11. ... so there be Dragons Dungeons & Dragons, am I right? 11
12. Caution Cryptographic API can be very easy to misuse What's different from dungeons like parse.y -vania is that you can actually hurt yourself. Do you like security breaches? I've done my research, but I don't consider myself a cryptography expert I don't have a PhD/Master in this field, so yeah... If you're not sure, please have your system audited by a security expert before going to production 12
13. Caution The level of production-readiness in this talk: The hpke gem: Almost ready Minor changes in the gem API are possible Mostly uses pretty safe stuff But not audited by an external security expert OpenSSL extension version of HPKE: Definitely not 13
14. Past Related Work Do Pure Ruby Dream of Encrypted Binary Protocol? / Yusuke Nakamura @ RubyKaigi 2021 Implementing QUIC in Ruby Talks about the pain of handling hex-encoded and raw strings in Ruby 14
16. HPKE Hybrid Public Key Encryption, RFC 9180 16
17. Hello Alice! Alice's private key Encrypt 6EB69570 08E03CE4 Hello Alice! Decrypt Alice's public key Alice Bob Recap: Public Key Encrypytion Alice generates key pair Bob encrypts with Public Key Alice can decrypt with Private Key https:/ /commons.wikimedia.org/wiki/File:Public_key_encryption.svg 17
18. What is HPKE In the past: Encrypt a session key using Public Key Cryptography like RSA Then send your messages using symmetric ciphers like AES You don't do everything with PKC because it's costly 18
19. What is HPKE The problem: There is lots of misuse and implementation bugs in cryptography PKCS#1 padding in RSA leading to Bleichenbacher's Oracle Attack Nonce reuse DSA/ECDSA: leading to leakage of private key in PlayStation 3's code signing AES's initialization vector re-use is everywhere! 19
20. What is HPKE You (typically) don't encrypt/decrypt with Elliptic Curve Cryptography You get key exchange (ECDH) and digital signatures (ECDSA) so the "encrypt with Public Key" trick doesn't work 20
21. What is HPKE HPKE solves this by: Using standardized/reviewed protocols to exchange keys and encrypt/decrypt Using best known cipher suites possible combination of cryptographic algorithms Using high-level APIs that prevent misuse 21
22. What is HPKE HPKE uses Agreement of symmetric keys using a Key Encapsulation Mechanism (KEM) that internally uses a Key Derivation Function (KDF) Then use the symmetric keys to perform Authenticated Encryption with Associated Data (AEAD) 22
23. How to use HPKE HPKE is a one-way protocol from Sender to Receiver Agreement of symmetric key using secret exporter is possible Receiver creates a public/private key pair, publishes public key Sender uses public key to encapsulate session key, then uses the session key to encrypt message Sender sends the encapsulation and the message Receiver uses the encapsulation to decapsulate session key, uses the session key to decrypt the message 23
24. So I wanted HPKE in Ruby... 24
25. What is needed for HPKE? Key Encapsulation Mechanism Diffie-Hellman Key Exchange use of Elliptic Curves: P-256, P-384, P-521, X25519, X448 Key Derivation Function HMAC-based Extract-and-Expand Key Derivation Function (HKDF) HMAC-SHA256, HMAC-SHA512 Authenticated Encryption with Associated Data AES-128-GCM, AES-256-GCM, ChaCha20-Poly1305 25
26. All of this is supported by OpenSSL, right? 26
27. Well, kinda 27
28. What we had readily available AES-128-GCM, AES-256-GCM, ChaCha20-Poly1305 test/openssl/test_cipher.rb ChaCha20-Poly1305 isn't in this test code, but uses the same API as AES-GCM HMAC-SHA256, HMAC-SHA512 Even with SHA256 and SHA512 only, I can implement HMAC Actually we have HKDF itself but we need to use some parts of HKDF and customize it 28
29. What we 'kinda' had Elliptic curves P-256, P-384, P-521 test/openssl/test_pkey_ec.rb X25519, X448 test/openssl/test_pkey.rb 29
30. Q: Really? These seem to be undocumented A: Yes. 30
31. Really? These seem to be undocumented Some APIs in OpenSSL are intentionally left undocumented to avoid misuses by people who are not well-versed in cryptography. Yes, misuse in cryptography can hurt yourself. 31
32. OpenSSL::PKey Set of classes that handle everything Public Key Cryptography, including RSA, DSA, Diffie-Hellman, and Elliptic Curve Cryptography. Wraps OpenSSL's EVP_PKEY struct. Note that X25519/X448 are actually ECC but does not use OpenSSL::PKey::EC . 32
33. Raw public/private key support for X25519/X448 Raw public/private key support for EC "kinda" existed 33
34. Raw public/private key support for X25519/X448 There was a pull request that worked on this But it was 3 years old and not working on some platforms So I started working on this by Fixing errors on the CI matrix Addressing unfixed review comments 34
35. https:/ /github.com/ruby/openssl/pull/ 646 my first Ruby C extension experience! 35
36. While I was at it... 36
37. OpenSSL::PKey is immutable in OpenSSL 3.0 https:/ /github.com/ruby/openssl/issues/619 37
38. So you can't do this pkey = OpenSSL::PKey::EC.new('prime256v1') pkey.private_key = SecureRandom.random_bytes(32) Note: A private key of P-256 elliptic curve consists of a 32 byte number (=scalar). 38
39. In OpenSSL 3.0, we were not just missing APIs to make X25519/X448 key pairs, we didn't even have APIs to make EC key pairs! 39
40. How can we create a key pair with a specified private key? 40
41. This was available der = # write something in ASN.1 DER format OpenSSL::PKey.read(der) 41
42. ASN.1 Sequence -> DER -> PKey::EC it's even in Ruby 3.3 preview1! 42
43. Enter ASN.1 Abstract Syntax Notation One Set of binary notation rules for encoding data structures DER (Distinguished Encoding Rules) is the ruleset to describe data structures in exactly one way, so as to ensure digital signatures produce a unique output PEM files encode certificates private keys written in DER with Base64 43
44. PKey -> DER -> ASN.1 Sequence pk = OpenSSL::PKey::EC.generate('prime256v1') pk.private_to_der.unpack1('H*') => "308187020100301306072a8648ce3d020106082a8648 ce3d030107046d306b02010104203baadea1f85b96b3 3b8a895ab2c44a4b72e827ff1b1ac23e7b756daef2f0 892ba14403420004b9f0779b47f432fd7df13b67cf21 0c58db276653a13b862db2eb6d47f3eabe9ac4317181 8e19e78f37fe5e86a7439d471d0f3442a2b727f9560a ff417d432391" (formatting just for visibility purposes) 44
45. PKey -> DER -> ASN.1 Sequence https:/ /lapo.it/asn1js/ 45
46. ASN.1 Sequence -> DER -> PKey::EC Do it backwards! def create_key_pair_from_secret(secret) asn1_seq = OpenSSL::ASN1.Sequence([ OpenSSL::ASN1.Integer(1), OpenSSL::ASN1.OctetString(secret), OpenSSL::ASN1.ObjectId(curve_name, 0, :EXPLICIT) ]) OpenSSL::PKey.read(asn1_seq.to_der) end 46
47. I did this in the HPKE gem for X25519/X448 too def create_key_pair_from_secret(secret) asn1_seq = OpenSSL::ASN1.Sequence([ OpenSSL::ASN1.Integer(0), OpenSSL::ASN1.Sequence([ OpenSSL::ASN1.ObjectId(asn1_oid) ]), OpenSSL::ASN1.OctetString("\x04\x20" + secret) ]) OpenSSL::PKey.read(asn1_seq.to_der) end 47
48. But does this look good? No, it's ugly! 48
49. API to generate PKeys with a private key is being worked on @ Issue #555 49
50. Now we have HPKE in Ruby GH: sylph01/hpke-rb gem install hpke now! note: Still in beta. I will likely tinker around with the API of the gem 50
52. Well actually, we had HPKE itself in OpenSSL 3.2 blog post in OpenSSL Blog: https:/ /www.openssl.org/blog/blog/2023/10/18/ossl-hpke/ 52
53. Why not write a Ruby wrapper for this? 53
54. so I did. GH: sylph01/openssl, hpke branch (note: this is super experimental territory, DO NOT use in production!) 54
55. How to develop C extensions GH: ruby/ruby/doc/extension.rdoc is your friend. You want to know how to: Convert C values into Ruby values and vice versa Wrap a C struct into a Ruby object 55
56. Handling HPKE Context OSSL_HPKE_CTX *OSSL_HPKE_CTX_new(int mode, OSSL_HPKE_SUITE suite, int role, OSSL_LIB_CTX *libctx, const char *propq); int OSSL_HPKE_encap(OSSL_HPKE_CTX *ctx, unsigned char *enc, size_t *enclen, const unsigned char *pub, size_t publen, const unsigned char *info, size_t infolen); int OSSL_HPKE_seal(OSSL_HPKE_CTX *ctx, unsigned char *ct, size_t *ctlen, const unsigned char *aad, size_t aadlen, const unsigned char *pt, size_t ptlen); https:/ /www.openssl.org/docs/manmaster/man3/OSSL_HPKE_CTX_new.html 56
57. Handling HPKE Context static void ossl_hpke_ctx_free(void *ptr) { OSSL_HPKE_CTX_free(ptr); } const rb_data_type_t ossl_hpke_ctx_type = { "OpenSSL/HPKE_CTX", { 0, ossl_hpke_ctx_free, }, 0, 0, RUBY_TYPED_FREE_IMMEDIATELY }; 57
58. Handling HPKE Context static VALUE hpke_ctx_new0(VALUE arg){ OSSL_HPKE_CTX *ctx = (OSSL_HPKE_CTX *)arg; VALUE obj; obj = rb_obj_alloc(cContext); RTYPEDDATA_DATA(obj) = ctx; return obj; } VALUE ossl_hpke_ctx_new(OSSL_HPKE_CTX *ctx){ VALUE obj; int status; obj = rb_protect(hpke_ctx_new0, (VALUE)ctx, &status); if (status) { OSSL_HPKE_CTX_free(ctx); rb_jump_tag(status); } return obj; } 58
59. Handling HPKE Context static VALUE ossl_hpke_ctx_alloc(VALUE klass) { return TypedData_Wrap_Struct(klass, &ossl_hpke_ctx_type, NULL); } void Init_ossl_hpke_ctx(void) { mHPKE = rb_define_module_under(mOSSL, "HPKE"); cContext = rb_define_class_under(mHPKE, "Context", rb_cObject); ... rb_define_alloc_func(cContext, ossl_hpke_ctx_alloc); } 59
60. Generating the key int OSSL_HPKE_keygen(OSSL_HPKE_SUITE suite, unsigned char *pub, size_t *publen, EVP_PKEY **priv, const unsigned char *ikm, size_t ikmlen, OSSL_LIB_CTX *libctx, const char *propq); priv is an EVP_PKEY so we can wrap that into OpenSSL::PKey pub is an unsigned char * = String, but which format? 60
61. Generating the key Apparently: for EC keys it's the equivalent of priv.public_key.to_octet_string(:uncompressed) for X25519/X448 keys it's the equivalent of priv.raw_public_key 61
62. Encapsulating the key VALUE ossl_hpke_encap(VALUE self, VALUE pub, VALUE info) { (snip definitions) GetHpkeCtx(self, sctx); // extract C pointer to context from object enclen = sizeof(enc); if (OSSL_HPKE_encap( sctx, enc, &enclen, (unsigned char*)RSTRING_PTR(pub), RSTRING_LEN(pub), (unsigned char*)RSTRING_PTR(info), RSTRING_LEN(info)) != 1) { ossl_raise(eHPKEError, "could not encap"); } enc_obj = rb_str_new((char *)enc, enclen); return enc_obj; } 62
63. How to develop (OpenSSL) ruby/openssl's CONTRIBUTING.md has a "Testing: With different versions of OpenSSL" section: git checkout openssl-3.2 OPENSSL_DIR=$HOME/.openssl/openssl-something ./Configure --prefix=$OPENSSL_DIR --libdir=lib enable- fips enable-trace '-Wl,-rpath,$(LIBRPATH)' -O0 -g3 -ggdb3 -gdwarf-5 make -j4 , make install 63
64. How to develop (OpenSSL) I wanted to peek inside the OSSL_HPKE_CTX , but I got an invalid use of incomplete typedef error. This is because the full typedef is not exposed to the public header files. I moved the internal definitions to public header file <openssl/hpke.h> then recompiled OpenSSL. 64
65. How to develop (OpenSSL gem) Then to build with that version: bundle exec rake clean bundle exec rake compile -- --with-openssl- dir=$OPENSSL_DIR irb -I lib -r openssl 65
66. How to develop (debugging) I wanted to see the C string in hex printed into the console, so: void rbdebug_print_hex(const unsigned char *str, size_t len) { VALUE rbstr; rbstr = rb_str_new((char *)str, len); rb_p(rb_funcall(rbstr, rb_intern("unpack1"), 1, rb_str_new_cstr("H*"))); } rb_p is your friend. Also rb_funcall is nice too. 66
67. Example irb(main):001:0> suite = OpenSSL::HPKE::Suite.new_with_names( :dhkem_p256_hkdf_sha256, :hkdf_sha256, :aes_128_gcm) => #<OpenSSL::HPKE::Suite:0x00007f5460d55b68 @aead_id=1, @kdf_id=1, @kem_id=16> irb(main):002:0> priv = OpenSSL::HPKE.keygen_with_suite(suite) irb(main):003:0> pub = priv.public_key => #<OpenSSL::PKey::EC::Point:0x00007f5460d67520 @group=#<OpenSSL::PKey::EC::Group:0x00007f5460d674d0>> Generate a public/private key pair 67
68. Example # sender irb(main):004:0> sctx = OpenSSL::HPKE::Context::Sender.new(:base, suite) => #<OpenSSL::HPKE::Context::Sender:0x00007f5460d22768 @aead_id=1, @kdf_id=1, @kem_id=16> irb(main):005:0> enc = sctx.encap(pub.to_octet_string(:uncompressed), "Some info") irb(main):006:0> ct = sctx.seal("\x01\x02\x03\x04\x05\x06\x07\x08", "a message not in a bottle") => "\x93\xC1\xDF7\x87\xA8\xBER\xC9&\xA4\xD1\xB7\x10\x8E..." Create sender context Encapsulate key into the public key of the receiver Using the generated shared secret, seal message before commit 4548dd0e it uses a different style of initialization API 68
69. Example # receiver irb(main):007:0> rctx = OpenSSL::HPKE::Context::Receiver.new(:base, suite) => #<OpenSSL::HPKE::Context::Receiver:0x00007f5460d64e88 @aead_id=1, @kdf_id=1, @kem_id=16> irb(main):008:0> rctx.decap(enc, priv, "Some info") irb(main):009:0> pt = rctx.open("\x01\x02\x03\x04\x05\x06\x07\x08", ct) => "a message not in a bottle" Receiver gets the encapsulation enc and sealed message ct Create receiver context Decapsulate key from enc with private key and derive shared secret Then using the derived shared secret, open message 69
70. By the way... Is anyone familiar with Ruby's encoding? I don't want the force_encoding but I kinda need it now irb(main):010:0> sctx = OpenSSL::HPKE::Context::Sender.new(:base, suite) irb(main):011:0> rctx = OpenSSL::HPKE::Context::Receiver.new(:base, suite) => #<OpenSSL::HPKE::Context::Receiver:0x00007f5460d2d8c0 @aead_id=1, @kdf_id=1, @kem_id=16> irb(main):012:0> enc = sctx.encap(pub.to_octet_string(:uncompressed), "Some info") => "\x04\x17M\b\xE0\x9D\xEEg\xE2..." irb(main):013:0> ct = sctx.seal("\x01\x02\x03\x04\x05\x06\x07\x08", " ニッホンゴー") => "E\x80\xEFP\x95Z9N\x1F\t..." irb(main):014:0> irb(main):015:0> rctx.decap(enc, priv, "Some info") => true irb(main):016:0> pt = rctx.open("\x01\x02\x03\x04\x05\x06\x07\x08", ct) irb(main):017:0> puts pt.force_encoding('UTF-8') ニッホンゴー => nil 70
71. Will this go into actual OpenSSL gem? This still needs a lot of work: This is limited to OpenSSL 3.2, so needs guards against older versions Currently supports Base mode only Hardcoded length values need to be fixed Is the C coding actually safe? 71
73. Conclusion 73
74. So here were my "Adventures in the Forgotten Realm" called OpenSSL 74
75. Why HPKE in Ruby? It's the building block for modern security/privacy protocols TLS Encrypted ClientHello Oblivious HTTP Messaging Layer Security 75
76. Why HPKE in Ruby? If we don't have the building block for modern networking protocols, people would not implement them ... and just go to Python / Go / Rust / whatever because they have the building blocks readily available. 76
77. We need modern cryptography in Ruby for Ruby to stay relevant 77
78. It's not for everyone ... but it surely is for someone 78
79. The adventure continues... HPKE is just a building block for other protocols I am working on implementing protocols that rely on HPKE Also now that I came back from the OpenSSL dungeon alive, I might continue digging into this "Forgotten Realm" 79
80. Shoutouts (@ mentions are in GitHub ID) Ruby OpenSSL maintainers, esp. @rhenium Past RubyKaigi speakers, esp. @unasuke and @shioimm HPKE implementers, esp. @dajiaji And to the organizers of RubyConf Taiwan 2023! 80
81. Questions? / Comments? Twitter: @s01 or Fediverse: @s01@ruby.social 81