Dynamic certificate internals with ngx_mruby #nagoyark03

Talk for Nagoya Ruby Kaigi 03 (http://regional.rubykaigi.org/nagoya03/) by OKUMURA Takahiro.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[Links in the slides]
mruby: https://github.com/mruby/mruby

200万 Webサイトを支える ロリポップ！と mruby:
https://speakerdeck.com/harasou/200mo-websaitowozhi-eru-roripotupu-to-mruby

haconiwa: https://github.com/haconiwa/haconiwa

ngx_mruby の Nginx::Var クラスの実装を理解する〜変数取得編
http://blog.hifumi.info/2016/11/07/ngx_mruby-nginx-var-using-method-missing/

How to build ngx_mruby:
https://github.com/matsumotory/ngx_mruby/wiki/Install
https://github.com/hsbt/ngx_mruby-package-builder
https://hub.docker.com/r/hfmgarden/ngx_mruby
https://github.com/giraffi/docker-nginx-mruby-base

100行あったmod_rewirteを ngx_mrubyで書き換えた話:
https://speakerdeck.com/buty4649/100xing-atutamod-rewirtewo-ngx-mrubydeshu-kihuan-etahua

goope: https://goope.jp

Too many domains and certificates:
https://gist.github.com/hfm/4a045a429f9303c90eac7c348d1a424a

SSL_CTX_set_cert_cb(): https://www.openssl.org/docs/man1.0.2/ssl/SSL_CTX_set_cert_cb.html

Announce that ngx_mruby supports dynamic certificates:
https://twitter.com/matsumotory/status/685341115814289408

ngx_mruby に mruby_ssl_handshake_handler を実装した:
http://blog.hifumi.info/2016/10/03/ngx_mruby-mruby_ssl_handshake_handler/

mruby のテスト用に MySQL 環境を自動で構築する mruby-test-mysqld を書いた
:
http://blog.hifumi.info/2016/09/06/mruby-test-mysqld/

How to test code with mruby:
http://www.slideshare.net/hsbt/20150525-testing-casualtalks

Transcript

1. Dynamic certificate internals with ngx_mruby OKUMURA Takahiro / GMO Pepabo, Inc. 2017.2.11 Nagoya Ruby Kaigi 03
2. Senior web operation engineer at GMO Pepabo, Inc. A maintainer of ngx_mruby. http://blog.hifumi.info hfm hfm OKUMURA
 Takahiro (okkun)
3. Today’s Agenda Introduce mruby/ngx_mruby How to build ngx_mruby How hosting too many domains & certificates Dynamic certificate with ngx_mruby
4. IUUQTTQFBLFSEFDLDPNIGNHNPIPTDPO EXTENDED
5. mruby
6. mruby •lightweight and embeddable implementation of the Ruby language •https://github.com/mruby/mruby
7. Pepabo mruby IUUQTTQFBLFSEFDLDPNIBSBTPVNPXFCTBJUPXP[IJFSVSPSJQPUVQVUPNSVCZ
8. Pepabo mruby
9. ngx_mruby
10. ngx_mruby •A fast and memory-efficient web server extension mechanism using mruby for nginx •Created by @matsumotory •An alternative to mod_mruby
11. ngx_mruby sample code location /hello_world { mruby_content_handler_code ‘ Nginx.rputs "Hello ngx_mruby world!" '; } $ curl http://127.0.0.1/hello_world Hello ngx_mruby world!
12. ngx_mruby sample code v = Nginx::Var.new Nginx.rputs "Date: #{v.date_local}" #=> Date: Friday, 14-Oct-2016 02:20:00 JST # Using ngx_http_geoip_module Nginx.rputs "Country: #{v.geoip_country_name}" #=> Country: Japan
13. ngx_mruby sample code v = Nginx::Var.new Nginx.rputs "Date: #{v.date_local}" #=> Date: Friday, 14-Oct-2016 02:20:00 JST # Using ngx_http_geoip_module Nginx.rputs "Country: #{v.geoip_country_name}" #=> Country: Japan Do you want to know Nginx::Var internal? => ngx_mruby ͷ Nginx::Var Ϋϥεͷ࣮૷Λཧղ͢Δʙม਺औಘฤ http://blog.hifumi.info/2016/11/07/ngx_mruby-nginx-var-using-method-missing/
14. How to build ngx_mruby •build.sh •hsbt/ngx_mruby-package-builder •Dockerfile •nginx-build
15. build.sh •Generate binary •https://github.com/matsumotory/ngx_mruby/wiki/Install
16. hsbt/ngx_mruby-package-builder •Providing RPM and DEB packages •CentOS 6, 7 •Ubuntu 14.04, 16.04 •https://github.com/hsbt/ngx_mruby-package-builder
17. Dockerfile •https://hub.docker.com/r/hfmgarden/ngx_mruby •Alpine Linux •Ubuntu 16.04
18. nginx-build •Created by @cubicdaiya •Use case •https://github.com/giraffi/docker-nginx-mruby-base
19. We use ngx_mruby in production. IUUQTTQFBLFSEFDLDPNCVUZYJOHBUVUBNPESFXJSUFXPOHYNSVCZEFTIVLJIVBOFUBIVB
20. Preliminary remarks mruby is good ngx_mruby is good We use ngx_mruby in production
21. How hosting too many domains and certificates Certificate Certificate Certificate
22. Website builder “Goope”
23. Custom domains with SSL certificates
24. Two problems •IP address per domain •Web server load too many certificates
25. Two problems •IP address per domain ➡ TLS SNI extension •Web server load too many certificates
26. TLS SNI extension TLS SNI sample-a.com Certificate IP Address Certificate IP Address Certificate IP Address sample-b.com sample-c.com IP Address Certificate Certificate Certificate sample-a.com sample-b.com sample-c.com
27. Two problems •IP address per domain ✓ TLS SNI extension •Web server load too many certificates
28. Two problems •IP address per domain ✓ TLS SNI extension •Web server load too many certificates
29. Too many domains and certificates https://a.jp application N https://b.jp Certificate Certificate c’mon
30. Too many domains and certificates https://a.jp application N https://b.jp https://z.jp … Certificate Certificate Certificate Certificate Certificate Certificate oh… Certificate Certificate Certificate
31. Too many domains and certificates https://a.jp application N https://b.jp https://z.jp https://aa.jp … … Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Oops… Certificate Certificate Certificate
32. Too many domains and certificates https://a.jp application N https://b.jp https://z.jp https://aa.jp … … Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate x 10,000 # ps axfo rss,cmd RSS CMD 2872 bash 256168 nginx (master) 258384 \_ nginx (worker) Certificate Certificate Certificate
33. Too many domains and certificates https://a.jp application N https://b.jp https://z.jp https://aa.jp … … Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate x 10,000 # ps axfo rss,cmd RSS CMD 2824 bash 512808 nginx (master) 511396 \_ nginx (worker) SIGHUP !? Certificate Certificate Certificate Certificate reproduce: https://gist.github.com/hfm/4a045a429f9303c90eac7c348d1a424a
34. Too many domains and certificates •Heavy load •Large amounts of memory
35. Two problems •IP address per domain ✓ TLS SNI extension •Web server load too many certificates
36. Two problems •IP address per domain ✓ TLS SNI extension •Web server load too many certificates ➡ Dynamic certificates
37. Dynamic certificate with ngx_mruby
38. SSL_CTX_set_cert_cb() •available in OpenSSL 1.0.2 and later •handle certificate callback function •It’s called before certificate will be used by client/server •https://www.openssl.org/docs/man1.0.2/ssl/SSL_CTX_set_cert_cb.html
39. SSL_CTX_set_cert_cb() Client Server ClientHello (server_name) ServerHello Certificate ɾ ɾ ɾ awesome_ssl_cert_handler
40. IUUQTUXJUUFSDPNNBUTVNPUPSZTUBUVT IUUQTHJUIVCDPNNBUTVNPUPSZOHY@NSVCZQVMM ngx_mruby supports it
41. mruby_ssl_handshake_handler_code # simple pattern (inline) mruby_ssl_handshake_handler_code ' ssl = Nginx::SSL.new ssl.certificate = "/path/to/#{ssl.servername}.crt" ssl.certificate_key = "/path/to/#{ssl.servername}.key" ’;
42. mruby_ssl_handshake_handler # simple pattern (file) mruby_ssl_handshake_handler /path/to/handler.rb cache; # /path/to/handler.rb ssl = Nginx::SSL.new ssl.certificate = "/path/to/#{ssl.servername}.crt" ssl.certificate_key = "/path/to/#{ssl.servername}.key"
43. mruby_ssl_handshake_handler # simple pattern (file) mruby_ssl_handshake_handler /path/to/handler.rb cache; # /path/to/handler.rb ssl = Nginx::SSL.new ssl.certificate = "/path/to/#{ssl.servername}.crt" ssl.certificate_key = "/path/to/#{ssl.servername}.key" => ngx_mruby ʹ mruby_ssl_handshake_handler Λ࣮૷ͨ͠ http://blog.hifumi.info/2016/10/03/ngx_mruby-mruby_ssl_handshake_handler/
44. Two problems •IP address per domain ✓ TLS SNI extention •Web server load too many certificates ✓ Dynamic certificates with ngx_mruby
45. The service architecture for dynamic certificate
46. Dynamic certificates architecture client application cache db 1. sent server_name 2. fetch crt/key pair from cache or db 3. proxy 2-a. set cache if missing direct pattern
47. Dynamic certificates architecture client application cache db 1. sent server_name 2. fetch crt/key pair 3. proxy internal api api pattern
48. worker handlers mruby_init_worker /path/to/mruby_init_worker.rb cache; mruby_exit_worker /path/to/mruby_exit_worker.rb cache;
49. mruby_init_worker.rb conf = Ini.load_file('config.ini') redis = Redis.new(conf['redis']['server'], 6379) Userdata.new(“redis_#{Process.pid}").redis_conn = redis mysql = MySQL::Database.new(conf[‘mysql’][‘host’], conf[‘mysql’][‘user’], conf[‘mysql’][‘pass’], conf[‘mysql’]['db']) Userdata.new(“mysql_#{Process.pid}").mysql_conn = mysql mruby_init_worker /path/to/mruby_init_worker.rb cache; mruby_exit_worker /path/to/mruby_exit_worker.rb cache;
50. Connecting with nginx workers nginx master worker worker redis mysql
51. mruby_exit_worker.rb # Close all connections redis = Userdata.new("redis_#{Process.pid}").redis_conn redis.close unless redis.nil? mysql = Userdata.new("mysql_#{Process.pid}").mysql_conn mysql.close unless mysql.nil? mruby_init_worker /path/to/mruby_init_worker.rb cache; mruby_exit_worker /path/to/mruby_exit_worker.rb cache;
52. Disconnecting with nginx workers nginx master worker worker redis mysql
53. Reconnecting with nginx workers nginx master worker worker redis mysql
54. Reconnecting with nginx workers nginx master worker worker redis mysql MySQL server has gone away × ×
55. Reconnecting with nginx workers nginx master worker worker redis mysql Reconnection def mysql_reconnect ... mysql = MySQL::Database.new(host, user, pass, db) Userdata.new("mysql_#{Process.pid}").mysql_conn = mysql end
56. Testing
57. hfm/mruby-test-mysqld •mruby porting from Test::mysqld of perl •mysqld runner for tests with mruby http://blog.hifumi.info/2016/09/06/mruby-test-mysqld/
58. Testing sample (1/2) if Object.const_defined?(:MTest) class DynCrtTest < MTest::Unit::TestCase def setup @mysqld = TestMysqld.new(database: 'foo') @mysql = MySQL::Database.new '127.0.0.1', 'root', '', 'foo', 3306 end def teardown @mysql.close @mysqld.stop end ...
59. Testing sample (2/2) if Object.const_defined?(:MTest) ... def test_dyn_crt ... end end MTest::Unit.new.run else ... end
60. IUUQXXXTMJEFTIBSFOFUITCUUFTUJOHDBTVBMUBMLT TFF
    BMTP
61. Summary We use ngx_mruby in production ngx_mruby supports dynamic certificates using SNI and OpenSSL 1.0.2