Dynamic certificate internals with ngx_mruby #nagoyark03

Talk for Nagoya Ruby Kaigi 03 (http://regional.rubykaigi.org/nagoya03/) by OKUMURA Takahiro.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[Links in the slides]
mruby: https://github.com/mruby/mruby

200万 Webサイトを支える ロリポップ！と mruby:
https://speakerdeck.com/harasou/200mo-websaitowozhi-eru-roripotupu-to-mruby

haconiwa: https://github.com/haconiwa/haconiwa

ngx_mruby の Nginx::Var クラスの実装を理解する〜変数取得編
http://blog.hifumi.info/2016/11/07/ngx_mruby-nginx-var-using-method-missing/

How to build ngx_mruby:
https://github.com/matsumotory/ngx_mruby/wiki/Install
https://github.com/hsbt/ngx_mruby-package-builder
https://hub.docker.com/r/hfmgarden/ngx_mruby
https://github.com/giraffi/docker-nginx-mruby-base

100行あったmod_rewirteを ngx_mrubyで書き換えた話:
https://speakerdeck.com/buty4649/100xing-atutamod-rewirtewo-ngx-mrubydeshu-kihuan-etahua

goope: https://goope.jp

Too many domains and certificates:
https://gist.github.com/hfm/4a045a429f9303c90eac7c348d1a424a

SSL_CTX_set_cert_cb(): https://www.openssl.org/docs/man1.0.2/ssl/SSL_CTX_set_cert_cb.html

Announce that ngx_mruby supports dynamic certificates:
https://twitter.com/matsumotory/status/685341115814289408

ngx_mruby に mruby_ssl_handshake_handler を実装した:
http://blog.hifumi.info/2016/10/03/ngx_mruby-mruby_ssl_handshake_handler/

mruby のテスト用に MySQL 環境を自動で構築する mruby-test-mysqld を書いた
:
http://blog.hifumi.info/2016/09/06/mruby-test-mysqld/

How to test code with mruby:
http://www.slideshare.net/hsbt/20150525-testing-casualtalks

Transcript

1. Dynamic certificate internals with
   ngx_mruby
   OKUMURA Takahiro / GMO Pepabo, Inc.
   2017.2.11 Nagoya Ruby Kaigi 03
   

   View Slide

2. Senior web operation engineer at
   GMO Pepabo, Inc.
   A maintainer of ngx_mruby.
   http://blog.hifumi.info
   hfm
   hfm
   OKUMURA

   Takahiro (okkun)
   

   View Slide

3. Today’s Agenda
   Introduce mruby/ngx_mruby
   How to build ngx_mruby
   How hosting too many domains & certificates
   Dynamic certificate with ngx_mruby
   

   View Slide

4. IUUQTTQFBLFSEFDLDPNIGNHNPIPTDPO
   EXTENDED
   

   View Slide

5. mruby
   

   View Slide

6. mruby
   •lightweight and embeddable implementation
   of the Ruby language
   •https://github.com/mruby/mruby
   

   View Slide

7. Pepabo mruby
   IUUQTTQFBLFSEFDLDPNIBSBTPVNPXFCTBJUPXP[IJFSVSPSJQPUVQVUPNSVCZ
   

   View Slide

8. Pepabo mruby
   

   View Slide

9. ngx_mruby
   

   View Slide

10. ngx_mruby
    •A fast and memory-efficient web server
    extension mechanism using mruby for nginx
    •Created by @matsumotory
    •An alternative to mod_mruby
    

    View Slide

11. ngx_mruby sample code
    location /hello_world {
    mruby_content_handler_code ‘
    Nginx.rputs "Hello ngx_mruby world!"
    ';
    }
    $ curl http://127.0.0.1/hello_world
    Hello ngx_mruby world!
    

    View Slide

12. ngx_mruby sample code
    v = Nginx::Var.new
    Nginx.rputs "Date: #{v.date_local}"
    #=> Date: Friday, 14-Oct-2016 02:20:00 JST
    # Using ngx_http_geoip_module
    Nginx.rputs "Country: #{v.geoip_country_name}"
    #=> Country: Japan
    

    View Slide

13. ngx_mruby sample code
    v = Nginx::Var.new
    Nginx.rputs "Date: #{v.date_local}"
    #=> Date: Friday, 14-Oct-2016 02:20:00 JST
    # Using ngx_http_geoip_module
    Nginx.rputs "Country: #{v.geoip_country_name}"
    #=> Country: Japan
    Do you want to know Nginx::Var internal?
    => ngx_mruby ͷ Nginx::Var Ϋϥεͷ࣮૷Λཧղ͢Δʙม਺औಘฤ
    http://blog.hifumi.info/2016/11/07/ngx_mruby-nginx-var-using-method-missing/
    

    View Slide

14. How to build ngx_mruby
    •build.sh
    •hsbt/ngx_mruby-package-builder
    •Dockerfile
    •nginx-build
    

    View Slide

15. build.sh
    •Generate binary
    •https://github.com/matsumotory/ngx_mruby/wiki/Install
    

    View Slide

16. hsbt/ngx_mruby-package-builder
    •Providing RPM and DEB packages
    •CentOS 6, 7
    •Ubuntu 14.04, 16.04
    •https://github.com/hsbt/ngx_mruby-package-builder
    

    View Slide

17. Dockerfile
    •https://hub.docker.com/r/hfmgarden/ngx_mruby
    •Alpine Linux
    •Ubuntu 16.04
    

    View Slide

18. nginx-build
    •Created by @cubicdaiya
    •Use case
    •https://github.com/giraffi/docker-nginx-mruby-base
    

    View Slide

19. We use ngx_mruby in production.
    IUUQTTQFBLFSEFDLDPNCVUZYJOHBUVUBNPESFXJSUFXPOHYNSVCZEFTIVLJIVBOFUBIVB
    

    View Slide

20. Preliminary remarks
    mruby is good
    ngx_mruby is good
    We use ngx_mruby in production
    

    View Slide

21. How hosting
    too many
    domains and certificates
    Certificate
    Certificate
    Certificate
    

    View Slide

22. Website builder “Goope”
    

    View Slide

23. Custom domains with SSL certificates
    

    View Slide

24. Two problems
    •IP address per domain
    •Web server load too many certificates
    

    View Slide

25. Two problems
    •IP address per domain
    ➡ TLS SNI extension
    •Web server load too many certificates
    

    View Slide

26. TLS SNI extension
    TLS
    SNI
    sample-a.com
    Certificate IP Address
    Certificate IP Address
    Certificate IP Address
    sample-b.com
    sample-c.com
    IP Address
    Certificate
    Certificate
    Certificate
    sample-a.com
    sample-b.com
    sample-c.com
    

    View Slide

27. Two problems
    •IP address per domain
    ✓ TLS SNI extension
    •Web server load too many certificates
    

    View Slide

28. Two problems
    •IP address per domain
    ✓ TLS SNI extension
    •Web server load too many certificates
    

    View Slide

29. Too many domains and certificates
    https://a.jp
    application
    N
    https://b.jp
    Certificate
    Certificate
    c’mon
    

    View Slide

30. Too many domains and certificates
    https://a.jp
    application
    N
    https://b.jp https://z.jp
    …
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    
    oh… Certificate
    Certificate
    Certificate
    

    View Slide

31. Too many domains and certificates
    https://a.jp
    application
    N
    https://b.jp https://z.jp https://aa.jp
    … …
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    
    Oops…
    
    Certificate
    Certificate
    Certificate
    

    View Slide

32. Too many domains and certificates
    https://a.jp
    application
    N
    https://b.jp https://z.jp https://aa.jp
    … …
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    
    x 10,000
    # ps axfo rss,cmd
    RSS CMD
    2872 bash
    256168 nginx (master)
    258384 \_ nginx (worker)
    Certificate
    Certificate
    Certificate
    
    

    View Slide

33. Too many domains and certificates
    https://a.jp
    application
    N
    https://b.jp https://z.jp https://aa.jp
    … …
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    Certificate
    
    x 10,000
    # ps axfo rss,cmd
    RSS CMD
    2824 bash
    512808 nginx (master)
    511396 \_ nginx (worker)
    SIGHUP
    !?
    Certificate
    Certificate
    Certificate
    Certificate
    
    reproduce: https://gist.github.com/hfm/4a045a429f9303c90eac7c348d1a424a
    

    View Slide

34. Too many domains and certificates
    •Heavy load
    •Large amounts of memory
    

    View Slide

35. Two problems
    •IP address per domain
    ✓ TLS SNI extension
    •Web server load too many certificates
    

    View Slide

36. Two problems
    •IP address per domain
    ✓ TLS SNI extension
    •Web server load too many certificates
    ➡ Dynamic certificates
    

    View Slide

37. Dynamic certificate
    with
    ngx_mruby
    

    View Slide

38. SSL_CTX_set_cert_cb()
    •available in OpenSSL 1.0.2 and later
    •handle certificate callback function
    •It’s called before certificate will be used by
    client/server
    •https://www.openssl.org/docs/man1.0.2/ssl/SSL_CTX_set_cert_cb.html
    

    View Slide

39. SSL_CTX_set_cert_cb()
    Client Server
    ClientHello
    (server_name)
    ServerHello
    Certificate
    ɾ
    ɾ
    ɾ
    awesome_ssl_cert_handler
    

    View Slide

40. IUUQTUXJUUFSDPNNBUTVNPUPSZTUBUVT IUUQTHJUIVCDPNNBUTVNPUPSZOHY@NSVCZQVMM
    ngx_mruby supports it
    

    View Slide

41. mruby_ssl_handshake_handler_code
    # simple pattern (inline)
    mruby_ssl_handshake_handler_code '
    ssl = Nginx::SSL.new
    ssl.certificate = "/path/to/#{ssl.servername}.crt"
    ssl.certificate_key = "/path/to/#{ssl.servername}.key"
    ’;
    

    View Slide

42. mruby_ssl_handshake_handler
    # simple pattern (file)
    mruby_ssl_handshake_handler /path/to/handler.rb cache;
    # /path/to/handler.rb
    ssl = Nginx::SSL.new
    ssl.certificate = "/path/to/#{ssl.servername}.crt"
    ssl.certificate_key = "/path/to/#{ssl.servername}.key"
    

    View Slide

43. mruby_ssl_handshake_handler
    # simple pattern (file)
    mruby_ssl_handshake_handler /path/to/handler.rb cache;
    # /path/to/handler.rb
    ssl = Nginx::SSL.new
    ssl.certificate = "/path/to/#{ssl.servername}.crt"
    ssl.certificate_key = "/path/to/#{ssl.servername}.key"
    => ngx_mruby ʹ mruby_ssl_handshake_handler Λ࣮૷ͨ͠
    http://blog.hifumi.info/2016/10/03/ngx_mruby-mruby_ssl_handshake_handler/
    

    View Slide

44. Two problems
    •IP address per domain
    ✓ TLS SNI extention
    •Web server load too many certificates
    ✓ Dynamic certificates with ngx_mruby
    

    View Slide

45. The service architecture
    for
    dynamic certificate
    

    View Slide

46. Dynamic certificates architecture
    client
    application
    cache
    db
    1. sent server_name
    2. fetch crt/key pair from cache or db
    3. proxy
    2-a. set cache if missing
    direct pattern
    

    View Slide

47. Dynamic certificates architecture
    client
    application
    cache
    db
    1. sent server_name
    2. fetch crt/key pair
    3. proxy
    internal api
    api pattern
    

    View Slide

48. worker handlers
    mruby_init_worker /path/to/mruby_init_worker.rb cache;
    mruby_exit_worker /path/to/mruby_exit_worker.rb cache;
    

    View Slide

49. mruby_init_worker.rb
    conf = Ini.load_file('config.ini')
    redis = Redis.new(conf['redis']['server'], 6379)
    Userdata.new(“redis_#{Process.pid}").redis_conn = redis
    mysql = MySQL::Database.new(conf[‘mysql’][‘host’],
    conf[‘mysql’][‘user’], conf[‘mysql’][‘pass’],
    conf[‘mysql’]['db'])
    Userdata.new(“mysql_#{Process.pid}").mysql_conn = mysql
    mruby_init_worker /path/to/mruby_init_worker.rb cache;
    mruby_exit_worker /path/to/mruby_exit_worker.rb cache;
    

    View Slide

50. Connecting with nginx workers
    nginx master
    worker
    worker
    redis
    mysql
    

    View Slide

51. mruby_exit_worker.rb
    # Close all connections
    redis = Userdata.new("redis_#{Process.pid}").redis_conn
    redis.close unless redis.nil?
    mysql = Userdata.new("mysql_#{Process.pid}").mysql_conn
    mysql.close unless mysql.nil?
    mruby_init_worker /path/to/mruby_init_worker.rb cache;
    mruby_exit_worker /path/to/mruby_exit_worker.rb cache;
    

    View Slide

52. Disconnecting with nginx workers
    nginx master
    worker
    worker
    redis
    mysql
    

    View Slide

53. Reconnecting with nginx workers
    nginx master
    worker
    worker
    redis
    mysql
    

    View Slide

54. Reconnecting with nginx workers
    nginx master
    worker
    worker
    redis
    mysql
    MySQL server has gone away
    ×
    ×
    

    View Slide

55. Reconnecting with nginx workers
    nginx master
    worker
    worker
    redis
    mysql
    Reconnection
    def mysql_reconnect
    ...
    mysql = MySQL::Database.new(host, user, pass, db)
    Userdata.new("mysql_#{Process.pid}").mysql_conn = mysql
    end
    

    View Slide

56. Testing
    

    View Slide

57. hfm/mruby-test-mysqld
    •mruby porting from Test::mysqld of perl
    •mysqld runner for tests with mruby
    http://blog.hifumi.info/2016/09/06/mruby-test-mysqld/
    

    View Slide

58. Testing sample (1/2)
    if Object.const_defined?(:MTest)
    class DynCrtTest < MTest::Unit::TestCase
    def setup
    @mysqld = TestMysqld.new(database: 'foo')
    @mysql = MySQL::Database.new '127.0.0.1', 'root',
    '', 'foo', 3306
    end
    def teardown
    @mysql.close
    @mysqld.stop
    end
    ...
    

    View Slide

59. Testing sample (2/2)
    if Object.const_defined?(:MTest)
    ...
    def test_dyn_crt
    ...
    end
    end
    MTest::Unit.new.run
    else
    ...
    end
    

    View Slide

60. IUUQXXXTMJEFTIBSFOFUITCUUFTUJOHDBTVBMUBMLT
    TFFBMTP
    

    View Slide

61. Summary
    We use ngx_mruby in production
    ngx_mruby supports dynamic certificates
    using SNI and OpenSSL 1.0.2
    

    View Slide