Tweet
Tweet

Slide 1

Slide 1 text

Dynamic certificate internals with ngx_mruby OKUMURA Takahiro / GMO Pepabo, Inc. 2017.2.11 Nagoya Ruby Kaigi 03

Slide 2

Slide 2 text

Senior web operation engineer at GMO Pepabo, Inc. A maintainer of ngx_mruby. http://blog.hifumi.info hfm hfm OKUMURA
 Takahiro (okkun)

Slide 3

Slide 3 text

Today’s Agenda Introduce mruby/ngx_mruby How to build ngx_mruby How hosting too many domains & certificates Dynamic certificate with ngx_mruby

Slide 4

Slide 4 text

IUUQTTQFBLFSEFDLDPNIGNHNPIPTDPO EXTENDED

Slide 5

Slide 5 text

mruby

Slide 6

Slide 6 text

mruby •lightweight and embeddable implementation of the Ruby language •https://github.com/mruby/mruby

Slide 7

Slide 7 text

Pepabo mruby IUUQTTQFBLFSEFDLDPNIBSBTPVNPXFCTBJUPXP[IJFSVSPSJQPUVQVUPNSVCZ

Slide 8

Slide 8 text

Pepabo mruby

Slide 9

Slide 9 text

ngx_mruby

Slide 10

Slide 10 text

ngx_mruby •A fast and memory-efficient web server extension mechanism using mruby for nginx •Created by @matsumotory •An alternative to mod_mruby

Slide 11

Slide 11 text

ngx_mruby sample code location /hello_world { mruby_content_handler_code ‘ Nginx.rputs "Hello ngx_mruby world!" '; } $ curl http://127.0.0.1/hello_world Hello ngx_mruby world!

Slide 12

Slide 12 text

ngx_mruby sample code v = Nginx::Var.new Nginx.rputs "Date: #{v.date_local}" #=> Date: Friday, 14-Oct-2016 02:20:00 JST # Using ngx_http_geoip_module Nginx.rputs "Country: #{v.geoip_country_name}" #=> Country: Japan

Slide 13

Slide 13 text

ngx_mruby sample code v = Nginx::Var.new Nginx.rputs "Date: #{v.date_local}" #=> Date: Friday, 14-Oct-2016 02:20:00 JST # Using ngx_http_geoip_module Nginx.rputs "Country: #{v.geoip_country_name}" #=> Country: Japan Do you want to know Nginx::Var internal? => ngx_mruby ͷ Nginx::Var Ϋϥεͷ࣮૷Λཧղ͢Δʙม਺औಘฤ http://blog.hifumi.info/2016/11/07/ngx_mruby-nginx-var-using-method-missing/

Slide 14

Slide 14 text

How to build ngx_mruby •build.sh •hsbt/ngx_mruby-package-builder •Dockerfile •nginx-build

Slide 15

Slide 15 text

build.sh •Generate binary •https://github.com/matsumotory/ngx_mruby/wiki/Install

Slide 16

Slide 16 text

hsbt/ngx_mruby-package-builder •Providing RPM and DEB packages •CentOS 6, 7 •Ubuntu 14.04, 16.04 •https://github.com/hsbt/ngx_mruby-package-builder

Slide 17

Slide 17 text

Dockerfile •https://hub.docker.com/r/hfmgarden/ngx_mruby •Alpine Linux •Ubuntu 16.04

Slide 18

Slide 18 text

nginx-build •Created by @cubicdaiya •Use case •https://github.com/giraffi/docker-nginx-mruby-base

Slide 19

Slide 19 text

We use ngx_mruby in production. IUUQTTQFBLFSEFDLDPNCVUZYJOHBUVUBNPESFXJSUFXPOHYNSVCZEFTIVLJIVBOFUBIVB

Slide 20

Slide 20 text

Preliminary remarks mruby is good ngx_mruby is good We use ngx_mruby in production

Slide 21

Slide 21 text

How hosting too many domains and certificates Certificate Certificate Certificate

Slide 22

Slide 22 text

Website builder “Goope”

Slide 23

Slide 23 text

Custom domains with SSL certificates

Slide 24

Slide 24 text

Two problems •IP address per domain •Web server load too many certificates

Slide 25

Slide 25 text

Two problems •IP address per domain ➡ TLS SNI extension •Web server load too many certificates

Slide 26

Slide 26 text

TLS SNI extension TLS SNI sample-a.com Certificate IP Address Certificate IP Address Certificate IP Address sample-b.com sample-c.com IP Address Certificate Certificate Certificate sample-a.com sample-b.com sample-c.com

Slide 27

Slide 27 text

Two problems •IP address per domain ✓ TLS SNI extension •Web server load too many certificates

Slide 28

Slide 28 text

Two problems •IP address per domain ✓ TLS SNI extension •Web server load too many certificates

Slide 29

Slide 29 text

Too many domains and certificates https://a.jp application N https://b.jp Certificate Certificate c’mon

Slide 30

Slide 30 text

Too many domains and certificates https://a.jp application N https://b.jp https://z.jp … Certificate Certificate Certificate Certificate Certificate Certificate oh… Certificate Certificate Certificate

Slide 31

Slide 31 text

Too many domains and certificates https://a.jp application N https://b.jp https://z.jp https://aa.jp … … Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Oops… Certificate Certificate Certificate

Slide 32

Slide 32 text

Too many domains and certificates https://a.jp application N https://b.jp https://z.jp https://aa.jp … … Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate x 10,000 # ps axfo rss,cmd RSS CMD 2872 bash 256168 nginx (master) 258384 \_ nginx (worker) Certificate Certificate Certificate

Slide 33

Slide 33 text

Too many domains and certificates https://a.jp application N https://b.jp https://z.jp https://aa.jp … … Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate Certificate x 10,000 # ps axfo rss,cmd RSS CMD 2824 bash 512808 nginx (master) 511396 \_ nginx (worker) SIGHUP !? Certificate Certificate Certificate Certificate reproduce: https://gist.github.com/hfm/4a045a429f9303c90eac7c348d1a424a

Slide 34

Slide 34 text

Too many domains and certificates •Heavy load •Large amounts of memory

Slide 35

Slide 35 text

Two problems •IP address per domain ✓ TLS SNI extension •Web server load too many certificates

Slide 36

Slide 36 text

Two problems •IP address per domain ✓ TLS SNI extension •Web server load too many certificates ➡ Dynamic certificates

Slide 37

Slide 37 text

Dynamic certificate with ngx_mruby

Slide 38

Slide 38 text

SSL_CTX_set_cert_cb() •available in OpenSSL 1.0.2 and later •handle certificate callback function •It’s called before certificate will be used by client/server •https://www.openssl.org/docs/man1.0.2/ssl/SSL_CTX_set_cert_cb.html

Slide 39

Slide 39 text

SSL_CTX_set_cert_cb() Client Server ClientHello (server_name) ServerHello Certificate ɾ ɾ ɾ awesome_ssl_cert_handler

Slide 40

Slide 40 text

IUUQTUXJUUFSDPNNBUTVNPUPSZTUBUVT IUUQTHJUIVCDPNNBUTVNPUPSZOHY@NSVCZQVMM ngx_mruby supports it

Slide 41

Slide 41 text

mruby_ssl_handshake_handler_code # simple pattern (inline) mruby_ssl_handshake_handler_code ' ssl = Nginx::SSL.new ssl.certificate = "/path/to/#{ssl.servername}.crt" ssl.certificate_key = "/path/to/#{ssl.servername}.key" ’;

Slide 42

Slide 42 text

mruby_ssl_handshake_handler # simple pattern (file) mruby_ssl_handshake_handler /path/to/handler.rb cache; # /path/to/handler.rb ssl = Nginx::SSL.new ssl.certificate = "/path/to/#{ssl.servername}.crt" ssl.certificate_key = "/path/to/#{ssl.servername}.key"

Slide 43

Slide 43 text

mruby_ssl_handshake_handler # simple pattern (file) mruby_ssl_handshake_handler /path/to/handler.rb cache; # /path/to/handler.rb ssl = Nginx::SSL.new ssl.certificate = "/path/to/#{ssl.servername}.crt" ssl.certificate_key = "/path/to/#{ssl.servername}.key" => ngx_mruby ʹ mruby_ssl_handshake_handler Λ࣮૷ͨ͠ http://blog.hifumi.info/2016/10/03/ngx_mruby-mruby_ssl_handshake_handler/

Slide 44

Slide 44 text

Two problems •IP address per domain ✓ TLS SNI extention •Web server load too many certificates ✓ Dynamic certificates with ngx_mruby

Slide 45

Slide 45 text

The service architecture for dynamic certificate

Slide 46

Slide 46 text

Dynamic certificates architecture client application cache db 1. sent server_name 2. fetch crt/key pair from cache or db 3. proxy 2-a. set cache if missing direct pattern

Slide 47

Slide 47 text

Dynamic certificates architecture client application cache db 1. sent server_name 2. fetch crt/key pair 3. proxy internal api api pattern

Slide 48

Slide 48 text

worker handlers mruby_init_worker /path/to/mruby_init_worker.rb cache; mruby_exit_worker /path/to/mruby_exit_worker.rb cache;

Slide 49

Slide 49 text

mruby_init_worker.rb conf = Ini.load_file('config.ini') redis = Redis.new(conf['redis']['server'], 6379) Userdata.new(“redis_#{Process.pid}").redis_conn = redis mysql = MySQL::Database.new(conf[‘mysql’][‘host’], conf[‘mysql’][‘user’], conf[‘mysql’][‘pass’], conf[‘mysql’]['db']) Userdata.new(“mysql_#{Process.pid}").mysql_conn = mysql mruby_init_worker /path/to/mruby_init_worker.rb cache; mruby_exit_worker /path/to/mruby_exit_worker.rb cache;

Slide 50

Slide 50 text

Connecting with nginx workers nginx master worker worker redis mysql

Slide 51

Slide 51 text

mruby_exit_worker.rb # Close all connections redis = Userdata.new("redis_#{Process.pid}").redis_conn redis.close unless redis.nil? mysql = Userdata.new("mysql_#{Process.pid}").mysql_conn mysql.close unless mysql.nil? mruby_init_worker /path/to/mruby_init_worker.rb cache; mruby_exit_worker /path/to/mruby_exit_worker.rb cache;

Slide 52

Slide 52 text

Disconnecting with nginx workers nginx master worker worker redis mysql

Slide 53

Slide 53 text

Reconnecting with nginx workers nginx master worker worker redis mysql

Slide 54

Slide 54 text

Reconnecting with nginx workers nginx master worker worker redis mysql MySQL server has gone away × ×

Slide 55

Slide 55 text

Reconnecting with nginx workers nginx master worker worker redis mysql Reconnection def mysql_reconnect ... mysql = MySQL::Database.new(host, user, pass, db) Userdata.new("mysql_#{Process.pid}").mysql_conn = mysql end

Slide 56

Slide 56 text

Testing

Slide 57

Slide 57 text

hfm/mruby-test-mysqld •mruby porting from Test::mysqld of perl •mysqld runner for tests with mruby http://blog.hifumi.info/2016/09/06/mruby-test-mysqld/

Slide 58

Slide 58 text

Testing sample (1/2) if Object.const_defined?(:MTest) class DynCrtTest < MTest::Unit::TestCase def setup @mysqld = TestMysqld.new(database: 'foo') @mysql = MySQL::Database.new '127.0.0.1', 'root', '', 'foo', 3306 end def teardown @mysql.close @mysqld.stop end ...

Slide 59

Slide 59 text

Testing sample (2/2) if Object.const_defined?(:MTest) ... def test_dyn_crt ... end end MTest::Unit.new.run else ... end

Slide 60

Slide 60 text

IUUQXXXTMJEFTIBSFOFUITCUUFTUJOHDBTVBMUBMLT TFFBMTP

Slide 61

Slide 61 text

Summary We use ngx_mruby in production ngx_mruby supports dynamic certificates using SNI and OpenSSL 1.0.2