Design pattern for embedding mruby into middleware

Slide 1

Slide 1 text

A design and implementation to connect middleware supporting Internet service with mruby Ryosuke Matsumoto / Pepabo R&D Institute, GMO Pepabo, Inc. 2018.06.02 RubyKaigi2018 Design pattern for embedding mruby into middleware

Slide 2

Slide 2 text

• Chief engineer at GMO Pepabo, inc. • Chief researcher at Pepabo R&D Institute • Ph.D of Informatics, Kyoto University • my interests: Operation technology, Internet foundation technology, System programming, Highly integrated multi- tenant architecture, Hosting service, Cloud platform, C, mruby, Rust • Engineering => Technology <= Research 2 Ryosuke Matsumoto @matsumotory

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

1. Introduction of embedding mruby into middleware 2. Design pattern for middleware 1. init / exit 2. each request 3. non-blocking 3. Conclusion 5 Table of Contents

Slide 6

Slide 6 text

1. Introduction of embedding mruby into middleware

Slide 7

Slide 7 text

• System management for network, OS and middleware • Hosting service, Cloud platform service • operation technology for middleware written by C • Apache, nginx, dovecot, postﬁx, sendmail, bind, proftp… • want to write operation tools for middleware using perl • want to write extension module for middleware • each middleware dependency • Apache module, nginx module, dovecot plugin…. 7 I’m a former web operation engineer

Slide 8

Slide 8 text

met mruby in April 2012

Slide 9

Slide 9 text

mruby / CRuby for middleware 9 NJEEMFXBSFJO$ NSVCZ NJEEMFXBSFNFUIPE JONSVCZ NJEEMFXBSFGVODUJPOTJO$ NJEEMFXBSFJO$3VCZ NJEEMFXBSFNFUIPE JO$3VCZ NSVCZGPSNJEEMFXBSF $3VCZGPSNJEEMFXBSF

Slide 10

Slide 10 text

middleware conﬁguration as code

Slide 11

Slide 11 text

basically embed mruby into middleware 11 NJEEMFXBSFQSPDFTT NSC@TUBUFNSVCZ NSVCZCZUFDPEF NJEEMFXBSFGVODUJPOTJO$ NSVCZ4DSJQUT lNJEEMFXBSFzXBTJNQMFNFOUFECZ$JOUIJTQSFTFOUBUJPO NBTUFS TFSWBOU

Slide 12

Slide 12 text

• mod_mruby / ngx_mruby (http server extension) • dovecot-mruby-plugin (imap server extension) • pmilter (milter server) • Trusterd HTTP/2 Web Server • h2o_mruby (Initial implementation with v1.4) • other system tools: rcon, capcon, pfds, virtualing, ab-mruby • developed the greatest number of mrbgem in mgem-list • also received Ph.D of informatics 12 my developed middleware with mruby

Slide 13

Slide 13 text

• middleware behavior scripting with mruby • middleware conﬁguration as code • limit Ruby methods by mruby • dynamically control middleware by session parameters • cooperate with microservices • record a session data and utilize the data • implement Service Mesh functions • HTTP(S)ɾTCP/UDP stream 13 Why embed mruby into middleware

Slide 14

Slide 14 text

• consider several viewpoints • memory management (allocate/free/leak) • performance • usability, operation / development technology • strategy for multi process / multi threading • ʮmruby = mrb_stateʯ in this presentation 14 How embed mruby into middleware

Slide 15

Slide 15 text

Basic multi threading strategy

Slide 16

Slide 16 text

Concurrency with native thread

Slide 17

Slide 17 text

multi-threading strategy for mruby 17 QSPDFTT UISFBE NSC@TUBUF CZUFDPEF UISFBE NSC@TUBUF CZUFDPEF QSPDFTT NSC@TUBUF UISFBE CZUFDPEF UISFBE CZUFDPEF QSPDFTT NSC@TUBUF CZUFDPEF UISFBE UISFBE

Slide 18

Slide 18 text

multi-threading strategy for mruby 18 QSPDFTT UISFBE NSC@TUBUF CZUFDPEF UISFBE NSC@TUBUF CZUFDPEF QSPDFTT NSC@TUBUF UISFBE CZUFDPEF UISFBE CZUFDPEF QSPDFTT NSC@TUBUF CZUFDPEF UISFBE UISFBE MPDL MPDL

Slide 19

Slide 19 text

• The basic strategy is embedding mruby into each thread • lock between threads in single mrb_state is complicated • A cost of creating mrb_state is much higher than creating thread • The cost becomes bottleneck each request to middleware • We want to create mrb_state on init phase of middleware • control only bytecode each request 19 considerations

Slide 20

Slide 20 text

• talk about design patterns for embedding mruby into middleware • try and error over and over again for 6 years • consider several viewpoints • memory management • performance • usability, operation / development technology 20 In this presentation

Slide 21

Slide 21 text

2. Design pattern for middleware

Slide 22

Slide 22 text

1. init / exit

Slide 23

Slide 23 text

init: independent mrb_state 23 NBTUFSQSPDFTT XPSLFSQSPDFTT NSC@TUBUF NSC@TUBUF CZUFDPEF CZUFDPEF HFOFSBUFDPEFBOESVOPONBTUFSJOJU GPSL DSFBUFNSC@TUBUFPONBTUFSJOJU NSC@TUBUF CZUFDPEF HFOFSBUFDPEFBOESVOPOXPSLFSJOJU DSFBUFNSC@TUBUFPOXPSLFSJOJU NVMUJQSPDFTTNPEFM

Slide 24

Slide 24 text

init: independent mrb_state 24 NBTUFSQSPDFTT XPSLFSQSPDFTT NSC@TUBUF NSC@TUBUF CZUFDPEF CZUFDPEF GSFFDPEFPONBTUFSJOJU GPSL DMPTFNSC@TUBUFPONBTUFSJOJU NSC@TUBUF CZUFDPEF GSFFDPEFPOXPSLFSJOJU DMPTFNSC@TUBUFPOXPSLFSJOJU NVMUJQSPDFTTNPEFM GSFF GSFF GSFF OPUIJOHUPEPPOXPSLFSBOENBTUFSFYJU

Slide 25

Slide 25 text

• Advantages • mruby scripts are executed completely independently • can save memory • Disadvantages • can’t pass objects to a request phase • can’t share objects with each other script 25 init: independent mrb_state

Slide 26

Slide 26 text

init: share mrb_state 26 NBTUFSQSPDFTT XPSLFSQSPDFTT NSC@TUBUF NSC@TUBUF CZUFDPEF CZUFDPEF CZUFDPEF HFOFSBUFDPEFBOESVOPONBTUFSJOJU POMZHFOFSBUFDPEFPOXPSLFSJOJU GPSL DSFBUFNSC@TUBUFPONBTUFSJOJU NVMUJQSPDFTTNPEFM

Slide 27

Slide 27 text

init: share mrb_state 27 NBTUFSQSPDFTT XPSLFSQSPDFTT NSC@TUBUF NSC@TUBUF CZUFDPEF CZUFDPEF CZUFDPEF GSFFDPEFPONBTUFSJOJU GPSL DMPTFNSC@TUBUFPONBTUFSJOJU NVMUJQSPDFTTNPEFM GSFFDPEFPOXPSLFSJOJU DMPTFNSC@TUBUFPOXPSLFSJOJU OPUIJOHUPEPPOXPSLFSBOENBTUFSFYJU

Slide 28

Slide 28 text

• Advantages • decrease a cost of creating mrb_state each script • can save memory • can share objects with each other script on init phase • Disadvantages • can’t pass objects to a request phase 28 init: share mrb_state

Slide 29

Slide 29 text

init: share mrb_state with request phase 29 NBTUFSQSPDFTT XPSLFSQSPDFTT NSC@TUBUF NSC@TUBUF CZUFDPEF CZUFDPEF CZUFDPEF HFOFSBUFDPEFBOESVOPONBTUFSJOJU POMZHFOFSBUFDPEFPOXPSLFSJOJU SFVTFUIFNSC@TUBUFFBDISFRVFTU GPSL DSFBUFNSC@TUBUFPONBTUFSJOJU NVMUJQSPDFTTNPEFM

Slide 30

Slide 30 text

exit: share mrb_state with request phase 30 NBTUFSQSPDFTT XPSLFSQSPDFTT NSC@TUBUF NSC@TUBUF CZUFDPEF CZUFDPEF CZUFDPEF GSFFDPEFPONBTUFSFYJU GPSL DMPTFNSC@TUBUFPONBTUFSFYJU NVMUJQSPDFTTNPEFM GSFFDPEFPOXPSLFSFYJU DMPTFNSC@TUBUFPOXPSLFSFYJU

Slide 31

Slide 31 text

• Advantages • decrease a cost of creating mrb_state each script • can share objects with each other script on init phase • can pass objects to a request phase • use DB object which connect the DB at init phase • Disadvantages • use more memory than other init pattern • complicated memory management each request 31 init: share mrb_state with request phase

Slide 32

Slide 32 text

2. each request

Slide 33

Slide 33 text

multi process model

Slide 34

Slide 34 text

XPSLFSQSPDFTT request: independent mrb_state 34 XPSLFSQSPDFTT NSC@TUBUF CZUFDPEF NSC@TUBUF CZUFDPEF 3FRVFTU XPSLFSQSPDFTT 3FRVFTU 3FTQPOTF 3FTQPOTF NSC@TUBUF CZUFDPEF 3FRVFTU NSC@TUBUF CZUFDPEF NSC@TUBUF CZUFDPEF GSFF GSFF DSFBUFNSC@TUBUFBOEHFOFSBUFDPEF BOESVOCZUFDPEFFBDISFRVFTU *OJU 3FRVFTU 3FTQPOTF

Slide 35

Slide 35 text

• Advantages • mruby scripts are executed completely independently • can manage memory easily and correctly • change script each request • Disadvantages • low performance because of creating mrb_state each request • can’t share objects with each other script each request 35 request: independent mrb_state

Slide 36

Slide 36 text

request: shared mrb_state 36 XPSLFSQSPDFTT XPSLFSQSPDFTT 3FRVFTU XPSLFSQSPDFTT 3FRVFTU 3FTQPOTF 3FTQPOTF *OJU 3FRVFTU 3FTQPOTF NSC@TUBUF NSC@TUBUF CZUFDPEF CZUFDPEF NSC@TUBUF CZUFDPEF CZUFDPEF CZUFDPEF 3FRVFTU BSFOB FYD BSFOB FYD BSFOB FYD DMFBOVQBSFOBBOEFYD PCKFDUTFBDISFRVFTU GSFF GSFF QBSTFTDSJQUBOEHFOFSBUFBOE SVOCZUFDPEFFBDISFRVFTU DSFBUFNSC@TUBUF POJOJUJBMJ[BUJPO

Slide 37

Slide 37 text

• Advantages • can share objects with each other script each request and init phase • change script each request • Disadvantages • slightly low performance because of compiling • memory management is very complicated because of compiling and freeing bytecode each request while sharing mrb_state 37 request: independent mrb_state

Slide 38

Slide 38 text

request: shared mrb_state and bytecode 38 XPSLFSQSPDFTT XPSLFSQSPDFTT 3FRVFTU XPSLFSQSPDFTT 3FTQPOTF *OJU 3FRVFTU 3FTQPOTF NSC@TUBUF CZUFDPEF CZUFDPEF CZUFDPEF NSC@TUBUF CZUFDPEF CZUFDPEF CZUFDPEF NSC@TUBUF CZUFDPEF CZUFDPEF CZUFDPEF 3FRVFTU BSFOB FYD BSFOB FYD BSFOB FYD DMFBOVQBSFOBBOEFYDPCKFDUT FBDISFRVFTU GSFFDPEFDPOUFYU FBDISFRVFTU DSFBUFNSC@TUBUF QBSTFTDSJQUBOEHFOFSBUF CZUFDPEFPOJOJUJBMJ[BUJPO SVOCZUFDPEFFBDISFRVFTU

Slide 39

Slide 39 text

• Advantages • high performance • can share objects with each other script each request and init phase • Disadvantages • can’t change script each request • memory management is a bit complicated 39 request: independent mrb_state

Slide 40

Slide 40 text

Performance for request pattern 40

Slide 41

Slide 41 text

multi threading model

Slide 42

Slide 42 text

multi-threading strategy for mruby 42 QSPDFTT UISFBE NSC@TUBUF CZUFDPEF UISFBE NSC@TUBUF CZUFDPEF QSPDFTT NSC@TUBUF UISFBE CZUFDPEF UISFBE CZUFDPEF QSPDFTT NSC@TUBUF CZUFDPEF UISFBE UISFBE

Slide 43

Slide 43 text

XPSLFSQSPDFTT request: independent mrb_state 43 XPSLFSQSPDFTT XPSLFSQSPDFTT DSFBUFNSC@TUBUFBOEHFOFSBUFCZUFDPEF BOESVOCZUFDPEFFBDISFRVFTU XPSLFSUISFBE XPSLFSUISFBE XPSLFSUISFBE NSC@TUBUF CZUFDPEF XPSLFSUISFBE XPSLFSUISFBE NSC@TUBUF CZUFDPEF XPSLFSUISFBE 3FRVFTU 3FTQPOTF NSC@TUBUF CZUFDPEF 3FRVFTU GSFF *OJU 3FRVFTU 3FTQPOTF NSC@TUBUF CZUFDPEF 3FRVFTU

Slide 44

Slide 44 text

request: share mrb_state 44 XPSLFSQSPDFTT *OJU 3FRVFTU 3FTQPOTF XPSLFSQSPDFTT XPSLFSQSPDFTT XPSLFSUISFBE NSC@TUBUF BSFOB FYD XPSLFSUISFBE NSC@TUBUF BSFOB FYD XPSLFSUISFBE NSC@TUBUF CZUFDPEF BSFOB FYD XPSLFSUISFBE NSC@TUBUF BSFOB FYD XPSLFSUISFBE NSC@TUBUF CZUFDPEF BSFOB FYD XPSLFSUISFBE NSC@TUBUF CZUFDPEF BSFOB FYD 3FTQPOTF 3FRVFTU 3FRVFTU DMFBOVQ BSFOBBOEFYD PCKFDUTFBDI SFRVFTU GSFF QBSTFTDSJQU BOEHFOFSBUF CZUFDPEFBOE SVOCZUFDPEF FBDISFRVFTU CZUFDPEF 3FRVFTU DSFBUFNSC@TUBUFPOJOJUJBMJ[BUJPO

Slide 45

Slide 45 text

request: share mrb_state and bytecode 45 XPSLFSQSPDFTT *OJU 3FRVFTU 3FTQPOTF XPSLFSQSPDFTT XPSLFSQSPDFTT XPSLFSUISFBE NSC@TUBUF CZUFDPEF BSFOB FYD CZUFDPEF XPSLFSUISFBE NSC@TUBUF CZUFDPEF BSFOB FYD CZUFDPEF XPSLFSUISFBE NSC@TUBUF CZUFDPEF BSFOB FYD CZUFDPEF XPSLFSUISFBE NSC@TUBUF CZUFDPEF BSFOB FYD CZUFDPEF XPSLFSUISFBE NSC@TUBUF CZUFDPEF BSFOB FYD CZUFDPEF XPSLFSUISFBE NSC@TUBUF CZUFDPEF BSFOB FYD CZUFDPEF 3FTQPOTF 3FRVFTU 3FRVFTU DMFBOVQ BSFOBBOEFYD PCKFDUTFBDI SFRVFTU 3FRVFTU GSFFDPEFDPOUFYU FBDISFRVFTU DSFBUFNSC@TUBUF QBSTFTDSJQUBOEHFOFSBUFCZUFDPEFPOJOJUJBMJ[BUJPO SVOCZUFDPEF FBDISFRVFTU

Slide 46

Slide 46 text

• multi process model • share mrb_state and bytecode • mod_mruby, ngx_mruby, postﬁx-mruby, dovecot- mruby, trusterd HTTP/2 server • multi threading model • share mrb_state and bytecode • H2O, nghttpd • independent mrb_state • pmilter 46 Summary: each request pattern

Slide 47

Slide 47 text

3. non-blocking

Slide 48

Slide 48 text

• want to accept as many requests simultaneously as possible in single process/thread (C10k) • want to use multi cpu core sufﬁciently • process blocking operation in non-blocking mode • File I/O, Network I/O, sleep… • monitor I/O state and notify status change like epoll() • generally implement suitable event loop for middleware 48 non-blocking middleware

Slide 49

Slide 49 text

• mruby blocks middleware processing generally • bottleneck of performance when accepting multiple requests simultaneously • Other responses are delayed in proportion to the time of processing of mruby blocking 49 non-blocking middleware with mruby

Slide 50

Slide 50 text

blocking each request with mruby 50 SFRVFTUQSPDFTTJOH NSVCZ NSVCZ SFRVFTUQSPDFTTJOH SFRVFTUQSPDFTTJOH NSVCZ DSFBUFSFTQPOTF DSFBUFSFTQPOTF DSFBUFSFTQPOTF TFOESFTQPOTF OPOCMPDLJOHNJEEMFXBSFMJLFOHJOYJOTJOHMFQSPDFTT SFDWSFRVFTU BUUIFTBNFUJNF

Slide 51

Slide 51 text

blocking each request with mruby 51 SFRVFTU NSVCZ NSVCZ SFTQPOTF SFRVFTU SFRVFTU SFTQPOTF SFTQPOTF NSVCZ TFOESFTQPOTF SFDWSFRVFTU BUUIFTBNFUJNF Other responses are delayed in proportion to the time of processing of mruby blocking OPOCMPDLJOHNJEEMFXBSFMJLFOHJOYJOTJOHMFQSPDFTT

Slide 52

Slide 52 text

Slide 53

Slide 53 text

non-blocking each request with mruby 53 SFRVFTU SFTQPOTF SFRVFTU SFRVFTU SFTQPOTF SFTQPOTF TFOESFTQPOTF SFDWSFRVFTU BUUIFTBNFUJNF CMPDLJOH PQFSBJUPO NSVCZ CMPDLJOH PQFSBJUPO NSVCZ NSVCZ CMPDLJOH PQFSBJUPO OPOCMPDLJOHNJEEMFXBSFMJLFOHJOYJOTJOHMFQSPDFTT

Slide 54

Slide 54 text

Slide 55

Slide 55 text

• change a blocking method of mruby to non-blocking • suspend mruby processing until the blocking method is completed • return to event loop of middleware from mruby • resume mruby processing by callback func in event loop 55 non-blocking middleware with mruby

Slide 56

Slide 56 text

3VCZ4DSJQUʢ3VCZXPSMEʣ 1SPD 'JCFSSFTVNF non-blocking each request with mruby 56 NJEEMFXBSF $XPSME NSC@TUBUF 'JCFS 3VCZCZUFDPEF JODMVEFECMPDLJOH NFUIPE DGVODPGCMPDLJOHNFUIPE SVOCMPDLJOHPQFSBUJPOXJUIOPOCMPDLJOHNPEF SVONSC@pCFS@ZJFME GSPN$BOETFUDBMMCBDL DGVODUPFWFOUMPPQPGNJEEMFXBSF DGVODPGNJEEMFXBSFPOFWFOUMPPQ QSPDFTTPUIFSSFRVFTUPOFWFOUMPPQ DBMMCBDLDGVOD SVOSFTVNFCZSVOOJOHQSPDPCKFDUGSPN$ DGVODPGNJEEMFXBSFXJUINSC@TUBUF SVOQSPDPCKFDUGSPN$ $VSSFOUNSVCZOPOCMPDLJOHNPEFM 3FRVFTU

Slide 57

Slide 57 text

My homework of RubyKaigi 2018 57 Show me your ideal design of non-blocking model using Fiber for middleware embedding mruby.

Slide 58

Slide 58 text

3VCZ4DSJQUʢ3VCZXPSMEʣ non-blocking each request with mruby 58 NJEEMFXBSF $XPSME NSC@TUBUF 3VCZCZUFDPEFJODMVEFE CMPDLJOHNFUIPEBOE'JCFS DGVODPGCMPDLJOHNFUIPE SVOCMPDLJOHPQFSBUJPOXJUIOPOCMPDLJOHNPEF SVONSC@pCFS@ZJFME GSPN$BOETFUDBMMCBDL DGVODUPFWFOUMPPQPGNJEEMFXBSF DGVODPGNJEEMFXBSFPOFWFOUMPPQ QSPDFTTPUIFSSFRVFTUPOFWFOUMPPQ DBMMCBDLDGVOD SVOSFTVNFCZNSC@pCFS@SFTVNF GSPN$ SFTFU'JCFSDPOUFYUUP DGVODPGNJEEMFXBSFXJUINSC@TUBUF SVOpCFSPCKFDU CZNSC@pCFS@SFTVNF GSPN$ NZ*EFBMNSVCZOPOCMPDLJOHNPEFM 3FRVFTU $VSSFOUNSVCZDBUDIFTTFHGBVMU

Slide 59

Slide 59 text

non-blocking sleep example of ngx_mruby v2

Slide 60

Slide 60 text

non-blocking sleep sample script 60

Slide 61

Slide 61 text

non-blocking sleep sample script 61

Slide 62

Slide 62 text

wrapped Ruby script by Fiber 62

Slide 63

Slide 63 text

wrapped Ruby script by ﬁber 63

Slide 64

Slide 64 text

run ﬁber object 64 SFUVSOUPFWFOUMPPQXIFOpCFSJTBMJWJOH

Slide 65

Slide 65 text

blocking method (Nginx::Async.sleep) 65 QSPDDFTTPUIFSSFRVFTUT VOUJMDBMMCBDLGVODUJPO

Slide 66

Slide 66 text

callback handler from event loop 66

Slide 67

Slide 67 text

now supports non-blocking http[s] client of ngx_mruby v2 in RubyKaigi 2018 !!

Slide 68

Slide 68 text

Slide 69

Slide 69 text

Special thanks for v2 developers 69 !QZBNB !TZV@DSFBN

Slide 70

Slide 70 text

3. Conclusion

Slide 71

Slide 71 text

• Why / How embed mruby into middleware • connect services dynamically and programmably • design pattern for embedding mruby into middleware • init /exit • each request • blocking / non-blocking • We will release ngx_mruby v2 in the near future • Happy hacking mruby! 71 Conclusion