Practical nginx module development〜C and Lua〜

by Tatsuhiko Kubo

Slide 1

Slide 1 text

Practical nginx module development ʙC and Luaʙ Tatsuhiko Kubo@cubicdaiya YAPC::Asia TOKYO 2015

Slide 2

Slide 2 text

@cubicdaiya / Tatsuhiko Kubo • Software Engineer @ Mercari, Inc. • Infrastructure Engineering • Skills • C, Go, Lua, nginx, … • OSS developer • ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, …

Slide 3

Slide 3 text

Activities on Github

Slide 4

Slide 4 text

Other Activities IUUQNP[BJDGNQPTUOHJOY

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty

Slide 8

Slide 8 text

Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty

Slide 9

Slide 9 text

nginx • Open Source Software • HTTP server for various purpose • Static content delivery • Reverse proxy ( HTTP / FastCGI / uWSGI / … ) • Load balancer • Cache • TLS Termination / SPDY Gateway • etc… • Mail proxy ( SMTP / POP3 / IMAP ) • TCP proxy

Slide 10

Slide 10 text

nginx’s architecture • event-driven • non-blocking • asynchronous • single-threaded ( multi-threaded partially) • multi-processes ( master / wokers / …) • modular

Slide 11

Slide 11 text

nginx’s architecture • event-driven • non-blocking • asynchronous • single-threaded ( multi-threaded partially) • multi-processes ( master / wokers / …) • modular

Slide 12

Slide 12 text

nginx is module-oriented • most features are designed as module • ngx_http_core_module • ngx_http_access_module • ngx_http_log_module • ngx_http_proxy_module • ngx_http_upstream_module • etc…

Slide 13

Slide 13 text

3rd party module for nginx • nginx-user can develop an original module • It is not required to modify source code of nginx • embed original module with ‘./conﬁgure’ $ cd nginx-1.9.x $ ./configure —-add-module=${ngx_module_path}

Slide 14

Slide 14 text

Examples

Slide 15

Slide 15 text

Example • ngx_http_empty_gif_module (Ofﬁcial) • return 1x1 GIF location /gif { empty_gif; } $ curl -I http://127.0.0.1/gif HTTP/1.1 200 OK Content-Length: 43 Content-Type: image/gif ... $

Slide 16

Slide 16 text

Example • ngx_small_light • Dynamic Image Transformation for nginx small_light on; location ~ small_light[^/]*/(.+)$ { set $file $1; rewrite ^ /$file; }

Slide 17

Slide 17 text

ngx_small_light • Dynamic Image Transformation for nginx • https://github.com/cubicdaiya/ngx_small_light • Provides various • Image Manipulations ( Resize / Crop / Composite / …) • Image Processing Libraries (Image Magick / Imlib2 / GD) • Image Formats (JPEG / GIF / PNG / WEBP)

Slide 18

Slide 18 text

Transform image ■URI for resize /small_light(dw=300,dh=200)/img/image.jpg →

Slide 19

Slide 19 text

Transform image on S3 proxy from nginx to S3 small_light on; location ~ small_light[^/]*/(.+)$ { set $file $1; rewrite ^ /$file; } location /img { proxy_pass http://; }

Slide 20

Slide 20 text

Adoption case https://github.com/wantedly/nginx-image-server

Slide 21

Slide 21 text

Adoption case https://speakerdeck.com/yano3/dynamic-image-transformation-server-okara-beta

Slide 22

Slide 22 text

Example • ngx_dynamic_upstream • Dynamic Upstream for nginx location / { allow xxx.xxx.xxx.xxx deny all; dynamic_upstream; }

Slide 23

Slide 23 text

ngx_dynamic_upstream • Dynamic Upstream for nginx • https://github.com/cubicdaiya/ngx_dynamic_upstream • Provides HTTP APIs for manipulating upstream for nginx • It is useful for • zero-down-time deployment • blue green deployment

Slide 24

Slide 24 text

Zero-down-time deployment on

Slide 25

Slide 25 text

ɾɾɾ ɾɾɾ "1* "QBDIFNPE@QIQ OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN HTTPS or SPDY OHJOY OHY@EZOBNJD@VQTUSFBN HTTP Mercari frontend "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ ɾɾɾ ɾɾɾ

Slide 26

Slide 26 text

"1* "QBDIFNPE@QIQ Previous deployment "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU rsync rsync rsync

Slide 27

Slide 27 text

Pros / Cons • Pros • Very Very Very simple • Cons • many 500 error happens at deployment!!!!!

Slide 28

Slide 28 text

"1* "QBDIFNPE@QIQ Current deployment "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ

Slide 29

Slide 29 text

"1* "QBDIFNPE@QIQ Current deployment "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Current deployment "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIFNPE@QIQ

Slide 32

Slide 32 text

"1* "QBDIFNPE@QIQ Current deployment "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Current deployment "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ

Slide 36

Slide 36 text

"1* "QBDIFNPE@QIQ Current deployment "1* "QBDIFNPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream

Slide 37

Slide 37 text

Slide 38

Slide 38 text

"1* "QBDIFNPE@QIQ Current deployment "1* "QBDIFNPE@QIQ ChatOps with Slack yes OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU

Slide 39

Slide 39 text

Current deployment ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ

Slide 40

Slide 40 text

"1* "QBDIFNPE@QIQ Current deployment "1* "QBDIFNPE@QIQ ChatOps with Slack yes OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN "1* "QBDIFNPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU

Slide 41

Slide 41 text

Current deployment ChatOps with Slack yes EFQMPZCPU OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN OHJOY OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ "1* "QBDIFNPE@QIQ

Slide 42

Slide 42 text

Hello, World!

Slide 43

Slide 43 text

Hello, World! • ngx_http_hello_world • https://github.com/cubicdaiya/ngx_http_hello_world location /hello { hello_world; } $ curl http://127.0.0.1/hello Hello, World! $

Slide 44

Slide 44 text

Layout of ngx_http_hello_world ngx_http_hello_world |———... |———config |———ngx_http_hello_world_module.c └──-...

Slide 45

Slide 45 text

config • module configuration file ngx_addon_name=ngx_http_hello_world_module HTTP_MODULES="$HTTP_MODULES \ ngx_http_hello_world_module” NGX_ADDON_SRCS="$NGX_ADDON_SRCS \ $ngx_addon_dir/ngx_http_hello_world_module.c”

Slide 46

Slide 46 text

Hello, World! • ngx_http_hello_world • return “Hello, World!” location /hello { hello_world; } $ curl http://127.0.0.1/hello Hello, World! $

Slide 47

Slide 47 text

Hello, World! • ngx_http_hello_world • return “Hello, World!” location /hello { hello_world; } $ curl http://127.0.0.1/hello Hello, World! $

Slide 48

Slide 48 text

Hello, World! • ngx_http_hello_world • return “Hello, World!” location /hello { hello_world; # original directive } $ curl http://127.0.0.1/hello Hello, World! $

Slide 49

Slide 49 text

Directive declaration struct ngx_command_s { ngx_str_t name; ngx_uint_t type; char *(*set)(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); ngx_uint_t conf; ngx_uint_t offset; void *post; };

Slide 50

Slide 50 text

Directive deﬁnition(hello_world) static ngx_command_t ngx_http_hello_world_commands[] = { { ngx_string("hello_world"), NGX_HTTP_LOC_CONF|NGX_CONF_NOARGS, ngx_http_hello_world, 0, 0, NULL }, ngx_null_command };

Slide 51

Slide 51 text

Module context typedef struct { ngx_int_t (*preconfiguration)(ngx_conf_t *cf); ngx_int_t (*postconfiguration)(ngx_conf_t *cf); void *(*create_main_conf)(ngx_conf_t *cf); char *(*init_main_conf)(ngx_conf_t *cf, void *conf); void *(*create_srv_conf)(ngx_conf_t *cf); char *(*merge_srv_conf)(ngx_conf_t *cf, void *prev, void *conf); void *(*create_loc_conf)(ngx_conf_t *cf); char *(*merge_loc_conf)(ngx_conf_t *cf, void *prev, void *conf); } ngx_http_module_t;

Slide 52

Slide 52 text

Module context static ngx_http_module_t ngx_http_hello_world_module_ctx = { NULL, /* preconfiguration */ NULL, /* postconfiguration */ NULL, /* create main configuration */ NULL, /* init main configuration */ NULL, /* create server configuration */ NULL, /* merge server configuration */ NULL, /* create location configuration */ NULL /* merge location configuration */ }; In this case, all is NULL.

Slide 53

Slide 53 text

Module deﬁnition ngx_module_t ngx_http_hello_world_module = { NGX_MODULE_V1, &ngx_http_hello_world_module_ctx, /* module context */ ngx_http_hello_world_commands, /* module directives */ NGX_HTTP_MODULE, /* module type */ NULL, /* init master */ NULL, /* init module */ NULL, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ NULL, /* exit process */ NULL, /* exit master */ NGX_MODULE_V1_PADDING };

Slide 54

Slide 54 text

Hook original handler static char *ngx_http_hello_world(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { ngx_http_core_loc_conf_t *clcf; clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module); clcf->handler = ngx_http_hello_world_handler; return NGX_CONF_OK; }

Slide 55

Slide 55 text

ngx_http_hello_world_handler() #define NGX_HTTP_HELLO_WORLD “Hello, World!” static ngx_int_t ngx_http_hello_world_handler(ngx_http_request_t *r) { ngx_int_t rc; ngx_chain_t out; ngx_buf_t *b; ngx_str_t body = ngx_string(NGX_HTTP_HELLO_WORLD); if (r->method != NGX_HTTP_GET && r->method != NGX_HTTP_HEAD) { return NGX_HTTP_NOT_ALLOWED; } if (r->headers_in.if_modified_since) { return NGX_HTTP_NOT_MODIFIED; } b = ngx_pcalloc(r->pool, sizeof(ngx_buf_t)); if (b == NULL) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } b->pos = body.data; b->last = b->pos + body.len; b->memory = 1; b->last_buf = 1; out.buf = b; out.next = NULL; ngx_str_set(&r->headers_out.content_type, "text/plain"); r->headers_out.status = NGX_HTTP_OK; r->headers_out.content_length_n = body.len; rc = ngx_http_send_header(r); if (rc == NGX_ERROR || rc > NGX_OK || r->header_only) { return rc; }

Slide 56

Slide 56 text

Embed ngx_http_hello_world into nginx $ ./confgiure \ —-add-module=../ngx_http_hello_world $ make $ sudo make install

Slide 57

Slide 57 text

Hook details…

Slide 58

Slide 58 text

Request processing phases in nginx TFSWFSSFXSJUF QPTUBDDFTT MPH pOEDPOpH SFXSJUF QPTUSFXSJUF QSFBDDFTT USZpMFT DPOUFOU Request BDDFTT try_ﬁles processing access-control response body generation logging rewrite in server ﬁnd location rewrite in location

Slide 59

Slide 59 text

Phase deﬁnitions typedef enum { NGX_HTTP_POST_READ_PHASE = 0, NGX_HTTP_SERVER_REWRITE_PHASE, NGX_HTTP_FIND_CONFIG_PHASE, NGX_HTTP_REWRITE_PHASE, NGX_HTTP_POST_REWRITE_PHASE, NGX_HTTP_PREACCESS_PHASE, NGX_HTTP_ACCESS_PHASE, NGX_HTTP_POST_ACCESS_PHASE, NGX_HTTP_TRY_FILES_PHASE, NGX_HTTP_CONTENT_PHASE, NGX_HTTP_LOG_PHASE } ngx_http_phases;

Slide 60

Slide 60 text

Hook handler to speciﬁed phases ngx_http_handler_pt *h; ngx_http_core_main_conf_t *cmcf; cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module); h = ngx_array_push(&cmcf->phases[NGX_HTTP_ACCESS_PHASE].handlers); if (h == NULL) { return NGX_ERROR; } *h = ngx_http_xxx_handler;

Slide 61

Slide 61 text

Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty

Slide 62

Slide 62 text

Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty

Slide 63

Slide 63 text

nginx API • memory-pool • string • array • hash-table • time • list • regular-expression • temporary-ﬁle • shared-memory • etc…

Slide 64

Slide 64 text

essence of nginx API programming • understand nginx architecture • event-driven • non-blocking • asynchronous • single-threaded • multi-process

Slide 65

Slide 65 text

essence of nginx API programming • memory management strategies • memory-pool • slab ( for shared-memory) • string with length • string is not always NULL-terminated

Slide 66

Slide 66 text

essence of nginx API programming • structs for HTTP • ngx_http_request_t • ngx_http_headers_in • ngx_http_headers_out • etc… ( See nginx/src/http/ngx_http_request.h )

Slide 67

Slide 67 text

Move to terminal…

Slide 68

Slide 68 text

Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty

Slide 69

Slide 69 text

Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty

Slide 70

Slide 70 text

Test::Nginx https://github.com/openresty/test-nginx

Slide 71

Slide 71 text

Test::Nginx • Testing framework for nginx • https://github.com/openresty/test-nginx • We are able to test nginx’s behavior with CLI • Install from CPAN $ cpanm Test::Nginx

Slide 72

Slide 72 text

test nginx’s behavior with Test::Nginx use Test::Nginx::Socket; plan tests => repeat_each() * 2 * blocks(); run_tests(); __DATA__ === TEST 1: hello world --- config location /hello { return 200 "Hello, World!\n"; } --- request GET /hello --- response_body Hello, World! --- error_code: 200

Slide 73

Slide 73 text

Slide 74

Slide 74 text

Slide 75

Slide 75 text

By the way, • C programming is difﬁcult and complicated • productivity is not high If scripting language is available on nginx, productivity would improve dramatically.

Slide 76

Slide 76 text

Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty

Slide 77

Slide 77 text

Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty

Slide 78

Slide 78 text

ngx_lua https://github.com/openresty/lua-nginx-moduleʙ

Slide 79

Slide 79 text

ngx_lua • embed Lua/LuaJIT into nginx • Lua scripting in nginx.conf • Hooking request processing phase with Lua • wrap nginx API with Lua • non-blocking APIs (e.g. ngx.location.capture())

Slide 80

Slide 80 text

• set_by_lua / set_by_lua_file • rewrite_by_lua / rewrite_by_lua_file • access_by_lua / access_by_lua_file • content_by_lua / content_by_lua_file • log_by_lua / log_by_lua_file Hook directives for request processing phases

Slide 81

Slide 81 text

Slide 82

Slide 82 text

Hook directives for request processing phases • set_by_lua / set_by_lua_file • rewrite_by_lua / rewrite_by_lua_file • access_by_lua / access_by_lua_file • content_by_lua / content_by_lua_file • log_by_lua / log_by_lua_file run Lua script-file

Slide 83

Slide 83 text

content_by_lua location /hello { content_by_lua “ngx.say(‘Hello, World!’)”; } ■ nginx.conf $ curl -s http://127.0.0.1/hello Hello, World! $ ■ run

Slide 84

Slide 84 text

content_by_lua_ﬁle location /hello { content_by_lua_file /etc/nginx/lua/hello.lua; } ■ nginx.conf ■ /etc/nginx/lua/hello.lua ngx.say(“Hello, World!”)

Slide 85

Slide 85 text

set_by_lua $result location /yapc { set $conference YAPC::Asia; set $location Tokyo; set $year 2015; set_by_lua $result ‘return ngx.var.conference .. “ ” .. ngx.var.location .. “ ” .. ngx.var.year ‘; content_by_lua ‘ngx.say(ngx.var.result)’; } ■ nginx.conf $ curl -s http://127.0.0.1/hello YAPC::Asia Tokyo 2015 $ ■ run

Slide 86

Slide 86 text

Slide 87

Slide 87 text

Other hook directives and phases IPPLOBNF QIBTF JOJU@CZ@MVB JOJU@CZ@MVB@pMF MPBEJOHOHJOYDPOG JOJU@XPSLFS@CZ@MVB JOJU@XPSLFS@CZ@MVB@pMF TUBSUJOHXPSLFSQSPDFTT IFBEFS@pMUFS@CZ@MVB IFBEFS@pMUFS@CZ@MVB@pMF pMUFSJOHIFBEFS CPEZ@pMUFS@CZ@MVB CPEZ@pMUFS@CZ@MVB@pMF pMUFSJOHCPEZ

Slide 88

Slide 88 text

ngx_lua APIs • ngx.say() / ngx.exit() / ngx.log / … • ngx.var.VARIABLE • ngx.shared.DICT • ngx.header.HEADER • etc…

Slide 89

Slide 89 text

ngx.say() ngx.say(“Hello, World!”) • output string to response body

Slide 90

Slide 90 text

ngx.exit() ngx.exit(ngx.HTTP_OK) • return HTTP status code

Slide 91

Slide 91 text

HTTP status code in ngx_lua • ngx.HTTP_OK (200) • ngx.HTTP_CREATED (201) • ngx.HTTP_MOVED_PERMANENTLY (301) • ngx.HTTP_MOVED_TEMPORARILY (302) • ngx.HTTP_NOT_MODIFIED (304) • ngx.HTTP_BAD_REQUEST (400) • ngx.HTTP_INTERNAL_SERVER_ERROR (500) • etc…

Slide 92

Slide 92 text

ngx.log() ngx.exit(ngx.ERR, “some error”) • output message to error.log • $arg1: log-level, $arg2: message

Slide 93

Slide 93 text

log-level in ngx_lua • ngx.EMERG • ngx.ALERT • ngx.CRIT • ngx.ERR • ngx.WARN • ngx.NOTICE • ngx.INFO • ngx.DEBUG

Slide 94

Slide 94 text

log-level in ngx_lua • ngx.EMERG • ngx.ALERT • ngx.CRIT • ngx.ERR • ngx.WARN • ngx.NOTICE • ngx.INFO • ngx.DEBUG log-level High Low

Slide 95

Slide 95 text

ngx.var.VARIABLE • read / overwrite nginx variables location / { set $fn “Tatsuhiko Kubo”; set $hn “@cubicdaiya”; content_by_lua ‘ local body = ngx.var.fn .. ngx.var.hn ngx.say(body) —- Tatsuhiko Kubo@cubicdaiya ‘; }

Slide 96

Slide 96 text

ngx.var.VARIABLE • creating new variable is not allowed location / { content_by_lua ‘ ngx.var.fn = “Tatsuhiko Kubo” ngx.var.hn = “@cubicdaiya” ‘; } runtime error: … variable “fn” not found for writing; maybe it is a built-in variable that is not changeable or … ɾ error.log

Slide 97

Slide 97 text

ngx.shared.DICT • dictionary with shared memory http { lua_shared_dict yapc 1m; init_by_lua ‘ ngx.shared.yapc:set(“presentator”, “@cubicdaiya”) ’; … server { … location / { content_by_lua ‘ ngx.say(ngx.shared.yapc:get(“presentator”)) ’; } } }

Slide 98

Slide 98 text

ngx.header.HEADER • response header reference and manipulation location / { content_by_lua ‘ local body = “{ \\”presentator\\” : \\”@cubicdaiya\\” }” —- assign application/json to Content-Type ngx.header.content_type = “application/json” ngx.say(body) ‘; }

Slide 99

Slide 99 text

Header reference and manipulation • ngx.req.get_headers() • ngx.req.set_header() • ngx.req.clear_header() • … • ngx.resp.get_headers() • ngx.header.HEADER • … ■ For request-header ■ For response-header

Slide 100

Slide 100 text

ngx.time() • return current timestamp local timestamp = ngx.time()

Slide 101

Slide 101 text

ngx_lua time APIs • ngx.time() • ngx.today() • ngx.now() • ngx.update_time() • ngx.parse_http_time() • ngx.localtime() • ngx.utctime() • ngx.cookie_time() • ngx.http_time()

Slide 102

Slide 102 text

Use ngx.time() instead of os.time() • os.time() always calls gettimeofday() • ngx.time() uses timer-cache of nginx • avoid system-call’s overhead • ngx.time() is more efﬁcient than os.time()

Slide 103

Slide 103 text

ngx.re.match() • Regular Expression matching • powered by PCRE local matches, err = ngx.re.match(“image/webp”, “([^/]+)\/(.+)”) -— matches[0] -> image/webp -— matches[1] -> image -— matches[2] -> webp

Slide 104

Slide 104 text

ngx.re.* (Regular Expression API) • ngx.re.ﬁnd() • ngx.re.match() • ngx.re.gmatch() • ngx.re.sub() • ngx.re.gsub()

Slide 105

Slide 105 text

• Use PCRE options(e.g. o, j) • o -> enable compile-once • j -> enable PCRE-JIT Optimize ngx.re.* local matches, err = ngx.re.match(“image/webp”, “([^/]+)\/(.+)”, “oj”)

Slide 106

Slide 106 text

• ngx.re is based PCRE • Powerful & Fat • Lua builtin-re(regular expression) • Simple & Light ngx.re.* VS Lua builtin-re If ‘re’ is simple, Lua builtin-re is faster.

Slide 107

Slide 107 text

ngx.location.capture() • send non-blocking sub-request to other location location /external { internal; proxy_pass http://external; } location / { content_by_lua ‘ —— non-blocking behavior local res = ngx.location.capture(“/external”) … ‘; }

Slide 108

Slide 108 text

By the way, • ngx_lua is very powerful • But setup and deployment is a little painful • Lua / LuaJIT instalation • Lua packages management

Slide 109

Slide 109 text

Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty

Slide 110

Slide 110 text

Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty

Slide 111

Slide 111 text

OpenResty http://openresty.org/

Slide 112

Slide 112 text

OpenResty • Web application framework bundled • nginx and 3rd party modules (e.g. ngx_lua) • Lua, LuaJIT • Resty modules(lua-resty-xxx…) • All-in-one superset of nginx

Slide 113

Slide 113 text

3rd party modules • lua-nginx-module (ngx_lua) • echo-nginx-module • headers-more-nginx-module • etc…

Slide 114

Slide 114 text

Resty modules • lua-resty-core • resty-cli • lua-resty-mysql / redis / memcached • etc…

Slide 115

Slide 115 text

lua-resty-core • pure Lua partial-re-implementation of ngx_lua APIs • accelerate ngx_lua APIs with LuaJIT • because ngx_lua APIs with C is not JIT compiled http { # accelerate ngx_lua APIs init_by_lua ‘require resty.core’; … }

Slide 116

Slide 116 text

lua-resty-cli • command-line utility for OpenResty • run OpenResty with CLI • all APIs are not always enabled $ resty -e ‘ngx.say(“Hello, World!”)’ Hello, World! $

Slide 117

Slide 117 text

lua-resty-mysql / redis / memcached • client drivers for mysql / redis / memcached • non-blocking behavior • Powered by ngx.socket

Slide 118

Slide 118 text

OpenResty on

Slide 119

Slide 119 text

PascalʙMercari app-event analysis baseʙ 0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF#JH2VFSZ Developer Data Sientist Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events transfer utilize events utilize events transfer

Slide 120

Slide 120 text

OpenResty’s role on Pascal • High performance event logger • receive events as JSON from devices • parse JSON • utilize events for BigQuery • log events into ﬁle every event-channel

Slide 121

Slide 121 text

References • The Architecture of Open Source Applications (Volume 2): nginx • http://www.aosabook.org/en/nginx.html • Nginx Internals • http://www.slideshare.net/joshzhu/nginx-internals • Emiller’s Guide To Nginx Module Development • http://www.evanmiller.org/nginx-modules-guide.html

Slide 122

Slide 122 text

References • openresty/lua-nginx-module • https://github.com/openresty/lua-nginx-module • Using ngx_lua / lua-nginx-module in pixiv • http://www.slideshare.net/harukayon/ngx-lua-public • ngx_mruby and ngx_lua • http://qiita.com/cubicdaiya/items/3913d3dddf6999984c5b

Slide 123

Slide 123 text

Thanks!