Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practical nginx module development〜C and Lua〜

Practical nginx module development〜C and Lua〜

Tatsuhiko Kubo

August 22, 2015
Tweet

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Transcript

  1. Practical nginx module development
    ʙC and Luaʙ
    Tatsuhiko [email protected]
    YAPC::Asia TOKYO 2015

    View Slide

  2. @cubicdaiya / Tatsuhiko Kubo
    • Software Engineer @ Mercari, Inc.
    • Infrastructure Engineering
    • Skills
    • C, Go, Lua, nginx, …
    • OSS developer
    • ngx_small_light, ngx_dynamic_upstream,
    nginx-build, slackboard, cachectl, gaurun, …

    View Slide

  3. Activities on Github

    View Slide

  4. Other Activities
    IUUQNP[BJDGNQPTUOHJOY

    View Slide

  5. View Slide

  6. View Slide

  7. Agenda
    • nginx http module development with C
    • nginx API programming
    • Test::Nginx
    • ngx_lua
    • OpenResty

    View Slide

  8. Agenda
    • nginx http module development with C
    • nginx API programming
    • Test::Nginx
    • ngx_lua
    • OpenResty

    View Slide

  9. nginx
    • Open Source Software
    • HTTP server for various purpose
    • Static content delivery
    • Reverse proxy ( HTTP / FastCGI / uWSGI / … )
    • Load balancer
    • Cache
    • TLS Termination / SPDY Gateway
    • etc…
    • Mail proxy ( SMTP / POP3 / IMAP )
    • TCP proxy

    View Slide

  10. nginx’s architecture
    • event-driven
    • non-blocking
    • asynchronous
    • single-threaded ( multi-threaded partially)
    • multi-processes ( master / wokers / …)
    • modular

    View Slide

  11. nginx’s architecture
    • event-driven
    • non-blocking
    • asynchronous
    • single-threaded ( multi-threaded partially)
    • multi-processes ( master / wokers / …)
    • modular

    View Slide

  12. nginx is module-oriented
    • most features are designed as module
    • ngx_http_core_module
    • ngx_http_access_module
    • ngx_http_log_module
    • ngx_http_proxy_module
    • ngx_http_upstream_module
    • etc…

    View Slide

  13. 3rd party module for nginx
    • nginx-user can develop an original module
    • It is not required to modify source code of
    nginx
    • embed original module with ‘./configure’
    $ cd nginx-1.9.x
    $ ./configure —-add-module=${ngx_module_path}

    View Slide

  14. Examples

    View Slide

  15. Example
    • ngx_http_empty_gif_module (Official)
    • return 1x1 GIF
    location /gif {
    empty_gif;
    }
    $ curl -I http://127.0.0.1/gif
    HTTP/1.1 200 OK
    Content-Length: 43
    Content-Type: image/gif
    ...
    $

    View Slide

  16. Example
    • ngx_small_light
    • Dynamic Image Transformation for nginx
    small_light on;
    location ~ small_light[^/]*/(.+)$ {
    set $file $1;
    rewrite ^ /$file;
    }

    View Slide

  17. ngx_small_light
    • Dynamic Image Transformation for nginx
    • https://github.com/cubicdaiya/ngx_small_light
    • Provides various
    • Image Manipulations ( Resize / Crop / Composite / …)
    • Image Processing Libraries (Image Magick / Imlib2 / GD)
    • Image Formats (JPEG / GIF / PNG / WEBP)

    View Slide

  18. Transform image
    ■URI for resize
    /small_light(dw=300,dh=200)/img/image.jpg

    View Slide

  19. Transform image on S3
    proxy from nginx to S3
    small_light on;
    location ~ small_light[^/]*/(.+)$ {
    set $file $1;
    rewrite ^ /$file;
    }
    location /img {
    proxy_pass http://;
    }

    View Slide

  20. Adoption case
    https://github.com/wantedly/nginx-image-server

    View Slide

  21. Adoption case
    https://speakerdeck.com/yano3/dynamic-image-transformation-server-okara-beta

    View Slide

  22. Example
    • ngx_dynamic_upstream
    • Dynamic Upstream for nginx
    location / {
    allow xxx.xxx.xxx.xxx
    deny all;
    dynamic_upstream;
    }

    View Slide

  23. ngx_dynamic_upstream
    • Dynamic Upstream for nginx
    • https://github.com/cubicdaiya/ngx_dynamic_upstream
    • Provides HTTP APIs for manipulating upstream for nginx
    • It is useful for
    • zero-down-time deployment
    • blue green deployment

    View Slide

  24. Zero-down-time deployment on

    View Slide

  25. ɾɾɾ ɾɾɾ
    "1* "[email protected]

    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    HTTPS or SPDY
    OHJOY

    [email protected]@VQTUSFBN
    HTTP
    Mercari frontend
    "1* "[email protected]
    "1* "[email protected]

    ɾɾɾ ɾɾɾ

    View Slide

  26. "1* "[email protected]

    Previous deployment
    "1* "[email protected]

    "1* "[email protected]

    ChatOps with Slack
    yes
    EFQMPZCPU
    rsync
    rsync
    rsync

    View Slide

  27. Pros / Cons
    • Pros
    • Very Very Very simple
    • Cons
    • many 500 error happens at deployment!!!!!

    View Slide

  28. "1* "[email protected]

    Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    "1* "[email protected]

    View Slide

  29. "1* "[email protected]

    Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    "1* "[email protected]

    ※rsync
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream

    View Slide

  30. "1* "[email protected]

    Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    "1* "[email protected]

    ※rsync
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream

    View Slide

  31. Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    "1* "[email protected]

    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    down
    "1* "[email protected]

    View Slide

  32. "1* "[email protected]

    Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    "1* "[email protected]

    rsync
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream

    View Slide

  33. Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    "1* "[email protected]

    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    up
    "1* "[email protected]

    View Slide

  34. "1* "[email protected]

    Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    "1* "[email protected]

    ※rsync
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream

    View Slide

  35. Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    down
    "1* "[email protected]

    "1* "[email protected]

    View Slide

  36. "1* "[email protected]

    Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    "1* "[email protected]

    rsync
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream

    View Slide

  37. Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    up
    "1* "[email protected]

    "1* "[email protected]

    View Slide

  38. "1* "[email protected]

    Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    "1* "[email protected]

    ※rsync
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    EFQMPZCPU

    View Slide

  39. Current deployment
    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    down
    "1* "[email protected]

    "1* "[email protected]

    "1* "[email protected]

    View Slide

  40. "1* "[email protected]

    Current deployment
    "1* "[email protected]

    ChatOps with Slack
    yes
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    "1* "[email protected]

    rsync
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    EFQMPZCPU

    View Slide

  41. Current deployment
    ChatOps with Slack
    yes
    EFQMPZCPU
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    OHJOY

    [email protected]@VQTUSFBN
    ※ rsync ——rsync-path=mercari_app_rsync(↓)
    #!/bin/sh
    mercari_app_ctl down # deactivate server on nginx upstream
    rsync $* # deploy
    mercari_app_ctl up # activate server on nginx upstream
    up
    "1* "[email protected]

    "1* "[email protected]

    "1* "[email protected]

    View Slide

  42. Hello, World!

    View Slide

  43. Hello, World!
    • ngx_http_hello_world
    • https://github.com/cubicdaiya/ngx_http_hello_world
    location /hello {
    hello_world;
    }
    $ curl http://127.0.0.1/hello
    Hello, World!
    $

    View Slide

  44. Layout of ngx_http_hello_world
    ngx_http_hello_world
    |———...
    |———config
    |———ngx_http_hello_world_module.c
    └──-...

    View Slide

  45. config
    • module configuration file
    ngx_addon_name=ngx_http_hello_world_module
    HTTP_MODULES="$HTTP_MODULES \
    ngx_http_hello_world_module”
    NGX_ADDON_SRCS="$NGX_ADDON_SRCS \
    $ngx_addon_dir/ngx_http_hello_world_module.c”

    View Slide

  46. Hello, World!
    • ngx_http_hello_world
    • return “Hello, World!”
    location /hello {
    hello_world;
    }
    $ curl http://127.0.0.1/hello
    Hello, World!
    $

    View Slide

  47. Hello, World!
    • ngx_http_hello_world
    • return “Hello, World!”
    location /hello {
    hello_world;
    }
    $ curl http://127.0.0.1/hello
    Hello, World!
    $

    View Slide

  48. Hello, World!
    • ngx_http_hello_world
    • return “Hello, World!”
    location /hello {
    hello_world; # original directive
    }
    $ curl http://127.0.0.1/hello
    Hello, World!
    $

    View Slide

  49. Directive declaration
    struct ngx_command_s {
    ngx_str_t name;
    ngx_uint_t type;
    char *(*set)(ngx_conf_t *cf, ngx_command_t *cmd, void *conf);
    ngx_uint_t conf;
    ngx_uint_t offset;
    void *post;
    };

    View Slide

  50. Directive definition(hello_world)
    static ngx_command_t ngx_http_hello_world_commands[] = {
    {
    ngx_string("hello_world"),
    NGX_HTTP_LOC_CONF|NGX_CONF_NOARGS,
    ngx_http_hello_world,
    0,
    0,
    NULL
    },
    ngx_null_command
    };

    View Slide

  51. Module context
    typedef struct {
    ngx_int_t (*preconfiguration)(ngx_conf_t *cf);
    ngx_int_t (*postconfiguration)(ngx_conf_t *cf);
    void *(*create_main_conf)(ngx_conf_t *cf);
    char *(*init_main_conf)(ngx_conf_t *cf, void *conf);
    void *(*create_srv_conf)(ngx_conf_t *cf);
    char *(*merge_srv_conf)(ngx_conf_t *cf, void *prev, void *conf);
    void *(*create_loc_conf)(ngx_conf_t *cf);
    char *(*merge_loc_conf)(ngx_conf_t *cf, void *prev, void *conf);
    } ngx_http_module_t;

    View Slide

  52. Module context
    static ngx_http_module_t ngx_http_hello_world_module_ctx = {
    NULL, /* preconfiguration */
    NULL, /* postconfiguration */
    NULL, /* create main configuration */
    NULL, /* init main configuration */
    NULL, /* create server configuration */
    NULL, /* merge server configuration */
    NULL, /* create location configuration */
    NULL /* merge location configuration */
    };
    In this case, all is NULL.

    View Slide

  53. Module definition
    ngx_module_t ngx_http_hello_world_module = {
    NGX_MODULE_V1,
    &ngx_http_hello_world_module_ctx, /* module context */
    ngx_http_hello_world_commands, /* module directives */
    NGX_HTTP_MODULE, /* module type */
    NULL, /* init master */
    NULL, /* init module */
    NULL, /* init process */
    NULL, /* init thread */
    NULL, /* exit thread */
    NULL, /* exit process */
    NULL, /* exit master */
    NGX_MODULE_V1_PADDING
    };

    View Slide

  54. Hook original handler
    static char *ngx_http_hello_world(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
    {
    ngx_http_core_loc_conf_t *clcf;
    clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module);
    clcf->handler = ngx_http_hello_world_handler;
    return NGX_CONF_OK;
    }

    View Slide

  55. ngx_http_hello_world_handler()
    #define NGX_HTTP_HELLO_WORLD “Hello, World!”
    static ngx_int_t ngx_http_hello_world_handler(ngx_http_request_t *r)
    {
    ngx_int_t rc;
    ngx_chain_t out;
    ngx_buf_t *b;
    ngx_str_t body = ngx_string(NGX_HTTP_HELLO_WORLD);
    if (r->method != NGX_HTTP_GET && r->method != NGX_HTTP_HEAD) {
    return NGX_HTTP_NOT_ALLOWED;
    }
    if (r->headers_in.if_modified_since) {
    return NGX_HTTP_NOT_MODIFIED;
    }
    b = ngx_pcalloc(r->pool, sizeof(ngx_buf_t));
    if (b == NULL) {
    return NGX_HTTP_INTERNAL_SERVER_ERROR;
    }
    b->pos = body.data;
    b->last = b->pos + body.len;
    b->memory = 1;
    b->last_buf = 1;
    out.buf = b;
    out.next = NULL;
    ngx_str_set(&r->headers_out.content_type, "text/plain");
    r->headers_out.status = NGX_HTTP_OK;
    r->headers_out.content_length_n = body.len;
    rc = ngx_http_send_header(r);
    if (rc == NGX_ERROR || rc > NGX_OK || r->header_only) {
    return rc;
    }

    View Slide

  56. Embed ngx_http_hello_world into nginx
    $ ./confgiure \
    —-add-module=../ngx_http_hello_world
    $ make
    $ sudo make install

    View Slide

  57. Hook details…

    View Slide

  58. Request processing phases in nginx
    TFSWFSSFXSJUF
    QPTUBDDFTT
    MPH
    pOEDPOpH
    SFXSJUF
    QPTUSFXSJUF
    QSFBDDFTT
    USZpMFT
    DPOUFOU
    Request
    BDDFTT
    try_files processing
    access-control
    response body generation
    logging
    rewrite in server
    find location
    rewrite in location

    View Slide

  59. Phase definitions
    typedef enum {
    NGX_HTTP_POST_READ_PHASE = 0,
    NGX_HTTP_SERVER_REWRITE_PHASE,
    NGX_HTTP_FIND_CONFIG_PHASE,
    NGX_HTTP_REWRITE_PHASE,
    NGX_HTTP_POST_REWRITE_PHASE,
    NGX_HTTP_PREACCESS_PHASE,
    NGX_HTTP_ACCESS_PHASE,
    NGX_HTTP_POST_ACCESS_PHASE,
    NGX_HTTP_TRY_FILES_PHASE,
    NGX_HTTP_CONTENT_PHASE,
    NGX_HTTP_LOG_PHASE
    } ngx_http_phases;

    View Slide

  60. Hook handler to specified phases
    ngx_http_handler_pt *h;
    ngx_http_core_main_conf_t *cmcf;
    cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module);
    h = ngx_array_push(&cmcf->phases[NGX_HTTP_ACCESS_PHASE].handlers);
    if (h == NULL) {
    return NGX_ERROR;
    }
    *h = ngx_http_xxx_handler;

    View Slide

  61. Agenda
    • nginx http module development with C
    • nginx API programming
    • Test::Nginx
    • ngx_lua
    • OpenResty

    View Slide

  62. Agenda
    • nginx http module development with C
    • nginx API programming
    • Test::Nginx
    • ngx_lua
    • OpenResty

    View Slide

  63. nginx API
    • memory-pool
    • string
    • array
    • hash-table
    • time
    • list
    • regular-expression
    • temporary-file
    • shared-memory
    • etc…

    View Slide

  64. essence of nginx API programming
    • understand nginx architecture
    • event-driven
    • non-blocking
    • asynchronous
    • single-threaded
    • multi-process

    View Slide

  65. essence of nginx API programming
    • memory management strategies
    • memory-pool
    • slab ( for shared-memory)
    • string with length
    • string is not always NULL-terminated

    View Slide

  66. essence of nginx API programming
    • structs for HTTP
    • ngx_http_request_t
    • ngx_http_headers_in
    • ngx_http_headers_out
    • etc… ( See nginx/src/http/ngx_http_request.h )

    View Slide

  67. Move to terminal…

    View Slide

  68. Agenda
    • nginx http module development with C
    • nginx API programming
    • Test::Nginx
    • ngx_lua
    • OpenResty

    View Slide

  69. Agenda
    • nginx http module development with C
    • nginx API programming
    • Test::Nginx
    • ngx_lua
    • OpenResty

    View Slide

  70. Test::Nginx
    https://github.com/openresty/test-nginx

    View Slide

  71. Test::Nginx
    • Testing framework for nginx
    • https://github.com/openresty/test-nginx
    • We are able to test nginx’s behavior with CLI
    • Install from CPAN
    $ cpanm Test::Nginx

    View Slide

  72. test nginx’s behavior with Test::Nginx
    use Test::Nginx::Socket;
    plan tests => repeat_each() * 2 * blocks();
    run_tests();
    __DATA__
    === TEST 1: hello world
    --- config
    location /hello {
    return 200 "Hello, World!\n";
    }
    --- request
    GET /hello
    --- response_body
    Hello, World!
    --- error_code: 200

    View Slide

  73. test nginx’s behavior with Test::Nginx
    use Test::Nginx::Socket;
    plan tests => repeat_each() * 2 * blocks();
    run_tests();
    __DATA__
    === TEST 1: hello world
    --- config
    location /hello {
    return 200 "Hello, World!\n";
    }
    --- request
    GET /hello
    --- response_body
    Hello, World!
    --- error_code: 200
    run tests

    View Slide

  74. test nginx’s behavior with Test::Nginx
    use Test::Nginx::Socket;
    plan tests => repeat_each() * 2 * blocks();
    run_tests();
    __DATA__
    === TEST 1: hello world
    --- config
    location /hello {
    return 200 "Hello, World!\n";
    }
    --- request
    GET /hello
    --- response_body
    Hello, World!
    --- error_code: 200
    run tests
    define behavior
    ɾnginx.conf
    ɾHTTP request
    ɾHTTP response

    View Slide

  75. By the way,
    • C programming is difficult and complicated
    • productivity is not high
    If scripting language is available on nginx,
    productivity would improve dramatically.

    View Slide

  76. Agenda
    • nginx http module development with C
    • nginx API programming
    • Test::Nginx
    • ngx_lua
    • OpenResty

    View Slide

  77. Agenda
    • nginx http module development with C
    • nginx API programming
    • Test::Nginx
    • ngx_lua
    • OpenResty

    View Slide

  78. ngx_lua
    https://github.com/openresty/lua-nginx-moduleʙ

    View Slide

  79. ngx_lua
    • embed Lua/LuaJIT into nginx
    • Lua scripting in nginx.conf
    • Hooking request processing phase with Lua
    • wrap nginx API with Lua
    • non-blocking APIs (e.g. ngx.location.capture())

    View Slide

  80. • set_by_lua / set_by_lua_file
    • rewrite_by_lua / rewrite_by_lua_file
    • access_by_lua / access_by_lua_file
    • content_by_lua / content_by_lua_file
    • log_by_lua / log_by_lua_file
    Hook directives for request processing phases

    View Slide

  81. • set_by_lua / set_by_lua_file
    • rewrite_by_lua / rewrite_by_lua_file
    • access_by_lua / access_by_lua_file
    • content_by_lua / content_by_lua_file
    • log_by_lua / log_by_lua_file
    run Lua script-string
    Hook directives for request processing phases

    View Slide

  82. Hook directives for request processing phases
    • set_by_lua / set_by_lua_file
    • rewrite_by_lua / rewrite_by_lua_file
    • access_by_lua / access_by_lua_file
    • content_by_lua / content_by_lua_file
    • log_by_lua / log_by_lua_file
    run Lua script-file

    View Slide

  83. content_by_lua
    location /hello {
    content_by_lua “ngx.say(‘Hello, World!’)”;
    }
    ■ nginx.conf
    $ curl -s http://127.0.0.1/hello
    Hello, World!
    $
    ■ run

    View Slide

  84. content_by_lua_file
    location /hello {
    content_by_lua_file /etc/nginx/lua/hello.lua;
    }
    ■ nginx.conf
    ■ /etc/nginx/lua/hello.lua
    ngx.say(“Hello, World!”)

    View Slide

  85. set_by_lua $result
    location /yapc {
    set $conference YAPC::Asia;
    set $location Tokyo;
    set $year 2015;
    set_by_lua $result ‘return
    ngx.var.conference .. “ ” ..
    ngx.var.location .. “ ” ..
    ngx.var.year
    ‘;
    content_by_lua ‘ngx.say(ngx.var.result)’;
    }
    ■ nginx.conf
    $ curl -s http://127.0.0.1/hello
    YAPC::Asia Tokyo 2015
    $
    ■ run

    View Slide

  86. Request processing phases in nginx
    TFSWFSSFXSJUF
    QPTUBDDFTT
    MPH
    pOEDPOpH
    SFXSJUF
    QPTUSFXSJUF
    QSFBDDFTT
    USZpMFT
    DPOUFOU
    Request
    BDDFTT
    try_files processing
    access-control
    response body generation
    logging
    rewrite in server
    find location
    rewrite in location
    set_by_lua*

    rewrite_by_lua*
    access_by_lua*
    content_by_lua*
    log_by_lua*

    View Slide

  87. View Slide

  88. ngx_lua APIs
    • ngx.say() / ngx.exit() / ngx.log / …
    • ngx.var.VARIABLE
    • ngx.shared.DICT
    • ngx.header.HEADER
    • etc…

    View Slide

  89. ngx.say()
    ngx.say(“Hello, World!”)
    • output string to response body

    View Slide

  90. ngx.exit()
    ngx.exit(ngx.HTTP_OK)
    • return HTTP status code

    View Slide

  91. HTTP status code in ngx_lua
    • ngx.HTTP_OK (200)
    • ngx.HTTP_CREATED (201)
    • ngx.HTTP_MOVED_PERMANENTLY (301)
    • ngx.HTTP_MOVED_TEMPORARILY (302)
    • ngx.HTTP_NOT_MODIFIED (304)
    • ngx.HTTP_BAD_REQUEST (400)
    • ngx.HTTP_INTERNAL_SERVER_ERROR (500)
    • etc…

    View Slide

  92. ngx.log()
    ngx.exit(ngx.ERR, “some error”)
    • output message to error.log
    • $arg1: log-level, $arg2: message

    View Slide

  93. log-level in ngx_lua
    • ngx.EMERG
    • ngx.ALERT
    • ngx.CRIT
    • ngx.ERR
    • ngx.WARN
    • ngx.NOTICE
    • ngx.INFO
    • ngx.DEBUG

    View Slide

  94. log-level in ngx_lua
    • ngx.EMERG
    • ngx.ALERT
    • ngx.CRIT
    • ngx.ERR
    • ngx.WARN
    • ngx.NOTICE
    • ngx.INFO
    • ngx.DEBUG
    log-level
    High
    Low

    View Slide

  95. ngx.var.VARIABLE
    • read / overwrite nginx variables
    location / {
    set $fn “Tatsuhiko Kubo”;
    set $hn “@cubicdaiya”;
    content_by_lua ‘
    local body = ngx.var.fn .. ngx.var.hn
    ngx.say(body) —- Tatsuhiko [email protected]
    ‘;
    }

    View Slide

  96. ngx.var.VARIABLE
    • creating new variable is not allowed
    location / {
    content_by_lua ‘
    ngx.var.fn = “Tatsuhiko Kubo”
    ngx.var.hn = “@cubicdaiya”
    ‘;
    }
    runtime error: … variable “fn” not found for writing;
    maybe it is a built-in variable that is not changeable
    or …
    ɾ error.log

    View Slide

  97. ngx.shared.DICT
    • dictionary with shared memory
    http {
    lua_shared_dict yapc 1m;
    init_by_lua ‘
    ngx.shared.yapc:set(“presentator”, “@cubicdaiya”)
    ’;

    server {

    location / {
    content_by_lua ‘
    ngx.say(ngx.shared.yapc:get(“presentator”))
    ’;
    }
    }
    }

    View Slide

  98. ngx.header.HEADER
    • response header reference and manipulation
    location / {
    content_by_lua ‘
    local body = “{ \\”presentator\\” : \\”@cubicdaiya\\” }”
    —- assign application/json to Content-Type
    ngx.header.content_type = “application/json”
    ngx.say(body)
    ‘;
    }

    View Slide

  99. Header reference and manipulation
    • ngx.req.get_headers()
    • ngx.req.set_header()
    • ngx.req.clear_header()
    • …
    • ngx.resp.get_headers()
    • ngx.header.HEADER
    • …
    ■ For request-header ■ For response-header

    View Slide

  100. ngx.time()
    • return current timestamp
    local timestamp = ngx.time()

    View Slide

  101. ngx_lua time APIs
    • ngx.time()
    • ngx.today()
    • ngx.now()
    • ngx.update_time()
    • ngx.parse_http_time()
    • ngx.localtime()
    • ngx.utctime()
    • ngx.cookie_time()
    • ngx.http_time()

    View Slide

  102. Use ngx.time() instead of os.time()
    • os.time() always calls gettimeofday()
    • ngx.time() uses timer-cache of nginx
    • avoid system-call’s overhead
    • ngx.time() is more efficient than os.time()

    View Slide

  103. ngx.re.match()
    • Regular Expression matching
    • powered by PCRE
    local matches, err = ngx.re.match(“image/webp”,
    “([^/]+)\/(.+)”)
    -— matches[0] -> image/webp
    -— matches[1] -> image
    -— matches[2] -> webp

    View Slide

  104. ngx.re.* (Regular Expression API)
    • ngx.re.find()
    • ngx.re.match()
    • ngx.re.gmatch()
    • ngx.re.sub()
    • ngx.re.gsub()

    View Slide

  105. • Use PCRE options(e.g. o, j)
    • o -> enable compile-once
    • j -> enable PCRE-JIT
    Optimize ngx.re.*
    local matches, err = ngx.re.match(“image/webp”,
    “([^/]+)\/(.+)”,
    “oj”)

    View Slide

  106. • ngx.re is based PCRE
    • Powerful & Fat
    • Lua builtin-re(regular expression)
    • Simple & Light
    ngx.re.* VS Lua builtin-re
    If ‘re’ is simple, Lua builtin-re is faster.

    View Slide

  107. ngx.location.capture()
    • send non-blocking sub-request to other location
    location /external {
    internal;
    proxy_pass http://external;
    }
    location / {
    content_by_lua ‘
    —— non-blocking behavior
    local res = ngx.location.capture(“/external”)

    ‘;
    }

    View Slide

  108. By the way,
    • ngx_lua is very powerful
    • But setup and deployment is a little painful
    • Lua / LuaJIT instalation
    • Lua packages management

    View Slide

  109. Agenda
    • nginx http module development with C
    • nginx API programming
    • Test::Nginx
    • ngx_lua
    • OpenResty

    View Slide

  110. Agenda
    • nginx http module development with C
    • nginx API programming
    • Test::Nginx
    • ngx_lua
    • OpenResty

    View Slide

  111. OpenResty
    http://openresty.org/

    View Slide

  112. OpenResty
    • Web application framework bundled
    • nginx and 3rd party modules (e.g. ngx_lua)
    • Lua, LuaJIT
    • Resty modules(lua-resty-xxx…)
    • All-in-one superset of nginx

    View Slide

  113. 3rd party modules
    • lua-nginx-module (ngx_lua)
    • echo-nginx-module
    • headers-more-nginx-module
    • etc…

    View Slide

  114. Resty modules
    • lua-resty-core
    • resty-cli
    • lua-resty-mysql / redis / memcached
    • etc…

    View Slide

  115. lua-resty-core
    • pure Lua partial-re-implementation of ngx_lua APIs
    • accelerate ngx_lua APIs with LuaJIT
    • because ngx_lua APIs with C is not JIT compiled
    http {
    # accelerate ngx_lua APIs
    init_by_lua ‘require resty.core’;

    }

    View Slide

  116. lua-resty-cli
    • command-line utility for OpenResty
    • run OpenResty with CLI
    • all APIs are not always enabled
    $ resty -e ‘ngx.say(“Hello, World!”)’
    Hello, World!
    $

    View Slide

  117. lua-resty-mysql / redis / memcached
    • client drivers for mysql / redis / memcached
    • non-blocking behavior
    • Powered by ngx.socket

    View Slide

  118. OpenResty on

    View Slide

  119. PascalʙMercari app-event analysis baseʙ
    0QFO3FTUZ
    0QFO3FTUZ
    0QFO3FTUZ
    (PPHMF#JH2VFSZ
    Developer Data Sientist
    Analyze
    by SQL
    send events
    send events
    send events
    Powered by
    cookpad/puree-(ios|android)
    utilize events
    transfer
    utilize events
    utilize events
    transfer

    View Slide

  120. OpenResty’s role on Pascal
    • High performance event logger
    • receive events as JSON from devices
    • parse JSON
    • utilize events for BigQuery
    • log events into file every event-channel

    View Slide

  121. References
    • The Architecture of Open Source Applications (Volume 2):
    nginx
    • http://www.aosabook.org/en/nginx.html
    • Nginx Internals
    • http://www.slideshare.net/joshzhu/nginx-internals
    • Emiller’s Guide To Nginx Module Development
    • http://www.evanmiller.org/nginx-modules-guide.html

    View Slide

  122. References
    • openresty/lua-nginx-module
    • https://github.com/openresty/lua-nginx-module
    • Using ngx_lua / lua-nginx-module in pixiv
    • http://www.slideshare.net/harukayon/ngx-lua-public
    • ngx_mruby and ngx_lua
    • http://qiita.com/cubicdaiya/items/3913d3dddf6999984c5b

    View Slide

  123. Thanks!

    View Slide