Practical nginx module development〜C and Lua〜

Practical nginx module development
ʙC and Luaʙ
Tatsuhiko [email protected]
YAPC::Asia TOKYO 2015

View Slide

@cubicdaiya / Tatsuhiko Kubo
• Software Engineer @ Mercari, Inc.
• Infrastructure Engineering
• Skills
• C, Go, Lua, nginx, …
• OSS developer
• ngx_small_light, ngx_dynamic_upstream,
nginx-build, slackboard, cachectl, gaurun, …

View Slide

Activities on Github

View Slide

Other Activities
IUUQNP[BJDGNQPTUOHJOY

View Slide

View Slide

Agenda
• nginx http module development with C
• nginx API programming
• Test::Nginx
• ngx_lua
• OpenResty

View Slide

nginx
• Open Source Software
• HTTP server for various purpose
• Static content delivery
• Reverse proxy ( HTTP / FastCGI / uWSGI / … )
• Load balancer
• Cache
• TLS Termination / SPDY Gateway
• etc…
• Mail proxy ( SMTP / POP3 / IMAP )
• TCP proxy

View Slide

nginx’s architecture
• event-driven
• non-blocking
• asynchronous
• single-threaded ( multi-threaded partially)
• multi-processes ( master / wokers / …)
• modular

View Slide

nginx is module-oriented
• most features are designed as module
• ngx_http_core_module
• ngx_http_access_module
• ngx_http_log_module
• ngx_http_proxy_module
• ngx_http_upstream_module
• etc…

View Slide

3rd party module for nginx
• nginx-user can develop an original module
• It is not required to modify source code of
nginx
• embed original module with ‘./conﬁgure’
$ cd nginx-1.9.x
$ ./configure —-add-module=${ngx_module_path}

View Slide

Examples

View Slide

Example
• ngx_http_empty_gif_module (Ofﬁcial)
• return 1x1 GIF
location /gif {
empty_gif;
}
$ curl -I http://127.0.0.1/gif
HTTP/1.1 200 OK
Content-Length: 43
Content-Type: image/gif
...
$

View Slide

Example
• ngx_small_light
• Dynamic Image Transformation for nginx
small_light on;
location ~ small_light[^/]*/(.+)$ {
set $file $1;
rewrite ^ /$file;
}

View Slide

ngx_small_light
• Dynamic Image Transformation for nginx
• https://github.com/cubicdaiya/ngx_small_light
• Provides various
• Image Manipulations ( Resize / Crop / Composite / …)
• Image Processing Libraries (Image Magick / Imlib2 / GD)
• Image Formats (JPEG / GIF / PNG / WEBP)

View Slide

Transform image
■URI for resize
/small_light(dw=300,dh=200)/img/image.jpg
→

View Slide

Transform image on S3
proxy from nginx to S3
small_light on;
location ~ small_light[^/]*/(.+)$ {
set $file $1;
rewrite ^ /$file;
}
location /img {
proxy_pass http://;
}

View Slide

Adoption case
https://github.com/wantedly/nginx-image-server

View Slide

Adoption case
https://speakerdeck.com/yano3/dynamic-image-transformation-server-okara-beta

View Slide

Example
• ngx_dynamic_upstream
• Dynamic Upstream for nginx
location / {
allow xxx.xxx.xxx.xxx
deny all;
dynamic_upstream;
}

View Slide

ngx_dynamic_upstream
• Dynamic Upstream for nginx
• https://github.com/cubicdaiya/ngx_dynamic_upstream
• Provides HTTP APIs for manipulating upstream for nginx
• It is useful for
• zero-down-time deployment
• blue green deployment

View Slide

Zero-down-time deployment on

View Slide

ɾɾɾ ɾɾɾ
"1* "[email protected]

OHJOY

[email protected]@VQTUSFBN
OHJOY

HTTPS or SPDY
OHJOY

HTTP
Mercari frontend

ɾɾɾ ɾɾɾ

View Slide


Previous deployment


ChatOps with Slack
yes
EFQMPZCPU
rsync
rsync
rsync

View Slide

Pros / Cons
• Pros
• Very Very Very simple
• Cons
• many 500 error happens at deployment!!!!!

View Slide


Current deployment

ChatOps with Slack
yes
EFQMPZCPU
OHJOY

OHJOY

OHJOY


View Slide


Current deployment

ChatOps with Slack
yes
EFQMPZCPU
OHJOY

OHJOY

OHJOY


※rsync
※ rsync ——rsync-path=mercari_app_rsync(↓)
#!/bin/sh
mercari_app_ctl down # deactivate server on nginx upstream
rsync $* # deploy
mercari_app_ctl up # activate server on nginx upstream

View Slide

Current deployment

ChatOps with Slack
yes
EFQMPZCPU
OHJOY

OHJOY

OHJOY


#!/bin/sh
rsync $* # deploy
down

View Slide


Current deployment

ChatOps with Slack
yes
EFQMPZCPU
OHJOY

OHJOY

OHJOY


rsync
#!/bin/sh
rsync $* # deploy

View Slide

Current deployment

ChatOps with Slack
yes
EFQMPZCPU
OHJOY

OHJOY

OHJOY


#!/bin/sh
rsync $* # deploy
up

View Slide


Current deployment

ChatOps with Slack
yes
EFQMPZCPU
OHJOY

OHJOY

OHJOY


※rsync
#!/bin/sh
rsync $* # deploy

View Slide

Current deployment

ChatOps with Slack
yes
EFQMPZCPU
OHJOY

OHJOY

OHJOY

#!/bin/sh
rsync $* # deploy
down


View Slide


Current deployment

ChatOps with Slack
yes
EFQMPZCPU
OHJOY

OHJOY

OHJOY


rsync
#!/bin/sh
rsync $* # deploy

View Slide

Current deployment

ChatOps with Slack
yes
EFQMPZCPU
OHJOY

OHJOY

OHJOY

#!/bin/sh
rsync $* # deploy
up


View Slide


Current deployment

ChatOps with Slack
yes
OHJOY

OHJOY

OHJOY


※rsync
#!/bin/sh
rsync $* # deploy
EFQMPZCPU

View Slide

Current deployment
ChatOps with Slack
yes
EFQMPZCPU
OHJOY

OHJOY

OHJOY

#!/bin/sh
rsync $* # deploy
down



View Slide


Current deployment

ChatOps with Slack
yes
OHJOY

OHJOY

OHJOY


rsync
#!/bin/sh
rsync $* # deploy
EFQMPZCPU

View Slide

Current deployment
ChatOps with Slack
yes
EFQMPZCPU
OHJOY

OHJOY

OHJOY

#!/bin/sh
rsync $* # deploy
up



View Slide

Hello, World!

View Slide

Hello, World!
• ngx_http_hello_world
• https://github.com/cubicdaiya/ngx_http_hello_world
location /hello {
hello_world;
}
$ curl http://127.0.0.1/hello
Hello, World!
$

View Slide

Layout of ngx_http_hello_world
ngx_http_hello_world
|———...
|———config
|———ngx_http_hello_world_module.c
└──-...

View Slide

config
• module configuration file
ngx_addon_name=ngx_http_hello_world_module
HTTP_MODULES="$HTTP_MODULES \
ngx_http_hello_world_module”
NGX_ADDON_SRCS="$NGX_ADDON_SRCS \
$ngx_addon_dir/ngx_http_hello_world_module.c”

View Slide

Hello, World!
• return “Hello, World!”
location /hello {
hello_world;
}
Hello, World!
$

View Slide

Hello, World!
• return “Hello, World!”
location /hello {
hello_world; # original directive
}
Hello, World!
$

View Slide

Directive declaration
struct ngx_command_s {
ngx_str_t name;
ngx_uint_t type;
char *(*set)(ngx_conf_t *cf, ngx_command_t *cmd, void *conf);
ngx_uint_t conf;
ngx_uint_t offset;
void *post;
};

View Slide

Directive deﬁnition(hello_world)
static ngx_command_t ngx_http_hello_world_commands[] = {
{
ngx_string("hello_world"),
NGX_HTTP_LOC_CONF|NGX_CONF_NOARGS,
ngx_http_hello_world,
0,
0,
NULL
},
ngx_null_command
};

View Slide

Module context
typedef struct {
ngx_int_t (*preconfiguration)(ngx_conf_t *cf);
ngx_int_t (*postconfiguration)(ngx_conf_t *cf);
void *(*create_main_conf)(ngx_conf_t *cf);
char *(*init_main_conf)(ngx_conf_t *cf, void *conf);
void *(*create_srv_conf)(ngx_conf_t *cf);
char *(*merge_srv_conf)(ngx_conf_t *cf, void *prev, void *conf);
void *(*create_loc_conf)(ngx_conf_t *cf);
char *(*merge_loc_conf)(ngx_conf_t *cf, void *prev, void *conf);
} ngx_http_module_t;

View Slide

Module context
static ngx_http_module_t ngx_http_hello_world_module_ctx = {
NULL, /* preconfiguration */
NULL, /* postconfiguration */
NULL, /* create main configuration */
NULL, /* init main configuration */
NULL, /* create server configuration */
NULL, /* merge server configuration */
NULL, /* create location configuration */
NULL /* merge location configuration */
};
In this case, all is NULL.

View Slide

Module deﬁnition
ngx_module_t ngx_http_hello_world_module = {
NGX_MODULE_V1,
&ngx_http_hello_world_module_ctx, /* module context */
ngx_http_hello_world_commands, /* module directives */
NGX_HTTP_MODULE, /* module type */
NULL, /* init master */
NULL, /* init module */
NULL, /* init process */
NULL, /* init thread */
NULL, /* exit thread */
NULL, /* exit process */
NULL, /* exit master */
NGX_MODULE_V1_PADDING
};

View Slide

Hook original handler
static char *ngx_http_hello_world(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
ngx_http_core_loc_conf_t *clcf;
clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module);
clcf->handler = ngx_http_hello_world_handler;
return NGX_CONF_OK;
}

View Slide

ngx_http_hello_world_handler()
#define NGX_HTTP_HELLO_WORLD “Hello, World!”
static ngx_int_t ngx_http_hello_world_handler(ngx_http_request_t *r)
{
ngx_int_t rc;
ngx_chain_t out;
ngx_buf_t *b;
ngx_str_t body = ngx_string(NGX_HTTP_HELLO_WORLD);
if (r->method != NGX_HTTP_GET && r->method != NGX_HTTP_HEAD) {
return NGX_HTTP_NOT_ALLOWED;
}
if (r->headers_in.if_modified_since) {
return NGX_HTTP_NOT_MODIFIED;
}
b = ngx_pcalloc(r->pool, sizeof(ngx_buf_t));
if (b == NULL) {
return NGX_HTTP_INTERNAL_SERVER_ERROR;
}
b->pos = body.data;
b->last = b->pos + body.len;
b->memory = 1;
b->last_buf = 1;
out.buf = b;
out.next = NULL;
ngx_str_set(&r->headers_out.content_type, "text/plain");
r->headers_out.status = NGX_HTTP_OK;
r->headers_out.content_length_n = body.len;
rc = ngx_http_send_header(r);
if (rc == NGX_ERROR || rc > NGX_OK || r->header_only) {
return rc;
}

View Slide

Embed ngx_http_hello_world into nginx
$ ./confgiure \
—-add-module=../ngx_http_hello_world
$ make
$ sudo make install

View Slide

Hook details…

View Slide

Request processing phases in nginx
TFSWFSSFXSJUF
QPTUBDDFTT
MPH
pOEDPOpH
SFXSJUF
QPTUSFXSJUF
QSFBDDFTT
USZpMFT
DPOUFOU
Request
BDDFTT
try_ﬁles processing
access-control
response body generation
logging
rewrite in server
ﬁnd location
rewrite in location

View Slide

Phase deﬁnitions
typedef enum {
NGX_HTTP_POST_READ_PHASE = 0,
NGX_HTTP_SERVER_REWRITE_PHASE,
NGX_HTTP_FIND_CONFIG_PHASE,
NGX_HTTP_REWRITE_PHASE,
NGX_HTTP_POST_REWRITE_PHASE,
NGX_HTTP_PREACCESS_PHASE,
NGX_HTTP_ACCESS_PHASE,
NGX_HTTP_POST_ACCESS_PHASE,
NGX_HTTP_TRY_FILES_PHASE,
NGX_HTTP_CONTENT_PHASE,
NGX_HTTP_LOG_PHASE
} ngx_http_phases;

View Slide

Hook handler to speciﬁed phases
ngx_http_handler_pt *h;
ngx_http_core_main_conf_t *cmcf;
cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module);
h = ngx_array_push(&cmcf->phases[NGX_HTTP_ACCESS_PHASE].handlers);
if (h == NULL) {
return NGX_ERROR;
}
*h = ngx_http_xxx_handler;

View Slide

Agenda
• Test::Nginx
• ngx_lua
• OpenResty

View Slide

nginx API
• memory-pool
• string
• array
• hash-table
• time
• list
• regular-expression
• temporary-ﬁle
• shared-memory
• etc…

View Slide

essence of nginx API programming
• understand nginx architecture
• event-driven
• non-blocking
• asynchronous
• single-threaded
• multi-process

View Slide

• memory management strategies
• memory-pool
• slab ( for shared-memory)
• string with length
• string is not always NULL-terminated

View Slide

• structs for HTTP
• ngx_http_request_t
• ngx_http_headers_in
• ngx_http_headers_out
• etc… ( See nginx/src/http/ngx_http_request.h )

View Slide

Move to terminal…

View Slide

Agenda
• Test::Nginx
• ngx_lua
• OpenResty

View Slide

Test::Nginx
https://github.com/openresty/test-nginx

View Slide

Test::Nginx
• Testing framework for nginx
• https://github.com/openresty/test-nginx
• We are able to test nginx’s behavior with CLI
• Install from CPAN
$ cpanm Test::Nginx

View Slide

test nginx’s behavior with Test::Nginx
use Test::Nginx::Socket;
plan tests => repeat_each() * 2 * blocks();
run_tests();
__DATA__
=== TEST 1: hello world
--- config
location /hello {
return 200 "Hello, World!\n";
}
--- request
GET /hello
--- response_body
Hello, World!
--- error_code: 200

View Slide

run_tests();
__DATA__
--- config
location /hello {
}
--- request
GET /hello
--- response_body
Hello, World!
--- error_code: 200
run tests

View Slide

run_tests();
__DATA__
--- config
location /hello {
}
--- request
GET /hello
--- response_body
Hello, World!
--- error_code: 200
run tests
deﬁne behavior
ɾnginx.conf
ɾHTTP request
ɾHTTP response

View Slide

By the way,
• C programming is difﬁcult and complicated
• productivity is not high
If scripting language is available on nginx,
productivity would improve dramatically.

View Slide

Agenda
• Test::Nginx
• ngx_lua
• OpenResty

View Slide

ngx_lua
https://github.com/openresty/lua-nginx-moduleʙ

View Slide

ngx_lua
• embed Lua/LuaJIT into nginx
• Lua scripting in nginx.conf
• Hooking request processing phase with Lua
• wrap nginx API with Lua
• non-blocking APIs (e.g. ngx.location.capture())

View Slide

• set_by_lua / set_by_lua_file
• rewrite_by_lua / rewrite_by_lua_file
• access_by_lua / access_by_lua_file
• content_by_lua / content_by_lua_file
• log_by_lua / log_by_lua_file
Hook directives for request processing phases

View Slide

run Lua script-string

View Slide

run Lua script-ﬁle

View Slide

content_by_lua
location /hello {
content_by_lua “ngx.say(‘Hello, World!’)”;
}
■ nginx.conf
$ curl -s http://127.0.0.1/hello
Hello, World!
$
■ run

View Slide

content_by_lua_ﬁle
location /hello {
content_by_lua_file /etc/nginx/lua/hello.lua;
}
■ nginx.conf
■ /etc/nginx/lua/hello.lua
ngx.say(“Hello, World!”)

View Slide

set_by_lua $result
location /yapc {
set $conference YAPC::Asia;
set $location Tokyo;
set $year 2015;
set_by_lua $result ‘return
ngx.var.conference .. “ ” ..
ngx.var.location .. “ ” ..
ngx.var.year
‘;
content_by_lua ‘ngx.say(ngx.var.result)’;
}
■ nginx.conf
$ curl -s http://127.0.0.1/hello
YAPC::Asia Tokyo 2015
$
■ run

View Slide

Request processing phases in nginx
TFSWFSSFXSJUF
QPTUBDDFTT
MPH
pOEDPOpH
SFXSJUF
QPTUSFXSJUF
QSFBDDFTT
USZpMFT
DPOUFOU
Request
BDDFTT
try_ﬁles processing
access-control
response body generation
logging
rewrite in server
ﬁnd location
rewrite in location
set_by_lua* 
rewrite_by_lua*
access_by_lua*
content_by_lua*
log_by_lua*

View Slide

Other hook directives and phases
IPPLOBNF QIBTF
[email protected]@MVB
[email protected]@[email protected]
MPBEJOHOHJOYDPOG
[email protected]@[email protected]@pMF
TUBSUJOHXPSLFSQSPDFTT
pMUFSJOHIFBEFS
pMUFSJOHCPEZ

View Slide

ngx_lua APIs
• ngx.say() / ngx.exit() / ngx.log / …
• ngx.var.VARIABLE
• ngx.shared.DICT
• ngx.header.HEADER
• etc…

View Slide

ngx.say()
ngx.say(“Hello, World!”)
• output string to response body

View Slide

ngx.exit()
ngx.exit(ngx.HTTP_OK)
• return HTTP status code

View Slide

HTTP status code in ngx_lua
• ngx.HTTP_OK (200)
• ngx.HTTP_CREATED (201)
• ngx.HTTP_MOVED_PERMANENTLY (301)
• ngx.HTTP_MOVED_TEMPORARILY (302)
• ngx.HTTP_NOT_MODIFIED (304)
• ngx.HTTP_BAD_REQUEST (400)
• ngx.HTTP_INTERNAL_SERVER_ERROR (500)
• etc…

View Slide

ngx.log()
ngx.exit(ngx.ERR, “some error”)
• output message to error.log
• $arg1: log-level, $arg2: message

View Slide

log-level in ngx_lua
• ngx.EMERG
• ngx.ALERT
• ngx.CRIT
• ngx.ERR
• ngx.WARN
• ngx.NOTICE
• ngx.INFO
• ngx.DEBUG

View Slide

log-level in ngx_lua
• ngx.EMERG
• ngx.ALERT
• ngx.CRIT
• ngx.ERR
• ngx.WARN
• ngx.NOTICE
• ngx.INFO
• ngx.DEBUG
log-level
High
Low

View Slide

ngx.var.VARIABLE
• read / overwrite nginx variables
location / {
set $fn “Tatsuhiko Kubo”;
set $hn “@cubicdaiya”;
content_by_lua ‘
local body = ngx.var.fn .. ngx.var.hn
ngx.say(body) —- Tatsuhiko [email protected]
‘;
}

View Slide

ngx.var.VARIABLE
• creating new variable is not allowed
location / {
content_by_lua ‘
ngx.var.fn = “Tatsuhiko Kubo”
ngx.var.hn = “@cubicdaiya”
‘;
}
runtime error: … variable “fn” not found for writing;
maybe it is a built-in variable that is not changeable
or …
ɾ error.log

View Slide

ngx.shared.DICT
• dictionary with shared memory
http {
lua_shared_dict yapc 1m;
init_by_lua ‘
ngx.shared.yapc:set(“presentator”, “@cubicdaiya”)
’;
…
server {
…
location / {
content_by_lua ‘
ngx.say(ngx.shared.yapc:get(“presentator”))
’;
}
}
}

View Slide

ngx.header.HEADER
• response header reference and manipulation
location / {
content_by_lua ‘
local body = “{ \\”presentator\\” : \\”@cubicdaiya\\” }”
—- assign application/json to Content-Type
ngx.header.content_type = “application/json”
ngx.say(body)
‘;
}

View Slide

Header reference and manipulation
• ngx.req.get_headers()
• ngx.req.set_header()
• ngx.req.clear_header()
• …
• ngx.resp.get_headers()
• ngx.header.HEADER
• …
■ For request-header ■ For response-header

View Slide

ngx.time()
• return current timestamp
local timestamp = ngx.time()

View Slide

ngx_lua time APIs
• ngx.time()
• ngx.today()
• ngx.now()
• ngx.update_time()
• ngx.parse_http_time()
• ngx.localtime()
• ngx.utctime()
• ngx.cookie_time()
• ngx.http_time()

View Slide

Use ngx.time() instead of os.time()
• os.time() always calls gettimeofday()
• ngx.time() uses timer-cache of nginx
• avoid system-call’s overhead
• ngx.time() is more efﬁcient than os.time()

View Slide

ngx.re.match()
• Regular Expression matching
• powered by PCRE
local matches, err = ngx.re.match(“image/webp”,
“([^/]+)\/(.+)”)
-— matches[0] -> image/webp
-— matches[1] -> image
-— matches[2] -> webp

View Slide

ngx.re.* (Regular Expression API)
• ngx.re.ﬁnd()
• ngx.re.match()
• ngx.re.gmatch()
• ngx.re.sub()
• ngx.re.gsub()

View Slide

• Use PCRE options(e.g. o, j)
• o -> enable compile-once
• j -> enable PCRE-JIT
Optimize ngx.re.*
local matches, err = ngx.re.match(“image/webp”,
“([^/]+)\/(.+)”,
“oj”)

View Slide

• ngx.re is based PCRE
• Powerful & Fat
• Lua builtin-re(regular expression)
• Simple & Light
ngx.re.* VS Lua builtin-re
If ‘re’ is simple, Lua builtin-re is faster.

View Slide

ngx.location.capture()
• send non-blocking sub-request to other location
location /external {
internal;
proxy_pass http://external;
}
location / {
content_by_lua ‘
—— non-blocking behavior
local res = ngx.location.capture(“/external”)
…
‘;
}

View Slide

By the way,
• ngx_lua is very powerful
• But setup and deployment is a little painful
• Lua / LuaJIT instalation
• Lua packages management

View Slide

Agenda
• Test::Nginx
• ngx_lua
• OpenResty

View Slide

OpenResty
http://openresty.org/

View Slide

OpenResty
• Web application framework bundled
• nginx and 3rd party modules (e.g. ngx_lua)
• Lua, LuaJIT
• Resty modules(lua-resty-xxx…)
• All-in-one superset of nginx

View Slide

3rd party modules
• lua-nginx-module (ngx_lua)
• echo-nginx-module
• headers-more-nginx-module
• etc…

View Slide

Resty modules
• lua-resty-core
• resty-cli
• lua-resty-mysql / redis / memcached
• etc…

View Slide

lua-resty-core
• pure Lua partial-re-implementation of ngx_lua APIs
• accelerate ngx_lua APIs with LuaJIT
• because ngx_lua APIs with C is not JIT compiled
http {
# accelerate ngx_lua APIs
init_by_lua ‘require resty.core’;
…
}

View Slide

lua-resty-cli
• command-line utility for OpenResty
• run OpenResty with CLI
• all APIs are not always enabled
$ resty -e ‘ngx.say(“Hello, World!”)’
Hello, World!
$

View Slide

lua-resty-mysql / redis / memcached
• client drivers for mysql / redis / memcached
• non-blocking behavior
• Powered by ngx.socket

View Slide

OpenResty on

View Slide

PascalʙMercari app-event analysis baseʙ
0QFO3FTUZ
0QFO3FTUZ
0QFO3FTUZ
(PPHMF#JH2VFSZ
Developer Data Sientist
Analyze
by SQL
send events
send events
send events
Powered by
cookpad/puree-(ios|android)
utilize events
transfer
utilize events
utilize events
transfer

View Slide

OpenResty’s role on Pascal
• High performance event logger
• receive events as JSON from devices
• parse JSON
• utilize events for BigQuery
• log events into ﬁle every event-channel

View Slide

References
• The Architecture of Open Source Applications (Volume 2):
nginx
• http://www.aosabook.org/en/nginx.html
• Nginx Internals
• http://www.slideshare.net/joshzhu/nginx-internals
• Emiller’s Guide To Nginx Module Development
• http://www.evanmiller.org/nginx-modules-guide.html

View Slide

References
• openresty/lua-nginx-module
• https://github.com/openresty/lua-nginx-module
• Using ngx_lua / lua-nginx-module in pixiv
• http://www.slideshare.net/harukayon/ngx-lua-public
• ngx_mruby and ngx_lua
• http://qiita.com/cubicdaiya/items/3913d3dddf6999984c5b

View Slide

Thanks!

View Slide

Practical nginx module development〜C and Lua〜

Practical nginx module development〜C and Lua〜

Tatsuhiko Kubo

More Decks by Tatsuhiko Kubo

Other Decks in Technology

Featured

Transcript