Reliable High-Performance HTTP Infrastructure with nginx and Lua

Slide 1

Slide 1 text

Reliable High-Performance HTTP Infrastructure with nginx and Lua Sean Cribbs Senior Principal Engineer, Comcast Cable @seancribbs

Slide 2

Slide 2 text

Background

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Consumer

Slide 5

Slide 5 text

Internal Consumer

Slide 6

Slide 6 text

Partner Internal Consumer

Slide 7

Slide 7 text

API Management

Slide 8

Slide 8 text

API Management access control capacity management

Slide 9

Slide 9 text

CodeBig 1 API Consumer

Slide 10

Slide 10 text

CodeBig 1 API Consumer CDN • traffic shaping • caching

Slide 11

Slide 11 text

CodeBig 1 API Consumer CDN • traffic shaping • caching • access control • rate limiting Vendor APIM

Slide 12

Slide 12 text

CodeBig 1 API Consumer CDN • traffic shaping • caching • access control • rate limiting Vendor APIM Internet Comcast LB • DNS RR • VIP

Slide 13

Slide 13 text

CodeBig 1 API Consumer CDN • traffic shaping • caching • access control • rate limiting Vendor APIM • DMZ intermediary • path-host mapping iAuth Internet Comcast LB • DNS RR • VIP

Slide 14

Slide 14 text

CodeBig 1 API Consumer CDN • traffic shaping • caching • access control • rate limiting Vendor APIM • DMZ intermediary • path-host mapping iAuth Internet Comcast Origin APIs LB • DNS RR • VIP

Slide 15

Slide 15 text

Challenges

Slide 16

Slide 16 text

Challenges visibility

Slide 17

Slide 17 text

Challenges visibility responsibility

Slide 18

Slide 18 text

Challenges visibility responsibility scope

Slide 19

Slide 19 text

Challenges visibility responsibility scope latency

Slide 20

Slide 20 text

Challenges visibility responsibility scope latency security

Slide 21

Slide 21 text

CodeBig 2

Slide 22

Slide 22 text

CodeBig 2 simplify architecture

Slide 23

Slide 23 text

CodeBig 2 simplify architecture increase visibility

Slide 24

Slide 24 text

CodeBig 2 simplify architecture increase visibility use open-source tools

Slide 25

Slide 25 text

Architecture

Slide 26

Slide 26 text

custom logic HTTP Proxy

Slide 27

Slide 27 text

Lua

Slide 28

Slide 28 text

nginx+Lua extension points init access header_filter log init_worker rewrite body_filter balancer ssl_certificate set content _by_lua _by_lua_file _by_lua_block +

Slide 29

Slide 29 text

CodeBig Request Phases init access header_filter log

Slide 30

Slide 30 text

CodeBig Request Phases init access header_filter log request flow

Slide 31

Slide 31 text

CodeBig Request Phases init access header_filter log Load code and configuration request flow

Slide 32

Slide 32 text

CodeBig Request Phases init access header_filter log Load code and configuration Authenticate Rate-limit Tweak request request flow

Slide 33

Slide 33 text

CodeBig Request Phases init access header_filter log Load code and configuration Authenticate Rate-limit Tweak request Tweak response request flow

Slide 34

Slide 34 text

CodeBig Request Phases init access header_filter log Load code and configuration Authenticate Rate-limit Tweak request Tweak response Clean up request flow

Slide 35

Slide 35 text

local setmetatable = setmetatable local _M = {} function _M:new(ctx, conf) local o = { _ctx = ctx, _conf = conf } o.super = self setmetatable(o, self) self.__index = self return o end function _M:access() return true end function _M:post_access() -‐-‐ nop end function _M:header_filter() -‐-‐ nop end function _M:log() -‐-‐ nop end return _M

Slide 36

Slide 36 text

for _, name in ipairs(conf.plugins) do -- load plugin by fully qualified name local plugin = require(name):new(ctx, conf) -- exit immediately upon first rejection local is_ok, err = plugin:access() if not is_ok then ngx.status = err.code ngx_say(err.error) ngx.var.access_error = err.error return ngx_exit(ngx.HTTP_OK) end insert(plugins, plugin) end for _, plugin in ipairs(plugins) do plugin:post_access() end

Slide 37

Slide 37 text

function _M.header_filter() local plugins = ngx.ctx.plugins or {} for _, plugin in ipairs(plugins) do plugin:header_filter() end end

Slide 38

Slide 38 text

# nginx.conf lua_package_path '/usr/share/?/init.lua;/usr/share/?.lua;;'; lua_shared_dict memory 50M; init_by_lua_block { codebig = require("codebig") codebig.init(“/path/to/configs“) }; # vhost.conf location / { access_by_lua 'return codebig.access("somehost")'; header_filter_by_lua 'return codebig.header_filter()'; log_by_lua 'return codebig.log()'; }

Slide 39

Slide 39 text

Lua ~ 3K LoC!!

Slide 40

Slide 40 text

Intra-Datacenter VIP

Slide 41

Slide 41 text

Intra-Datacenter VIP haproxy haproxy …

Slide 42

Slide 42 text

Intra-Datacenter VIP … haproxy haproxy …

Slide 43

Slide 43 text

Intra-Datacenter VIP Origin APIs … haproxy haproxy …

Slide 44

Slide 44 text

Cross-Datacenter DC1 DC2 DC3 vod vod acct acct entry-‐vip-‐dc1. A 10.1.0.1 vod-‐dc1. CNAME entry-‐vip-‐dc1. vod. CNAME vod-‐dc1. entry-‐vip-‐dc2. A 10.2.0.1 entry-‐vip-‐dc3. A 10.3.0.1 vod-‐dc2. CNAME entry-‐vip-‐dc2. VIP VIP VIP vod-‐dc1-‐fo. CNAME entry-‐vip-‐dc1.

Slide 45

Slide 45 text

Capacity Management

Slide 46

Slide 46 text

N = XR

Slide 47

Slide 47 text

N = XR # concurrent requests

Slide 48

Slide 48 text

N = XR # concurrent requests transaction rate

Slide 49

Slide 49 text

N = XR # concurrent requests transaction rate response time

Slide 50

Slide 50 text

N = XR # concurrent requests transaction rate response time Little’s Law

Slide 51

Slide 51 text

client origin APIM 2 req/s 1s N = XR = 2 req/s x 1s = 2 concurrent

Slide 52

Slide 52 text

client origin APIM 2 req/s 10s N = XR = 2 req/s x 10s = 20 concurrent

Slide 53

Slide 53 text

client origin APIM 2 req/s 10s N = XR = 2 req/s x 10s = 20 concurrent

Slide 54

Slide 54 text

Concurrent Request Limiting lua_shared_dict memory 50M; access_by_lua … +1 log_by_lua … -‐1

Slide 55

Slide 55 text

Deployment

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

Configs in VCS Playbooks in VCS config templates API configs vault  (keys) vip.conf vhost.lua vhost.conf vhost.conf vhost.json nginx.conf ssh

Slide 58

Slide 58 text

Results

Slide 59

Slide 59 text

Performance

Slide 60

Slide 60 text

switch Performance

Slide 61

Slide 61 text

switch mean 99th Performance

Slide 62

Slide 62 text

switch ~10x mean 99th Performance

Slide 63

Slide 63 text

Stability

Slide 64

Slide 64 text

switch Stability

Slide 65

Slide 65 text

Impact index=codebig host=*.cimops.net source="/var/log/nginx/access.log" | eval d = request_time -‐ upstream_response_time |   timechart span=1m perc99(d) max(d)

Slide 66

Slide 66 text

Impact seconds index=codebig host=*.cimops.net source="/var/log/nginx/access.log" | eval d = request_time -‐ upstream_response_time |   timechart span=1m perc99(d) max(d)

Slide 67

Slide 67 text

Impact seconds index=codebig host=*.cimops.net source="/var/log/nginx/access.log" | eval d = request_time -‐ upstream_response_time |   timechart span=1m perc99(d) max(d) 99th

Slide 68

Slide 68 text

Impact seconds index=codebig host=*.cimops.net source="/var/log/nginx/access.log" | eval d = request_time -‐ upstream_response_time |   timechart span=1m perc99(d) max(d) 99th max

Slide 69

Slide 69 text

Successes

Slide 70

Slide 70 text

Successes great performance improvements

Slide 71

Slide 71 text

Successes great performance improvements hosting ~400 endpoints

Slide 72

Slide 72 text

Successes great performance improvements hosting ~400 endpoints > 367MM requests a day

Slide 73

Slide 73 text

Successes great performance improvements hosting ~400 endpoints > 367MM requests a day prevented upstream downtime

Slide 74

Slide 74 text

Challenges

Slide 75

Slide 75 text

Challenges 3rd-party Lua ecosystem

Slide 76

Slide 76 text

Challenges 3rd-party Lua ecosystem not self-service yet

Slide 77

Slide 77 text

Challenges 3rd-party Lua ecosystem not self-service yet configuration file size

Slide 78

Slide 78 text

Challenges 3rd-party Lua ecosystem not self-service yet configuration file size kernel tuning

Slide 79

Slide 79 text

Challenges 3rd-party Lua ecosystem not self-service yet configuration file size kernel tuning owning availability

Slide 80

Slide 80 text

Conclusion

Slide 81

Slide 81 text

Conclusion NGINX + Lua for HTTP middleware

Slide 82

Slide 82 text

Conclusion NGINX + Lua for HTTP middleware Automated deployment pipeline

Slide 83

Slide 83 text

Conclusion NGINX + Lua for HTTP middleware Automated deployment pipeline Concurrent request limiting

Slide 84

Slide 84 text

Conclusion NGINX + Lua for HTTP middleware Automated deployment pipeline Concurrent request limiting Operational flexibility

Slide 85

Slide 85 text

Thanks