Save 37% off PRO during our Black Friday Sale! »

Reliable High-Performance HTTP Infrastructure with nginx and Lua

Reliable High-Performance HTTP Infrastructure with nginx and Lua

We recently replaced a proprietary API management solution with an in-house implementation built with nginx and Lua that is more robust, higher performance, and has greater visibility. Learn about our development process and the overall architecture that allowed us to write high-level code while enjoying native code performance, and how we leveraged other open source tools like Vagrant, Ansible, and OpenStack to build an automation-rich delivery pipeline. We will also take an in-depth look at our capacity management approach that differs from the rate limiting concept prevalent in the API community.

Ce461b57b6a1f65ce5b6cc1c124673e3?s=128

Sean Cribbs

April 11, 2016
Tweet

Transcript

  1. Reliable High-Performance HTTP Infrastructure with nginx and Lua Sean Cribbs

    Senior Principal Engineer, Comcast Cable @seancribbs
  2. Background

  3. None
  4. Consumer

  5. Internal Consumer

  6. Partner Internal Consumer

  7. API Management

  8. API Management access control capacity management

  9. CodeBig 1 API Consumer

  10. CodeBig 1 API Consumer CDN • traffic shaping • caching

  11. CodeBig 1 API Consumer CDN • traffic shaping • caching

    • access control • rate limiting Vendor APIM
  12. CodeBig 1 API Consumer CDN • traffic shaping • caching

    • access control • rate limiting Vendor APIM Internet Comcast LB • DNS RR • VIP
  13. CodeBig 1 API Consumer CDN • traffic shaping • caching

    • access control • rate limiting Vendor APIM • DMZ intermediary • path-host mapping iAuth Internet Comcast LB • DNS RR • VIP
  14. CodeBig 1 API Consumer CDN • traffic shaping • caching

    • access control • rate limiting Vendor APIM • DMZ intermediary • path-host mapping iAuth Internet Comcast Origin APIs LB • DNS RR • VIP
  15. Challenges

  16. Challenges visibility

  17. Challenges visibility responsibility

  18. Challenges visibility responsibility scope

  19. Challenges visibility responsibility scope latency

  20. Challenges visibility responsibility scope latency security

  21. CodeBig 2

  22. CodeBig 2 simplify architecture

  23. CodeBig 2 simplify architecture increase visibility

  24. CodeBig 2 simplify architecture increase visibility use open-source tools

  25. Architecture

  26. custom  logic HTTP Proxy

  27. Lua

  28. nginx+Lua extension points init access header_filter log init_worker rewrite body_filter

    balancer ssl_certificate set content _by_lua   _by_lua_file     _by_lua_block +
  29. CodeBig Request Phases init access header_filter log

  30. CodeBig Request Phases init access header_filter log request flow

  31. CodeBig Request Phases init access header_filter log Load code and

    configuration request flow
  32. CodeBig Request Phases init access header_filter log Load code and

    configuration Authenticate Rate-limit Tweak request request flow
  33. CodeBig Request Phases init access header_filter log Load code and

    configuration Authenticate Rate-limit Tweak request Tweak response request flow
  34. CodeBig Request Phases init access header_filter log Load code and

    configuration Authenticate Rate-limit Tweak request Tweak response Clean up request flow
  35. local  setmetatable  =  setmetatable   local  _M  =  {}  

    function  _M:new(ctx,  conf)          local  o  =  {                  _ctx  =  ctx,                  _conf  =  conf          }          o.super  =  self          setmetatable(o,  self)          self.__index  =  self          return  o   end   function  _M:access()          return  true   end   function  _M:post_access()          -­‐-­‐  nop   end   function  _M:header_filter()          -­‐-­‐  nop   end   function  _M:log()          -­‐-­‐  nop   end   return  _M
  36. for _, name in ipairs(conf.plugins) do -- load plugin by

    fully qualified name local plugin = require(name):new(ctx, conf) -- exit immediately upon first rejection local is_ok, err = plugin:access() if not is_ok then ngx.status = err.code ngx_say(err.error) ngx.var.access_error = err.error return ngx_exit(ngx.HTTP_OK) end insert(plugins, plugin) end for _, plugin in ipairs(plugins) do plugin:post_access() end
  37. function _M.header_filter() local plugins = ngx.ctx.plugins or {} for _,

    plugin in ipairs(plugins) do plugin:header_filter() end end
  38. # nginx.conf lua_package_path '/usr/share/?/init.lua;/usr/share/?.lua;;'; lua_shared_dict memory 50M; init_by_lua_block { codebig

    = require("codebig") codebig.init(“/path/to/configs“) }; # vhost.conf location / { access_by_lua 'return codebig.access("somehost")'; header_filter_by_lua 'return codebig.header_filter()'; log_by_lua 'return codebig.log()'; }
  39. Lua ~ 3K LoC!!

  40. Intra-Datacenter VIP

  41. Intra-Datacenter VIP haproxy haproxy …

  42. Intra-Datacenter VIP … haproxy haproxy …

  43. Intra-Datacenter VIP Origin APIs … haproxy haproxy …

  44. Cross-Datacenter DC1 DC2 DC3 vod vod acct acct entry-­‐vip-­‐dc1.  A

                 10.1.0.1 vod-­‐dc1.              CNAME      entry-­‐vip-­‐dc1. vod.                      CNAME      vod-­‐dc1. entry-­‐vip-­‐dc2.  A              10.2.0.1 entry-­‐vip-­‐dc3.  A              10.3.0.1 vod-­‐dc2.              CNAME      entry-­‐vip-­‐dc2. VIP VIP VIP vod-­‐dc1-­‐fo.        CNAME      entry-­‐vip-­‐dc1.
  45. Capacity Management

  46. N = XR

  47. N = XR # concurrent requests

  48. N = XR # concurrent requests transaction rate

  49. N = XR # concurrent requests transaction rate response time

  50. N = XR # concurrent requests transaction rate response time

    Little’s Law
  51. client origin APIM 2 req/s 1s N = XR =

    2 req/s x 1s = 2 concurrent
  52. client origin APIM 2 req/s 10s N = XR =

    2 req/s x 10s = 20 concurrent
  53. client origin APIM 2 req/s 10s N = XR =

    2 req/s x 10s = 20 concurrent
  54. Concurrent Request Limiting lua_shared_dict            

     memory    50M;   access_by_lua          …      +1   log_by_lua                …      -­‐1
  55. Deployment

  56. None
  57. Configs in VCS Playbooks in VCS config templates API configs

    vault
 (keys) vip.conf vhost.lua vhost.conf vhost.conf vhost.json nginx.conf ssh
  58. Results

  59. Performance

  60. switch Performance

  61. switch mean 99th Performance

  62. switch ~10x mean 99th Performance

  63. Stability

  64. switch Stability

  65. Impact index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =  request_time

     -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d)
  66. Impact seconds index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =

     request_time  -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d)
  67. Impact seconds index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =

     request_time  -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d) 99th
  68. Impact seconds index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =

     request_time  -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d) 99th max
  69. Successes

  70. Successes great performance improvements

  71. Successes great performance improvements hosting ~400 endpoints

  72. Successes great performance improvements hosting ~400 endpoints > 367MM requests

    a day
  73. Successes great performance improvements hosting ~400 endpoints > 367MM requests

    a day prevented upstream downtime
  74. Challenges

  75. Challenges 3rd-party Lua ecosystem

  76. Challenges 3rd-party Lua ecosystem not self-service yet

  77. Challenges 3rd-party Lua ecosystem not self-service yet configuration file size

  78. Challenges 3rd-party Lua ecosystem not self-service yet configuration file size

    kernel tuning
  79. Challenges 3rd-party Lua ecosystem not self-service yet configuration file size

    kernel tuning owning availability
  80. Conclusion

  81. Conclusion NGINX + Lua for HTTP middleware

  82. Conclusion NGINX + Lua for HTTP middleware Automated deployment pipeline

  83. Conclusion NGINX + Lua for HTTP middleware Automated deployment pipeline

    Concurrent request limiting
  84. Conclusion NGINX + Lua for HTTP middleware Automated deployment pipeline

    Concurrent request limiting Operational flexibility
  85. Thanks