Large-scale Certificate Management on Multi-tenant Web Servers

2b692bd83f4418103142a053ecf5ff59?s=47 MATSUMOTO Ryosuke
July 23, 2018
780

Large-scale Certificate Management on Multi-tenant Web Servers

Ryosuke Matsumoto, Kenji Rikitake*1, Kentaro Kuribayashi
Pepabo R&D Institute, GMO Pepabo, Inc. / *1 KRPEO

2018.07.23 2018 42nd IEEE International Conference on Computer Software & Applications

[abstract] In large-scale certificate management on multitenant web servers, preloading a large number of certificates for managing a large number of hosts under the single server process results in increasing the required memory usage due to the respective page table entry manipulation, which may be poor resource efficiency and reduced capacity. To solve this issue, we propose a method to dynamically load the certificates bound to the hostnames found during the SSL/TLS handshake sequences without preloading, provided the Server Name Indication (SNI) extension is available. We implement
the function of choosing the respective certificates with the ngx_mruby module which extend Web server functions using mruby with small memory footprint while maintaining the execution speed. We also evaluated the proposed method on a Web hosting service of authors’ place of an employer.

2b692bd83f4418103142a053ecf5ff59?s=128

MATSUMOTO Ryosuke

July 23, 2018
Tweet

Transcript

  1. Ryosuke Matsumoto, Kenji Rikitake*1, Kentaro Kuribayashi Pepabo R&D Institute, GMO

    Pepabo, Inc. / *1 KRPEO 2018.07.23 2018 42nd IEEE International Conference on Computer Software & Applications Large-scale Certificate Management on Multi-tenant Web Servers
  2. 1.Introduction 2.Related works 3.Proposed method 4.Evaluation and Consideration in production

    5.Conclusion 2 Table of Contents
  3. 1. Introduction

  4. • Free Domain-validated (DV) certificates such as Let’s Encrypt •

    Supporting HTTPS becomes relatively low cost • Supporting HTTPS has become an urgent task by Web hosting companies • 4 Background
  5. • To provide at a low price by reducing the

    hardware cost and operation cost • A server process was shared by a large number of hosts • The number of server processes does not depend on the number of hosts • Supporting HTTPS has become an urgent task on the highly integrated multi-tenant architecture • 5 Web server based on the multi-tenant architecture
  6. • needs to load the secret key paired with the

    server certificate for each host at the server process startup • the highly integrated multi-tenant architecture don’t take advantage of reducing the hardware cost and operation cos • takes a lot of time to start the server process • the memory usage of the server process increases 6 To communicate with HTTPS, existing Web server
  7. Greatly reduces both the startup time and the memory consumption

    of the Web server process • by dynamically acquiring corresponding server certificate and secret key data each request • in a highly integrated multi-tenant Web server • implement the new feature of ngx_mruby that can handle the loading phase of certificates 7 Propose large-scale certificate management method
  8. 2. Related works

  9. • A typical application service of highly integrated multi- tenant

    architecture • shares server resources among multiple hosts and provides an HTTP server function for each hostname 9 Web hosting service
  10. • The function that is identified by a Fully Qualified

    Domain Name (FQDN) and serves the corresponding content • 10 What is “host”
  11. • can accommodate tens of thousands of hosts as a

    highly integrated multi-tenant architecture • adopts the virtual host method which processes multiple hosts by a single server process group • such as “VirtualHost” configuration of Apache httpd • In a production environment, a single server process may accommodate more than several tens of thousands of hosts • 11 What is highly integrated multi-tenant architecture?
  12. • Statically preloading method • loads the certificate associated with

    the hostname into a memory at the server process startup • reads out the certificate corresponding to an IP address/port or a hostname from memory at each SSL/TLS handshake 12 Existing server certificate management
  13. • Web hosting services of our employer use over two

    million domains • the loading time of configurations and certificates data at the server process startup greatly increases • the memory usage of the server process greatly increases 13 Issues for the existing method
  14. • Reverse proxy for the TLS termination • many system

    adapts the TLS termination • The system needs to first perform TLS communication on the reverse proxy • proxy to the hostnames of all the hosts accommodated in a large number of hosting servers (backend servers) • These increasing resources may cause a serious problem X Related issues
  15. 3. Proposed method

  16. 1. To support the Server Name Indication (SNI) extension to

    accommodate hosts 2. To avoid loading all Web server certificates for faster startup of the server processes 3. To ensure that the memory usage of the Web server process is independent of the number of hosts 15 Three requirements for resolve the issues
  17. dynamically loading the associated server certificates during each SSL/TLS handshake

  18. • the server certificate and secret key of the request

    are dynamically loaded from data-store like database, file system or API • based on the requested hostname during the SSL/TLS handshake when an HTTPS request comes 17 Proposed method
  19. • the startup time of the web server process does

    not depend on the number of hosts and memory usage, and does not increase • adding more hosts by changing the configuration does not require the server process reloading • dynamically analyze the certificate location from the hostname 18 Advantage of proposed method
  20. Architecture of proposed method 19 'JMF,74 IUUQE
 QSPDFTT $MJFOU 4UBSU44-5-4IBOETIBLFGPS

    FYBNQMFDPN w SVO UIF GVODUJPO BDDPSEJOH UP UIF  TFSWFSOBNFFBDI44-5-4IBOETIBLF w MPBE UIF TFSWFS DFSUJpDBUF BOE TFDSFU LFZEZOBNJDBMMZGSPN'JMFPS,74
  21. Implementation

  22. • Add TLS handshake hook to ngx_mruby • ngx_mruby can

    extend nginx scripting with mruby and process at high speed with less memory usage • calls back an extension function of SSL/TLS handshake behavior using SSL_CTX_set_ cert_cb() • such as custom loading server certificates and secret keys during SSL/TLS handshake • 21 ngx_mruby v1.16.0ʢFeb 2016ʣ
  23. 22 Dynamic Server Certificate Management System

  24. 23 'JMFCBTFE$POpHVSBUJPO&YBQNQMF DFSUJpDBUFBOEDFSUJpDBUF@LFZNFUIPETQBTTpMF PGBDFSUJpDBUFBOEBQSJWBUFLFZ 

  25. X DFSUJpDBUF@EBUBBOEDFSUJpDBUF@LFZ@EBUBNFUIPETQBTTEBUB PGBDFSUJpDBUFBOEBQSJWBUFLFZUIFNTFMWFT OPUBpMF  ,74CBTFE$POpHSBUJPO 5IFTFSWFSDFSUJpDBUFTBSFTUPSFE JO,FZ7BMVF4UPSF ,74 

  26. • Pros • memory usage is independent of the number

    of hosts • faster startup of the server process • loading new certificate data without reloading server • Cons • cost of dynamic loading each TLS handshake X Pros and Cons of proposed method
  27. 4. Evaluation and consideration in the production environment

  28. 1. Verification of startup time of existing methods 2. Performance

    evaluation of the proposed method 3. Evaluation in production 25 Evaluation and consideration
  29. 1. Verification of startup time of existing methods

  30. Experimental environment 27 4QFDJpDBUJPOT $16 *OUFM9FPO&W()[DPSF .FNPSZ (CZUFT 4FSWFS /&$&YQSFTT3G&

    04 $FOU04-JOVY,FSOFM
  31. • nginx version 1.11.13 • generated server certificates and secret

    keys of the key length of 4096 bits 28 Startup time for one hundred thousand hosts 4QFDJpDBUJPO &YJTUJOHNFUIPE 1SPQPTFENFUIPE SFBEUJNFPGTUBSUVQ TFD  VTFSDQVUJNFPGTUBSUVQ TFD  TZTUFNDQVUJNFPGTUBSUVQ TFD 
  32. 2. Performance evaluation of the proposed method

  33. •set one certificate to be read in the configuration of

    the existing method and the proposed method •Existing method use TLS configuration of nginx by default •Proposed method also acquires the certificate from the KVS using the requested hostname as the key • 30 Performance evaluation settings
  34. •sent 5 million requests and measured RPS(Request Per Sec) while

    changing the number of simultaneous connections •uses index.html of 612 bytes enclosed with nginx by default. •cipher suites: ECDHE-RSA-AES128-GCM-SHA256 31 Benchmark environment
  35. Performance comparison 32 TJNVMUBOFPVT DPOOFDUJPOT &YJTJUJOHNFUIPE QSFMPBE SFRTFD 1SPQPTFENFUIPE EZOBNJDMPBE

    SFRTFD             5IFSFJTBMNPTUOPQFSGPSNBODFEJ⒎FSFODFCFUXFFOQSFMPBEJOHBOE EZOBNJDMPBEJOHNFUIPE 
  36. • the process of dynamically loading a certificate is almost

    negligible • the cost of encryption and compound processing in SSL/ TLS handshake is very large • the difference of the result is also less than 1% 33 Consideration for performance comparison
  37. 3. Evaluation in production

  38. • existing method: from March 4 to April 4 with

    Apache • proposed method: from July 22 to August 22 with nginx • the total number of certificates • Request per seconds, CPU usage, memory usage • Same specifications of server hardware • compared the transition with the same kind of measured values during the one month 35 Compared the transition for one month of 2017
  39. Web hosting system in production 36 BEPQUTQSPQPTFENFUIPE

  40. Reverse proxy specification in production 37 4QFDJpDBUJPO $16 *OUFM9FPO&W()[UISFBE .FNPSZ

    (CZUFT 4FSWFS /&$&YQSFTT&F. 04 $FOU04
  41. Premise:The number of certificate in a month 38 0 5000

    10000 15000 20000 25000 1 6 11 16 21 26 31 The number of cer-ficates day The number of cer-ficate in a month dynamic load preload JODSFBTFTCZBCPVUJOPOFNPOUI JOUIFQSFMPBEJOHNFUIPE JODSFBTFECZNPSFUIBO JOPOFNPOUI
  42. Premise:The transition of RPS in a month 39 NPSFUIBOTJYUJNFT TJODFMPBECBMBODJOHSBUFXBTDIBOHFEBGUFSUIFNFUIPESFQMBDFNFOU

  43. The transition of CPU usage in a month 40 5IFSFJTMJUUMFEJ⒎FSFODF

    CFUXFFOUIFQSPQPTFENFUIPEBOEUIFFYJTUJOHNFUIPE
  44. The transition of memory usage in a month 41 UIFQSPQPTFENFUIPEEPFTOPUTJHOJpDBOUMZJODSFBTFJONFNPSZVTBHF

    EFQFOEJOHPOUIFOVNCFSPGDFSUJpDBUFT  NFNPSZVTBHFJODSFBTFTCZBCPVU(#ZUFT BMPOHXJUIUIFOVNCFSPGDFSUJpDBUFTJODSFBTFTCZBCPVU
  45. 4. The discussion of the evaluation result

  46. • To process 20000 certificates by the existing method •

    50 GBytes of additional memory is required • The proposed method can process 20000 certificates with about 3 GBytes X The discussion of the evaluation result
  47. • Existing method: requires 500 GBytes memory • requires more

    than 15 servers with 32 Gbytes memory • Proposed method: even one server can process • can process 20000 certificates with about 3 GBytes • As the number of 10000 certificates increased, the amount of memory used hardly increased The transition of memory usage X the number of certificates reaches 200,000
  48. Conclusion

  49. • Existing method in highly-integrated multi-tenant • the loading time

    of configurations and certificates data at the server process startup greatly increases • the memory usage of the server process greatly increases • Proposed method solve the loading time and the memory usage problems using dynamic loading each TLS handshake 43 Conclusion
  50. • The experimental results show that performance does not cause

    a problem in practical use about dynamically loading • Resource usage can be greatly reduced in production • We conclude that the proposed method is one promising method of a practical system design for supporting HTTPS of highly integrated multi-tenant architecture 44 Conclusion