Large-scale Certificate Management on Multi-tenant Web Servers

Ryosuke Matsumoto, Kenji Rikitake*1, Kentaro Kuribayashi
Pepabo R&D Institute, GMO Pepabo, Inc. / *1 KRPEO

2018.07.23 2018 42nd IEEE International Conference on Computer Software & Applications

[abstract] In large-scale certificate management on multitenant web servers, preloading a large number of certificates for managing a large number of hosts under the single server process results in increasing the required memory usage due to the respective page table entry manipulation, which may be poor resource efficiency and reduced capacity. To solve this issue, we propose a method to dynamically load the certificates bound to the hostnames found during the SSL/TLS handshake sequences without preloading, provided the Server Name Indication (SNI) extension is available. We implement
the function of choosing the respective certificates with the ngx_mruby module which extend Web server functions using mruby with small memory footprint while maintaining the execution speed. We also evaluated the proposed method on a Web hosting service of authors’ place of an employer.

Transcript

1. Ryosuke Matsumoto, Kenji Rikitake*1, Kentaro Kuribayashi Pepabo R&D Institute, GMO Pepabo, Inc. / *1 KRPEO 2018.07.23 2018 42nd IEEE International Conference on Computer Software & Applications Large-scale Certificate Management on Multi-tenant Web Servers
2. 1.Introduction 2.Related works 3.Proposed method 4.Evaluation and Consideration in production 5.Conclusion 2 Table of Contents
3. 1. Introduction
4. • Free Domain-validated (DV) certificates such as Let’s Encrypt • Supporting HTTPS becomes relatively low cost • Supporting HTTPS has become an urgent task by Web hosting companies • 4 Background
5. • To provide at a low price by reducing the hardware cost and operation cost • A server process was shared by a large number of hosts • The number of server processes does not depend on the number of hosts • Supporting HTTPS has become an urgent task on the highly integrated multi-tenant architecture • 5 Web server based on the multi-tenant architecture
6. • needs to load the secret key paired with the server certificate for each host at the server process startup • the highly integrated multi-tenant architecture don’t take advantage of reducing the hardware cost and operation cos • takes a lot of time to start the server process • the memory usage of the server process increases 6 To communicate with HTTPS, existing Web server
7. Greatly reduces both the startup time and the memory consumption of the Web server process • by dynamically acquiring corresponding server certificate and secret key data each request • in a highly integrated multi-tenant Web server • implement the new feature of ngx_mruby that can handle the loading phase of certificates 7 Propose large-scale certificate management method
8. 2. Related works
9. • A typical application service of highly integrated multi- tenant architecture • shares server resources among multiple hosts and provides an HTTP server function for each hostname 9 Web hosting service
10. • The function that is identified by a Fully Qualified Domain Name (FQDN) and serves the corresponding content • 10 What is “host”
11. • can accommodate tens of thousands of hosts as a highly integrated multi-tenant architecture • adopts the virtual host method which processes multiple hosts by a single server process group • such as “VirtualHost” configuration of Apache httpd • In a production environment, a single server process may accommodate more than several tens of thousands of hosts • 11 What is highly integrated multi-tenant architecture?
12. • Statically preloading method • loads the certificate associated with the hostname into a memory at the server process startup • reads out the certificate corresponding to an IP address/port or a hostname from memory at each SSL/TLS handshake 12 Existing server certificate management
13. • Web hosting services of our employer use over two million domains • the loading time of configurations and certificates data at the server process startup greatly increases • the memory usage of the server process greatly increases 13 Issues for the existing method
14. • Reverse proxy for the TLS termination • many system adapts the TLS termination • The system needs to first perform TLS communication on the reverse proxy • proxy to the hostnames of all the hosts accommodated in a large number of hosting servers (backend servers) • These increasing resources may cause a serious problem X Related issues
15. 3. Proposed method
16. 1. To support the Server Name Indication (SNI) extension to accommodate hosts 2. To avoid loading all Web server certificates for faster startup of the server processes 3. To ensure that the memory usage of the Web server process is independent of the number of hosts 15 Three requirements for resolve the issues
17. dynamically loading the associated server certificates during each SSL/TLS handshake 

18. • the server certificate and secret key of the request are dynamically loaded from data-store like database, file system or API • based on the requested hostname during the SSL/TLS handshake when an HTTPS request comes 17 Proposed method
19. • the startup time of the web server process does not depend on the number of hosts and memory usage, and does not increase • adding more hosts by changing the configuration does not require the server process reloading • dynamically analyze the certificate location from the hostname 18 Advantage of proposed method
20. Architecture of proposed method 19 'JMF,74 IUUQE
 QSPDFTT $MJFOU 4UBSU
    44-5-4
    IBOETIBLF
    GPS
    FYBNQMFDPN w SVO
    UIF
    GVODUJPO
    BDDPSEJOH
    UP
    UIF
    
    TFSWFSOBNF
    FBDI
    44-5-4
    IBOETIBLF
    w MPBE
    UIF
    TFSWFS
    DFSUJpDBUF
    BOE
    TFDSFU
    LFZ
    EZOBNJDBMMZ
    GSPN
    'JMF
    PS
    ,74
21. Implementation
22. • Add TLS handshake hook to ngx_mruby • ngx_mruby can extend nginx scripting with mruby and process at high speed with less memory usage • calls back an extension function of SSL/TLS handshake behavior using SSL_CTX_set_ cert_cb() • such as custom loading server certificates and secret keys during SSL/TLS handshake • 21 ngx_mruby v1.16.0ʢFeb 2016ʣ
23. 22 Dynamic Server Certificate Management System
24. 23 'JMFCBTFE
    $POpHVSBUJPO
    &YBQNQMF DFSUJpDBUF
    BOE
    DFSUJpDBUF@LFZ
    NFUIPET
    QBTT
    pMF
    PG
    B
    DFSUJpDBUF
    BOE
    B
    QSJWBUF
    LFZ
    
    
25. X DFSUJpDBUF@EBUB
    BOE
    DFSUJpDBUF@LFZ@EBUB
    NFUIPET
    QBTT
    EBUB
    PG
    B
    DFSUJpDBUF
    BOE
    B
    QSJWBUF
    LFZ
    UIFNTFMWFT
    OPU
    B
    pMF
    
    ,74CBTFE
    $POpHSBUJPO 5IF
    TFSWFS
    DFSUJpDBUFT
    BSF
    TUPSFE
    JO
    ,FZ7BMVF
    4UPSF
    ,74
    
26. • Pros • memory usage is independent of the number of hosts • faster startup of the server process • loading new certificate data without reloading server • Cons • cost of dynamic loading each TLS handshake X Pros and Cons of proposed method
27. 4. Evaluation and consideration in the production environment
28. 1. Verification of startup time of existing methods 2. Performance evaluation of the proposed method 3. Evaluation in production 25 Evaluation and consideration
29. 1. Verification of startup time of existing methods
30. Experimental environment 27 4QFDJpDBUJPOT $16 *OUFM
    9FPO
    &
    W
    ()[
    
    DPSF .FNPSZ 
    (CZUFT 4FSWFS /&$
    &YQSFTT3G& 04 $FOU04
    -JOVY
    ,FSOFM
    
31. • nginx version 1.11.13 • generated server certificates and secret keys of the key length of 4096 bits 28 Startup time for one hundred thousand hosts 4QFDJpDBUJPO &YJTUJOH
    NFUIPE 1SPQPTFE
    NFUIPE SFBE
    UJNF
    PG
    TUBSUVQ 
    TFD  VTFS
    DQV
    UJNF
    PG
    TUBSUVQ 
    TFD  TZTUFN
    DQV
    UJNF
    PG
    TUBSUVQ 
    TFD 
32. 2. Performance evaluation of the proposed method
33. •set one certificate to be read in the configuration of the existing method and the proposed method •Existing method use TLS configuration of nginx by default •Proposed method also acquires the certificate from the KVS using the requested hostname as the key • 30 Performance evaluation settings
34. •sent 5 million requests and measured RPS(Request Per Sec) while changing the number of simultaneous connections •uses index.html of 612 bytes enclosed with nginx by default. •cipher suites: ECDHE-RSA-AES128-GCM-SHA256 31 Benchmark environment
35. Performance comparison 32 TJNVMUBOFPVT
    DPOOFDUJPOT
    &YJTJUJOH
    NFUIPE
    QSFMPBE
    SFRTFD 1SPQPTFE
    NFUIPE
    EZOBNJD
    MPBE
    SFRTFD             5IFSF
    JT
    BMNPTU
    OP
    QFSGPSNBODF
    EJ⒎FSFODF
    CFUXFFO
    QSFMPBEJOH
    BOE
    EZOBNJD
    MPBEJOH
    NFUIPE
    
    
36. • the process of dynamically loading a certificate is almost negligible • the cost of encryption and compound processing in SSL/ TLS handshake is very large • the difference of the result is also less than 1% 33 Consideration for performance comparison
37. 3. Evaluation in production
38. • existing method: from March 4 to April 4 with Apache • proposed method: from July 22 to August 22 with nginx • the total number of certificates • Request per seconds, CPU usage, memory usage • Same specifications of server hardware • compared the transition with the same kind of measured values during the one month 35 Compared the transition for one month of 2017
39. Web hosting system in production 36 BEPQUT
    QSPQPTFE
    NFUIPE
40. Reverse proxy specification in production 37 4QFDJpDBUJPO $16 *OUFM
    9FPO
    &
    W
    ()[
    
    UISFBE .FNPSZ 
    (CZUFT 4FSWFS /&$
    &YQSFTT&F. 04 $FOU04
41. Premise:The number of certificate in a month 38 0 5000 10000 15000 20000 25000 1 6 11 16 21 26 31 The number of cer-ficates
    day
    The number of cer-ficate in a month
    dynamic load preload JODSFBTFT
    CZ
    BCPVU
    
    JO
    POF
    NPOUI
    
    JO
    UIF
    QSFMPBEJOH
    NFUIPE JODSFBTFE
    CZ
    NPSF
    UIBO
     
    JO
    POF
    NPOUI
    
42. Premise:The transition of RPS in a month 39 NPSF
    UIBO
    TJY
    UJNFT
    TJODF
    MPBE
    CBMBODJOH
    SBUF
    XBT
    DIBOHFE
    BGUFS
    UIF
    NFUIPE
    SFQMBDFNFOU
43. The transition of CPU usage in a month 40 5IFSF
    JT
    MJUUMF
    EJ⒎FSFODF
    
    CFUXFFO
    UIF
    QSPQPTFE
    NFUIPE
    BOE
    UIF
    FYJTUJOH
    NFUIPE
    
44. The transition of memory usage in a month 41 UIF
    QSPQPTFE
    NFUIPE
    EPFT
    OPU
    TJHOJpDBOUMZ
    JODSFBTF
    JO
    NFNPSZ
    VTBHF
    EFQFOEJOH
    PO
    UIF
    OVNCFS
    PG
    DFSUJpDBUFT
    
    NFNPSZ
    VTBHF
    JODSFBTFT
    CZ
    BCPVU
    
    (#ZUFT
    
    BMPOH
    XJUI
    UIF
    OVNCFS
    PG
    DFSUJpDBUFT
    JODSFBTFT
    CZ
    BCPVU
    
    
45. 4. The discussion of the evaluation result
46. • To process 20000 certificates by the existing method • 50 GBytes of additional memory is required • The proposed method can process 20000 certificates with about 3 GBytes X The discussion of the evaluation result
47. • Existing method: requires 500 GBytes memory • requires more than 15 servers with 32 Gbytes memory • Proposed method: even one server can process • can process 20000 certificates with about 3 GBytes • As the number of 10000 certificates increased, the amount of memory used hardly increased The transition of memory usage X the number of certificates reaches 200,000
48. Conclusion
49. • Existing method in highly-integrated multi-tenant • the loading time of configurations and certificates data at the server process startup greatly increases • the memory usage of the server process greatly increases • Proposed method solve the loading time and the memory usage problems using dynamic loading each TLS handshake 43 Conclusion
50. • The experimental results show that performance does not cause a problem in practical use about dynamically loading • Resource usage can be greatly reduced in production • We conclude that the proposed method is one promising method of a practical system design for supporting HTTPS of highly integrated multi-tenant architecture 44 Conclusion