Large-scale Certificate Management on Multi-tenant Web Servers

Ryosuke Matsumoto, Kenji Rikitake*1, Kentaro Kuribayashi
Pepabo R&D Institute, GMO Pepabo, Inc. / *1 KRPEO

2018.07.23 2018 42nd IEEE International Conference on Computer Software & Applications

[abstract] In large-scale certificate management on multitenant web servers, preloading a large number of certificates for managing a large number of hosts under the single server process results in increasing the required memory usage due to the respective page table entry manipulation, which may be poor resource efficiency and reduced capacity. To solve this issue, we propose a method to dynamically load the certificates bound to the hostnames found during the SSL/TLS handshake sequences without preloading, provided the Server Name Indication (SNI) extension is available. We implement
the function of choosing the respective certificates with the ngx_mruby module which extend Web server functions using mruby with small memory footprint while maintaining the execution speed. We also evaluated the proposed method on a Web hosting service of authors’ place of an employer.

Transcript

1. Ryosuke Matsumoto, Kenji Rikitake*1, Kentaro Kuribayashi
   Pepabo R&D Institute, GMO Pepabo, Inc. / *1 KRPEO
   2018.07.23 2018 42nd IEEE International Conference on Computer Software & Applications
   Large-scale Certificate Management
   on Multi-tenant Web Servers
   

   View Slide

2. 1.Introduction
   2.Related works
   3.Proposed method
   4.Evaluation and Consideration in production
   5.Conclusion
   2
   Table of Contents
   

   View Slide

3. 1.
   Introduction
   

   View Slide

4. • Free Domain-validated (DV) certificates such as Let’s Encrypt
   • Supporting HTTPS becomes relatively low cost
   • Supporting HTTPS has become an urgent task by Web
   hosting companies
   •
   4
   Background
   

   View Slide

5. • To provide at a low price by reducing the hardware cost and
   operation cost
   • A server process was shared by a large number of hosts
   • The number of server processes does not depend on the
   number of hosts
   • Supporting HTTPS has become an urgent task on the highly
   integrated multi-tenant architecture
   •
   5
   Web server based on the multi-tenant architecture
   

   View Slide

6. • needs to load the secret key paired with the server
   certificate for each host at the server process startup
   • the highly integrated multi-tenant architecture don’t take
   advantage of reducing the hardware cost and operation cos
   • takes a lot of time to start the server process
   • the memory usage of the server process increases
   6
   To communicate with HTTPS, existing Web server
   

   View Slide

7. Greatly reduces both the startup time and the memory
   consumption of the Web server process
   • by dynamically acquiring corresponding server certificate
   and secret key data each request
   • in a highly integrated multi-tenant Web server
   • implement the new feature of ngx_mruby that can handle
   the loading phase of certificates
   7
   Propose large-scale certificate management method
   

   View Slide

8. 2.
   Related works
   

   View Slide

9. • A typical application service of highly integrated multi-
   tenant architecture
   • shares server resources among multiple hosts and provides
   an HTTP server function for each hostname
   9
   Web hosting service
   

   View Slide

10. • The function that is identified by a Fully Qualified Domain
    Name (FQDN) and serves the corresponding content
    •
    10
    What is “host”
    

    View Slide

11. • can accommodate tens of thousands of hosts as a highly
    integrated multi-tenant architecture
    • adopts the virtual host method which processes multiple
    hosts by a single server process group
    • such as “VirtualHost” configuration of Apache httpd
    • In a production environment, a single server process may
    accommodate more than several tens of thousands of hosts
    •
    11
    What is highly integrated multi-tenant architecture?
    

    View Slide

12. • Statically preloading method
    • loads the certificate associated with the hostname into a
    memory at the server process startup
    • reads out the certificate corresponding to an IP address/port
    or a hostname from memory at each SSL/TLS handshake
    12
    Existing server certificate management
    

    View Slide

13. • Web hosting services of our employer use over two million
    domains
    • the loading time of configurations and certificates data at
    the server process startup greatly increases
    • the memory usage of the server process greatly increases
    13
    Issues for the existing method
    

    View Slide

14. • Reverse proxy for the TLS termination
    • many system adapts the TLS termination
    • The system needs to first perform TLS communication on
    the reverse proxy
    • proxy to the hostnames of all the hosts accommodated in
    a large number of hosting servers (backend servers)
    • These increasing resources may cause a serious problem
    X
    Related issues
    

    View Slide

15. 3.
    Proposed method
    

    View Slide

16. 1. To support the Server Name Indication (SNI) extension to
    accommodate hosts
    2. To avoid loading all Web server certificates for faster
    startup of the server processes
    3. To ensure that the memory usage of the Web server
    process is independent of the number of hosts
    15
    Three requirements for resolve the issues
    

    View Slide

17. dynamically loading the associated
    server certificates during each SSL/TLS
    handshake
    

    

    View Slide

18. • the server certificate and secret key of the request are
    dynamically loaded from data-store like database, file
    system or API
    • based on the requested hostname during the SSL/TLS
    handshake when an HTTPS request comes
    17
    Proposed method
    

    View Slide

19. • the startup time of the web server process does not depend
    on the number of hosts and memory usage, and does not
    increase
    • adding more hosts by changing the configuration does not
    require the server process reloading
    • dynamically analyze the certificate location from the
    hostname
    18
    Advantage of proposed method
    

    View Slide

20. Architecture of proposed method
    19
    'JMF,74
    IUUQE

    QSPDFTT
    $MJFOU
    4UBSU44-5-4IBOETIBLFGPS
    FYBNQMFDPN
    w SVO UIF GVODUJPO BDDPSEJOH UP UIF
    
    TFSWFSOBNFFBDI44-5-4IBOETIBLF
    w MPBE UIF TFSWFS DFSUJpDBUF BOE TFDSFU
    LFZEZOBNJDBMMZGSPN'JMFPS,74
    

    View Slide

21. Implementation
    

    View Slide

22. • Add TLS handshake hook to ngx_mruby
    • ngx_mruby can extend nginx scripting with mruby and
    process at high speed with less memory usage
    • calls back an extension function of SSL/TLS handshake
    behavior using SSL_CTX_set_ cert_cb()
    • such as custom loading server certificates and secret keys
    during SSL/TLS handshake
    •
    21
    ngx_mruby v1.16.0ʢFeb 2016ʣ
    

    View Slide

23. 22
    Dynamic Server Certificate Management System
    

    View Slide

24. 23
    'JMFCBTFE$POpHVSBUJPO&YBQNQMF
    DFSUJpDBUFBOEDFSUJpDBUF@LFZNFUIPETQBTTpMF
    PGBDFSUJpDBUFBOEBQSJWBUFLFZ
    
    

    View Slide

25. X
    DFSUJpDBUF@EBUBBOEDFSUJpDBUF@LFZ@EBUBNFUIPETQBTTEBUB
    PGBDFSUJpDBUFBOEBQSJWBUFLFZUIFNTFMWFT OPUBpMF
    
    ,74CBTFE$POpHSBUJPO
    5IFTFSWFSDFSUJpDBUFTBSFTUPSFE
    JO,FZ7BMVF4UPSF ,74
    
    

    View Slide

26. • Pros
    • memory usage is independent of the number of hosts
    • faster startup of the server process
    • loading new certificate data without reloading server
    • Cons
    • cost of dynamic loading each TLS handshake
    X
    Pros and Cons of proposed method
    

    View Slide

27. 4.
    Evaluation and consideration
    in the production environment
    

    View Slide

28. 1. Verification of startup time of existing methods
    2. Performance evaluation of the proposed method
    3. Evaluation in production
    25
    Evaluation and consideration
    

    View Slide

29. 1. Verification of startup time
    of existing methods
    

    View Slide

30. Experimental environment
    27
    4QFDJpDBUJPOT
    $16 *OUFM9FPO&W()[DPSF
    .FNPSZ (CZUFT
    4FSWFS /&$&YQSFTT3G&
    04 $FOU04-JOVY,FSOFM
    

    View Slide

31. • nginx version 1.11.13
    • generated server certificates and secret keys of the key
    length of 4096 bits
    28
    Startup time for one hundred thousand hosts
    4QFDJpDBUJPO &YJTUJOHNFUIPE 1SPQPTFENFUIPE
    SFBEUJNFPGTUBSUVQ TFD
    VTFSDQVUJNFPGTUBSUVQ TFD
    TZTUFNDQVUJNFPGTUBSUVQ TFD
    

    View Slide

32. 2. Performance evaluation of the
    proposed method
    

    View Slide

33. •set one certificate to be read in the configuration of
    the existing method and the proposed method
    •Existing method use TLS configuration of nginx by
    default
    •Proposed method also acquires the certificate from
    the KVS using the requested hostname as the key
    •
    30
    Performance evaluation settings
    

    View Slide

34. •sent 5 million requests and measured RPS(Request Per
    Sec) while changing the number of simultaneous
    connections
    •uses index.html of 612 bytes enclosed with nginx by
    default.
    •cipher suites: ECDHE-RSA-AES128-GCM-SHA256
    31
    Benchmark environment
    

    View Slide

35. Performance comparison
    32
    TJNVMUBOFPVT
    DPOOFDUJPOT
    &YJTJUJOHNFUIPE
    QSFMPBE
    SFRTFD
    
    1SPQPTFENFUIPE
    EZOBNJDMPBE
    SFRTFD
    
    
    
    
    
    5IFSFJTBMNPTUOPQFSGPSNBODFEJ⒎FSFODFCFUXFFOQSFMPBEJOHBOE
    EZOBNJDMPBEJOHNFUIPE
    
    

    View Slide

36. • the process of dynamically loading a certificate is almost
    negligible
    • the cost of encryption and compound processing in SSL/
    TLS handshake is very large
    • the difference of the result is also less than 1%
    33
    Consideration for performance comparison
    

    View Slide

37. 3. Evaluation in production
    

    View Slide

38. • existing method: from March 4 to April 4 with Apache
    • proposed method: from July 22 to August 22 with nginx
    • the total number of certificates
    • Request per seconds, CPU usage, memory usage
    • Same specifications of server hardware
    • compared the transition with the same kind of measured
    values during the one month
    35
    Compared the transition for one month of 2017
    

    View Slide

39. Web hosting system in production
    36
    BEPQUTQSPQPTFENFUIPE
    

    View Slide

40. Reverse proxy specification in production
    37
    4QFDJpDBUJPO
    $16 *OUFM9FPO&W()[UISFBE
    .FNPSZ (CZUFT
    4FSWFS /&$&YQSFTT&F.
    04 $FOU04
    

    View Slide

41. Premise:The number of certificate in a month
    38
    0
    5000
    10000
    15000
    20000
    25000
    1 6 11 16 21 26 31
    The number of cer-ficates
    day
    The number of cer-ficate in a month
    dynamic load preload
    JODSFBTFTCZBCPVUJOPOFNPOUI
    JOUIFQSFMPBEJOHNFUIPE
    JODSFBTFECZNPSFUIBO JOPOFNPOUI
    

    View Slide

42. Premise:The transition of RPS in a month
    39
    NPSFUIBOTJYUJNFT
    TJODFMPBECBMBODJOHSBUFXBTDIBOHFEBGUFSUIFNFUIPESFQMBDFNFOU
    

    View Slide

43. The transition of CPU usage in a month
    40
    5IFSFJTMJUUMFEJ⒎FSFODF
    CFUXFFOUIFQSPQPTFENFUIPEBOEUIFFYJTUJOHNFUIPE
    

    View Slide

44. The transition of memory usage in a month
    41
    UIFQSPQPTFENFUIPEEPFTOPUTJHOJpDBOUMZJODSFBTFJONFNPSZVTBHF
    EFQFOEJOHPOUIFOVNCFSPGDFSUJpDBUFT
    
    NFNPSZVTBHFJODSFBTFTCZBCPVU(#ZUFT
    BMPOHXJUIUIFOVNCFSPGDFSUJpDBUFTJODSFBTFTCZBCPVU
    

    View Slide

45. 4. The discussion of the evaluation result
    

    View Slide

46. • To process 20000 certificates by the existing method
    • 50 GBytes of additional memory is required
    • The proposed method can process 20000 certificates with
    about 3 GBytes
    X
    The discussion of the evaluation result
    

    View Slide

47. • Existing method: requires 500 GBytes memory
    • requires more than 15 servers with 32 Gbytes memory
    • Proposed method: even one server can process
    • can process 20000 certificates with about 3 GBytes
    • As the number of 10000 certificates increased, the amount
    of memory used hardly increased The transition of
    memory usage
    X
    the number of certificates reaches 200,000
    

    View Slide

48. Conclusion
    

    View Slide

49. • Existing method in highly-integrated multi-tenant
    • the loading time of configurations and certificates data at
    the server process startup greatly increases
    • the memory usage of the server process greatly increases
    • Proposed method solve the loading time and the memory
    usage problems using dynamic loading each TLS handshake
    43
    Conclusion
    

    View Slide

50. • The experimental results show that performance does not
    cause a problem in practical use about dynamically loading
    • Resource usage can be greatly reduced in production
    • We conclude that the proposed method is one promising
    method of a practical system design for supporting HTTPS
    of highly integrated multi-tenant architecture
    44
    Conclusion
    

    View Slide