Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Large-scale Certificate Management on Multi-tenant Web Servers

MATSUMOTO Ryosuke
July 23, 2018
5.3k

Large-scale Certificate Management on Multi-tenant Web Servers

Ryosuke Matsumoto, Kenji Rikitake*1, Kentaro Kuribayashi
Pepabo R&D Institute, GMO Pepabo, Inc. / *1 KRPEO

2018.07.23 2018 42nd IEEE International Conference on Computer Software & Applications

[abstract] In large-scale certificate management on multitenant web servers, preloading a large number of certificates for managing a large number of hosts under the single server process results in increasing the required memory usage due to the respective page table entry manipulation, which may be poor resource efficiency and reduced capacity. To solve this issue, we propose a method to dynamically load the certificates bound to the hostnames found during the SSL/TLS handshake sequences without preloading, provided the Server Name Indication (SNI) extension is available. We implement
the function of choosing the respective certificates with the ngx_mruby module which extend Web server functions using mruby with small memory footprint while maintaining the execution speed. We also evaluated the proposed method on a Web hosting service of authors’ place of an employer.

MATSUMOTO Ryosuke

July 23, 2018
Tweet

More Decks by MATSUMOTO Ryosuke

Transcript

  1. Ryosuke Matsumoto, Kenji Rikitake*1, Kentaro Kuribayashi
    Pepabo R&D Institute, GMO Pepabo, Inc. / *1 KRPEO
    2018.07.23 2018 42nd IEEE International Conference on Computer Software & Applications
    Large-scale Certificate Management
    on Multi-tenant Web Servers

    View full-size slide

  2. 1.Introduction
    2.Related works
    3.Proposed method
    4.Evaluation and Consideration in production
    5.Conclusion
    2
    Table of Contents

    View full-size slide

  3. 1.
    Introduction

    View full-size slide

  4. • Free Domain-validated (DV) certificates such as Let’s Encrypt
    • Supporting HTTPS becomes relatively low cost
    • Supporting HTTPS has become an urgent task by Web
    hosting companies

    4
    Background

    View full-size slide

  5. • To provide at a low price by reducing the hardware cost and
    operation cost
    • A server process was shared by a large number of hosts
    • The number of server processes does not depend on the
    number of hosts
    • Supporting HTTPS has become an urgent task on the highly
    integrated multi-tenant architecture

    5
    Web server based on the multi-tenant architecture

    View full-size slide

  6. • needs to load the secret key paired with the server
    certificate for each host at the server process startup
    • the highly integrated multi-tenant architecture don’t take
    advantage of reducing the hardware cost and operation cos
    • takes a lot of time to start the server process
    • the memory usage of the server process increases
    6
    To communicate with HTTPS, existing Web server

    View full-size slide

  7. Greatly reduces both the startup time and the memory
    consumption of the Web server process
    • by dynamically acquiring corresponding server certificate
    and secret key data each request
    • in a highly integrated multi-tenant Web server
    • implement the new feature of ngx_mruby that can handle
    the loading phase of certificates
    7
    Propose large-scale certificate management method

    View full-size slide

  8. 2.
    Related works

    View full-size slide

  9. • A typical application service of highly integrated multi-
    tenant architecture
    • shares server resources among multiple hosts and provides
    an HTTP server function for each hostname
    9
    Web hosting service

    View full-size slide

  10. • The function that is identified by a Fully Qualified Domain
    Name (FQDN) and serves the corresponding content

    10
    What is “host”

    View full-size slide

  11. • can accommodate tens of thousands of hosts as a highly
    integrated multi-tenant architecture
    • adopts the virtual host method which processes multiple
    hosts by a single server process group
    • such as “VirtualHost” configuration of Apache httpd
    • In a production environment, a single server process may
    accommodate more than several tens of thousands of hosts

    11
    What is highly integrated multi-tenant architecture?

    View full-size slide

  12. • Statically preloading method
    • loads the certificate associated with the hostname into a
    memory at the server process startup
    • reads out the certificate corresponding to an IP address/port
    or a hostname from memory at each SSL/TLS handshake
    12
    Existing server certificate management

    View full-size slide

  13. • Web hosting services of our employer use over two million
    domains
    • the loading time of configurations and certificates data at
    the server process startup greatly increases
    • the memory usage of the server process greatly increases
    13
    Issues for the existing method

    View full-size slide

  14. • Reverse proxy for the TLS termination
    • many system adapts the TLS termination
    • The system needs to first perform TLS communication on
    the reverse proxy
    • proxy to the hostnames of all the hosts accommodated in
    a large number of hosting servers (backend servers)
    • These increasing resources may cause a serious problem
    X
    Related issues

    View full-size slide

  15. 3.
    Proposed method

    View full-size slide

  16. 1. To support the Server Name Indication (SNI) extension to
    accommodate hosts
    2. To avoid loading all Web server certificates for faster
    startup of the server processes
    3. To ensure that the memory usage of the Web server
    process is independent of the number of hosts
    15
    Three requirements for resolve the issues

    View full-size slide

  17. dynamically loading the associated
    server certificates during each SSL/TLS
    handshake

    View full-size slide

  18. • the server certificate and secret key of the request are
    dynamically loaded from data-store like database, file
    system or API
    • based on the requested hostname during the SSL/TLS
    handshake when an HTTPS request comes
    17
    Proposed method

    View full-size slide

  19. • the startup time of the web server process does not depend
    on the number of hosts and memory usage, and does not
    increase
    • adding more hosts by changing the configuration does not
    require the server process reloading
    • dynamically analyze the certificate location from the
    hostname
    18
    Advantage of proposed method

    View full-size slide

  20. Architecture of proposed method
    19
    'JMF,74
    IUUQE

    QSPDFTT
    $MJFOU
    4UBSU44-5-4IBOETIBLFGPS
    FYBNQMFDPN
    w SVO UIF GVODUJPO BDDPSEJOH UP UIF

    TFSWFSOBNFFBDI44-5-4IBOETIBLF
    w MPBE UIF TFSWFS DFSUJpDBUF BOE TFDSFU
    LFZEZOBNJDBMMZGSPN'JMFPS,74

    View full-size slide

  21. Implementation

    View full-size slide

  22. • Add TLS handshake hook to ngx_mruby
    • ngx_mruby can extend nginx scripting with mruby and
    process at high speed with less memory usage
    • calls back an extension function of SSL/TLS handshake
    behavior using SSL_CTX_set_ cert_cb()
    • such as custom loading server certificates and secret keys
    during SSL/TLS handshake

    21
    ngx_mruby v1.16.0ʢFeb 2016ʣ

    View full-size slide

  23. 22
    Dynamic Server Certificate Management System

    View full-size slide

  24. 23
    'JMFCBTFE$POpHVSBUJPO&YBQNQMF
    DFSUJpDBUFBOEDFSUJpDBUF@LFZNFUIPETQBTTpMF
    PGBDFSUJpDBUFBOEBQSJWBUFLFZ

    View full-size slide

  25. X
    DFSUJpDBUF@EBUBBOEDFSUJpDBUF@LFZ@EBUBNFUIPETQBTTEBUB
    PGBDFSUJpDBUFBOEBQSJWBUFLFZUIFNTFMWFT OPUBpMF

    ,74CBTFE$POpHSBUJPO
    5IFTFSWFSDFSUJpDBUFTBSFTUPSFE
    JO,FZ7BMVF4UPSF ,74

    View full-size slide

  26. • Pros
    • memory usage is independent of the number of hosts
    • faster startup of the server process
    • loading new certificate data without reloading server
    • Cons
    • cost of dynamic loading each TLS handshake
    X
    Pros and Cons of proposed method

    View full-size slide

  27. 4.
    Evaluation and consideration
    in the production environment

    View full-size slide

  28. 1. Verification of startup time of existing methods
    2. Performance evaluation of the proposed method
    3. Evaluation in production
    25
    Evaluation and consideration

    View full-size slide

  29. 1. Verification of startup time
    of existing methods

    View full-size slide

  30. Experimental environment
    27
    4QFDJpDBUJPOT
    $16 *OUFM9FPO&W()[DPSF
    .FNPSZ (CZUFT
    4FSWFS /&$&YQSFTT3G&
    04 $FOU04-JOVY,FSOFM

    View full-size slide

  31. • nginx version 1.11.13
    • generated server certificates and secret keys of the key
    length of 4096 bits
    28
    Startup time for one hundred thousand hosts
    4QFDJpDBUJPO &YJTUJOHNFUIPE 1SPQPTFENFUIPE
    SFBEUJNFPGTUBSUVQ TFD
    VTFSDQVUJNFPGTUBSUVQ TFD
    TZTUFNDQVUJNFPGTUBSUVQ TFD

    View full-size slide

  32. 2. Performance evaluation of the
    proposed method

    View full-size slide

  33. •set one certificate to be read in the configuration of
    the existing method and the proposed method
    •Existing method use TLS configuration of nginx by
    default
    •Proposed method also acquires the certificate from
    the KVS using the requested hostname as the key

    30
    Performance evaluation settings

    View full-size slide

  34. •sent 5 million requests and measured RPS(Request Per
    Sec) while changing the number of simultaneous
    connections
    •uses index.html of 612 bytes enclosed with nginx by
    default.
    •cipher suites: ECDHE-RSA-AES128-GCM-SHA256
    31
    Benchmark environment

    View full-size slide

  35. Performance comparison
    32
    TJNVMUBOFPVT
    DPOOFDUJPOT
    &YJTJUJOHNFUIPE
    QSFMPBE
    SFRTFD

    1SPQPTFENFUIPE
    EZOBNJDMPBE
    SFRTFD





    5IFSFJTBMNPTUOPQFSGPSNBODFEJ⒎FSFODFCFUXFFOQSFMPBEJOHBOE
    EZOBNJDMPBEJOHNFUIPE

    View full-size slide

  36. • the process of dynamically loading a certificate is almost
    negligible
    • the cost of encryption and compound processing in SSL/
    TLS handshake is very large
    • the difference of the result is also less than 1%
    33
    Consideration for performance comparison

    View full-size slide

  37. 3. Evaluation in production

    View full-size slide

  38. • existing method: from March 4 to April 4 with Apache
    • proposed method: from July 22 to August 22 with nginx
    • the total number of certificates
    • Request per seconds, CPU usage, memory usage
    • Same specifications of server hardware
    • compared the transition with the same kind of measured
    values during the one month
    35
    Compared the transition for one month of 2017

    View full-size slide

  39. Web hosting system in production
    36
    BEPQUTQSPQPTFENFUIPE

    View full-size slide

  40. Reverse proxy specification in production
    37
    4QFDJpDBUJPO
    $16 *OUFM9FPO&W()[UISFBE
    .FNPSZ (CZUFT
    4FSWFS /&$&YQSFTT&F.
    04 $FOU04

    View full-size slide

  41. Premise:The number of certificate in a month
    38
    0
    5000
    10000
    15000
    20000
    25000
    1 6 11 16 21 26 31
    The number of cer-ficates
    day
    The number of cer-ficate in a month
    dynamic load preload
    JODSFBTFTCZBCPVUJOPOFNPOUI
    JOUIFQSFMPBEJOHNFUIPE
    JODSFBTFECZNPSFUIBO JOPOFNPOUI

    View full-size slide

  42. Premise:The transition of RPS in a month
    39
    NPSFUIBOTJYUJNFT
    TJODFMPBECBMBODJOHSBUFXBTDIBOHFEBGUFSUIFNFUIPESFQMBDFNFOU

    View full-size slide

  43. The transition of CPU usage in a month
    40
    5IFSFJTMJUUMFEJ⒎FSFODF
    CFUXFFOUIFQSPQPTFENFUIPEBOEUIFFYJTUJOHNFUIPE

    View full-size slide

  44. The transition of memory usage in a month
    41
    UIFQSPQPTFENFUIPEEPFTOPUTJHOJpDBOUMZJODSFBTFJONFNPSZVTBHF
    EFQFOEJOHPOUIFOVNCFSPGDFSUJpDBUFT

    NFNPSZVTBHFJODSFBTFTCZBCPVU(#ZUFT
    BMPOHXJUIUIFOVNCFSPGDFSUJpDBUFTJODSFBTFTCZBCPVU

    View full-size slide

  45. 4. The discussion of the evaluation result

    View full-size slide

  46. • To process 20000 certificates by the existing method
    • 50 GBytes of additional memory is required
    • The proposed method can process 20000 certificates with
    about 3 GBytes
    X
    The discussion of the evaluation result

    View full-size slide

  47. • Existing method: requires 500 GBytes memory
    • requires more than 15 servers with 32 Gbytes memory
    • Proposed method: even one server can process
    • can process 20000 certificates with about 3 GBytes
    • As the number of 10000 certificates increased, the amount
    of memory used hardly increased The transition of
    memory usage
    X
    the number of certificates reaches 200,000

    View full-size slide

  48. • Existing method in highly-integrated multi-tenant
    • the loading time of configurations and certificates data at
    the server process startup greatly increases
    • the memory usage of the server process greatly increases
    • Proposed method solve the loading time and the memory
    usage problems using dynamic loading each TLS handshake
    43
    Conclusion

    View full-size slide

  49. • The experimental results show that performance does not
    cause a problem in practical use about dynamically loading
    • Resource usage can be greatly reduced in production
    • We conclude that the proposed method is one promising
    method of a practical system design for supporting HTTPS
    of highly integrated multi-tenant architecture
    44
    Conclusion

    View full-size slide