Large-scale Certificate Management on Multi-tenant Web Servers

Slide 1

Slide 1 text

Ryosuke Matsumoto, Kenji Rikitake*1, Kentaro Kuribayashi Pepabo R&D Institute, GMO Pepabo, Inc. / *1 KRPEO 2018.07.23 2018 42nd IEEE International Conference on Computer Software & Applications Large-scale Certiﬁcate Management on Multi-tenant Web Servers

Slide 2

Slide 2 text

1.Introduction 2.Related works 3.Proposed method 4.Evaluation and Consideration in production 5.Conclusion 2 Table of Contents

Slide 3

Slide 3 text

1. Introduction

Slide 4

Slide 4 text

• Free Domain-validated (DV) certiﬁcates such as Let’s Encrypt • Supporting HTTPS becomes relatively low cost • Supporting HTTPS has become an urgent task by Web hosting companies • 4 Background

Slide 5

Slide 5 text

• To provide at a low price by reducing the hardware cost and operation cost • A server process was shared by a large number of hosts • The number of server processes does not depend on the number of hosts • Supporting HTTPS has become an urgent task on the highly integrated multi-tenant architecture • 5 Web server based on the multi-tenant architecture

Slide 6

Slide 6 text

• needs to load the secret key paired with the server certiﬁcate for each host at the server process startup • the highly integrated multi-tenant architecture don’t take advantage of reducing the hardware cost and operation cos • takes a lot of time to start the server process • the memory usage of the server process increases 6 To communicate with HTTPS, existing Web server

Slide 7

Slide 7 text

Greatly reduces both the startup time and the memory consumption of the Web server process • by dynamically acquiring corresponding server certificate and secret key data each request • in a highly integrated multi-tenant Web server • implement the new feature of ngx_mruby that can handle the loading phase of certificates 7 Propose large-scale certificate management method

Slide 8

Slide 8 text

2. Related works

Slide 9

Slide 9 text

• A typical application service of highly integrated multi- tenant architecture • shares server resources among multiple hosts and provides an HTTP server function for each hostname 9 Web hosting service

Slide 10

Slide 10 text

• The function that is identiﬁed by a Fully Qualiﬁed Domain Name (FQDN) and serves the corresponding content • 10 What is “host”

Slide 11

Slide 11 text

• can accommodate tens of thousands of hosts as a highly integrated multi-tenant architecture • adopts the virtual host method which processes multiple hosts by a single server process group • such as “VirtualHost” conﬁguration of Apache httpd • In a production environment, a single server process may accommodate more than several tens of thousands of hosts • 11 What is highly integrated multi-tenant architecture?

Slide 12

Slide 12 text

• Statically preloading method • loads the certificate associated with the hostname into a memory at the server process startup • reads out the certificate corresponding to an IP address/port or a hostname from memory at each SSL/TLS handshake 12 Existing server certificate management

Slide 13

Slide 13 text

• Web hosting services of our employer use over two million domains • the loading time of conﬁgurations and certiﬁcates data at the server process startup greatly increases • the memory usage of the server process greatly increases 13 Issues for the existing method

Slide 14

Slide 14 text

• Reverse proxy for the TLS termination • many system adapts the TLS termination • The system needs to ﬁrst perform TLS communication on the reverse proxy • proxy to the hostnames of all the hosts accommodated in a large number of hosting servers (backend servers) • These increasing resources may cause a serious problem X Related issues

Slide 15

Slide 15 text

3. Proposed method

Slide 16

Slide 16 text

1. To support the Server Name Indication (SNI) extension to accommodate hosts 2. To avoid loading all Web server certiﬁcates for faster startup of the server processes 3. To ensure that the memory usage of the Web server process is independent of the number of hosts 15 Three requirements for resolve the issues

Slide 17

Slide 17 text

dynamically loading the associated server certiﬁcates during each SSL/TLS handshake  

Slide 18

Slide 18 text

• the server certiﬁcate and secret key of the request are dynamically loaded from data-store like database, ﬁle system or API • based on the requested hostname during the SSL/TLS handshake when an HTTPS request comes 17 Proposed method

Slide 19

Slide 19 text

• the startup time of the web server process does not depend on the number of hosts and memory usage, and does not increase • adding more hosts by changing the conﬁguration does not require the server process reloading • dynamically analyze the certiﬁcate location from the hostname 18 Advantage of proposed method

Slide 20

Slide 20 text

Architecture of proposed method 19 'JMF,74 IUUQE  QSPDFTT $MJFOU 4UBSU44-5-4IBOETIBLFGPS FYBNQMFDPN w SVO UIF GVODUJPO BDDPSEJOH UP UIF TFSWFSOBNFFBDI44-5-4IBOETIBLF w MPBE UIF TFSWFS DFSUJpDBUF BOE TFDSFU LFZEZOBNJDBMMZGSPN'JMFPS,74

Slide 21

Slide 21 text

Implementation

Slide 22

Slide 22 text

• Add TLS handshake hook to ngx_mruby • ngx_mruby can extend nginx scripting with mruby and process at high speed with less memory usage • calls back an extension function of SSL/TLS handshake behavior using SSL_CTX_set_ cert_cb() • such as custom loading server certiﬁcates and secret keys during SSL/TLS handshake • 21 ngx_mruby v1.16.0ʢFeb 2016ʣ

Slide 23

Slide 23 text

22 Dynamic Server Certiﬁcate Management System

Slide 24

Slide 24 text

23 'JMFCBTFE$POpHVSBUJPO&YBQNQMF DFSUJpDBUFBOEDFSUJpDBUF@LFZNFUIPETQBTTpMF PGBDFSUJpDBUFBOEBQSJWBUFLFZ

Slide 25

Slide 25 text

X DFSUJpDBUF@EBUBBOEDFSUJpDBUF@LFZ@EBUBNFUIPETQBTTEBUB PGBDFSUJpDBUFBOEBQSJWBUFLFZUIFNTFMWFT OPUBpMF ,74CBTFE$POpHSBUJPO 5IFTFSWFSDFSUJpDBUFTBSFTUPSFE JO,FZ7BMVF4UPSF ,74

Slide 26

Slide 26 text

• Pros • memory usage is independent of the number of hosts • faster startup of the server process • loading new certiﬁcate data without reloading server • Cons • cost of dynamic loading each TLS handshake X Pros and Cons of proposed method

Slide 27

Slide 27 text

4. Evaluation and consideration in the production environment

Slide 28

Slide 28 text

1. Veriﬁcation of startup time of existing methods 2. Performance evaluation of the proposed method 3. Evaluation in production 25 Evaluation and consideration

Slide 29

Slide 29 text

1. Veriﬁcation of startup time of existing methods

Slide 30

Slide 30 text

Experimental environment 27 4QFDJpDBUJPOT $16 *OUFM9FPO&W()[DPSF .FNPSZ (CZUFT 4FSWFS /&$&YQSFTT3G& 04 $FOU04-JOVY,FSOFM

Slide 31

Slide 31 text

• nginx version 1.11.13 • generated server certiﬁcates and secret keys of the key length of 4096 bits 28 Startup time for one hundred thousand hosts 4QFDJpDBUJPO &YJTUJOHNFUIPE 1SPQPTFENFUIPE SFBEUJNFPGTUBSUVQ TFD VTFSDQVUJNFPGTUBSUVQ TFD TZTUFNDQVUJNFPGTUBSUVQ TFD

Slide 32

Slide 32 text

2. Performance evaluation of the proposed method

Slide 33

Slide 33 text

•set one certificate to be read in the configuration of the existing method and the proposed method •Existing method use TLS configuration of nginx by default •Proposed method also acquires the certificate from the KVS using the requested hostname as the key • 30 Performance evaluation settings

Slide 34

Slide 34 text

•sent 5 million requests and measured RPS(Request Per Sec) while changing the number of simultaneous connections •uses index.html of 612 bytes enclosed with nginx by default. •cipher suites: ECDHE-RSA-AES128-GCM-SHA256 31 Benchmark environment

Slide 35

Slide 35 text

Performance comparison 32 TJNVMUBOFPVT DPOOFDUJPOT &YJTJUJOHNFUIPE QSFMPBE SFRTFD 1SPQPTFENFUIPE EZOBNJDMPBE SFRTFD 5IFSFJTBMNPTUOPQFSGPSNBODFEJ⒎FSFODFCFUXFFOQSFMPBEJOHBOE EZOBNJDMPBEJOHNFUIPE

Slide 36

Slide 36 text

• the process of dynamically loading a certiﬁcate is almost negligible • the cost of encryption and compound processing in SSL/ TLS handshake is very large • the difference of the result is also less than 1% 33 Consideration for performance comparison

Slide 37

Slide 37 text

3. Evaluation in production

Slide 38

Slide 38 text

• existing method: from March 4 to April 4 with Apache • proposed method: from July 22 to August 22 with nginx • the total number of certiﬁcates • Request per seconds, CPU usage, memory usage • Same speciﬁcations of server hardware • compared the transition with the same kind of measured values during the one month 35 Compared the transition for one month of 2017

Slide 39

Slide 39 text

Web hosting system in production 36 BEPQUTQSPQPTFENFUIPE

Slide 40

Slide 40 text

Reverse proxy speciﬁcation in production 37 4QFDJpDBUJPO $16 *OUFM9FPO&W()[UISFBE .FNPSZ (CZUFT 4FSWFS /&$&YQSFTT&F. 04 $FOU04

Slide 41

Slide 41 text

Premise:The number of certificate in a month 38 0 5000 10000 15000 20000 25000 1 6 11 16 21 26 31 The number of cer-ficates day The number of cer-ficate in a month dynamic load preload JODSFBTFTCZBCPVUJOPOFNPOUI JOUIFQSFMPBEJOHNFUIPE JODSFBTFECZNPSFUIBO JOPOFNPOUI

Slide 42

Slide 42 text

Premise:The transition of RPS in a month 39 NPSFUIBOTJYUJNFT TJODFMPBECBMBODJOHSBUFXBTDIBOHFEBGUFSUIFNFUIPESFQMBDFNFOU

Slide 43

Slide 43 text

The transition of CPU usage in a month 40 5IFSFJTMJUUMFEJ⒎FSFODF CFUXFFOUIFQSPQPTFENFUIPEBOEUIFFYJTUJOHNFUIPE

Slide 44

Slide 44 text

The transition of memory usage in a month 41 UIFQSPQPTFENFUIPEEPFTOPUTJHOJpDBOUMZJODSFBTFJONFNPSZVTBHF EFQFOEJOHPOUIFOVNCFSPGDFSUJpDBUFT NFNPSZVTBHFJODSFBTFTCZBCPVU(#ZUFT BMPOHXJUIUIFOVNCFSPGDFSUJpDBUFTJODSFBTFTCZBCPVU

Slide 45

Slide 45 text

4. The discussion of the evaluation result

Slide 46

Slide 46 text

• To process 20000 certiﬁcates by the existing method • 50 GBytes of additional memory is required • The proposed method can process 20000 certiﬁcates with about 3 GBytes X The discussion of the evaluation result

Slide 47

Slide 47 text

• Existing method: requires 500 GBytes memory • requires more than 15 servers with 32 Gbytes memory • Proposed method: even one server can process • can process 20000 certificates with about 3 GBytes • As the number of 10000 certificates increased, the amount of memory used hardly increased The transition of memory usage X the number of certificates reaches 200,000

Slide 48

Slide 48 text

Conclusion

Slide 49

Slide 49 text

• Existing method in highly-integrated multi-tenant • the loading time of conﬁgurations and certiﬁcates data at the server process startup greatly increases • the memory usage of the server process greatly increases • Proposed method solve the loading time and the memory usage problems using dynamic loading each TLS handshake 43 Conclusion

Slide 50

Slide 50 text

• The experimental results show that performance does not cause a problem in practical use about dynamically loading • Resource usage can be greatly reduced in production • We conclude that the proposed method is one promising method of a practical system design for supporting HTTPS of highly integrated multi-tenant architecture 44 Conclusion