実践nginx〜メルカリの場合〜

by Tatsuhiko Kubo

Slide 1

Slide 1 text

࣮ફnginx ʙϝϧΧϦͷ৔߹ʙ Tatsuhiko Kubo@cubicdaiya ALM@2015/04/21

Slide 2

Slide 2 text

ࣗݾ঺հ • ٱอୡ඙(Tatsuhiko Kubo) • bokko@cubicdaiya • Software Engineer in Infrastructure Engineering • Mercari, Inc. • Favorites: Go, C, nginx

Slide 3

Slide 3 text

OSS࡞ͬͨΓίϯτϦϏϡʔτͨ͠Γ

Slide 4

Slide 4 text

Agenda ϝϧΧϦͰͷnginxͷ׆༻ࣄྫʹ͍ͭͯ

Slide 5

Slide 5 text

nginx • ੈքͰೋ൪໨ʹར༻͞Ε͍ͯΔOSSͷHTTPαʔό • C10Kʹ଱͑ΒΕΔΞʔΩςΫνϟ • Πϕϯτۦಈ • ϊϯϒϩοΩϯάI/O • ඇಉظI/O • ܰྔͰߴ଎

Slide 6

Slide 6 text

ϝϧΧϦͰΑ͋͘Δޫܠ ࣮ࡍʹ͸͋Δఔ౓ू໿͍ͯ͠·͢

Slide 7

Slide 7 text

Agenda • Reverse proxy • SSL termination • SPDY gateway • L7 load balancer

Slide 8

Slide 8 text

Agenda • Reverse proxy • SSL termination • SPDY gateway • L7 load balancer

Slide 9

Slide 9 text

ϦόʔεϓϩΩγͱͯ͠ͷnginx • ϦΫΤετͷϩΪϯά • ΞΫηε੍ޚ • ίϯςϯπͷѹॖɾΩϟογϡ • όοϑΝϦϯά • etc…

Slide 10

Slide 10 text

nginx.confͷઃఆྫ server { listen 443 ssl spdy; server_name xxx.yyy; # ϓϩΩγઃఆ proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Connection ""; proxy_http_version 1.1; proxy_buffers 50 8k; # ੩తίϯςϯπ͸nginxͰฦ͢ location ~ /(styles|js|images)/ { root /usr/share/zabbix; expires 30d; } # Zabbix(༻ͷgoogle_auth_proxy)΁ϓϩΩγ location / { proxy_pass http://google_auth_proxy_for_zabbix; } }

Slide 11

Slide 11 text

લஈʹnginxΛஔ͘ϝϦοτ • HTTP or HTTPSαʔόʹඞཁͳλεΫΛҰ௨Γ͜ͳͤΔ • ΞϓϦέʔγϣϯαʔό(e.g. Unicorn)ʹ଍Γͳ͍ػೳΛิ׬ • ωοτϫʔΫϨΠςϯγͷվળ • KeepAlive • gzipѹॖ • TLS Session (Cache | Tickets)ɺOCSP Stapling • SPDY • etc…

Slide 12

Slide 12 text

Agenda • Reverse proxy • SSL termination • SPDY gateway • L7 load balancer

Slide 13

Slide 13 text

HTTPS௨৴ͷߴ଎Խ • ҰൠʹHTTPS௨৴Ͱ͸TCP 3-way handshakeʹՃ͑ͯ TLS 3-way handshake͕ൃੜ͢ΔͷͰHTTP௨৴ΑΓ஗͘ͳΔ • HTTPS௨৴ߴ଎ԽͷͨΊͷࡾछͷਆث • TLS Session Cache • TLS Session Tickets • OCSP Stapling

Slide 14

Slide 14 text

TLS Session Cache • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ Ωϟογϡ • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ • ࣍ճͷTLSϋϯυγΣΠΫΛলུ • CPUͷϦιʔεͷ࡟ݮ΍ϨΠςϯγͷղফʹޮՌ͕͋Δ

Slide 15

Slide 15 text

TLS Session Cache with nginx

Slide 16

Slide 16 text

TLS Session Tickets • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥΠΞϯ τʹ౉͢ • νέοτΛݩʹTLSηογϣϯΛ࠶։ • HTTPSαʔόෳ਺୆Ͱηογϣϯ৘ใΛڞ༗Ͱ͖Δ • εϚϗͩͱαϙʔτ͍ͯ͠Δ୺຤͕গͳ͍…

Slide 17

Slide 17 text

TLS Session Tickets with nginx

Slide 18

Slide 18 text

OCSP Stapling • OCSPʹΑΔSSLূ໌ॻͷࣦޮ֬ೝΛαʔόଆͰߦͬ ͯΩϟογϡ • ΫϥΠΞϯτଆͰ΍ΔͱTLSϋϯυγΣΠΫ࣌ʹϨΠ ςϯγ͕ൃੜ͢Δ • ΍ͬͺΓεϚϗͩͱ͋Μ·ΓରԠͯ͠ͳ͍ • Google Chrome for iOSͩͱରԠͯͨ͠

Slide 19

Slide 19 text

OCSP Stapling with nginx ssl_stapling on; ssl_stapling_verify on; ssl_trusted_certificate /etc/nginx/ssl/xxx.yyy.ocsp.crt; resolver xxx.xxx.xxx.xxx valid=30s; resolver_timeout 5s;

Slide 20

Slide 20 text

Agenda • Reverse proxy • SSL termination • SPDY gateway • L7 load balancer

Slide 21

Slide 21 text

SPDY with nginx

Slide 22

Slide 22 text

Ͱɺ࣮ࡍͷޮՌ͸ͱݴ͏ͱ

Slide 23

Slide 23 text

TLS Session (Cache|Tickets) ಋೖλΠϛϯά

Slide 24

Slide 24 text

TLS Session (Cache|Tickets) ಋೖλΠϛϯά

Slide 25

Slide 25 text

SPDY 41%:ಋೖͨ͠೔ͷϐʔΫ

Slide 26

Slide 26 text

΋͏ͪΐͬͱ۩ମతͳྫ

Slide 27

Slide 27 text

೔ຊ͔Βւ֎ͷZabbix dashboardʹΞΫηε

Slide 28

Slide 28 text

Client • MacBookPro • Google Chrome • HTTP/2༗ޮ • ϒϥ΢βΩϟογϡ͸ৗʹແޮˣ

Slide 29

Slide 29 text

Server(ॳظঢ়ଶ) • Apache(prefork) + mod_php • த਎͸Zabbix • KeepAlive Off • gzipѹॖແޮ • TLS Session Cache & Tickets༗ޮ

Slide 30

Slide 30 text

Server(ॳظঢ়ଶ) "QBDIF 1PSU HTTPS Server (PPHMF"VUI1SPYZ "QBDIF 1PSU SSLऴ୺ Ϣʔβೝূ Zabbix

Slide 31

Slide 31 text

ύϑΥʔϚϯε(ॳظঢ়ଶ) ߲໨໊ ܭଌ஋ લճ ͱͷൺֱ ॳظঢ়ଶ ͱͷൺֱ 3FDFJWFESFRVFTUT %BUB5SBOTGFS ,# -PBE5JNF TFD %0.$POUFOU-PBE FE5JNF TFD

Slide 32

Slide 32 text

νϡʔχϯά ͦͷ1 KeepAlive On KeepAliveΛ༗ޮʹ͢Δ

Slide 33

Slide 33 text

ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ ͱͷൺֱ ॳظঢ়ଶ ͱͷൺֱ 3FDFJWFESFRVFTUT %BUB5SBOTGFS ,# ,# ,# -PBE5JNF TFD TFD TFD %0.$POUFOU-PBE FE5JNF TFD TFD TFD

Slide 34

Slide 34 text

νϡʔχϯά ͦͷ2 AddOutputFilterByType DEFLATE text/html AddOutputFilterByType DEFLATE text/css AddOutputFilterByType DEFLATE text/js AddOutputFilterByType DEFLATE text/javascript AddOutputFilterByType DEFLATE application/javascript AddOutputFilterByType DEFLATE application/json-rpc gzipѹॖΛ༗ޮʹ͢Δ

Slide 35

Slide 35 text

ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ ͱͷൺֱ ॳظঢ়ଶ ͱͷൺֱ 3FDFJWFESFRVFTUT %BUB5SBOTGFS ,# ,# ,# -PBE5JNF TFD TFD TFD %0.$POUFOU-PBEF E5JNF TFD TFD TFD

Slide 36

Slide 36 text

νϡʔχϯάͦͷ3 "QBDIF 1PSU HTTPS Server (PPHMF"VUI1SPYZ "QBDIF 1PSU SSLऴ୺ Ϣʔβೝূ Zabbix OHJOY 1PSU SSLऴ୺ΛnginxͰߦ͏

Slide 37

Slide 37 text

νϡʔχϯάͦͷ3 # nginx.conf # in main context worker_processes auto; tcp_nopush on; keepalive_timeout 65s; open_file_cache max=1000 inactive=20s; ssl_session_cache shared:SSL:30m; gzip on; gzip_comp_level 9; gzip_types text/css text/plain text/js text/javascript application/javascript application/json-rpc; # in event context accept_mutex_delay 100ms; # in event context ੩తϑΝΠϧ͸શ෦nginxͰ഑৴͢Δ

Slide 38

Slide 38 text

ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ ͱͷൺֱ ॳظঢ়ଶ ͱͷൺֱ 3FDFJWFESFRVFTUT %BUB5SBOTGFS ,# ,# ,# -PBE5JNF TFD TFD TFD %0.$POUFOU-PBEF E5JNF TFD TFD TFD

Slide 39

Slide 39 text

νϡʔχϯάͦͷ4 listen 443 ssl spdy; SPDY/3.1Λ༗ޮʹ͢Δ

Slide 40

Slide 40 text

ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ ͱͷൺֱ ॳظঢ়ଶ ͱͷൺֱ 3FDFJWFESFRVFTUT %BUB5SBOTGFS ,# ,# ,# -PBE5JNF TFD TFD TFD %0.$POUFOU-PBEF E5JNF TFD TFD TFD ࠷ऴతʹWebϖʔδͷϩʔυ͕࣌ؒ4ඵ͔Β1ඵʹ

Slide 41

Slide 41 text

Agenda • Reverse proxy • SSL termination • SPDY gateway • L7 load balancer

Slide 42

Slide 42 text

OHJOY

Slide 43

Slide 43 text

൚༻ϓογϡ௨஌γεςϜ PS CBUDIαʔό

Slide 44

Slide 44 text

·ͱΊ • nginx͸ • ϝϧΧϦ಺ͰαʔϏεɺࣾ಺γεςϜͰ͍ΖΜ ͳՕॴʹڬΜͰ׆༻͍ͯ͠·͢ • L7ϩʔυόϥϯαʔɺϦόϓϩɺSSLऴ୺αʔ όͱͯ͠ͱͯ΋༏ल • ࠓޙ΋ར༻Օॴ͕૿͑Δ༧ఆ