実践nginx〜メルカリの場合〜

 実践nginx〜メルカリの場合〜

5d74d743eabd2bf7d4d2f68b9d3c727d?s=128

Tatsuhiko Kubo

April 22, 2015
Tweet

Transcript

  1. ࣮ફnginx ʙϝϧΧϦͷ৔߹ʙ Tatsuhiko Kubo@cubicdaiya ALM@2015/04/21

  2. ࣗݾ঺հ • ٱอୡ඙(Tatsuhiko Kubo) • bokko@cubicdaiya • Software Engineer in

    Infrastructure Engineering • Mercari, Inc. • Favorites: Go, C, nginx
  3. OSS࡞ͬͨΓίϯτϦϏϡʔτͨ͠Γ

  4. Agenda ϝϧΧϦͰͷnginxͷ׆༻ࣄྫʹ͍ͭͯ

  5. nginx • ੈքͰೋ൪໨ʹར༻͞Ε͍ͯΔOSSͷHTTPαʔό • C10Kʹ଱͑ΒΕΔΞʔΩςΫνϟ • Πϕϯτۦಈ • ϊϯϒϩοΩϯάI/O •

    ඇಉظI/O • ܰྔͰߴ଎
  6. ϝϧΧϦͰΑ͋͘Δޫܠ ࣮ࡍʹ͸͋Δఔ౓ू໿͍ͯ͠·͢

  7. Agenda • Reverse proxy • SSL termination • SPDY gateway

    • L7 load balancer
  8. Agenda • Reverse proxy • SSL termination • SPDY gateway

    • L7 load balancer
  9. ϦόʔεϓϩΩγͱͯ͠ͷnginx • ϦΫΤετͷϩΪϯά • ΞΫηε੍ޚ • ίϯςϯπͷѹॖɾΩϟογϡ • όοϑΝϦϯά •

    etc…
  10. nginx.confͷઃఆྫ server { listen 443 ssl spdy; server_name xxx.yyy; #

    ϓϩΩγઃఆ proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Connection ""; proxy_http_version 1.1; proxy_buffers 50 8k; # ੩తίϯςϯπ͸nginxͰฦ͢ location ~ /(styles|js|images)/ { root /usr/share/zabbix; expires 30d; } # Zabbix(༻ͷgoogle_auth_proxy)΁ϓϩΩγ location / { proxy_pass http://google_auth_proxy_for_zabbix; } }
  11. લஈʹnginxΛஔ͘ϝϦοτ • HTTP or HTTPSαʔόʹඞཁͳλεΫΛҰ௨Γ͜ͳͤΔ • ΞϓϦέʔγϣϯαʔό(e.g. Unicorn)ʹ଍Γͳ͍ػೳΛิ׬ • ωοτϫʔΫϨΠςϯγͷվળ

    • KeepAlive • gzipѹॖ • TLS Session (Cache | Tickets)ɺOCSP Stapling • SPDY • etc…
  12. Agenda • Reverse proxy • SSL termination • SPDY gateway

    • L7 load balancer
  13. HTTPS௨৴ͷߴ଎Խ • ҰൠʹHTTPS௨৴Ͱ͸TCP 3-way handshakeʹՃ͑ͯ TLS 3-way handshake͕ൃੜ͢ΔͷͰHTTP௨৴ΑΓ஗͘ͳΔ • HTTPS௨৴ߴ଎ԽͷͨΊͷࡾछͷਆث

    • TLS Session Cache • TLS Session Tickets • OCSP Stapling
  14. TLS Session Cache • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ Ωϟογϡ • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ • ࣍ճͷTLSϋϯυγΣΠΫΛলུ

    • CPUͷϦιʔεͷ࡟ݮ΍ϨΠςϯγͷղফʹޮՌ͕͋Δ
  15. TLS Session Cache with nginx

  16. TLS Session Tickets • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥΠΞϯ τʹ౉͢ • νέοτΛݩʹTLSηογϣϯΛ࠶։ • HTTPSαʔόෳ਺୆Ͱηογϣϯ৘ใΛڞ༗Ͱ͖Δ

    • εϚϗͩͱαϙʔτ͍ͯ͠Δ୺຤͕গͳ͍…
  17. TLS Session Tickets with nginx

  18. OCSP Stapling • OCSPʹΑΔSSLূ໌ॻͷࣦޮ֬ೝΛαʔόଆͰߦͬ ͯΩϟογϡ • ΫϥΠΞϯτଆͰ΍ΔͱTLSϋϯυγΣΠΫ࣌ʹϨΠ ςϯγ͕ൃੜ͢Δ • ΍ͬͺΓεϚϗͩͱ͋Μ·ΓରԠͯ͠ͳ͍

    • Google Chrome for iOSͩͱରԠͯͨ͠
  19. OCSP Stapling with nginx ssl_stapling on; ssl_stapling_verify on; ssl_trusted_certificate /etc/nginx/ssl/xxx.yyy.ocsp.crt;

    resolver xxx.xxx.xxx.xxx valid=30s; resolver_timeout 5s;
  20. Agenda • Reverse proxy • SSL termination • SPDY gateway

    • L7 load balancer
  21. SPDY with nginx

  22. Ͱɺ࣮ࡍͷޮՌ͸ͱݴ͏ͱ

  23. TLS Session (Cache|Tickets) ಋೖλΠϛϯά

  24. TLS Session (Cache|Tickets) ಋೖλΠϛϯά

  25. SPDY 41%:ಋೖͨ͠೔ͷϐʔΫ

  26. ΋͏ͪΐͬͱ۩ମతͳྫ

  27. ೔ຊ͔Βւ֎ͷZabbix dashboardʹΞΫηε

  28. Client • MacBookPro • Google Chrome • HTTP/2༗ޮ • ϒϥ΢βΩϟογϡ͸ৗʹແޮˣ

  29. Server(ॳظঢ়ଶ) • Apache(prefork) + mod_php • த਎͸Zabbix • KeepAlive Off

    • gzipѹॖແޮ • TLS Session Cache & Tickets༗ޮ
  30. Server(ॳظঢ়ଶ) "QBDIF 1PSU HTTPS Server (PPHMF"VUI1SPYZ "QBDIF 1PSU SSLऴ୺ Ϣʔβೝূ

    Zabbix
  31. ύϑΥʔϚϯε(ॳظঢ়ଶ) ߲໨໊ ܭଌ஋ લճ ͱͷൺֱ ॳظঢ়ଶ ͱͷൺֱ 3FDFJWFESFRVFTUT  

     %BUB5SBOTGFS ,#   -PBE5JNF TFD   %0.$POUFOU-PBE FE5JNF TFD  
  32. νϡʔχϯά ͦͷ1 KeepAlive On KeepAliveΛ༗ޮʹ͢Δ

  33. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ ͱͷൺֱ ॳظঢ়ଶ ͱͷൺֱ 3FDFJWFESFRVFTUT  

     %BUB5SBOTGFS ,# ,# ,# -PBE5JNF TFD TFD TFD %0.$POUFOU-PBE FE5JNF TFD TFD TFD
  34. νϡʔχϯά ͦͷ2 <IfModule mod_deflate.c> AddOutputFilterByType DEFLATE text/html AddOutputFilterByType DEFLATE text/css

    AddOutputFilterByType DEFLATE text/js AddOutputFilterByType DEFLATE text/javascript AddOutputFilterByType DEFLATE application/javascript AddOutputFilterByType DEFLATE application/json-rpc </IfModule> gzipѹॖΛ༗ޮʹ͢Δ
  35. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ ͱͷൺֱ ॳظঢ়ଶ ͱͷൺֱ 3FDFJWFESFRVFTUT  

     %BUB5SBOTGFS ,# ,# ,# -PBE5JNF TFD TFD TFD %0.$POUFOU-PBEF E5JNF TFD TFD TFD
  36. νϡʔχϯάͦͷ3 "QBDIF 1PSU HTTPS Server (PPHMF"VUI1SPYZ "QBDIF 1PSU SSLऴ୺ Ϣʔβೝূ

    Zabbix OHJOY 1PSU SSLऴ୺ΛnginxͰߦ͏
  37. νϡʔχϯάͦͷ3 # nginx.conf # in main context worker_processes auto; tcp_nopush

    on; keepalive_timeout 65s; open_file_cache max=1000 inactive=20s; ssl_session_cache shared:SSL:30m; gzip on; gzip_comp_level 9; gzip_types text/css text/plain text/js text/javascript application/javascript application/json-rpc; # in event context accept_mutex_delay 100ms; # in event context ੩తϑΝΠϧ͸શ෦nginxͰ഑৴͢Δ
  38. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ ͱͷൺֱ ॳظঢ়ଶ ͱͷൺֱ 3FDFJWFESFRVFTUT  

     %BUB5SBOTGFS ,# ,# ,# -PBE5JNF TFD TFD TFD %0.$POUFOU-PBEF E5JNF TFD TFD TFD
  39. νϡʔχϯάͦͷ4 listen 443 ssl spdy; SPDY/3.1Λ༗ޮʹ͢Δ

  40. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ ͱͷൺֱ ॳظঢ়ଶ ͱͷൺֱ 3FDFJWFESFRVFTUT  

     %BUB5SBOTGFS ,# ,# ,# -PBE5JNF TFD TFD TFD %0.$POUFOU-PBEF E5JNF TFD TFD TFD ࠷ऴతʹWebϖʔδͷϩʔυ͕࣌ؒ4ඵ͔Β1ඵʹ
  41. Agenda • Reverse proxy • SSL termination • SPDY gateway

    • L7 load balancer
  42. OHJOY

  43. ൚༻ϓογϡ௨஌γεςϜ PS CBUDIαʔό

  44. ·ͱΊ • nginx͸ • ϝϧΧϦ಺ͰαʔϏεɺࣾ಺γεςϜͰ͍ΖΜ ͳՕॴʹڬΜͰ׆༻͍ͯ͠·͢ • L7ϩʔυόϥϯαʔɺϦόϓϩɺSSLऴ୺αʔ όͱͯ͠ͱͯ΋༏ल •

    ࠓޙ΋ར༻Օॴ͕૿͑Δ༧ఆ