実践nginx〜メルカリの場合〜

Transcript

1. ࣮ફnginx ʙϝϧΧϦͷ৔߹ʙ Tatsuhiko Kubo@cubicdaiya ALM@2015/04/21

2. ࣗݾ঺հ • ٱอୡ඙(Tatsuhiko Kubo) • bokko@cubicdaiya • Software Engineer in

   Infrastructure Engineering • Mercari, Inc. • Favorites: Go, C, nginx

3. OSS࡞ͬͨΓίϯτϦϏϡʔτͨ͠Γ

4. Agenda ϝϧΧϦͰͷnginxͷ׆༻ࣄྫʹ͍ͭͯ

5. nginx • ੈքͰೋ൪໨ʹར༻͞Ε͍ͯΔOSSͷHTTPαʔό • C10Kʹ଱͑ΒΕΔΞʔΩςΫνϟ • Πϕϯτۦಈ • ϊϯϒϩοΩϯάI/O •

   ඇಉظI/O • ܰྔͰߴ଎

6. ϝϧΧϦͰΑ͋͘Δޫܠ ࣮ࡍʹ͸͋Δఔ౓ू໿͍ͯ͠·͢

7. Agenda • Reverse proxy • SSL termination • SPDY gateway

   • L7 load balancer

8. Agenda • Reverse proxy • SSL termination • SPDY gateway

   • L7 load balancer

9. ϦόʔεϓϩΩγͱͯ͠ͷnginx • ϦΫΤετͷϩΪϯά • ΞΫηε੍ޚ • ίϯςϯπͷѹॖɾΩϟογϡ • όοϑΝϦϯά •

   etc…

10. nginx.confͷઃఆྫ server { listen 443 ssl spdy; server_name xxx.yyy; #

    ϓϩΩγઃఆ proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header Connection ""; proxy_http_version 1.1; proxy_buffers 50 8k; # ੩తίϯςϯπ͸nginxͰฦ͢ location ~ /(styles|js|images)/ { root /usr/share/zabbix; expires 30d; } # Zabbix(༻ͷgoogle_auth_proxy)΁ϓϩΩγ location / { proxy_pass http://google_auth_proxy_for_zabbix; } }

11. લஈʹnginxΛஔ͘ϝϦοτ • HTTP or HTTPSαʔόʹඞཁͳλεΫΛҰ௨Γ͜ͳͤΔ • ΞϓϦέʔγϣϯαʔό(e.g. Unicorn)ʹ଍Γͳ͍ػೳΛิ׬ • ωοτϫʔΫϨΠςϯγͷվળ

    • KeepAlive • gzipѹॖ • TLS Session (Cache | Tickets)ɺOCSP Stapling • SPDY • etc…

12. Agenda • Reverse proxy • SSL termination • SPDY gateway

    • L7 load balancer

13. HTTPS௨৴ͷߴ଎Խ • ҰൠʹHTTPS௨৴Ͱ͸TCP 3-way handshakeʹՃ͑ͯ TLS 3-way handshake͕ൃੜ͢ΔͷͰHTTP௨৴ΑΓ஗͘ͳΔ • HTTPS௨৴ߴ଎ԽͷͨΊͷࡾछͷਆث

    • TLS Session Cache • TLS Session Tickets • OCSP Stapling

14. TLS Session Cache • TLSϋϯυγΣΠΫͷηογϣϯ৘ใΛαʔόʹ Ωϟογϡ • nginxͰ͸ڞ༗ϝϞϦ্ʹΩϟογϡ͞ΕΔ • ࣍ճͷTLSϋϯυγΣΠΫΛলུ

    • CPUͷϦιʔεͷ࡟ݮ΍ϨΠςϯγͷղফʹޮՌ͕͋Δ

15. TLS Session Cache with nginx

16. TLS Session Tickets • ҉߸Խͨ͠ηογϣϯ৘ใ(νέοτ)ΛΫϥΠΞϯ τʹ౉͢ • νέοτΛݩʹTLSηογϣϯΛ࠶։ • HTTPSαʔόෳ਺୆Ͱηογϣϯ৘ใΛڞ༗Ͱ͖Δ

    • εϚϗͩͱαϙʔτ͍ͯ͠Δ୺຤͕গͳ͍…

17. TLS Session Tickets with nginx

18. OCSP Stapling • OCSPʹΑΔSSLূ໌ॻͷࣦޮ֬ೝΛαʔόଆͰߦͬ ͯΩϟογϡ • ΫϥΠΞϯτଆͰ΍ΔͱTLSϋϯυγΣΠΫ࣌ʹϨΠ ςϯγ͕ൃੜ͢Δ • ΍ͬͺΓεϚϗͩͱ͋Μ·ΓରԠͯ͠ͳ͍

    • Google Chrome for iOSͩͱରԠͯͨ͠

19. OCSP Stapling with nginx ssl_stapling on; ssl_stapling_verify on; ssl_trusted_certificate /etc/nginx/ssl/xxx.yyy.ocsp.crt;

    resolver xxx.xxx.xxx.xxx valid=30s; resolver_timeout 5s;

20. Agenda • Reverse proxy • SSL termination • SPDY gateway

    • L7 load balancer

21. SPDY with nginx

22. Ͱɺ࣮ࡍͷޮՌ͸ͱݴ͏ͱ

23. TLS Session (Cache|Tickets) ಋೖλΠϛϯά

24. TLS Session (Cache|Tickets) ಋೖλΠϛϯά

25. SPDY 41%:ಋೖͨ͠೔ͷϐʔΫ

26. ΋͏ͪΐͬͱ۩ମతͳྫ

27. ೔ຊ͔Βւ֎ͷZabbix dashboardʹΞΫηε

28. Client • MacBookPro • Google Chrome • HTTP/2༗ޮ • ϒϥ΢βΩϟογϡ͸ৗʹແޮˣ

29. Server(ॳظঢ়ଶ) • Apache(prefork) + mod_php • த਎͸Zabbix • KeepAlive Off

    • gzipѹॖແޮ • TLS Session Cache & Tickets༗ޮ

30. Server(ॳظঢ়ଶ) "QBDIF
    1PSU HTTPS Server (PPHMF
    "VUI
    1SPYZ "QBDIF
    1PSU SSLऴ୺ Ϣʔβೝূ

    Zabbix

31. ύϑΥʔϚϯε(ॳظঢ়ଶ) ߲໨໊ ܭଌ஋ લճ
    ͱͷൺֱ ॳظঢ়ଶ
    ͱͷൺֱ 3FDFJWFE
    SFRVFTUT  

     %BUB
    5SBOTGFS ,#   -PBE
    5JNF TFD   %0.$POUFOU-PBE FE
    5JNF TFD  

32. νϡʔχϯά ͦͷ1 KeepAlive On KeepAliveΛ༗ޮʹ͢Δ

33. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ
    ͱͷൺֱ ॳظঢ়ଶ
    ͱͷൺֱ 3FDFJWFE
    SFRVFTUT  

     %BUB
    5SBOTGFS ,# ,# ,# -PBE
    5JNF TFD TFD TFD %0.$POUFOU-PBE FE
    5JNF TFD TFD TFD

34. νϡʔχϯά ͦͷ2 <IfModule mod_deflate.c> AddOutputFilterByType DEFLATE text/html AddOutputFilterByType DEFLATE text/css

    AddOutputFilterByType DEFLATE text/js AddOutputFilterByType DEFLATE text/javascript AddOutputFilterByType DEFLATE application/javascript AddOutputFilterByType DEFLATE application/json-rpc </IfModule> gzipѹॖΛ༗ޮʹ͢Δ

35. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ
    ͱͷൺֱ ॳظঢ়ଶ
    ͱͷൺֱ 3FDFJWFE
    SFRVFTUT  

     %BUB
    5SBOTGFS ,# ,# ,# -PBE
    5JNF TFD TFD TFD %0.$POUFOU-PBEF E
    5JNF TFD TFD TFD

36. νϡʔχϯάͦͷ3 "QBDIF
    1PSU HTTPS Server (PPHMF
    "VUI
    1SPYZ "QBDIF
    1PSU SSLऴ୺ Ϣʔβೝূ

    Zabbix OHJOY
    1PSU SSLऴ୺ΛnginxͰߦ͏

37. νϡʔχϯάͦͷ3 # nginx.conf # in main context worker_processes auto; tcp_nopush

    on; keepalive_timeout 65s; open_file_cache max=1000 inactive=20s; ssl_session_cache shared:SSL:30m; gzip on; gzip_comp_level 9; gzip_types text/css text/plain text/js text/javascript application/javascript application/json-rpc; # in event context accept_mutex_delay 100ms; # in event context ੩తϑΝΠϧ͸શ෦nginxͰ഑৴͢Δ

38. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ
    ͱͷൺֱ ॳظঢ়ଶ
    ͱͷൺֱ 3FDFJWFE
    SFRVFTUT  

     %BUB
    5SBOTGFS ,# ,# ,# -PBE
    5JNF TFD TFD TFD %0.$POUFOU-PBEF E
    5JNF TFD TFD TFD

39. νϡʔχϯάͦͷ4 listen 443 ssl spdy; SPDY/3.1Λ༗ޮʹ͢Δ

40. ύϑΥʔϚϯε ߲໨໊ ܭଌ஋ લճ
    ͱͷൺֱ ॳظঢ়ଶ
    ͱͷൺֱ 3FDFJWFE
    SFRVFTUT  

     %BUB
    5SBOTGFS ,# ,# ,# -PBE
    5JNF TFD TFD TFD %0.$POUFOU-PBEF E
    5JNF TFD TFD TFD ࠷ऴతʹWebϖʔδͷϩʔυ͕࣌ؒ4ඵ͔Β1ඵʹ

41. Agenda • Reverse proxy • SSL termination • SPDY gateway

    • L7 load balancer

42. OHJOY

43. ൚༻ϓογϡ௨஌γεςϜ PS
    CBUDIαʔό

44. ·ͱΊ • nginx͸ • ϝϧΧϦ಺ͰαʔϏεɺࣾ಺γεςϜͰ͍ΖΜ ͳՕॴʹڬΜͰ׆༻͍ͯ͠·͢ • L7ϩʔυόϥϯαʔɺϦόϓϩɺSSLऴ୺αʔ όͱͯ͠ͱͯ΋༏ल •

    ࠓޙ΋ར༻Օॴ͕૿͑Δ༧ఆ