C1M and NGINX

use case for OS Tuning for high concurrency NGINX

Transcript

1. * C1M and NGINX · FEBRUARY 25, 2014
2. * DISQUS Who We Are We are the web’s community of communities. Online communities are where people go to connect with their passions. We link all those communities so it’s easier to discover and discuss new stuff worth talking about. NGINX User Summit 2014
3. * DISQUS By the numbers • 1.3 billion unique visitors • 10 billion page views • 500 million users engaged in discussions • 3 million communities • 25 million comments NGINX User Summit 2014
4. * DISQUS Realtime NGINX User Summit 2014
5. * DISQUS Realtime Users Disqus (Python) Realtime (Go) NGINX NGINX NGINX NGINX PushStream (NGINX) NGINX User Summit 2014
6. * DISQUS PushStream • A pure stream http push technology for NGINX • Supports EventSource, WebSocket, Long Polling, and Forever Iframe • Authored by Wandenberg Peixoto NGINX User Summit 2014
7. * DISQUS Normal Day • 3200 connection/s • 1 million connections • 150k packet/s TX and 130k packet/s RX • 150 mbit/s TX and 80 mbit/s RX • <15ms delay end-to-end NGINX User Summit 2014
8. * DISQUS NGINX User Summit 2014
9. * DISQUS [16717132.196640] Out of socket memory NGINX User Summit 2014
10. * DISQUS [175253.640329] TCP: Possible SYN flooding on port 80. Sending cookies. NGINX User Summit 2014
11. * DISQUS [2814258.430156] TCP: time wait bucket table overflow NGINX User Summit 2014
12. * DISQUS sysctl net.core.netdev_max_backlog=65535 net.core.optmem_max=20480 net.core.rmem_default=16384 net.core.rmem_max=16777216 net.core.somaxconn=65535 net.core.wmem_default=32768 net.core.wmem_max=16777216 net.ipv4.tcp_fin_timeout=15 net.ipv4.tcp_keepalive_intvl=30 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_time=120 net.ipv4.tcp_max_orphans=262144 net.ipv4.tcp_max_syn_backlog=524288 net.ipv4.tcp_max_tw_buckets=524288 net.ipv4.tcp_mem=1572864 1835008 2097152 net.ipv4.tcp_no_metrics_save=1 net.ipv4.tcp_orphan_retries=0 net.ipv4.tcp_rmem=4096 16384 16777216 net.ipv4.tcp_synack_retries=2 net.ipv4.tcp_syncookies=1 net.ipv4.tcp_syn_retries=2 net.ipv4.tcp_wmem=4096 32768 16777216 NGINX User Summit 2014
13. * DISQUS NGINX listen 80 backlog=32768 rcvbuf=4194304 sndbuf=16777216; worker_processes 8; worker_connections 262143; tcp_nodelay on; tcp_nopush off; lingering_time 5s; lingering_timeout 1s; client_header_timeout 5s; reset_timedout_connection on; send_timeout 9s; push_stream_subscriber_connection_ttl 90s; push_stream_ping_message_interval 10s; NGINX User Summit 2014
14. * DISQUS NGINX User Summit 2014
15. * DISQUS gzip By default, nginx is very generous with the buffers it allocates for gzip. A healthy, 264kb per connection. Observe the content-length of responses and tune gzip_window and gzip_hash accordingly. The difference of shrinking pre-allocation to only 32kb with: gzip_window 2k; gzip_hash 4k; NGINX User Summit 2014
16. * DISQUS Links • https://github.com/wandenberg/nginx-push-stream-module • http://bit.ly/disqus-nginx-push-stream • http://www.dctrwatson.com/2013/05/nginx-gzip-high-concurrency-and-memory/ • http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html NGINX User Summit 2014
17. * Sound interesting? We’re hiring http://disqus.com/jobs/