C1M and NGINX - Speaker Deck

Slide 1

Slide 1 text

* C1M and NGINX · FEBRUARY 25, 2014

Slide 2

Slide 2 text

* DISQUS Who We Are We are the web’s community of communities. Online communities are where people go to connect with their passions. We link all those communities so it’s easier to discover and discuss new stuff worth talking about. NGINX User Summit 2014

Slide 3

Slide 3 text

* DISQUS By the numbers ● 1.3 billion unique visitors ● 10 billion page views ● 500 million users engaged in discussions ● 3 million communities ● 25 million comments NGINX User Summit 2014

Slide 4

Slide 4 text

* DISQUS Realtime NGINX User Summit 2014

Slide 5

Slide 5 text

* DISQUS Realtime Users Disqus (Python) Realtime (Go) NGINX NGINX NGINX NGINX PushStream (NGINX) NGINX User Summit 2014

Slide 6

Slide 6 text

* DISQUS PushStream ● A pure stream http push technology for NGINX ● Supports EventSource, WebSocket, Long Polling, and Forever Iframe ● Authored by Wandenberg Peixoto NGINX User Summit 2014

Slide 7

Slide 7 text

* DISQUS Normal Day ● 3200 connection/s ● 1 million connections ● 150k packet/s TX and 130k packet/s RX ● 150 mbit/s TX and 80 mbit/s RX ● <15ms delay end-to-end NGINX User Summit 2014

Slide 8

Slide 8 text

* DISQUS NGINX User Summit 2014

Slide 9

Slide 9 text

* DISQUS [16717132.196640] Out of socket memory NGINX User Summit 2014

Slide 10

Slide 10 text

* DISQUS [175253.640329] TCP: Possible SYN flooding on port 80. Sending cookies. NGINX User Summit 2014

Slide 11

Slide 11 text

* DISQUS [2814258.430156] TCP: time wait bucket table overflow NGINX User Summit 2014

Slide 12

Slide 12 text

* DISQUS sysctl net.core.netdev_max_backlog=65535 net.core.optmem_max=20480 net.core.rmem_default=16384 net.core.rmem_max=16777216 net.core.somaxconn=65535 net.core.wmem_default=32768 net.core.wmem_max=16777216 net.ipv4.tcp_fin_timeout=15 net.ipv4.tcp_keepalive_intvl=30 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_keepalive_time=120 net.ipv4.tcp_max_orphans=262144 net.ipv4.tcp_max_syn_backlog=524288 net.ipv4.tcp_max_tw_buckets=524288 net.ipv4.tcp_mem=1572864 1835008 2097152 net.ipv4.tcp_no_metrics_save=1 net.ipv4.tcp_orphan_retries=0 net.ipv4.tcp_rmem=4096 16384 16777216 net.ipv4.tcp_synack_retries=2 net.ipv4.tcp_syncookies=1 net.ipv4.tcp_syn_retries=2 net.ipv4.tcp_wmem=4096 32768 16777216 NGINX User Summit 2014

Slide 13

Slide 13 text

* DISQUS NGINX listen 80 backlog=32768 rcvbuf=4194304 sndbuf=16777216; worker_processes 8; worker_connections 262143; tcp_nodelay on; tcp_nopush off; lingering_time 5s; lingering_timeout 1s; client_header_timeout 5s; reset_timedout_connection on; send_timeout 9s; push_stream_subscriber_connection_ttl 90s; push_stream_ping_message_interval 10s; NGINX User Summit 2014

Slide 14

Slide 14 text

* DISQUS NGINX User Summit 2014

Slide 15

Slide 15 text

* DISQUS gzip By default, nginx is very generous with the buffers it allocates for gzip. A healthy, 264kb per connection. Observe the content-length of responses and tune gzip_window and gzip_hash accordingly. The difference of shrinking pre-allocation to only 32kb with: gzip_window 2k; gzip_hash 4k; NGINX User Summit 2014

Slide 16

Slide 16 text

* DISQUS Links ● https://github.com/wandenberg/nginx-push-stream-module ● http://bit.ly/disqus-nginx-push-stream ● http://www.dctrwatson.com/2013/05/nginx-gzip-high-concurrency-and-memory/ ● http://vincent.bernat.im/en/blog/2014-tcp-time-wait-state-linux.html NGINX User Summit 2014

Slide 17

Slide 17 text

* Sound interesting? We’re hiring http://disqus.com/jobs/