TCP Performance for Mobile Applications

TCP Performance For Mobile Applications Vladimir Kirillov @darkproger

Networking Stack

Data Link Network Transport Application Session

Data Link Network Transport Application Session WiFi Edge 3G LTE
IP TCP HTTP TLS

Data Link Network Transport Application Session WiFi Edge 3G LTE
IP TCP HTTP TLS Level Protocol API / Implementation hardware kernel SOCK_STREAM (Http|NS)URLConnection OpenSSL

Protocol API / Implementation Introspection WiFi Edge 3G LTE IP
TCP HTTP TLS hardware kernel SOCK_STREAM (Http|NS)URLConnection OpenSSL gdb ptrace socket API bpf(4) LSF dtrace

capturing iPhone traffic % udid=$(system_profiler SPUSBDataType \ | awk '/iPhone/{go=1}
/Serial/ {if (go) print $3; go=0}') 276cb9530201bcehelloworldcd55560ed015d00 % rvictl -s $udid Starting device 276cb9530201bcehelloworldcd55560ed015d00 [SUCCEEDED] % ifconfig rvi0 rvi0: flags=3005<UP,DEBUG,LINK0,LINK1> mtu 0

capturing Android traffic # adb connect 192.168.56.100 # adb shell
shell@android:/ $ su Test prop su allows access thanks to androVM.su.bypass property shell@android:/ # tcpdump -i eth1

tcpdump -i lo0 -w t.pcap -s0 & nc -l 5000
& echo hello | nc localhost 5000 kill %1

# tcpdump -r t.pcap -nnvv -tttt -K 'tcp port 5000'
2012-11-24 12:23:35.511134 IP6 (hlim 64, next-header TCP (6) payload length: 44) ::1.51734 > ::1.5000: Flags [ S], seq 453038127, win 65535, options [mss 16324,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 0 2012-11-24 12:23:35.511175 IP6 (hlim 64, next-header TCP (6) payload length: 20) ::1.5000 > ::1.51734: Flags [ R.], seq 0, ack 453038128, win 0, length 0 2012-11-24 12:23:35.511226 IP (tos 0x0, ttl 64, id 8400, offset 0, flags [DF], proto TCP (6), length 64) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [ S], seq 2527137802, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 0 2012-11-24 12:23:35.511276 IP (tos 0x0, ttl 64, id 58311, offset 0, flags [DF], proto TCP (6), length 64) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [ S.], seq 494520280, ack 2527137803, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr 303407352,sackOK,eol], length 0 2012-11-24 12:23:35.511287 IP (tos 0x0, ttl 64, id 47796, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511298 IP (tos 0x0, ttl 64, id 52186, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511332 IP (tos 0x0, ttl 64, id 31417, offset 0, flags [DF], proto TCP (6), length 58) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [ P.], seq 1:7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 6 2012-11-24 12:23:35.511351 IP (tos 0x0, ttl 64, id 29060, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511354 IP (tos 0x0, ttl 64, id 4019, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 7, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511367 IP (tos 0x0, ttl 64, id 20879, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511378 IP (tos 0x0, ttl 64, id 59633, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511388 IP (tos 0x0, ttl 64, id 56794, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [F.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 17 packets captured

# tcpdump -r t.pcap -nnvv -tttt -K 'tcp port 5000'
2012-11-24 12:23:35.511134 IP6 (hlim 64, next-header TCP (6) payload length: 44) ::1.51734 > ::1.5000: Flags [ S], seq 453038127, win 65535, options [mss 16324,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 0 2012-11-24 12:23:35.511175 IP6 (hlim 64, next-header TCP (6) payload length: 20) ::1.5000 > ::1.51734: Flags [ R.], seq 0, ack 453038128, win 0, length 0 2012-11-24 12:23:35.511226 IP (tos 0x0, ttl 64, id 8400, offset 0, flags [DF], proto TCP (6), length 64) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [ S], seq 2527137802, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr 0,sackOK,eol], length 0 2012-11-24 12:23:35.511276 IP (tos 0x0, ttl 64, id 58311, offset 0, flags [DF], proto TCP (6), length 64) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [ S.], seq 494520280, ack 2527137803, win 65535, options [mss 16344,nop,wscale 4,nop,nop,TS val 303407352 ecr 303407352,sackOK,eol], length 0 2012-11-24 12:23:35.511287 IP (tos 0x0, ttl 64, id 47796, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511298 IP (tos 0x0, ttl 64, id 52186, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511332 IP (tos 0x0, ttl 64, id 31417, offset 0, flags [DF], proto TCP (6), length 58) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [ P.], seq 1:7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 6 2012-11-24 12:23:35.511351 IP (tos 0x0, ttl 64, id 29060, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511354 IP (tos 0x0, ttl 64, id 4019, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 7, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511367 IP (tos 0x0, ttl 64, id 20879, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511378 IP (tos 0x0, ttl 64, id 59633, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.51735 > 127.0.0.1.5000: Flags [F.], seq 7, ack 1, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 2012-11-24 12:23:35.511388 IP (tos 0x0, ttl 64, id 56794, offset 0, flags [DF], proto TCP (6), length 52) 127.0.0.1.5000 > 127.0.0.1.51735: Flags [F.], seq 1, ack 8, win 9186, options [nop,nop,TS val 303407352 ecr 303407352], length 0 17 packets captured ^^ % stat -f %z t.pcap 1306

% tcptrace t.pcap 17 packets seen, 17 TCP packets traced
elapsed wallclock time: 0:00:00.001344, 12648 pkts/sec analyzed trace file elapsed time: 0:00:00.000305 TCP connection info: 1: localhost:52132 - localhost:5000 (a2b) 1> 1< (reset) 2: localhost:52133 - localhost:5000 (c2d) 8> 7< (complete) (reset)

% tcptrace -o2 -l t.pcap ... adv wind scale: 4
adv wind scale: 4 req sack: Y req sack: Y sacks sent: 0 sacks sent: 0 urgent data pkts: 0 pkts urgent data pkts: 0 pkts urgent data bytes: 0 bytes urgent data bytes: 0 bytes mss requested: 16344 bytes mss requested: 16344 bytes max segm size: 6 bytes max segm size: 0 bytes min segm size: 6 bytes min segm size: 0 bytes avg segm size: 5 bytes avg segm size: 0 bytes max win adv:146976 bytes max win adv: 146976 bytes min win adv:146976 bytes min win adv: 146976 bytes zero win adv: 0 times zero win adv: 0 times avg win adv: 146976 bytes avg win adv: 122480 bytes initial window:6 bytes initial window: 0 bytes initial window: 1 pkts initial window: 0 pkts ttl stream length: 6 bytes ttl stream length: 1 bytes mi ssed data: 0 bytes missed data: 1 bytes truncated data: 0 bytes truncated data: 0 bytes truncated packets: 0 pkts truncated packets: 0 pkts data xmit time: 0.000 secs data xmit time: 0.000 secs idletime max: 0.1 ms idletime max: 0.0 ms throughput: 27027 Bps throughput: 0 Bps

endpoint SO_RCVBUF SO_SNDBUF endpoint SO_RCVBUF SO_SNDBUF

endpoint SO_RCVBUF SO_SNDBUF endpoint SO_RCVBUF SO_SNDBUF SEG SEG

endpoint SO_RCVBUF SO_SNDBUF endpoint SO_RCVBUF SO_SNDBUF LATENCY SEG SEG BANDWIDTH
2 * LATENCY = RTT

Latency • Time from one endpoint to another • Each
connection spans multiple links • latency = sum (lat foreach link) • RTT = 2 * latency

Bandwidth • Number of bytes a link can handle •
bw = min (bw foreach link)

Bandwidth Delay Product BDP = RTT * BANDWIDTH

sender window receiver window client SO_RCVBUF SO_SNDBUF server SO_RCVBUF SO_SNDBUF

SEG SEG SEG SEG

SEG SEG SEG SEG SEG SEG SEG SEG receiver window SEG SEG SEG SEG SEG SEG

TCP byte stream • stateful • ordered • reliable •
managed

IP TCP HTTP TLS has state no state paired

IP TCP HTTP TLS 1 RTT SYN SYN,ACK ACK

TLS "Oh, a SSL certiﬁcate warning. I'll read it carefully
and understand the possible implications before proceeding.” -- no User, ever.

TLS "Oh, a SSL library. I'll understand carefully its semantics
and will not break authentication.” -- unknown developer.

TLS % openssl s_client - showcerts -connect internet.velcom.by:443

CONNECTED(00000003) depth=3 Thawte Premium Server CA verify error:num=19:self signed certificate
in certificate chain verify return:0

Certificate chain 0 s:/C=BY/ST=Minsk/L=Minsk/O=FE Velcom/ CN=internet.velcom.by i:/C=US/O=Thawte, Inc./CN=Thawte SSL CA
-----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

1 s:/C=US/O=Thawte, Inc./CN=Thawte SSL CA i:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006
thawte, Inc. - For authorized use only/CN=thawte Primary Root CA -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

2 s:/C=US/O=thawte, Inc./OU=Certification Services Division/OU=(c) 2006 thawte, Inc. - For
authorized use only/CN=thawte Primary Root CA i:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/ CN=Thawte Premium Server CA/emailAddress=premium- [email protected] -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

3 s:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/ CN=Thawte Premium
Server CA/emailAddress=premium- [email protected] i:/C=ZA/ST=Western Cape/L=Cape Town/O=Thawte Consulting cc/OU=Certification Services Division/ CN=Thawte Premium Server CA/emailAddress=premium- [email protected] -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE-----

Server certificate subject=/C=BY/ST=Minsk/L=Minsk/O=FE Velcom/ CN=internet.velcom.by issuer=/C=US/O=Thawte, Inc./CN=Thawte SSL CA SSL
handshake has read 4736 bytes and written 328 bytes

TLS % openssl s_client -showcerts - connect ciklum.com:443 CONNECTED(00000003) depth=0
/C=UA/OU=Domain Control Validated/CN=*. ciklum.net verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 /C=UA/OU=Domain Control Validated/CN=*.ciklum.net verify error:num=27:certificate not trusted verify return:1 depth=0 /C=UA/OU=Domain Control Validated/CN=*.ciklum.net verify error:num=21:unable to verify the first certificate verify return:1 --- Certificate chain 0 s:/C=UA/OU=Domain Control Validated/CN=*.ciklum.net i:/O=AlphaSSL/CN=AlphaSSL CA - G2 ... Server certificate subject=/C=UA/OU=Domain Control Validated/CN=*.ciklum.net issuer=/O=AlphaSSL/CN=AlphaSSL CA - G2 SSL handshake has read 1854 bytes and written 328 bytes

IP TCP HTTP TLS 1 RTT 2 RTTs SYN SYN,ACK
ACK, ClientHello ServerHello, Certiﬁcate ClientKEX, ChangeCipherSpec ChangeCipherSpec,Finished

It takes 4 RTTs to serve a HTTPS request IP
TCP HTTP TLS 1 RTT 2 RTTs SYN SYN,ACK ACK, ClientHello ServerHello, Certiﬁcate ClientKEX, ChangeCipherSpec ChangeCipherSpec,Finished GET OK 1 RTT

TCP Reliability

SEG SEG SEG SEG SEG SEG SEG SEG receiver window A C K A C K A C K A C K

AirPort Express router router router sender window receiver window client
SO_RCVBUF SO_SNDBUF server SO_RCVBUF SO_SNDBUF SEG SEG SEG SEG SEG SEG SEG SEG receiver window A C K A C K A C K A C K

SO_RCVBUF SO_SNDBUF server SO_RCVBUF SO_SNDBUF SEG SEG SEG SEG SEG SEG SEG SEG receiver window A C K A C K A C K A C K retransmit on timeout (~200ms)

TCP Congestion Control

SO_RCVBUF sender window receiver window server SO_RCVBUF SO_SNDBUF SEG client
SO_SNDBUF SEG SEG SEG SEG SEG SEG SEG SEG SEG SEG

sender window receiver window AirPort Express overloaded router router router
receiver window client SO_RCVBUF SO_SNDBUF server SO_RCVBUF SO_SNDBUF SEG SEG SEG SEG SEG SEG A C K A C K A C K ^^^ What congestion control is actually designed for

SO_RCVBUF SO_SNDBUF server SO_RCVBUF SO_SNDBUF SEG SEG SEG SEG SEG SEG SEG SEG receiver window A C K A C K A C K A C K S E G S E G S E G S E G ^^^ What actually happens on mobile devices Crappy Wi-Fi

TCP Artifacts • Nagle algorithm while (1) write(fd, “5”, 1);
(telnet syndrom) Delayed ACK http://www.stuartcheshire.org/ papers/NagleDelayedAck/

TCP Artifacts •SO_OOBINLINE •TCP URG •RFC 6093

API Issues

API Issues Async NSURLConnection UIScrollView CFRunLoopAddCommonMode

SO_RCVBUF sender window receiver window SO_RCVBUF server SO_RCVBUF SO_SNDBUF SEG
SEG client SO_SNDBUF SEG SEG SEG SEG SEG SEG SEG SEG

Congestion Avoidance TCP Reno •Additive Increase •Multiplicative Decrease •Slow Start

Android

# cat /proc/sys/net/ipv4/tcp_slow_start_after_idle 1 # cat /proc/sys/net/ipv4/tcp_no_metrics_save 0 # echo
0 > /proc/sys/net/ipv4/tcp_slow_start_after_idle # echo 1 > /proc/sys/net/ipv4/tcp_no_metrics_save

# find /proc/sys/net/ipv4 | grep cong | xargs -tn1 cat
cat /proc/sys/net/ipv4/tcp_allowed_congestion_control cubic reno cat /proc/sys/net/ipv4/tcp_available_congestion_control cubic reno cat /proc/sys/net/ipv4/tcp_congestion_control cubic

# ip route show default via 192.168.56.1 dev eth1 initcwnd
10 initrwnd 10

Sockets • setsockopt(2) • adjust window size • socket buffer
sizes • TCP_NODELAY (Nagle) • etc •getsockopt(2) • monitoring • low-latency responding to socket events • do not let the buffer stay full

getsockopt(SOL_TCP, TCP_INFO) ESTAB 0 176 10.1.1.1:22 10.1.1.2:61984 users:(("sshd",18989,3)) ! mem:(r0,w1168,f2928,t0)
ts sack bic wscale:4,5 rto:280 rtt:56.25/7.5 ato:40 cwnd:8 ssthresh:7 send 1.6Mbps rcv_rtt:50 rcv_space:14480 #include <linux/tcp.h> iproute2

Speedup

Do not create connections!

for _i in $(seq 10); ssh -f thailand cat

for _i in $(seq 10); ssh \ -o 'ControlMaster yes'
\ -f thailand cat

Responsive UI • Instagram • VK • best UI •
worst reliability

Steroids • TCP Fast Open • Linux 3.6 • HAProxy

Steroids • TCP/NC • TCP and math (maths) • http://dspace.mit.edu/openaccess-
disseminate/1721.1/58796

Scheduling, Algorithms • TCP Westwood+ (LFN) • TCP Veno (Wi-Fi)
• http://www.apan.net/meetings/ honolulu2004/materials/engineering/ APAN_ppt.pdf •CONF_TCP_CONG_VENO

Steroids • TLS False Start • TLS NPN • Next
Protocol Negotiation • HTTP Pipelining • SPDY

Research • https://github.com/proger/iproute2 ss -I • https://github.com/proger/captcp • tcptrace •
tcpﬂow • monitoring

kthxbai @darkproger http://kirillov.im

TCP Performance for Mobile Applications

TCP Performance for Mobile Applications

More Decks by Volodymyr Kyrylov

Other Decks in Programming

Featured

Transcript