Threads @ JavaZone 2018

Transcript

1. Threads Adam Dubiel
2. Being abstract is something profoundly different from being vague … The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise. Edsger W. Dijkstra
3. 12 mln users 700+ microservices 600+ engineers
8. JVM starts
9. VM threads come to life (~20 threads)
10. spring context boots up (+~15 threads)
11. traffic starts flowing (+~80 threads)
12. The process is born
13. systemd(1) |-agetty(1338) |-sshd(1133)---sshd(1790)---bash(1791)
14. java -version
15. systemd(1) |-agetty(1338) |-sshd(1133)-...-bash(1791)-java(1876)
16. systemd(1) |-agetty(1338) |-java(1876)
17. Process Memory T1 T2 TN
18. Threads in JVM
19. pstree -Ap
20. systemd(1) |-agetty(1338) |-java(1876)-+-{java}(1893) | |-{java}(1892) | |-{java}(1896) | |-{java}(1890) | |-{java}(1882) | |-{java}(1881) | |-{java}(1885) | |-{java}(1886) | |-{java}(1887) | |-{java}(1879) | |-{java}(1880) | |-{java}(1897) | |-{java}(1898) | |-{java}(1889)
21. JVM threads == system threads
22. pthread_t tid; int ret = pthread_create(&tid, &attr, (void* (*)(void*)) thread_native_entry, thread); http://hg.openjdk.java.net/jdk10/jdk10/hotspot/file/5ab7a67bc155/src/os/linux/vm/os_linux.cpp#l731
23. If JVM threads are system threads.. observable using system tools scheduled using system scheduler
24. ls /proc/1876/task 1876 1893 1892 1896 1890 1882 1881 1885 1886 1887 1879 1880 1897 1898 1889
25. ps H -p <pid> -o pid,tid,pmem,pcpu,time,comm
26. top -H -p <pid>
27. ps H -p <pid> -o pid,tid,pmem,pcpu,time,comm process id thread id
29. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
30. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
31. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
32. How to match sys & JVM thread?
33. jstack <pid>
34. jcmd <pid> Thread.print
35. "pool-1-thread-10" #19 prio=5 os_prio=31 tid=0x00007fcd05868000 nid=0x6103 runnable [0x000070000ffc5000]
36. "pool-1-thread-10" #19 prio=5 os_prio=31 tid=0x00007fcd05868000 nid=0x6103 runnable [0x000070000ffc5000]
37. pool-1-thread-10 tid=0x00007fcd05868000 nid=0x6103 hex(pthreads id) hex(tid)
38. pool-1-thread-10 nid=0x6103
39. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
40. hex(8521) = 2149 jstack <pid> | grep -i 2149
41. jstack <pid> | grep `printf %x 8521`
42. "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd7ec01f800 nid=0x2149 runnable hex(8521)
43. Java 8 http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/87ee5ee27509/src/os/linux/vm/os_linux.cpp#l5038 void os::set_native_thread_name(const char *name) { // Not yet implemented. return; }
44. Java 9+ http://hg.openjdk.java.net/jdk10/jdk10/hotspot/file/5ab7a67bc155/src/os/linux/vm/os_linux.cpp#l5038 void os::set_native_thread_name(const char *name) { if (Linux::_pthread_setname_np) { char buf [16]; // according to glibc manpage, 16 chars incl. '/0' snprintf(buf, sizeof(buf), "%s", name); buf[sizeof(buf) - 1] = '\0'; const int rc = Linux::_pthread_setname_np(pthread_self(), buf); // ERANGE should not happen; all other errors should just be ignored. assert(rc != ERANGE, "pthread_setname_np failed"); } }
45. Thread name propagation Java 9+ Thread.setThreadName() is propagated to pthreads 15 chars limit Java thread names visible in system tools
46. systemd(1) |-agetty(1338) |-java(1876)-+-{C1 CompilerThre}(1893) | |-{C2 CompilerThre}(1892) | |-{Common-Cleaner}(1896) | |-{Finalizer}(1890) | |-{G1 Conc#0}(1882) | |-{G1 Main Marker}(1881) | |-{G1 Refine#0}(1885) | |-{G1 Refine#1}(1886) | |-{G1 Young RemSet}(1887) | |-{GC Thread#0}(1879) | |-{GC Thread#1}(1880) | |-{Service Thread}(1895) | |-{XNIO-1 Accept}(1904) | |-{XNIO-1 I/O-1}(1903)
47. PID TID %MEM %CPU TIME COMMAND 3799 3799 15.9 0.0 00:00:00 java 3799 3800 15.9 2.5 00:00:08 java 3799 3804 15.9 10.0 00:00:00 GC Thread#0 3799 3805 15.9 10.0 00:00:00 GC Thread#1 3799 3806 15.9 0.0 00:00:00 G1 Main Marker 3799 3807 15.9 0.0 00:00:00 G1 Conc#0 3799 3808 15.9 0.0 00:00:00 G1 Refine#0 3799 3809 15.9 0.0 00:00:00 G1 Refine#1 3799 3810 15.9 0.0 00:00:00 G1 Young RemSet 3799 3811 15.9 0.0 00:00:00 VM Thread
49. Cost of a thread
50. Cost of thread stack memory - 1MB by default: explicit context switches: implicit safepointing: implicit gc roots: implicit
51. Thread stack
52. Heap Offheap thread stacks live here
53. java -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -version
54. java -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -version
55. Thread (reserved=18548KB, committed=18548KB) (thread #18) (stack: reserved=18468KB, committed=18468KB) (malloc=59KB #101) (arena=21KB #34)
56. Thread (reserved=18548KB, committed=18548KB) (thread #18) (stack: reserved=18468KB, committed=18468KB) (malloc=59KB #101) (arena=21KB #34) number of threads memory taken from the system 18468KB / 18 = 1024KB
57. total app memory = heap size + thread count * stack size + ...
58. How many threads can i spawn? Heap Offheap RAM
59. How many threads can i spawn? Heap Offheap RAM 2GB 500MB
60. How many threads can i spawn? Heap Offheap RAM 2GB 512MB 300MB for stuff.. 1.2GB for threads
61. How many threads can i spawn? 1.2GB / 1MB = 1200
64. VSZ vs RES
65. VSZ vs RES virtual memory size address space process can access doesn't mean there is enough physical memory
66. VSZ vs RES resident set size memory acutaly used by the process the only metric showing real memory usage
68. core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 7873 max locked memory (kbytes, -l) 16384 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7873 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
69. core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 7873 max locked memory (kbytes, -l) 16384 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7873 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
70. Why less is more?
71. CPU CPU CPU CPU waiting..
72. Cost of thread stack memory - 1MB by default: explicit context switches: implicit safepointing: implicit gc roots: implicit
73. Context switch
74. core L3 cache L1 cache L2 cache shared exclusive
75. core L3 cache L1 cache L2 cache <1 ns ~4 ns ~20-40 ns ~100 ns RAM
76. core RAM data flows L2 -> L1
77. core RAM another thread this data belonged to other thread - probably useless for new one
78. Context switches up to 30 micro seconds each observable via perf and vmstat
79. CPU CPU CPU CPU
80. Back to the app...
82. VM threads come to life (~20 threads)
83. G1 GC threads VMThread C1 & C2 PeriodicTaskThread
84. traffic starts flowing (+~80 threads) spring context boots up (+~15 threads)
85. Managing threads
86. new Thread() Executors.fixedThreadPool()
87. Spring default: SimpleAsyncTaskExecutor TaskExecutor implementation that fires up a new Thread for each task, executing it asynchronously. By default, the number of concurrent threads is unlimited. This implementation does not reuse threads!
88. There is hope! http://my-little-pony-friendship-is-magic-rakoon1.wikia.com/wiki/Radiant_Hope
89. Thread pool
90. Thread pool what happens when it's full?
91. Thread pool how big is it?
92. Thread pool
93. new ThreadPoolExecutor( corePoolSize, maxPoolSize, keepAliveTime, keepAliveTimeUnit, taskQueue, threadFactory, rejectionPolicy )
94. new ThreadPoolExecutor( corePoolSize, maxPoolSize, keepAliveTime, keepAliveTimeUnit, taskQueue, threadFactory, rejectionPolicy ) task queue implementation
95. new ThreadPoolExecutor( corePoolSize, maxPoolSize, keepAliveTime, keepAliveTimeUnit, taskQueue, threadFactory, rejectionPolicy ) what happens when queue is full
96. Rejection: Happy path Thread Pool Caller Thread
97. Rejection: Default unhappy path Thread Pool Caller Thread
98. Rejection: CallerRunsPolicy Thread Pool Caller Thread
99. Rejection: AbortPolicy Thread Pool Caller Thread Rejected Execution Exception
100. ThreadPoolExecutor.DiscardPolicy Thread Pool Caller Thread /dev/null
101. ThreadPoolExecutor.DiscardOldestPolicy Thread Pool Caller Thread /dev/null
102. traffic starts flowing (+~80 threads)
105. once used threads are kept alive
106. ThreadPoolExecutor.prestartAllCoreThreads()
107. traffic starts flowing (+~80 threads) spring context boots up (+~15 threads)
108. -Xlog:thread+os:file=/tmp/threads.log
109. Key takeaways JVM threads are system threads can be observed using system tools WebFlux, because less is should be more tune your thread pools
110. github.com/adamdubiel @dubieladam