Project Loom Broadening Concurrency

Mark Paluch Spring Data Project Lead @ VMware R2DBC Spec Lead Lettuce Redis Driver Maintainer Paramedic & SAR Dogs

Sept 2020, https://paluch.biz/blog/182-experimenting-with-project-loom-eap-and-spring-webmvc.html „We’ve been there, already“

Tomcat + Virtual Threads Tomcat + Kernel Threads

What is Project Loom • Initiative to introduce virtual threads to the Java Platform. Virtual threads are lightweight threads that dramatically reduce the effort of writing, maintaining, and observing high-throughput concurrent applications.

What is Project Loom • Initiative to introduce Introduction of virtual threads and concurrency API to the Java Platform. Virtual threads are lightweight threads that eliminate I/O wait. dramatically reduce the effort of writing, maintaining, and observing high-throughput concurrent applications.

What Project Loom actually is • Introduction of virtual threads and concurrency API to the Java Platform. Virtual threads are lightweight threads that eliminate I/O wait.

Dramatically reduce the effort of writing, maintaining, and observing concurrent applications • True if you run on a Virtual Thread • Future.get(), Thread.sleep(), Lock.lock() are good friends again • No need for future chaining, using complex reactive or asynchronous API • Not so, if you accidentally run on a Kernel Thread

Overview

Platform Thread • Renamed from Kernel Thread • new Thread(…)

Virtual Thread • Package-private subtype of java.lang.Thread • Provides customized behavior • Subject to be executed on carrier Thread

Carrier Threads • Separate Fork/Join Pool of Platform Threads • Default size: Runtime.availableProcessors() • Configurable through system properties (jdk.virtualThreadScheduler.parallelism, maxPoolSize, minRunnable)

Virtual Threads API • Extended Thread API Thread.ofVirtual().start(() -> {…}); Executors.newVirtualThreadPerTaskExecutor() Thread.currentThread().isVirtual() • Implemented almost entirely in Java

try(var e = Executors.newVirtualThreadPerTaskExecutor()){ for (int j = 0; j < 100; j++) { e.execute(() -> Thread.sleep(100)); } } // implicit wait

No content

Parking & Unparking • On each blocking operation • If blocked, capture stack and store it • Carrier thread free to perform other work (unpark next virtual thread)

No content

Blocking in synchronized code • Leads to Carrier Thread Pinning • Carrier Thread bound to virtual thread until unblock • synchronized is a JVM behavior, cannot be controlled from Java code • I/O, Locks, Future.get() in synchronized leads to Thread Pinning

Thread Type Selection • I/O-heavy, shared-resources with Java Locks, Concurrency Utils • Virtual Threads • Computation-heavy, synchronized workloads • Platform Threads

Observability, Tooling, Trouble Shooting • Java Flight Recorder Events • Virtual Thread start/end, pinned, submission failed • Debugging: IntelliJ announced tooling support for later this year • Pinned thread monitoring • -Djdk.tracePinnedThreads=full|short

Demo

Virtual Thread Usage

Usage of Virtual Thread API • In every piece that could block or wait in Java • Thread.sleep • Locks, ConcurrentHashMap • I/O (Socket) • Rewrite of several components as pre-requisite for Loom to avoid thread pinning

Usage Pattern • Kernel Threads („Platform Threads“) • Expensive • Thread Pooling • Virtual Threads • Cheap • New Virtual Thread per Runnable

Benchmarks

JMH Benchmarks Benchmark Mode Cnt Score Error Units VirtualThreadsBenchmark.createVirtualThreads thrpt 5 14488555,637 ± 1127513,241 ops/s VirtualThreadsBenchmark.runAndAwait thrpt 5 455494,235 ± 22033,286 ops/s PlatformThreadsBenchmark.createPlatformThreads thrpt 5 19094,736 ± 677,739 ops/s PlatformThreadsBenchmark.runAndAwait thrpt 5 15807,802 ± 7337,906 ops/s

JMH Benchmarks Benchmark Mode Cnt Score Error Units VirtualThreadsBenchmark.createVirtualThreads thrpt 5 14488555,637 ± 1127513,241 ops/s VirtualThreadsBenchmark.runAndAwait thrpt 5 455494,235 ± 22033,286 ops/s PlatformThreadsBenchmark.createPlatformThreads thrpt 5 19094,736 ± 677,739 ops/s PlatformThreadsBenchmark.runAndAwait thrpt 5 15807,802 ± 7337,906 ops/s ThreadPoolExecutorBenchmark.runAndAwait thrpt 5 461335,862 ± 54427,010 ops/s

Conclusion

What does that mean? • If the CPU availability is the limit: You cannot increase concurrency • If the Runnable wait is the limit: You can increase concurrency

Loom Overhead • Hard to say. What’s the typical application?

Scalability Boundaries • I/O in synchronized Code • Connection Pool Sizes • Databases • HTTP/JMS/… Connection Pools • File handles • number of open files

Pitfalls • System.out.println(…) • Attempts to obtain a lock on the I/O • Measurement and benchmark correctness („What do you measure?“) • Visible and invisible Thread Pinning

Client/Server Workloads • Workload to be served by virtual thread (carrier threads must be available) • System must have sufficient resources to serve the request (e.g. connection pools). Also a constraint for platform thread usage. • Request to server via I/O • During wait, virtual thread is parked • Client must not pin kernel thread (see carrier thread availability)

Computation Workloads • Incoming request/event to be served by virtual thread (carrier threads must be available) • System performs a computation based on in-memory resources • Response is returned

Call to Action • Investigate: What is limiting your application scalability? • Investigate: What workloads do you run? • Experiment: Run your application on Virtual Threads and benchmark it! • Observe: What libraries lead to thread pinning?

Maybe. „Will Loom help with concurrency and scalability in my app?“

Thank You. github.com/mp911de @mp911de paluch.biz