Virtual Threads: Project Loom’s Scalability Revolution

Threads, which have always been an integral part of Java, have their limits: You cannot start more than a few thousand without jeopardizing system stability. For highly scalable applications, we had to resort to asynchronous programming. But asynchronous code is difficult to write, read and debug.

For years, a pioneering solution was being worked on within Project Loom: With almost 100,000 changed lines of code, the final version of virtual threads finally appeared in Java 21.

These require orders of magnitude fewer resources than conventional threads, especially for blocking operations. Instead of thousands, we now have millions of threads at our disposal. This allows us to write highly scalable applications in the traditional thread-per-request style that are easier to maintain, test, and debug than asynchronous applications.

Using an example application, I will take you from the limitations of classic concurrent programming to asynchronous approaches and the possible applications and functionality of virtual threads. With a final overview of limitations and potential pitfalls, as well as some tips on migrating existing applications, you will be well-equipped to use virtual threads in your daily work.

Transcript

1. Redefining Concurrency: A Deep Dive into Virtual Threads Sven Woltmann
2. Copyright © 2023, HappyCoders.eu What are Virtual Threads?
3. Get product data Get availability in warehouse Available? Get delivery time from supplier Yes No Copyright © 2023, HappyCoders.eu API endpoint called Return product data, shipping after X days Return product data, shipping today
4. @GET @Path("/product/{productId}") public ProductPageResponse getProduct(@PathParam("productId") String productId) { Product product = productService.getProduct(productId) .orElseThrow(NotFoundException::new); boolean available = warehouseService.isAvailable(productId); int shipsInDays = available ? 0 : supplierService.getDeliveryTime(product.supplier(), productId); return new ProductPageResponse(product, shipsInDays); } Copyright © 2023, HappyCoders.eu
5. Idle 100% 100% Idle 100% Idle 100% Copyright © 2023, HappyCoders.eu @GET @Path("/product/{productId}") public ProductPageResponse getProduct(@PathParam("productId") String productId) { Product product = productService.getProduct(productId) .orElseThrow(NotFoundException::new); boolean available = warehouseService.isAvailable(productId); int shipsInDays = available ? 0 : supplierService.getDeliveryTime(product.supplier(), productId); return new ProductPageResponse(product, shipsInDays); }
6. Idle 100% 100% Idle 100% Idle 100% ns ms ns ms ns ms ns CPU busy: in the order of nanoseconds CPU idle: in the order of milliseconds → CPU Core Busy: 0.0001% → CPU Busy: 0.00001% Copyright © 2023, HappyCoders.eu Spring Boot: 200 threads → CPU Busy: 0.002%
7. Copyright © 2023, HappyCoders.eu How can we better utilize the CPU? Solution A: More threads
8. Copyright © 2023, HappyCoders.eu 20,000 200 threads → CPU busy: 0.002% 20,000 threads → CPU busy: 0.2%
9. Copyright © 2023, HappyCoders.eu Java OS Thread 1 Thread 2 Thread 3 Thread n ⋮ OS Thread 1 OS Thread 2 OS Thread 3 OS Thread n ⋮
10. Copyright © 2023, HappyCoders.eu Price of Operating System Threads • 1 MB reserved / 32-64 KB committed for stack • Start time: ~ 1ms • High price for context switches
11. Copyright © 2023, HappyCoders.eu How can we better utilize the CPU? Solution A: More threads
12. Copyright © 2023, HappyCoders.eu How can we better utilize the CPU? Solution A: More threads ↓ ⊕ Easy to set up ⊖ Limited number
13. Copyright © 2023, HappyCoders.eu How can we better utilize the CPU? Solution A: More threads Solution B: More requests per thread ↓ Asynchronous ↓ ⊕ Easy to set up ⊖ Limited number
14. @GET @Path("/product/{productId}") public ProductPageResponse getProduct(@PathParam("productId") String productId) { Product product = productService.getProduct(productId) .orElseThrow(NotFoundException::new); boolean available = warehouseService.isAvailable(productId); int shipsInDays = available ? 0 : supplierService.getDeliveryTime(product.supplier(), productId); return new ProductPageResponse(product, shipsInDays); } Copyright © 2023, HappyCoders.eu
15. @GET @Path("/product/{productId}") public CompletionStage<Response> getProduct(@PathParam("productId") String productId) { return productService .getProductAsync(productId) .thenCompose( product -> { if (product.isEmpty()) { return CompletableFuture.completedFuture(Response.status(Status.NOT_FOUND).build()); } return warehouseService .isAvailableAsync(productId) .thenCompose( available -> available ? CompletableFuture.completedFuture(0) : supplierService.getDeliveryTimeAsync( product.get().supplier(), productId)) .thenApply( daysUntilShippable -> Response.ok(new ProductPageResponse(product.get(), daysUntilShippable)) .build()); }); }
16. Copyright © 2023, HappyCoders.eu How can we better utilize the CPU? Solution A: More threads Solution B: More requests per thread ↓ Asynchronous ↓ ⊕ Easy to set up ⊖ Limited number
17. Copyright © 2023, HappyCoders.eu How can we better utilize the CPU? Solution A: More threads Solution B: More requests per thread ↓ Asynchronous ↓ ⊕ Proven, Mature ⊖ High cost ↓ ⊕ Easy to set up ⊖ Limited number
18. Copyright © 2023, HappyCoders.eu How can we better utilize the CPU? Solution A: More threads Solution B: More requests per thread Solution C: Cheaper threads ↓ Asynchronous ↓ ⊕ Proven, Mature ⊖ High cost ↓ Virtual threads ↓ ⊕ Easy to set up ⊖ Limited number
19. Copyright © 2023, HappyCoders.eu What are Virtual Threads?
20. Copyright © 2023, HappyCoders.eu Java OS Platform Thread 1 Platform Thread 2 Platform Thread 3 Platform Thread n ⋮ OS Thread 1 OS Thread 2 OS Thread 3 OS Thread n ⋮
21. Copyright © 2023, HappyCoders.eu Java OS Virtual Thread 2 Virtual Thread 3 Virtual Thread 4 Virtual Thread n ⋮ OS Thread 1 OS Thread 2 OS Thread m ⋮ Carrier Thread 1 Carrier Thread 2 ⋮ Carrier Thread m Virtual Thread 1
22. Copyright © 2023, HappyCoders.eu Java OS Virtual Thread 2 Virtual Thread 3 Virtual Thread 4 Virtual Thread n ⋮ OS Thread 1 OS Thread 2 OS Thread m ⋮ Carrier Thread 1 Carrier Thread 2 ⋮ Carrier Thread m Virtual Thread 1
23. Copyright © 2023, HappyCoders.eu Java OS Virtual Thread 2 Virtual Thread 3 Virtual Thread 4 Virtual Thread n ⋮ OS Thread 1 OS Thread 2 OS Thread m ⋮ Carrier Thread 1 Carrier Thread 2 ⋮ Carrier Thread m Virtual Thread 1
24. Copyright © 2023, HappyCoders.eu VT 1 Idle 100% 100% Idle 100% Idle 100% Idle 100% 100% Idle 100% Idle 100% Idle 100% 100% Idle 100% Idle 100% VT 1 VT 1 VT 1 VT 2 VT 2 VT 2 VT 2 VT 3 VT 3 VT 3 VT 3 VT 1: VT 2: VT 3: Carrier Thread 1:
25. Copyright © 2023, HappyCoders.eu Live Coding: How to Create Virtual Threads?
26. Copyright © 2023, HappyCoders.eu 10,000,000 200 threads → CPU busy: 0.002% 20,000 threads → CPU busy: 0.2% 10,000,000 threads → CPU busy: 100%
27. Copyright © 2023, HappyCoders.eu How can we better utilize the CPU? Solution A: More threads Solution B: More requests per thread Solution C: Cheaper threads ↓ Asynchronous ↓ Virtual threads ↓ ⊕ Proven, Mature ⊖ High cost ↓ ⊕ Easy to set up ⊖ Limited number
28. Copyright © 2023, HappyCoders.eu How can we better utilize the CPU? Solution A: More threads Solution B: More requests per thread Solution C: Cheaper threads ↓ Asynchronous ↓ Virtual threads ✓ ↓ ⊕ Proven, Mature ⊖ High cost ↓ ⊕ Easy to set up ⊖ Limited number ↓ ⊕ Easy to implement ⊕ Scales
29. Copyright © 2023, HappyCoders.eu Jakarta EE 11 (Quarkus since 2.10.0) @GET @Path("/product/{productId}") @RunOnVirtualThread public ProductPageResponse getProduct(@PathParam("productId") String productId) { Product product = productService.getProduct(productId) .orElseThrow(NotFoundException::new); boolean available = warehouseService.isAvailable(productId); int shipsInDays = available ? 0 : supplierService.getDeliveryTime(product.supplier(), productId); return new ProductPageResponse(product, shipsInDays); }
30. Copyright © 2023, HappyCoders.eu @Bean(TaskExecutionAutoConfiguration.APPLICATION_TASK_EXECUTOR_BEAN_NAME) public AsyncTaskExecutor asyncTaskExecutor() { return new TaskExecutorAdapter(Executors.newVirtualThreadPerTaskExecutor()); } @Bean public TomcatProtocolHandlerCustomizer<?> protocolHandlerVirtualThreadExecutorCustomizer() { return protocolHandler -> { protocolHandler.setExecutor(Executors.newVirtualThreadPerTaskExecutor()); }; } Spring
31. Copyright © 2023, HappyCoders.eu How do Virtual Threads work under the hood?
32. Carrier Thread Pool (“ForkJoinPool-1”) Continuation Copyright © 2023, HappyCoders.eu Task: handleRequest(...) validateRequest(...) getFromApi(...) httpGet(...) connect(...) Continuation.yield(...) send(...) receive(...) ... ... run() Work Queue Carrier Thread 1 Work Queue Carrier Thread 2 Heap Stack Stack IP
33. Copyright © 2023, HappyCoders.eu Show how Thread.sleep() works
34. Copyright © 2023, HappyCoders.eu Advantages of Virtual Threads - They can be created much faster (<1 µs vs. ~1 ms) - Less memory with shallow stacks (~1 KB vs. 32-64 KB) - Context switches are fast - Blocking is cheap • They are cheap • Familiarity - java.lang.Thread API and ExecutorService API - Traditional blocking thread-per-request-style - Debugging, observing, profiling with existing tools
35. Copyright © 2023, HappyCoders.eu What are Virtual Threads Not? • They are not faster threads • They are not preemptive • They are not suitable for parallel computing • They are not higher-level abstractions
36. Copyright © 2023, HappyCoders.eu Limitations of Virtual Threads - File I/O - Object.wait() • Some blocking operations do not (yet) unmount a virtual thread: • Pinning: - synchronized - Native code → Both operations will temporarily increase the number of carrier threads → Diagnostics: -Djdk.tracePinnedThread=full/short • No lock information in thread dumps → Use ReentrantLock around blocking operations
37. Copyright © 2023, HappyCoders.eu Thread Dumps • Traditional thread dumps do not include virtual threads! jcmd <pid> Thread.print • New thread dump formats: jcmd <pid> Thread.dump_to_file -format=plain <file> jcmd <pid> Thread.dump_to_file -format=json <file>
38. Copyright © 2023, HappyCoders.eu Differences Virtual / Platform Threads • Virtual threads are daemon threads. • Their priority is normal and cannot be changed. • They do not belong to a thread group.
39. Copyright © 2023, HappyCoders.eu Other Things to Consider • Don’t pool virtual threads • Use semaphores to limit access to resources • Much of the virtual thread code is written in Java → Let the JVM warm up when performing tests • Use virtual threads for what they were made for: Lots of tasks that contain mainly blocking operations • Don’t just flip the switch – test! • A virtual thread with a deep stack will use as much memory as a platform thread with a deep stack
40. Copyright © 2023, HappyCoders.eu Outlook - What’s Coming? • Compressed Frames • Structured Concurrency • Scoped Values • File I/O • synchronized not pinning
41. Copyright © 2023, HappyCoders.eu The End
42. Copyright © 2023, HappyCoders.eu GitHub repositories: https://github.com/SvenWoltmann/virtual-threads https://github.com/SvenWoltmann/virtual-threads-quarkus https://github.com/SvenWoltmann/virtual-threads-spring JEP 444: Virtual Threads: https://openjdk.org/jeps/444 Virtual Threads in Java: https://www.happycoders.eu/java/virtual-threads/ https://twitter.com/svenwoltmann https://youtube.com/c/happycoders Links The Challenges of Introducing Virtual Threads: https://www.youtube.com/watch?v=WsCJYQDPrrE Continuations - Under the Covers: https://www.youtube.com/watch?v=6nRS6UiN7X0 https://www.happycoders.eu/ https://github.com/SvenWoltmann https://linkedin.com/in/sven-woltmann/