Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Virtual Threads in Action!

Virtual Threads in Action!

Presented at DevBcn Conference 2023

Dmitry Alexandrov

July 07, 2023
Tweet

More Decks by Dmitry Alexandrov

Other Decks in Technology

Transcript

  1. Safe Harbor Copyright © 2023, Oracle and/or its affiliates Statement

    The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. 2
  2. Copyright © 2023, Oracle and/or its affiliates Nice to meet

    you Dmitry Aleksandrov Writing code at Oracle [email protected] @bercut2000(@c.im) 3
  3. Threads in Java are expensive Copyright © 2023, Oracle and/or

    its affiliates • 2 Mb. per thread memory consumption • ~ 1 Ms. to start a thread • Context switch ~100 ns. (different for each OS) On an “infinite machine” you will need several minutes just to start them… 5
  4. Project Loom Copyright © 2023, Oracle and/or its affiliates Carrier

    Threads Virtual Threads • Not a Green thread (who know what is the difference) 10
  5. Project Loom Copyright © 2023, Oracle and/or its affiliates Carrier

    Threads Virtual Threads • Not a Green thread (who know what is the difference) • Virtual threads are mounted over Carrier Threads 11
  6. Project Loom Copyright © 2023, Oracle and/or its affiliates Carrier

    Threads Virtual Threads • Not a Green thread (who know what is the difference) • Virtual threads are mounted over Carrier Threads • When a virtual thread blocks, it is unmounted from the carrier, making room for others. 12
  7. Project Loom Copyright © 2023, Oracle and/or its affiliates Carrier

    Threads Virtual Threads • Not a Green thread (who know what is the difference) • Virtual threads are mounted over Carrier Threads • When a virtual thread blocks, it is unmounted from the carrier, making room for others. • Once the virtual thread is unlocked, the scheduler will direct it to the real carrier thread 13
  8. Project Loom Copyright © 2023, Oracle and/or its affiliates Carrier

    Threads Virtual Threads • Not a Green thread (who know what is the difference) • Virtual threads are mounted over Carrier Threads • When a virtual thread blocks, it is unmounted from the carrier, making room for others. • Once the virtual thread is unlocked, the scheduler will direct it to the real carrier thread • ForkJoinPool used for Carrier threads 14
  9. Project Loom Copyright © 2023, Oracle and/or its affiliates Carrier

    Threads Virtual Threads • Not a Green thread (who know what is the difference) • Virtual threads are mounted over Carrier Threads • When a virtual thread blocks, it is unmounted from the carrier, making room for others. • Once the virtual thread is unlocked, the scheduler will direct it to the real carrier thread • ForkJoinPool used for Carrier threads • It has all the same properties as the threads we are used to 15
  10. Project Loom Copyright © 2023, Oracle and/or its affiliates Carrier

    Threads Virtual Threads • Not a Green thread (who know what is the difference) • Virtual threads are mounted over Carrier Threads • When a virtual thread blocks, it is unmounted from the carrier, making room for others. • Once the virtual thread is unlocked, the scheduler will direct it to the real carrier thread • ForkJoinPool used for Carrier threads • It has all the same properties as the threads we are used to • Implemented with Continuations (Continuations.yield(), no external API) 16
  11. Virtual Threads Copyright © 2023, Oracle and/or its affiliates •

    There can be millions of threads • Always deamon threads • Always have group “VirtualThreads” • Have no permission with a SecurityManager (well.. It is deprecated) 17
  12. API Copyright © 2023, Oracle and/or its affiliates • Executors.newVirtualThreadExecutor()

    • Executors.newVirtualThreadPerTaskExecutor() • Thread.startVirtualThread(Runnable) – run virtual thread • Thread.ofVirtual() – builder • Thread.ofVirtual().factory() – thread factory • Thread.isVirtual() – instance method 18
  13. Ok, we heard that on some other talks Copyright ©

    2023, Oracle and/or its affiliates Where is the “in action” part? 19
  14. • Framework for developing cloud-native microservices • Server, but not

    a Servlet container • As little dependence on 3rd party libraries as possible • Always sync with the latest Java release • Multiple flavors for different usage • K8s friendly Copyright © 2023, Oracle and/or its affiliates What is Helidon: 21
  15. • Simplicity • Transparency • No tricks and cheating •

    Try not to use external dependencies • Use latest version of Java • Adopt the latest Java language features • Encourage customers to upgrade Copyright © 2023, Oracle and/or its affiliates Helidon Philosophy 22
  16. Copyright © 2023, Oracle and/or its affiliates Helidon History 23

    Helidon 1.x 2019 Java 8 MicroProfile 3.2 Helidon 2.x 2020 Java 11 MicroProfile 3.3 javax.* Helidon 3.x 2022 Java 17 MicroProfile 5.0 jakarta.* Helidon 4.x-⍶ 2022 Java 20 MicroProfile 6.0 Helidon 4.x 2021 Java 21 MicroProfile 6.0
  17. Copyright © 2023, Oracle and/or its affiliates Helidon Packaging 24

    Container Application OS Java Runtime > java -jar helidon-app.jar Executable Jar Container Application OS Custom Image > helidon-app-jri/bin/start Custom Image Container Native App OS > ./helidon-app GraalVM Native Image
  18. Copyright © 2023, Oracle and/or its affiliates Helidon SE •

    follows the functional style • is reactive and non-blocking • is very Fast • has a tiny Footprint • is transparent, no “magic” Helidon MP • Is declarative style (Jakarta EE, Spring Boot) • Based on MicroProfile plus some Jakarta EE components • Is fast • Has a small Footprint • Uses Annotations & Dependency Injection Routing routing = Routing.builder() .get("/hello", (req, res) -> res.send( "Hello World")) .build(); WebServer.create(routing) .start(); @Path("hello") public class HelloWorld { @GET public String hello() { return "Hello World"; } } Choose your way! 25
  19. Copyright © 2023, Oracle and/or its affiliates Helidon Architecture (1.x

    to 3.x) 26 Netty Helidon MP Helidon SE CDI JAX-RS WebServer Config Reactive Security
  20. • Reactive Programming Model • Reactive Streams • CompletionStage •

    Non-blocking, backpressure • For highly concurrent services • Feature parity with HelidonMP Copyright © 2023, Oracle and/or its affiliates Helidon SE 27
  21. Helidon Reactive Engine Copyright © 2023, Oracle and/or its affiliates

    Helidon has its own set of reactive operators that have no dependencies outside of the Helidon ecosystem. These operators can be used with java.util.concurrent.Flow based reactive streams. Stream processing operator chain can be easily constructed by io.helidon.common.reactive.Multi, or io.helidon.common.reactive.Single for streams with single value. <dependency> <groupId>io.helidon.common</groupId> <artifactId>helidon-common-reactive</artifactId> </dependency> 28
  22. Java 9+ Copyright © 2023, Oracle and/or its affiliates java.util.concurrent.Flow

    API consists of four basic interfaces: • Subscriber: The Subscriber subscribes to Publisher for callbacks. • Publisher: The Publisher publishes the stream of data items to the registered subscribers. • Subscription: The link between publisher and subscriber. • Processor: The processor sits between Publisher and Subscriber, and transforms one stream to another. It is a combination of both Iterator and Observer patterns 29
  23. Helidon Reactive Engine Copyright © 2023, Oracle and/or its affiliates

    AtomicInteger sum = new AtomicInteger(); Multi.just("1", "2", "3", "4", "5") .limit(3) .map(Integer::parseInt) .forEach(sum::addAndGet); System.out.println("Sum: " + sum.get()); 30 Single.just("1") .map(Integer::parseInt) .map(i -> i + 5) .whenComplete((i, t) -> System.out.println("Result: " + i)); > Result: 6
  24. Helidon Reactive Engine Operators Copyright © 2023, Oracle and/or its

    affiliates fromIterable of ofNullable iterate generate failed concat coupled limit peek filter map flatMap flatMapIterable flatMapCompletionStage flatMapRSPublisher takeWhile dropWhile skip distinct via onError onErrorResume onErrorResumeWitt onErrorResumeWithRsPub onComplete onTerminate to toList collect forEach ignore reduce cancel findFirst 31
  25. Reactive WebServer Copyright © 2023, Oracle and/or its affiliates public

    static void main(String[] args) { WebServer webServer = WebServer .create(Routing.builder() .any((req, res) -> res.send("It works!"))) .start() .await(10, TimeUnit.SECONDS); System.out.println("Server started at: http://localhost:" + webServer.port()); } 33
  26. DBClient Copyright © 2023, Oracle and/or its affiliates Async work

    with a Sync driver dbClient.inTransaction(tx -> tx .createQuery("SELECT name FROM Pokemons WHERE id = :id") .addParam("id", 1) .execute() ); 35
  27. WebClient Copyright © 2023, Oracle and/or its affiliates Async client:

    WebClient.builder() .baseUri("http://localhost") .addReader(JsonpSupport.reader()) .addWriter(JsonpSupport.writer()) .addMediaService(JsonpSupport.create()) .build(); 36
  28. Copyright © 2023, Oracle and/or its affiliates 37 • Custom

    subscribers is a pitfall • Steep learning curve • Hard to get right™ ◦ Exceptionally ◦ Troubleshooting ◦ No useful stack traces ◦ More than one task in parallel is tough • Using blocking code requires executor services • “Callback Hell” Reactive Programming
  29. Copyright © 2023, Oracle and/or its affiliates Helidon Architecture (1.x

    to 3.x) 38 Netty Helidon MP Helidon SE CDI JAX-RS WebServer Config Reactive Security
  30. Copyright © 2023, Oracle and/or its affiliates • Set of

    specifications for Microservices • Core Jakarta EE APIS + MicroProfile specific APIs • “Source code” compatibility • Multiple Vendors
  31. Copyright © 2023, Oracle and/or its affiliates MicroProfile Config MicroProfile

    Metrics MicroProfile Health Check MicroProfile Tracing MicroProfile Fault Tolerance MicroProfile JWT Auth MicroProfile REST Client MicroProfile Open API Jakarta RESTful Web Services Jakarta JSON Processing Jakarta JSON Binding Jakarta CDI Jakarta Persistence Jakarta Transactions MicroProfile Reactive Streams Operators MicroProfile Reactive Messaging CORS Jakarta WebSocket gRPC Server & Client Jakarta Annotations MicroProfile GraphQL MicroProfile LRA Components: 40 MicroProfile MP standalone Jakarta EE Helidon Specific Integrations
  32. Copyright © 2023, Oracle and/or its affiliates Trade offs 43

    + • Reactive • Great performance • High concurrency – • Scaffolding • Hard to debug • Hard to maintain • Hard to learn Helidon MP Helidon SE + • Standard • Easy to write • Easy to maintain • Easy to debug • Easy to learn – • Blocking • Average performance • Limited by quantity of threads
  33. Copyright © 2023, Oracle and/or its affiliates Or 44 HIGH

    PERFORMANCE + COMPLEXITY SIMPLICITY + AVERAGE PERFORMANCE In other words:
  34. Helidon + Virtual Threads Copyright © 2023, Oracle and/or its

    affiliates Perfect match for our workload! 46
  35. Virtual Threads Based WebServer Copyright © 2023, Oracle and/or its

    affiliates • The world’s first framework based on virtual threads! 48
  36. Virtual Threads Based WebServer Copyright © 2023, Oracle and/or its

    affiliates • The world’s first framework based on virtual threads! • Scalability of asynchronous programming models with the simplicity of synchronous code 49
  37. Virtual Threads Based WebServer Copyright © 2023, Oracle and/or its

    affiliates • The world’s first framework based on virtual threads! • Scalability of asynchronous programming models with the simplicity of synchronous code • Built from the ground up in tight collaboration with the Java team 50
  38. Virtual Threads Based WebServer Copyright © 2023, Oracle and/or its

    affiliates • The world’s first framework based on virtual threads! • Scalability of asynchronous programming models with the simplicity of synchronous code • Built from the ground up in tight collaboration with the Java team • Contains Nima Web Server plus additional libraries (observability, testing, etc.) 51
  39. Virtual Threads Based WebServer Copyright © 2023, Oracle and/or its

    affiliates • The world’s first framework based on virtual threads! • Scalability of asynchronous programming models with the simplicity of synchronous code • Built from the ground up in tight collaboration with the Java team • Contains Nima Web Server plus additional libraries (observability, testing, etc.) • Performance comparable to Netty 52
  40. Virtual Threads Based WebServer Copyright © 2023, Oracle and/or its

    affiliates • The world’s first framework based on virtual threads! • Scalability of asynchronous programming models with the simplicity of synchronous code • Built from the ground up in tight collaboration with the Java team • Contains Nima Web Server plus additional libraries (observability, testing, etc.) • Performance comparable to Netty • Will be a heart of next major Helidon release (planned in 2023) 53
  41. Virtual Threads Based WebServer Copyright © 2023, Oracle and/or its

    affiliates • The world’s first framework based on virtual threads! • Scalability of asynchronous programming models with the simplicity of synchronous code • Built from the ground up in tight collaboration with the Java team • Contains Nima Web Server plus additional libraries (observability, testing, etc.) • Performance comparable to Netty • Will be a heart of next major Helidon release (planned in 2023) • Alpha-6 version is available in Maven Central 54
  42. Copyright © 2023, Oracle and/or its affiliates Helidon Architecture (4.x)

    55 Helidon MP Helidon Níma CDI JAX-RS WebServer Config Security
  43. Virtual Threads Based WebServer Copyright © 2023, Oracle and/or its

    affiliates • HTTP/1.1 protocol with full pipelining support • HTTP/2 protocol • GRPC protocol • WebSocket protocol • Unit and integration testing support • TLS and mTLS support (ALPN for HTTP/2) • Extensible ◦ Other protocols (even non-HTTP) ◦ HTTP based protocols (Upgrade from HTTP/1, HTTP/2) 56
  44. Virtual Threads WebServer Copyright © 2023, Oracle and/or its affiliates

    • Access Log support • CORS support • Static Content • OpenTelemetry tracing • Observability endpoint • Configuration information - /observe/config • Application information - /observe/info • Health checks - /observe/health • Other coming… 57
  45. New Java Features Copyright © 2023, Oracle and/or its affiliates

    • Virtual threads – for performant blocking code • Enhanced switch – cleaner code • Sealed classes – for cleaner API • System.logger – no slf4j, simple replacement 58
  46. Copyright © 2023, Oracle and/or its affiliates Config Health Checks

    Metrics Security Tracing CORS WebSocket WebServer gRPC Server Fault Tolerance WebClient 59 • Nima Core • Helidon Shared • In Progress Tips and Tricks
  47. Why? That’s why! Copyright © 2023, Oracle and/or its affiliates

    60 Helidon 4 Níma WebServer Helidon SE Reactive
  48. Helidon MP apps Copyright © 2023, Oracle and/or its affiliates

    Your Helidon 3 MP will run on Helidon 4 MP almost no change! 63
  49. Helidon MP apps Copyright © 2023, Oracle and/or its affiliates

    Your Helidon 3 MP will run on Helidon 4 MP almost no change! With all benefits of Níma Server and Virtual Threads. 64
  50. Helidon MP apps Copyright © 2023, Oracle and/or its affiliates

    Your Helidon 3 MP will run on Helidon 4 MP almost no change! With all benefits of Níma Server and Virtual Threads. With a Reactive Performance! 65
  51. What we do not solve Copyright © 2023, Oracle and/or

    its affiliates • “Obstructing” of a thread is not possible in either blocking or reactive frameworks • Obstruction – long term, full utilization of a thread, requiring usage of yield • Reactive: designed to handle short non-blocking tasks, obstruction degrades performance heavily • Blocking: when obstructed, consumes the pinned thread of Fork-join pool, if used concurrently, degrades server performance • Solution: use a custom executor service ◦ Reactive: complete a CompletableFuture when processing done ◦ Blocking: block until processing done 68
  52. Challenges Copyright © 2023, Oracle and/or its affiliates 69 •

    To re-use or not (byte buffers/byte arrays) ◦ Due to the (huge) number of virtual threads, using a single component to cache buffers for re-use is not efficient. We have achieved higher throughput with discarding them (and let GC do its work) than with reuse. Similar results with native byte buffers and heap byte buffers • Asynchronous writes ◦ By default, we write to sockets asynchronously. On Linux, this provides higher performance when HTTP/1.1 pipelining is used (up to 3x). When not using pipelining, there is no additional advantage to this. So the async writes are configurable and can be disabled.
  53. Challenges Copyright © 2023, Oracle and/or its affiliates 70 •

    Blocking or non-blocking sockets/socket channels ◦ After a lot of testing and validation with Java team, we found out that the best performance is achieved with blocking sockets – e.g. we use ServerSocket in blocking mode to listen for connections and use “old school” approach of accepting a socket and running a new thread to process it (of course the thread is virtual) • Learn to code blocking! ◦ A lot of times we are used to asynchronous coding; forget about it, just block! ◦ Code is easier to read, cleaner and easier to troubleshoot.
  54. Challenges Copyright © 2023, Oracle and/or its affiliates 71 •

    HTTP/2 is hard ◦ Nevertheless we found a way to provide unified HTTP routing (regardless of version), with support to version specific routes ◦ Connection/stream interaction is the only complex threading scenario, that can cause race conditions • Grpc ◦ We do not need to use Netty grpc, so we can now serve grpc on same port as other protocols!
  55. Good to know Copyright © 2023, Oracle and/or its affiliates

    72 When virtual thread cannot unmount from carrier • OS level limitations (some file systems ops) • JDK limitations (Object.wait()) These operations cannot unmount from the carrier thread. JVM compensates by temporary increasing number of threads in the ForkJoinPool.
  56. Good to know Copyright © 2023, Oracle and/or its affiliates

    73 When virtual thread is pinned to carrier • Synchronized blocks • Native methods and foreign functions These operations may hinder application scalability, as the scheduler does not compensate by expanding its parallelism. These operations should be guarded using Reentrant Lock and other constructs from java.util.concurrent instead of synchronized
  57. Lessons learned Copyright © 2023, Oracle and/or its affiliates 74

    • “unlearn” asynchronous programming • Everything is easier to write, easier to read, easier to maintain, easier to debug • We will need to wait for a few libraries to catch up (Connection pools, messaging etc.) • Refactor synchronized to java.util.concurrent locking • Think about thread locals (ScopedValues) • Virtual Threads are the best improvement in Java (EVER so far)
  58. Keep an eye on Copyright © 2023, Oracle and/or its

    affiliates 75 • JEP-428: Structured Concurrency ◦ Preview in Java 19 • JEP-429: Scoped Values ◦ Preview in Java 20 • JEP-444: Virtual threads in Java 21 ◦ Not experimental feature
  59. Copyright © 2023, Oracle and/or its affiliates Or 77 HIGH

    PERFORMANCE + COMPLEXITY SIMPLICITY + AVERAGE PERFORMANCE Remember this slide?
  60. Copyright © 2023, Oracle and/or its affiliates 78 SIMPLICITY +

    HIGH PERFORMANCE Now you can have both:
  61. The main “take away” Copyright © 2023, Oracle and/or its

    affiliates 80 Complexity Performance Virtual threads and Helidon Níma:
  62. Helidon Copyright © 2023, Oracle and/or its affiliates Demo 87

    https://apexapps.oracle.com/pls/apex/r/db pm/livelabs/view-workshop?wid=3353&cle ar=RR,180&session=14645993487863 https://otube.oracle.com/media/First%20Helido n%20%D0%90pplication/1_g44nb71y
  63. Copyright © 2023, Oracle and/or its affiliates Helidon Support 88

    • Java 11 • Netty • Helidon CLI • JEP-290 configuration • MicroProfile 3.3 • javax.* packages Helidon 3.2.1 Helidon 4 (Alpha-6) Helidon 2.6.1 • Java 17 • Netty • helidon.io/starter • JEP-290 enforced • MicroProfile 5.0 • jakarta.* packages • Java 21 • Níma • Virtual Threads • MicroProfile 6
  64. • Open source Support ◦ GitHub issues ◦ Slack ◦

    StackOverFlow • Commercial Support through Oracle Support of WLS, Coherence and Verrazzano Copyright © 2023, Oracle and/or its affiliates Helidon Support 89