Optimizing Java Applications for Serverless

This deck captures the different knobs to turn for a JVM application and the trade offs to make.

Full documentation here: https://cloud.google.com/run/docs/tips/java

Transcript

1. Optimizing Java apps for Serverless...
   

   View Slide

2. 2
   @saturnism @gcpcloud
   Mostly startup speed & memory usage
   

   View Slide

3. 3
   @saturnism @gcpcloud
   It's a trade off though…
   Faster startup != more efficient
   

   View Slide

4. 4
   @saturnism @gcpcloud
   JVM traditionally optimize for...
   Long Running, High throughput, High concurrency
   

   View Slide

5. 5
   @saturnism @gcpcloud
   So we…
   Cache, Connection Pool, JIT
   

   View Slide

6. 6
   @saturnism @gcpcloud
   Minimize Memory Usages
   Minimize JVM startup time
   Minimize Framework init time
   Minimize Application init time
   

   View Slide

7. Container Image
   

   View Slide

8. 8
   @saturnism @gcpcloud
   Use Jib
   Jib produces optimized container image for fast reads
   Trade off - almost none
   Some frameworks require additional configuration
   Works out of the box with Spring Boot though
   

   View Slide

9. JVM
   

   View Slide

10. 10
    @saturnism @gcpcloud
    Java -XX:+TieredCompilation -XX:TieredStopAtLevel=1
    Turn off optimization compiler
    Trade off - no optimization for hot paths
    Turn it off means no speed up...
    

    View Slide

11. 11
    @saturnism @gcpcloud
    java -noverify
    Don't verify classes
    Trade off - undetected malicious edits to bytecode
    JVM might crash
    Worse, you might get hacked
    

    View Slide

12. 12
    @saturnism @gcpcloud
    java -Xss256k
    Reduce thread stack size, and thus non-heap memory
    Trade off - StackOverflowError
    Need to find optimal value
    

    View Slide

13. 13
    @saturnism @gcpcloud
    Use OpenJDK 8u142 or above
    Container awareness
    Trade off - none, just do it! If you don't...
    See wrong # of CPUs, use more than you should
    See wrong amount of Memory, get OOMKilled
    

    View Slide

14. 14
    @saturnism @gcpcloud
    Dealing with OOMKilled
    Need to understand Memory usage
    Heap + Non-Heap =
    Heap + (Metaspace + Code Cache + Thread stack + ...)
    

    View Slide

15. 15
    @saturnism @gcpcloud
    Native Memory Tracking
    -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics
    (Doesn't work when set via JAVA_TOOL_OPTIONS - must be part of the argument)
    

    View Slide

16. 16
    @saturnism @gcpcloud
    Use Memory Calculator
    Non-heap usage estimated O(classes)
    https://github.com/cloudfoundry/java-buildpack-memory-calculator
    

    View Slide

17. 17
    @saturnism @gcpcloud
    Understand Memory Usage and Accounting
    JVM
    Heap
    Non-Heap
    tmpfs
    Write
    filesystem
    Read
    Memory Accounting
    - JVM
    - Heap
    - Non-Heap
    - Code Cache
    - Thread
    - GC
    - …
    - tmpfs (/tmp/)
    - Filesystem buffers/cache
    - What system reads - JVM, JDK, .so
    lib, ...
    

    View Slide

18. Threads
    

    View Slide

19. 19
    @saturnism @gcpcloud
    Reduce # of Threads
    Tomcat defaults to 10 min spare threads, 200 max threads
    More threads = more memory too (stack size, -Xss)
    Trade off - lower throughput/concurrency
    Cloud Run max at 80 concurrent reqs per instance anyways...
    

    View Slide

20. 20
    @saturnism @gcpcloud
    Pending Requests
    Most thread-per-connection apps queues up
    requests/connections that it cannot immediately serve.
    Understand what a single app instance is capable of
    Reduce this number
    

    View Slide

21. 21
    @saturnism @gcpcloud
    Use Reactive / Webflux
    Asynchronous, non-blocking, high concurrency
    Defaults to CPU cores * 2 threads
    Use less memory
    Trade off - if you block...
    Then your exhaust the few threads
    Worse, if new worker threads are created in unbounded pool...
    

    View Slide

22. 22
    @saturnism @gcpcloud
    Avoid Background Tasks
    Cloud Run throttles CPU if instances has no active requests
    Something to be aware of...
    What's running in the background?
    Scheduled tasks, @Timer, @Async, message receivers
    Metrics senders, trace senders, ...
    

    View Slide

23. 23
    @saturnism @gcpcloud
    Connection Pools
    Set Max Instances
    Cloud Run Max Instances x Connections per Instance < Max Connections Allowed
    Something to be aware of...
    Connection eviction may not happen if CPU throttled
    

    View Slide

24. Application / Framework
    

    View Slide

25. 25
    @saturnism @gcpcloud
    Use Spring Boot 2.2.x or up
    spring.main.lazy-initialization=true
    Trade off
    No tradeoffs. Spring Boot 2.2.x implemented a number of optimizations
    For < 2.2, consider disabling JMX, actuator, ...
    

    View Slide

26. 26
    @saturnism @gcpcloud
    Use 2-CPU
    gcloud run deploy … --cpu=2
    Trade off - Cost
    Spring Boot utilizes multiple threads for initialization
    Single CPU allocation may cause contention
    

    View Slide

27. 27
    @saturnism @gcpcloud
    Spring Boot Lazy Initialization
    spring.main.lazy-initialization=true
    Trade off - initialization on first request
    First request may be slower
    Use if there is a readiness check (GKE, App Engine)
    

    View Slide

28. 28
    @saturnism @gcpcloud
    Use Context Indexer
    
    org.springframework
    spring-context-indexer
    true
    
    Trade off - slower if few components
    May help in the case of deep component/bean graphs
    

    View Slide

29. 29
    @saturnism @gcpcloud
    Make sure Dev Tool is not in Production
    Trade off - no trade offs
    Spring Boot Dev Tool helps w/ local development, but is not
    needed in Production
    If using Jib, make sure this is not included
    

    View Slide

30. 30
    @saturnism @gcpcloud
    Application Lifecycle
    Trade off - no trade offs
    In App Engine, implement /_ah/health and /_ah/warmup
    Prevents app receiving requests prematurely
    

    View Slide

31. Native Image
    

    View Slide

32. 32
    @saturnism @gcpcloud
    To Graal or not to Graal?
    More accurately, to native image or not?
    

    View Slide

33. 33
    @saturnism @gcpcloud
    Trade Offs!
    Compile time
    GC
    JIT
    Throughput
    

    View Slide

34. 34
    @saturnism @gcpcloud
    Frameworks support for Native Image
    Quarkus
    Micronaut
    Spring Boot (Experimental)
    More than framework - libraries need to support them too
    

    View Slide

35. 35
    @saturnism @gcpcloud
    Additional Resources
    Cloud Run Java Optimization Guide https://cloud.google.com/run/docs/tips/java
    Spring Boot in a Container (Dave Syer) https://spring.io/blog/2018/11/08/spring-boot-in-a-container
    How Fast is Spring? (Dave Syer) https://spring.io/blog/2018/12/12/how-fast-is-spring
    OpenJDK and Containers (Christine Flood) https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers/
    Running Spring Boot App in GraalVM (Sebastian Deluze) https://www.youtube.com/watch?v=3eoAxphAUIg
    Spring to Kubernetes Faster and Easier (Ray Tsang) https://saturnism.me/talk/kubernetes-spring-java-best-practices/
    

    View Slide

36. Thanks!
    @saturnism | saturnism.me
    

    View Slide