JVM Tuning in a rush

When performance hits rock-bottom everybody (and their dog) is called upon and all of a sudden developers should have been responsible for last half a year or so and code with performance in mind (and deadlines, but that of course goes unsaid). So, here I'm talking about what can a dev do to meet those unreasonable demands and what might he do anticipating them.

Strictly JVM, mostly Sun Hotspot impl, but number of points can be used to other JVMs as well.

With more time, I usually show also tools available both on OS and JDK side.

Transcript

1. JVM Tuning in a rush
   Tomasz Borek, Symentis
   

   View Slide

2. • Infected in childhood
   • From me parents
   • Amstrad, ElWro Junior
   • Games! Doh!
   • Mem pages in DOS anyone?
   • In IT
   • ETL, crawlers, web-app, archi
   • IAAS, SAAS, own servers
   • Java 4 – 7, GNU/Linux, SQLs
   • Ardent “activist”
   Tomasz Borek
   

   View Slide

3. Can be found in the net! :P
   https://lafkblogs.wordpress.com/
   https://twitter.com/lafk_pl
   https://pl.linkedin.com/in/tjborek
   GeeCON, ChamberConf, Confitura, Lambda Days, LambdaCon,
   Java Developer Days, JavaDay Lviv, JavaDay Kiev
   

   View Slide

4. Tomek in IT groups
   .com
   http://java.pl Kraków
   Poznań
   Praga
   Sopot
   http://geecon.org
   

   View Slide

5. Prod hits rock bottom
   ●
   Everybody is kung-fu fighting
   ●
   Must be quick like lightning
   

   View Slide

6. So! Being a dev, you need:
   1.To know your traffic and your app and your infra
   

   View Slide

7. So! Being a dev, you need:
   1.To know your traffic and your app and your infra
   2.To know recent changes
   

   View Slide

8. So! Being a dev, you need:
   1.To know your traffic and your app and your infra
   2.To know recent changes
   3.To know your limits (sometimes comes from point 1)
   

   View Slide

9. So! Being a dev, you need:
   1.To know your traffic and your app and your infra
   2.To know recent changes
   3.To know your limits (sometimes comes from point 1)
   4.To have access to monitoring
   

   View Slide

10. So! Being a dev, you need:
    1.To know your traffic and your app and your infra
    2.To know recent changes
    3.To know your limits (sometimes comes from point 1)
    4.To have access to monitoring
    5.To adjust pooling (both thread and connection)
    

    View Slide

11. So! Being a dev, you need:
    1.To know your traffic and your app and your infra
    2.To know recent changes
    3.To know your limits (sometimes comes from point 1)
    4.To have access to monitoring
    5.To adjust pooling (both thread and connection)
    6.To peruse logs and pull out anomalies
    

    View Slide

12. So! Being a dev, you need:
    1.To know your traffic and your app and your infra
    2.To know recent changes
    3.To know your limits (sometimes comes from point 1)
    4.To have access to monitoring
    5.To adjust pooling (both thread and connection)
    6.To peruse logs and pull out anomalies
    7.TO TUNE JVM
    

    View Slide

13. So! Being a dev, you need:
    1.To know your traffic and your app and your infra
    2.To know recent changes
    3.To know your limits (sometimes comes from point 1)
    4.To have access to monitoring
    5.To adjust pooling (both thread and connection)
    6.To peruse logs and pull out anomalies
    7.TO TUNE JVM
    

    View Slide

14. JVM? Meaning?
    • Sun Hotspot
    • IBM
    • JRockit
    • IceTea
    • OpenJDK
    • TCK decides
    

    View Slide

15. JVM? Meaning?
    • Sun Hotspot
    • IBM
    • JRockit
    • IceTea
    • OpenJDK
    • TCK decides
    

    View Slide

16. JVM? Meaning?
    • Sun Hotspot – but which version?
    • IBM
    • JRockit
    • IceTea
    • OpenJDK
    • TCK decides
    

    View Slide

17. JVM? Meaning?
    • Sun Hotspot – but which version? And architecture?
    • IBM
    • JRockit
    • IceTea
    • OpenJDK
    • TCK decides
    

    View Slide

18. So! JVM tuning?
    1.To adjust GC, based on it's logs
    

    View Slide

19. So! JVM tuning?
    1.To adjust GC, based on it's logs
    2.To adjust (native) heap
    

    View Slide

20. So! JVM tuning?
    1.To adjust GC, based on it's logs
    2.To adjust (native) heap
    3.Tweaking JVM starting flags
    

    View Slide

21. So! JVM tuning?
    1.To adjust GC, based on it's logs
    2.To adjust (native) heap
    3.Tweaking JVM starting flags
    4.To use TOOLS
    

    View Slide

22. Today!
    ●
    JVM tuning
    ●
    Diagnosing performance problems
    ●
    Tools
    ●
    All in a rush
    

    View Slide

23. JVM tuning
    

    View Slide

24. Takeaway #1
    JVM is a process
    

    View Slide

25. Being a process means
    ●
    OS architecture enforces JVM architecture
    ●
    Your memory is limited by OS and other processes
    – Heap is one
    – C-heap is another!
    ●
    IO-wise and thread-wise:
    – Your threads and files opened are limited by ulimit
    ●
    File descriptors!
    

    View Slide

26. OOM flavours?
    

    View Slide

27. OOM flavours
    ●
    Out of HeapSpace
    

    View Slide

28. OOM flavours
    ●
    Out of HeapSpace
    ●
    PermGen error
    

    View Slide

29. OOM flavours
    ●
    Out of HeapSpace
    ●
    PermGen error
    ●
    Unable to create native thread
    

    View Slide

30. OOM flavours
    ●
    Out of HeapSpace
    ●
    PermGen error
    ●
    Unable to create native thread
    ●
    There are others, these are most popular
    

    View Slide

31. API says OutOfMemoryError is
    ●
    Thrown when the Java Virtual Machine cannot
    allocate an object because it is out of memory, and
    no more memory could be made available by the
    garbage collector. OutOfMemoryError objects may
    be constructed by the virtual machine as if
    suppression were disabled and/or the stack trace
    was not writable.
    

    View Slide

32. API says OutOfMemoryError is
    ●
    Thrown when the Java Virtual Machine cannot
    allocate an object because it is out of memory,
    and no more memory could be made available
    by the garbage collector. OutOfMemoryError
    objects may be constructed by the virtual machine
    as if suppression were disabled and/or the stack
    trace was not writable.
    

    View Slide

33. API says OutOfMemoryError is
    ●
    Due to failure in allocating an object after garbage
    collection.
    ●
    Nothing about threads.
    ●
    Nothing about PermGen.
    

    View Slide

34. How much memory JVM uses?
    ●
    Say you start it with Xmx == Xms == 1GB
    

    View Slide

35. How much memory JVM uses?
    ●
    Say you start it with Xmx == Xms == 1GB
    ●
    1GB?
    

    View Slide

36. How much memory JVM uses?
    ●
    Say you start it with Xmx == Xms == 1GB
    ●
    1GB?
    ●
    Around 1GB?
    

    View Slide

37. How much memory JVM uses?
    ●
    Say you start it with Xmx == Xms == 1GB
    ●
    1GB?
    ●
    Around 1GB?
    ●
    If we talk about Java HEAP
    

    View Slide

38. Memory management
    Entire RAM
    ●
    32 bits: can address less than 4GB.
    ●
    64 bits: can address… theoretically 16 EB.
    – Still – much more!
    ●
    There's much more to 32vs64 but that's for another time
    

    View Slide

39. Memory management
    Entire RAM
    What's left of RAM – C-HEAP
    JVM heap
    -Xmx
    ●
    -Xms – minimal heap size
    ●
    -Xmx – maximal heap size
    ●
    When both should be set to same value?
    

    View Slide

40. Memory management
    Entire RAM
    What's left of RAM – C-HEAP
    JVM heap
    What's left – C-HEAP
    JVM ops
    JVM heap
    -Xmx
    PermGen
    

    View Slide

41. PermGen
    ●
    Permanent Generation
    – Early Java
    – for JDK classes
    – and Strings
    ●
    Back then – good idea
    ●
    Now removed
    – JDK 7 – Strings no longer in PermGen
    – JDK 8 – no PermGen at all
    

    View Slide

42. PermGen
    ●
    Permanent Generation
    – Early Java
    – for JDK classes
    – and Strings
    ●
    Back then – good idea
    ●
    Now removed
    – JDK 7 – Strings no longer in PermGen
    – JDK 8 – no PermGen at all
    Quick fix?
    Increase PermGen size.
    

    View Slide

43. Memory management
    What's left – C-HEAP
    JVM ops
    JVM heap
    -Xmx
    PermGen
    

    View Slide

44. GC
    ●
    When it runs?
    

    View Slide

45. GC
    ●
    When it runs?
    ●
    Minor collection?
    

    View Slide

46. GC
    ●
    When it runs?
    ●
    Minor collection?
    ●
    Major collection?
    

    View Slide

47. GC
    ●
    When it runs?
    ●
    Minor collection?
    ●
    Major collection?
    ●
    How can a dev tune GC?
    

    View Slide

48. Generational GC
    Tenured – where long-lived are promoted
    Eden – infants that die quickly
    ●
    Studies showed:
    – most objects die young
    – Some live really long
    ●
    Ergo: short- and long-lived division
    ●
    Minor collection: Eden
    ●
    Major collection: whole heap
    

    View Slide

49. Generational GC - mechanics
    ●
    In fact Eden has also 2 Survivor spaces
    – To handle locality
    – Helps compress after freeing space
    – First promotion to survivor, then to tenured
    ●
    Flags tell:
    – How many GCs object must survive to be promoted
    – How large Eden / Tenured / Survivors spaces are
    – What is logged (how details GC logs are)
    – Many, MANY more
    

    View Slide

50. Memory management trade-offs
    ●
    Large heap – large full GC – small native and C-heap?
    ●
    Smaller heap – minor GC may be enough
    ●
    Make sure your objects die young and only really long
    lived reach tenured
    

    View Slide

51. JVM tuning
    1.To adjust GC, based on it's logs
    2.To adjust (native) heap
    3.Tweaking JVM starting flags
    4.To use TOOLS – later
    

    View Slide

52. Takeaway #2
    Log GC
    

    View Slide

53. Takeaway #3
    GC tuning is a trade-off
    

    View Slide

54. Diagnosing performance problems
    

    View Slide

55. „The Box”
    • Heinz Kabutz, Kirk Pepperdine
    • Top - bottom
    

    View Slide

56. „The Box”
    • TRAFFIC: ?
    TRAFFIC
    

    View Slide

57. „The Box”
    • TRAFFIC: how is app used?
    TRAFFIC: people,
    automatic
    

    View Slide

58. „The Box”
    • TRAFFIC: how is app used?
    • CODE: ?
    TRAFFIC: people,
    automatic
    Application CODE
    

    View Slide

59. „The Box”
    • TRAFFIC: how is app used?
    • CODE: threads, algos, data structures...
    TRAFFIC: people,
    automatic
    CODE: threads, data
    structs, algo
    

    View Slide

60. „The Box”
    • TRAFFIC: how is app used?
    • CODE: threads, algos, data structures...
    • JVM: ?
    TRAFFIC: people,
    automatic
    CODE: threads, data
    structs, algo
    JVM
    

    View Slide

61. „The Box”
    • TRAFFIC: how is app used?
    • CODE: threads, algos, data structures...
    • JVM: ?
    TRAFFIC: people,
    automatic
    CODE: threads, data
    structs, algo
    JVM: flags, GC
    

    View Slide

62. „The Box”
    • TRAFFIC: how is app used?
    • CODE: threads, algos, data structures...
    • JVM: starting flags, GC
    • OS: ?
    TRAFFIC: people,
    automatic
    CODE: threads, data
    structs, algo
    JVM: flags, GC
    OS
    

    View Slide

63. „The Box”
    • TRAFFIC: how is app used?
    • CODE: threads, algos, data structures...
    • JVM: starting flags, GC
    • OS: ulimit, FS, config, archi, other procs…
    TRAFFIC: people,
    automatic
    CODE: threads, data
    structs, algo
    JVM: flags, GC
    OS: FS, config, limits
    

    View Slide

64. „The Box”
    • TRAFFIC: how is app used?
    • CODE: threads, algos, data structures...
    • JVM: starting flags, GC
    • OS: ulimit, FS, config, archi, other procs…
    • VIRT: ?
    TRAFFIC: people,
    automatic
    CODE: threads, data
    structs, algo
    JVM: flags, GC
    OS: FS, config, limits
    VIRT
    

    View Slide

65. „The Box”
    • TRAFFIC: how is app used?
    • CODE: threads, algos, data structures...
    • JVM: starting flags, GC
    • OS: ulimit, FS, config, archi, other procs…
    • VIRT: hell depends!
    TRAFFIC: people,
    automatic
    CODE: threads, data
    structs, algo
    JVM: flags, GC
    OS: FS, config, limits
    VIRT
    

    View Slide

66. „The Box”
    • TRAFFIC: how is app used?
    • CODE: threads, algos, data structures...
    • JVM: starting flags, GC
    • OS: ulimit, FS, config, archi, other procs…
    • VIRT: hell depends!
    • HARDWARE: ?
    TRAFFIC: people,
    automatic
    CODE: threads, data
    structs, algo
    JVM: flags, GC
    OS: FS, config, limits
    VIRT
    HARDWARE
    

    View Slide

67. „The Box”
    • TRAFFIC: how is app used?
    • CODE: threads, algos, data structures...
    • JVM: starting flags, GC
    • OS: ulimit, FS, config, archi, other procs…
    • VIRT: hell depends!
    • HARDWARE: 32 vs 64, RAM, BIOS, drivers…
    TRAFFIC: people,
    automatic
    CODE: threads, data
    structs, algo
    JVM: flags, GC
    OS: FS, config, limits
    VIRT
    HARDWARE
    

    View Slide

68. Brendan Gregg – USE
    • Utilization, Saturation, Errors
    • Use – how much in use is your resource
    • Saturation – how many requests await (waiting queue)
    • Errors – what errors are thrown by resource
    • http://www.brendangregg.com/usemethod.html
    • Create a checklist for each important resource, for
    finding out Utilization, Saturation and Errors and you'll
    know how to soon find out what is going on with
    resource
    

    View Slide

69. Takeaway #1
    The Box
    

    View Slide

70. Takeaway #2
    GNU/Linux perf? Brendan Gregg
    

    View Slide

71. TOOLS
    

    View Slide

72. View Slide

73. View Slide

74. Takeaway #1
    GNU/Linux surely has a tool for that
    

    View Slide

75. How to find your java process?
    ●
    ps … | grep java
    ●
    

    View Slide

76. How to find your java process?
    ●
    ps … | grep java
    ●
    pgrep java
    

    View Slide

77. How to find your java process?
    ●
    ps … | grep java
    ●
    pgrep java
    ●
    jps
    

    View Slide

78. How to change flags on-the-fly?
    ●
    jinfo
    

    View Slide

79. How to dump threads or heap?
    ●
    kill -3
    

    View Slide

80. How to dump threads or heap?
    ●
    kill -3
    ●
    jstack
    

    View Slide

81. How to dump threads or heap?
    ●
    kill -3
    ●
    jstack
    ●
    jhat (heap)
    

    View Slide

82. jvisualvm
    ●
    Deserves it's own slide
    ●
    Profiler, sampler
    ●
    Monitor (heap, threads, etc.)
    ●
    Calls GC, does dumps
    

    View Slide

83. SUMMARIZING
    ●
    For being rushed – prepare in advance
    – Monitoring, logs, ceilings, etc.
    ●
    Log GC
    ●
    JVM is a process – all process limits in your OS apply
    ●
    The Box
    ●
    Brendan Gregg
    ●
    GNU/Linux – tools unparalleled
    

    View Slide

84. ?
    

    View Slide