JVM Tuning in a rush

JVM Tuning in a rush

When performance hits rock-bottom everybody (and their dog) is called upon and all of a sudden developers should have been responsible for last half a year or so and code with performance in mind (and deadlines, but that of course goes unsaid). So, here I'm talking about what can a dev do to meet those unreasonable demands and what might he do anticipating them.

Strictly JVM, mostly Sun Hotspot impl, but number of points can be used to other JVMs as well.

With more time, I usually show also tools available both on OS and JDK side.

E72ba25d250c41a7d0716953f8ec3f4b?s=128

Tomasz Borek

July 28, 2015
Tweet

Transcript

  1. JVM Tuning in a rush Tomasz Borek, Symentis

  2. • Infected in childhood • From me parents • Amstrad,

    ElWro Junior • Games! Doh! • Mem pages in DOS anyone? • In IT • ETL, crawlers, web-app, archi • IAAS, SAAS, own servers • Java 4 – 7, GNU/Linux, SQLs • Ardent “activist” Tomasz Borek
  3. Can be found in the net! :P https://lafkblogs.wordpress.com/ https://twitter.com/lafk_pl https://pl.linkedin.com/in/tjborek

    GeeCON, ChamberConf, Confitura, Lambda Days, LambdaCon, Java Developer Days, JavaDay Lviv, JavaDay Kiev
  4. Tomek in IT groups .com http://java.pl Kraków Poznań Praga Sopot

    http://geecon.org
  5. Prod hits rock bottom • Everybody is kung-fu fighting •

    Must be quick like lightning
  6. So! Being a dev, you need: 1.To know your traffic

    and your app and your infra
  7. So! Being a dev, you need: 1.To know your traffic

    and your app and your infra 2.To know recent changes
  8. So! Being a dev, you need: 1.To know your traffic

    and your app and your infra 2.To know recent changes 3.To know your limits (sometimes comes from point 1)
  9. So! Being a dev, you need: 1.To know your traffic

    and your app and your infra 2.To know recent changes 3.To know your limits (sometimes comes from point 1) 4.To have access to monitoring
  10. So! Being a dev, you need: 1.To know your traffic

    and your app and your infra 2.To know recent changes 3.To know your limits (sometimes comes from point 1) 4.To have access to monitoring 5.To adjust pooling (both thread and connection)
  11. So! Being a dev, you need: 1.To know your traffic

    and your app and your infra 2.To know recent changes 3.To know your limits (sometimes comes from point 1) 4.To have access to monitoring 5.To adjust pooling (both thread and connection) 6.To peruse logs and pull out anomalies
  12. So! Being a dev, you need: 1.To know your traffic

    and your app and your infra 2.To know recent changes 3.To know your limits (sometimes comes from point 1) 4.To have access to monitoring 5.To adjust pooling (both thread and connection) 6.To peruse logs and pull out anomalies 7.TO TUNE JVM
  13. So! Being a dev, you need: 1.To know your traffic

    and your app and your infra 2.To know recent changes 3.To know your limits (sometimes comes from point 1) 4.To have access to monitoring 5.To adjust pooling (both thread and connection) 6.To peruse logs and pull out anomalies 7.TO TUNE JVM
  14. JVM? Meaning? • Sun Hotspot • IBM • JRockit •

    IceTea • OpenJDK • TCK decides
  15. JVM? Meaning? • Sun Hotspot • IBM • JRockit •

    IceTea • OpenJDK • TCK decides
  16. JVM? Meaning? • Sun Hotspot – but which version? •

    IBM • JRockit • IceTea • OpenJDK • TCK decides
  17. JVM? Meaning? • Sun Hotspot – but which version? And

    architecture? • IBM • JRockit • IceTea • OpenJDK • TCK decides
  18. So! JVM tuning? 1.To adjust GC, based on it's logs

  19. So! JVM tuning? 1.To adjust GC, based on it's logs

    2.To adjust (native) heap
  20. So! JVM tuning? 1.To adjust GC, based on it's logs

    2.To adjust (native) heap 3.Tweaking JVM starting flags
  21. So! JVM tuning? 1.To adjust GC, based on it's logs

    2.To adjust (native) heap 3.Tweaking JVM starting flags 4.To use TOOLS
  22. Today! • JVM tuning • Diagnosing performance problems • Tools

    • All in a rush
  23. JVM tuning

  24. Takeaway #1 JVM is a process

  25. Being a process means • OS architecture enforces JVM architecture

    • Your memory is limited by OS and other processes – Heap is one – C-heap is another! • IO-wise and thread-wise: – Your threads and files opened are limited by ulimit • File descriptors!
  26. OOM flavours?

  27. OOM flavours • Out of HeapSpace

  28. OOM flavours • Out of HeapSpace • PermGen error

  29. OOM flavours • Out of HeapSpace • PermGen error •

    Unable to create native thread
  30. OOM flavours • Out of HeapSpace • PermGen error •

    Unable to create native thread • There are others, these are most popular
  31. API says OutOfMemoryError is • Thrown when the Java Virtual

    Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector. OutOfMemoryError objects may be constructed by the virtual machine as if suppression were disabled and/or the stack trace was not writable.
  32. API says OutOfMemoryError is • Thrown when the Java Virtual

    Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector. OutOfMemoryError objects may be constructed by the virtual machine as if suppression were disabled and/or the stack trace was not writable.
  33. API says OutOfMemoryError is • Due to failure in allocating

    an object after garbage collection. • Nothing about threads. • Nothing about PermGen.
  34. How much memory JVM uses? • Say you start it

    with Xmx == Xms == 1GB
  35. How much memory JVM uses? • Say you start it

    with Xmx == Xms == 1GB • 1GB?
  36. How much memory JVM uses? • Say you start it

    with Xmx == Xms == 1GB • 1GB? • Around 1GB?
  37. How much memory JVM uses? • Say you start it

    with Xmx == Xms == 1GB • 1GB? • Around 1GB? • If we talk about Java HEAP
  38. Memory management Entire RAM • 32 bits: can address less

    than 4GB. • 64 bits: can address… theoretically 16 EB. – Still – much more! • There's much more to 32vs64 but that's for another time
  39. Memory management Entire RAM What's left of RAM – C-HEAP

    JVM heap -Xmx • -Xms – minimal heap size • -Xmx – maximal heap size • When both should be set to same value?
  40. Memory management Entire RAM What's left of RAM – C-HEAP

    JVM heap What's left – C-HEAP JVM ops JVM heap -Xmx PermGen
  41. PermGen • Permanent Generation – Early Java – for JDK

    classes – and Strings • Back then – good idea • Now removed – JDK 7 – Strings no longer in PermGen – JDK 8 – no PermGen at all
  42. PermGen • Permanent Generation – Early Java – for JDK

    classes – and Strings • Back then – good idea • Now removed – JDK 7 – Strings no longer in PermGen – JDK 8 – no PermGen at all Quick fix? Increase PermGen size.
  43. Memory management What's left – C-HEAP JVM ops JVM heap

    -Xmx PermGen
  44. GC • When it runs?

  45. GC • When it runs? • Minor collection?

  46. GC • When it runs? • Minor collection? • Major

    collection?
  47. GC • When it runs? • Minor collection? • Major

    collection? • How can a dev tune GC?
  48. Generational GC Tenured – where long-lived are promoted Eden –

    infants that die quickly • Studies showed: – most objects die young – Some live really long • Ergo: short- and long-lived division • Minor collection: Eden • Major collection: whole heap
  49. Generational GC - mechanics • In fact Eden has also

    2 Survivor spaces – To handle locality – Helps compress after freeing space – First promotion to survivor, then to tenured • Flags tell: – How many GCs object must survive to be promoted – How large Eden / Tenured / Survivors spaces are – What is logged (how details GC logs are) – Many, MANY more
  50. Memory management trade-offs • Large heap – large full GC

    – small native and C-heap? • Smaller heap – minor GC may be enough • Make sure your objects die young and only really long lived reach tenured
  51. JVM tuning 1.To adjust GC, based on it's logs 2.To

    adjust (native) heap 3.Tweaking JVM starting flags 4.To use TOOLS – later
  52. Takeaway #2 Log GC

  53. Takeaway #3 GC tuning is a trade-off

  54. Diagnosing performance problems

  55. „The Box” • Heinz Kabutz, Kirk Pepperdine • Top -

    bottom
  56. „The Box” • TRAFFIC: ? TRAFFIC

  57. „The Box” • TRAFFIC: how is app used? TRAFFIC: people,

    automatic
  58. „The Box” • TRAFFIC: how is app used? • CODE:

    ? TRAFFIC: people, automatic Application CODE
  59. „The Box” • TRAFFIC: how is app used? • CODE:

    threads, algos, data structures... TRAFFIC: people, automatic CODE: threads, data structs, algo
  60. „The Box” • TRAFFIC: how is app used? • CODE:

    threads, algos, data structures... • JVM: ? TRAFFIC: people, automatic CODE: threads, data structs, algo JVM
  61. „The Box” • TRAFFIC: how is app used? • CODE:

    threads, algos, data structures... • JVM: ? TRAFFIC: people, automatic CODE: threads, data structs, algo JVM: flags, GC
  62. „The Box” • TRAFFIC: how is app used? • CODE:

    threads, algos, data structures... • JVM: starting flags, GC • OS: ? TRAFFIC: people, automatic CODE: threads, data structs, algo JVM: flags, GC OS
  63. „The Box” • TRAFFIC: how is app used? • CODE:

    threads, algos, data structures... • JVM: starting flags, GC • OS: ulimit, FS, config, archi, other procs… TRAFFIC: people, automatic CODE: threads, data structs, algo JVM: flags, GC OS: FS, config, limits
  64. „The Box” • TRAFFIC: how is app used? • CODE:

    threads, algos, data structures... • JVM: starting flags, GC • OS: ulimit, FS, config, archi, other procs… • VIRT: ? TRAFFIC: people, automatic CODE: threads, data structs, algo JVM: flags, GC OS: FS, config, limits VIRT
  65. „The Box” • TRAFFIC: how is app used? • CODE:

    threads, algos, data structures... • JVM: starting flags, GC • OS: ulimit, FS, config, archi, other procs… • VIRT: hell depends! TRAFFIC: people, automatic CODE: threads, data structs, algo JVM: flags, GC OS: FS, config, limits VIRT
  66. „The Box” • TRAFFIC: how is app used? • CODE:

    threads, algos, data structures... • JVM: starting flags, GC • OS: ulimit, FS, config, archi, other procs… • VIRT: hell depends! • HARDWARE: ? TRAFFIC: people, automatic CODE: threads, data structs, algo JVM: flags, GC OS: FS, config, limits VIRT HARDWARE
  67. „The Box” • TRAFFIC: how is app used? • CODE:

    threads, algos, data structures... • JVM: starting flags, GC • OS: ulimit, FS, config, archi, other procs… • VIRT: hell depends! • HARDWARE: 32 vs 64, RAM, BIOS, drivers… TRAFFIC: people, automatic CODE: threads, data structs, algo JVM: flags, GC OS: FS, config, limits VIRT HARDWARE
  68. Brendan Gregg – USE • Utilization, Saturation, Errors • Use

    – how much in use is your resource • Saturation – how many requests await (waiting queue) • Errors – what errors are thrown by resource • http://www.brendangregg.com/usemethod.html • Create a checklist for each important resource, for finding out Utilization, Saturation and Errors and you'll know how to soon find out what is going on with resource
  69. Takeaway #1 The Box

  70. Takeaway #2 GNU/Linux perf? Brendan Gregg

  71. TOOLS

  72. None
  73. None
  74. Takeaway #1 GNU/Linux surely has a tool for that

  75. How to find your java process? • ps … |

    grep java •
  76. How to find your java process? • ps … |

    grep java • pgrep java
  77. How to find your java process? • ps … |

    grep java • pgrep java • jps
  78. How to change flags on-the-fly? • jinfo

  79. How to dump threads or heap? • kill -3

  80. How to dump threads or heap? • kill -3 •

    jstack
  81. How to dump threads or heap? • kill -3 •

    jstack • jhat (heap)
  82. jvisualvm • Deserves it's own slide • Profiler, sampler •

    Monitor (heap, threads, etc.) • Calls GC, does dumps
  83. SUMMARIZING • For being rushed – prepare in advance –

    Monitoring, logs, ceilings, etc. • Log GC • JVM is a process – all process limits in your OS apply • The Box • Brendan Gregg • GNU/Linux – tools unparalleled
  84. ?