Pro Yearly is on sale from $80 to $50! »

Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)

E16bc9c356b65d61ee1d74c8f06ae35b?s=47 Colin Jones
October 29, 2015
Programming
1
630

Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)

A session from JavaOne 2015

E16bc9c356b65d61ee1d74c8f06ae35b?s=128

Colin Jones

October 29, 2015
Tweet

More Decks by Colin Jones

See All by Colin Jones
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
0
96
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
1
1.1k
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
0
320
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
2
220
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
3
450
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
0
86
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
1
190
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
0
99
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
2
140

Other Decks in Programming

See All in Programming
8cc27471bcdeb2858fca212586a208b4?s=48 takahia1988
0
370
B337bccaf042748ab81c81ea623bad40?s=48 regtan
0
180
72a2082c6a4dd79ad68befb3db911616?s=48 mraible
PRO
1
340
15934fa2aa7b2ce21f091e9b7cffa856?s=48 manfredsteyer
PRO
0
200
7ea0eec719c20e8d7880bfdb35f78b4e?s=48 alcaeus
1
110
Acd2f6f7498ea41881e161e191aa7c02?s=48 yattom
3
450
2ddd3d3e0d92ad45884bdcbef3a514a8?s=48 chiaoteni
2
350
37b01da9150c1e789f35771b06d36890?s=48 morimorihoge
4
730
B58b002245bdcc345dac78d426e6332b?s=48 junki555
4
1k
4578d99b560b2a470e05288ef6766ac2?s=48 paulk
0
670
73b0a25ed03f649ec63ded82ce4ac7cb?s=48 nunulk
0
140
C0e29dade6ec33081689ac135a269b45?s=48 crgarridos
1
150

Featured

See All Featured
168ccec72eee0530b818d44f3fedaacf?s=48 shlominoach
175
7.3k
C620790ae5bf5b50c245b2e0ef95f338?s=48 deanohume
295
27k
F29327647a9cff5c69618bae420792ea?s=48 tenderlove
49
3.3k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
168
7k
C44e1f7e22c3f23cff7bc130871047ef?s=48 eileencodes
113
25k
5b9fe87ec1faa67a4599782930f45ec9?s=48 sstephenson
144
12k
95d9f37115fbb0d13135d0f77132f856?s=48 morganepeng
92
13k
C7393b7ba7ec9c8890dd77d209fbb3c9?s=48 maltzj
500
36k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
498
130k
A0517b08532aec8ab2aae3f65d40ad86?s=48 hannesfritz
27
910
1b8ad785acdd1ce1c99914b1c2a4e10e?s=48 sugarenia
232
810k
Fde6c3a50ca518a909ef35c22c0ee0e4?s=48 destraynor
146
19k

Transcript

  1. Beyond top(1) Command-Line Monitoring on the JVM Colin Jones @trptcolin

    8th Light
  2. None

  3. What to expect

  4. command-line tooling

  5. on the JVM

  6. introspection & serviceability

  7. --all-flags=false

  8. war stories

  9. real-life usage (well, re-enacted anyway)

  10. A long time ago in a software shop far, far

    away…

  11. Things are going pretty well

  12. What does this thing look like? app-architecture Postgres Web /

    API Application Server Load Balancer Periodic Job Application Server 3rd-party Service A 3rd-party Service B Monitored email account End users: native mobile app Admin users: desktop browsers

  13. But strange things are afoot

  14. the server sometimes gets really slow

  15. the team has to manually restart the application server

  16. incident response time is ~5 minutes

  17. Yes, strange things are afoot

  18. Pain, frustration, anger

  19. Just the facts

  20. sometimes, things get slow

  21. all requests seem to be affected

  22. the JVM stays up

  23. restart the JVM and everything is fine

  24. What could it be?

  25. Demo

  26. More facts, please!

  27. constant full GCs

  28. what’s in the heap

  29. what application code was running

  30. The right tools for the job

  31. vmstat system-level: CPU, memory, disk, context switching

  32. top per-process: CPU & memory

  33. jps what’s our PID?

  34. jstack status of all threads (right now-ish!)

  35. jcmd what can’t it do?! jcmd [PID] help (sorry, JVM

    6 users: see jinfo/jmap/jstack)

  36. jstat GC classloader compiler

  37. Mystery solved!

  38. Now “just” fix it

  39. idea 1: eliminate the leak

  40. idea 2: eliminate the cache altogether?

  41. idea 3: delete the feature

  42. idea 4: full-text search engine

  43. So we’re good now… until the next incident

  44. Lessons

  45. “it’s slow” could mean lots of things

  46. “high CPU” could mean lots of things

  47. collecting data is crucial in a crisis

  48. reproducing the issue helps me sleep at night

  49. Other “right tools for the job”

  50. Heap analyzers

  51. Profilers

  52. Constant monitoring & alerting

  53. Dynamic tracing

  54. Learning more

  55. Books

  56. operators are standing by! man jstat man jstack jcmd [PID]

    help [COMMAND] etc.

  57. Thank you! Colin Jones @trptcolin 8th Light