Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)

E16bc9c356b65d61ee1d74c8f06ae35b?s=47 Colin Jones
October 29, 2015
Programming
1
630

Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)

A session from JavaOne 2015

E16bc9c356b65d61ee1d74c8f06ae35b?s=128

Colin Jones

October 29, 2015
Tweet

More Decks by Colin Jones

See All by Colin Jones
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
0
98
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
1
1.1k
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
0
330
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
2
220
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
3
450
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
0
97
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
1
200
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
0
99
E16bc9c356b65d61ee1d74c8f06ae35b?s=48 trptcolin
2
140

Other Decks in Programming

See All in Programming
091c8449decb2f3549c695b5f903865a?s=48 sh_akira
0
220
9bf923e39671cde83584e3e926296c13?s=48 kishikawakatsumi
1
530
15bd11f2c2c5e3dd854153d03a102b0d?s=48 standfm
0
380
1d5ca7d8f17f15de4272e1ec7ef7abac?s=48 watilde
0
780
Df4978f14401325586e9e286b140ac4c?s=48 n1215
2
330
15934fa2aa7b2ce21f091e9b7cffa856?s=48 manfredsteyer
PRO
0
190
E7151ab8219e76672f7a7691dd2c88e6?s=48 seike460
3
470
E0a89ea965a4ac4b5f091d6bd9e8d8ea?s=48 takahiro_komatsu1982
0
120
64e95d2d8051ee58501f21987f46b9e7?s=48 chck
5
2.3k
922f51d458d6ffee8578b9d3ab0a52b6?s=48 ug
0
520
924d7de2a5a7102a597728e5edbbff78?s=48 kekyo
PRO
1
130
2dea59f12b07aa47a8604dc21ba27036?s=48 izm
0
470

Featured

See All Featured
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
33
3.6k
A16eb159db895e3b01d3dc95767ad595?s=48 jonyablonski
30
1.5k
E195ae45320d9202eaa01c9f1d31a416?s=48 marktimemedia
10
550
15f2b8221f247985a27a627d2ece72f0?s=48 pauljervisheath
194
15k
24fc194843a71f10949be18d5a692682?s=48 gr2m
85
11k
48c098b6579220a67a6ba949be6e4b5a?s=48 revolveconf
202
9.8k
D8fc2580cfaca035f666d9e4ee79a7f7?s=48 mongodb
23
4k
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
167
7.5k
91cefdf141106fa10674aae74ce05890?s=48 yeseniaperezcruz
303
32k
B6a8f005f39d23ffc930508ac9da68b9?s=48 vanstee
118
4.9k
465724d73fe3a92c0879fdfb43a3a6f3?s=48 philhawksworth
192
17k
5b6a2bd86ef643786a381a212e0ef2f1?s=48 geoffreycrofte
31
1.2k

Transcript

  1. Beyond top(1) Command-Line Monitoring on the JVM Colin Jones @trptcolin

    8th Light
  2. None

  3. What to expect

  4. command-line tooling

  5. on the JVM

  6. introspection & serviceability

  7. --all-flags=false

  8. war stories

  9. real-life usage (well, re-enacted anyway)

  10. A long time ago in a software shop far, far

    away…

  11. Things are going pretty well

  12. What does this thing look like? app-architecture Postgres Web /

    API Application Server Load Balancer Periodic Job Application Server 3rd-party Service A 3rd-party Service B Monitored email account End users: native mobile app Admin users: desktop browsers

  13. But strange things are afoot

  14. the server sometimes gets really slow

  15. the team has to manually restart the application server

  16. incident response time is ~5 minutes

  17. Yes, strange things are afoot

  18. Pain, frustration, anger

  19. Just the facts

  20. sometimes, things get slow

  21. all requests seem to be affected

  22. the JVM stays up

  23. restart the JVM and everything is fine

  24. What could it be?

  25. Demo

  26. More facts, please!

  27. constant full GCs

  28. what’s in the heap

  29. what application code was running

  30. The right tools for the job

  31. vmstat system-level: CPU, memory, disk, context switching

  32. top per-process: CPU & memory

  33. jps what’s our PID?

  34. jstack status of all threads (right now-ish!)

  35. jcmd what can’t it do?! jcmd [PID] help (sorry, JVM

    6 users: see jinfo/jmap/jstack)

  36. jstat GC classloader compiler

  37. Mystery solved!

  38. Now “just” fix it

  39. idea 1: eliminate the leak

  40. idea 2: eliminate the cache altogether?

  41. idea 3: delete the feature

  42. idea 4: full-text search engine

  43. So we’re good now… until the next incident

  44. Lessons

  45. “it’s slow” could mean lots of things

  46. “high CPU” could mean lots of things

  47. collecting data is crucial in a crisis

  48. reproducing the issue helps me sleep at night

  49. Other “right tools for the job”

  50. Heap analyzers

  51. Profilers

  52. Constant monitoring & alerting

  53. Dynamic tracing

  54. Learning more

  55. Books

  56. operators are standing by! man jstat man jstack jcmd [PID]

    help [COMMAND] etc.

  57. Thank you! Colin Jones @trptcolin 8th Light