Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)
Search
Colin Jones
October 29, 2015
Programming
1
650
Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)
A session from JavaOne 2015
Colin Jones
October 29, 2015
Tweet
Share
More Decks by Colin Jones
See All by Colin Jones
A Bug's Life: What if `select` is Broken After All?
trptcolin
0
130
Underestimated costs of microservice architectures
trptcolin
3
1.4k
FP vs. OOP: Beyond the Bikeshed
trptcolin
0
390
Diving into the Details with DTrace! (RubyConf 2016 edition)
trptcolin
2
290
Diving into the Details with DTrace
trptcolin
3
480
Adopting FP: the good, the familiar, and the unknown
trptcolin
0
140
Finding out what's *really* going on, with DTrace!
trptcolin
1
290
Beyond top: Command-Line Monitoring on the JVM (ClojureRemote)
trptcolin
0
120
ZooKeeper: Wait-free coordination for Internet-scale systems
trptcolin
2
160
Other Decks in Programming
See All in Programming
Ruby GitHub Packages
bkuhlmann
0
630
Random\Randomizer クラスで日常のあれこれを解決しよう! / Random\Randomizer class solves familiar trouble
cocoeyes02
0
250
MetricKitで予期せぬ終了を検知する話 / Detect unexpected termination with MetricKit
nekowen
1
190
ADRを一年運用してみた/adr_after_a_year
hanhan1978
7
2.4k
使ってみよう Azure AI Document Intelligence
kosmosebi
2
320
Prepare for Jakarta EE 11 - Performance and Developer Productivity
ivargrimstad
0
810
CA.swift19 恋するAIアプリ開発の裏側
oskmr
0
360
FigmaとPHPで作る1ミリたりとも表示崩れしない最強の帳票印刷ソリューション
ttskch
43
19k
PHP8.3の機能を振り返る / Review of PHP 8.3 features
seike460
PRO
1
110
GitHub Copilotのススメ
marcy731
1
200
Elm 0.19.0 Changes
bkuhlmann
0
490
Amazon SQSコンシューマー疎結合への旅 - 出張! #DevelopersIO IT技術ブログの中の人が語る勉強会 #3
quiver
0
270
Featured
See All Featured
Rails Girls Zürich Keynote
gr2m
91
13k
Practical Orchestrator
shlominoach
182
9.7k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
187
16k
Adopting Sorbet at Scale
ufuk
68
8.6k
Unsuck your backbone
ammeep
663
57k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
125
32k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
322
20k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
25
2.3k
RailsConf 2023
tenderlove
4
540
A Tale of Four Properties
chriscoyier
151
22k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
40
4.4k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
116
18k
Transcript
Beyond top(1) Command-Line Monitoring on the JVM Colin Jones @trptcolin
8th Light
None
What to expect
command-line tooling
on the JVM
introspection & serviceability
--all-flags=false
war stories
real-life usage (well, re-enacted anyway)
A long time ago in a software shop far, far
away…
Things are going pretty well
What does this thing look like? app-architecture Postgres Web /
API Application Server Load Balancer Periodic Job Application Server 3rd-party Service A 3rd-party Service B Monitored email account End users: native mobile app Admin users: desktop browsers
But strange things are afoot
the server sometimes gets really slow
the team has to manually restart the application server
incident response time is ~5 minutes
Yes, strange things are afoot
Pain, frustration, anger
Just the facts
sometimes, things get slow
all requests seem to be affected
the JVM stays up
restart the JVM and everything is fine
What could it be?
Demo
More facts, please!
constant full GCs
what’s in the heap
what application code was running
The right tools for the job
vmstat system-level: CPU, memory, disk, context switching
top per-process: CPU & memory
jps what’s our PID?
jstack status of all threads (right now-ish!)
jcmd what can’t it do?! jcmd [PID] help (sorry, JVM
6 users: see jinfo/jmap/jstack)
jstat GC classloader compiler
Mystery solved!
Now “just” fix it
idea 1: eliminate the leak
idea 2: eliminate the cache altogether?
idea 3: delete the feature
idea 4: full-text search engine
So we’re good now… until the next incident
Lessons
“it’s slow” could mean lots of things
“high CPU” could mean lots of things
collecting data is crucial in a crisis
reproducing the issue helps me sleep at night
Other “right tools for the job”
Heap analyzers
Profilers
Constant monitoring & alerting
Dynamic tracing
Learning more
Books
operators are standing by! man jstat man jstack jcmd [PID]
help [COMMAND] etc.
Thank you! Colin Jones @trptcolin 8th Light