Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)
Search
Colin Jones
October 29, 2015
Programming
1
650
Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)
A session from JavaOne 2015
Colin Jones
October 29, 2015
Tweet
Share
More Decks by Colin Jones
See All by Colin Jones
A Bug's Life: What if `select` is Broken After All?
trptcolin
0
140
Underestimated costs of microservice architectures
trptcolin
3
1.5k
FP vs. OOP: Beyond the Bikeshed
trptcolin
0
390
Diving into the Details with DTrace! (RubyConf 2016 edition)
trptcolin
2
330
Diving into the Details with DTrace
trptcolin
3
480
Adopting FP: the good, the familiar, and the unknown
trptcolin
0
150
Finding out what's *really* going on, with DTrace!
trptcolin
1
300
Beyond top: Command-Line Monitoring on the JVM (ClojureRemote)
trptcolin
0
120
ZooKeeper: Wait-free coordination for Internet-scale systems
trptcolin
2
160
Other Decks in Programming
See All in Programming
今こそ始める、CDKコンストラクトライブラリ開発 ― 入門から実践まで
tmokmss
1
930
I/O Extended Android in Korea 2024 ~ Whats new in Android development tools
pluu
0
250
はしめてのプログラミングとロボット制御
watawatavoltage
0
290
SDCon2024: Enabling DevOps and Team Topologies thru architecture: architecting for fast flow
cer
PRO
0
780
しくじり先生 Image Matching Challenge 2024 編
goosehaaan
0
810
12年前の『型システム入門』翻訳の思い出話
mame
11
1.2k
Findy - エンジニア向け会社紹介 / Findy Letter for Engineers
findyinc
2
81k
CSC307 Lecture 12
javiergs
PRO
0
220
Temporalを取り巻く仕様を整理する
sajikix
0
110
SRE チーム立ち上げ前に考えたこと・取り組んだこと / Considerations and Preparations Before Establishing an SRE Team
mackey0225
3
320
入社1ヶ月でここまでやった!Findy Toolsインフラ支援の最適化
rvirus0817
6
1.4k
最古の関数型言語「Lisp」ことはじめ / lisp_in_kamiyama
uhooi
1
190
Featured
See All Featured
Bash Introduction
62gerente
607
210k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
24
1.8k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
662
120k
How GitHub (no longer) Works
holman
305
140k
Docker and Python
trallard
37
2.9k
Debugging Ruby Performance
tmm1
71
11k
Rails Girls Zürich Keynote
gr2m
93
13k
Making the Leap to Tech Lead
cromwellryan
127
8.7k
Product Roadmaps are Hard
iamctodd
PRO
48
10k
Visualization
eitanlees
139
14k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
189
16k
Leading Effective Engineering Teams 2024
addyosmani
3
300
Transcript
Beyond top(1) Command-Line Monitoring on the JVM Colin Jones @trptcolin
8th Light
None
What to expect
command-line tooling
on the JVM
introspection & serviceability
--all-flags=false
war stories
real-life usage (well, re-enacted anyway)
A long time ago in a software shop far, far
away…
Things are going pretty well
What does this thing look like? app-architecture Postgres Web /
API Application Server Load Balancer Periodic Job Application Server 3rd-party Service A 3rd-party Service B Monitored email account End users: native mobile app Admin users: desktop browsers
But strange things are afoot
the server sometimes gets really slow
the team has to manually restart the application server
incident response time is ~5 minutes
Yes, strange things are afoot
Pain, frustration, anger
Just the facts
sometimes, things get slow
all requests seem to be affected
the JVM stays up
restart the JVM and everything is fine
What could it be?
Demo
More facts, please!
constant full GCs
what’s in the heap
what application code was running
The right tools for the job
vmstat system-level: CPU, memory, disk, context switching
top per-process: CPU & memory
jps what’s our PID?
jstack status of all threads (right now-ish!)
jcmd what can’t it do?! jcmd [PID] help (sorry, JVM
6 users: see jinfo/jmap/jstack)
jstat GC classloader compiler
Mystery solved!
Now “just” fix it
idea 1: eliminate the leak
idea 2: eliminate the cache altogether?
idea 3: delete the feature
idea 4: full-text search engine
So we’re good now… until the next incident
Lessons
“it’s slow” could mean lots of things
“high CPU” could mean lots of things
collecting data is crucial in a crisis
reproducing the issue helps me sleep at night
Other “right tools for the job”
Heap analyzers
Profilers
Constant monitoring & alerting
Dynamic tracing
Learning more
Books
operators are standing by! man jstat man jstack jcmd [PID]
help [COMMAND] etc.
Thank you! Colin Jones @trptcolin 8th Light