Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Beyond top: Command-Line Monitoring on the JVM ...
Search
Colin Jones
October 29, 2015
Programming
1
650
Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)
A session from JavaOne 2015
Colin Jones
October 29, 2015
Tweet
Share
More Decks by Colin Jones
See All by Colin Jones
A Bug's Life: What if `select` is Broken After All?
trptcolin
0
140
Underestimated costs of microservice architectures
trptcolin
3
1.5k
FP vs. OOP: Beyond the Bikeshed
trptcolin
0
390
Diving into the Details with DTrace! (RubyConf 2016 edition)
trptcolin
2
350
Diving into the Details with DTrace
trptcolin
3
490
Adopting FP: the good, the familiar, and the unknown
trptcolin
0
160
Finding out what's *really* going on, with DTrace!
trptcolin
1
320
Beyond top: Command-Line Monitoring on the JVM (ClojureRemote)
trptcolin
0
120
ZooKeeper: Wait-free coordination for Internet-scale systems
trptcolin
2
160
Other Decks in Programming
See All in Programming
「今のプロジェクトいろいろ大変なんですよ、app/services とかもあって……」/After Kaigi on Rails 2024 LT Night
junk0612
5
2.1k
RubyLSPのマルチバイト文字対応
notfounds
0
120
.NET のための通信フレームワーク MagicOnion 入門 / Introduction to MagicOnion
mayuki
1
1.5k
よくできたテンプレート言語として TypeScript + JSX を利用する試み / Using TypeScript + JSX outside of Web Frontend #TSKaigiKansai
izumin5210
6
1.7k
Generative AI Use Cases JP (略称:GenU)奮闘記
hideg
1
290
Make Impossible States Impossibleを 意識してReactのPropsを設計しよう
ikumatadokoro
0
170
見せてあげますよ、「本物のLaravel批判」ってやつを。
77web
7
7.7k
as(型アサーション)を書く前にできること
marokanatani
10
2.6k
Realtime API 入門
riofujimon
0
150
watsonx.ai Dojo #4 生成AIを使ったアプリ開発、応用編
oniak3ibm
PRO
1
100
[Do iOS '24] Ship your app on a Friday...and enjoy your weekend!
polpielladev
0
100
AWS IaCの注目アップデート 2024年10月版
konokenj
3
3.3k
Featured
See All Featured
Why Our Code Smells
bkeepers
PRO
334
57k
10 Git Anti Patterns You Should be Aware of
lemiorhan
654
59k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
16
2.1k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
506
140k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
4
370
Site-Speed That Sticks
csswizardry
0
24
Agile that works and the tools we love
rasmusluckow
327
21k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
38
1.8k
4 Signs Your Business is Dying
shpigford
180
21k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
232
17k
The Cult of Friendly URLs
andyhume
78
6k
A designer walks into a library…
pauljervisheath
204
24k
Transcript
Beyond top(1) Command-Line Monitoring on the JVM Colin Jones @trptcolin
8th Light
None
What to expect
command-line tooling
on the JVM
introspection & serviceability
--all-flags=false
war stories
real-life usage (well, re-enacted anyway)
A long time ago in a software shop far, far
away…
Things are going pretty well
What does this thing look like? app-architecture Postgres Web /
API Application Server Load Balancer Periodic Job Application Server 3rd-party Service A 3rd-party Service B Monitored email account End users: native mobile app Admin users: desktop browsers
But strange things are afoot
the server sometimes gets really slow
the team has to manually restart the application server
incident response time is ~5 minutes
Yes, strange things are afoot
Pain, frustration, anger
Just the facts
sometimes, things get slow
all requests seem to be affected
the JVM stays up
restart the JVM and everything is fine
What could it be?
Demo
More facts, please!
constant full GCs
what’s in the heap
what application code was running
The right tools for the job
vmstat system-level: CPU, memory, disk, context switching
top per-process: CPU & memory
jps what’s our PID?
jstack status of all threads (right now-ish!)
jcmd what can’t it do?! jcmd [PID] help (sorry, JVM
6 users: see jinfo/jmap/jstack)
jstat GC classloader compiler
Mystery solved!
Now “just” fix it
idea 1: eliminate the leak
idea 2: eliminate the cache altogether?
idea 3: delete the feature
idea 4: full-text search engine
So we’re good now… until the next incident
Lessons
“it’s slow” could mean lots of things
“high CPU” could mean lots of things
collecting data is crucial in a crisis
reproducing the issue helps me sleep at night
Other “right tools for the job”
Heap analyzers
Profilers
Constant monitoring & alerting
Dynamic tracing
Learning more
Books
operators are standing by! man jstat man jstack jcmd [PID]
help [COMMAND] etc.
Thank you! Colin Jones @trptcolin 8th Light