Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Beyond top: Command-Line Monitoring on the JVM ...
Search
Colin Jones
October 29, 2015
Programming
1
670
Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)
A session from JavaOne 2015
Colin Jones
October 29, 2015
Tweet
Share
More Decks by Colin Jones
See All by Colin Jones
A Bug's Life: What if `select` is Broken After All?
trptcolin
0
150
Underestimated costs of microservice architectures
trptcolin
3
1.5k
FP vs. OOP: Beyond the Bikeshed
trptcolin
0
420
Diving into the Details with DTrace! (RubyConf 2016 edition)
trptcolin
2
480
Diving into the Details with DTrace
trptcolin
3
510
Adopting FP: the good, the familiar, and the unknown
trptcolin
0
230
Finding out what's *really* going on, with DTrace!
trptcolin
1
370
Beyond top: Command-Line Monitoring on the JVM (ClojureRemote)
trptcolin
0
130
ZooKeeper: Wait-free coordination for Internet-scale systems
trptcolin
2
180
Other Decks in Programming
See All in Programming
デミカツ切り抜きで面倒くさいことはPythonにやらせよう
aokswork3
0
260
Ktorで簡単AIアプリケーション
tsukakei
0
110
AIと人間の共創開発!OSSで試行錯誤した開発スタイル
mae616
2
810
NixOS + Kubernetesで構築する自宅サーバーのすべて
ichi_h3
0
1.2k
Migration to Signals, Resource API, and NgRx Signal Store
manfredsteyer
PRO
0
120
他言語経験者が Golangci-lint を最初のコーディングメンターにした話 / How Golangci-lint Became My First Coding Mentor: A Story from a Polyglot Programmer
uma31
0
430
GC25 Recap: The Code You Reviewed is Not the Code You Built / #newt_gophercon_tour
mazrean
0
110
Developer Joy - The New Paradigm
hollycummins
1
370
When Dependencies Fail: Building Antifragile Applications in a Fragile World
selcukusta
0
110
エンジニアインターン「Treasure」とHonoの2年、そして未来へ / Our Journey with Hono Two Years at Treasure and Beyond
carta_engineering
0
430
CSC509 Lecture 06
javiergs
PRO
0
270
Six and a half ridiculous things to do with Quarkus
hollycummins
0
210
Featured
See All Featured
Code Reviewing Like a Champion
maltzj
526
40k
The Straight Up "How To Draw Better" Workshop
denniskardys
238
140k
The Language of Interfaces
destraynor
162
25k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
285
14k
KATA
mclloyd
PRO
32
15k
Speed Design
sergeychernyshev
32
1.2k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
132
19k
Testing 201, or: Great Expectations
jmmastey
45
7.7k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
230
22k
4 Signs Your Business is Dying
shpigford
185
22k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
249
1.3M
Unsuck your backbone
ammeep
671
58k
Transcript
Beyond top(1) Command-Line Monitoring on the JVM Colin Jones @trptcolin
8th Light
None
What to expect
command-line tooling
on the JVM
introspection & serviceability
--all-flags=false
war stories
real-life usage (well, re-enacted anyway)
A long time ago in a software shop far, far
away…
Things are going pretty well
What does this thing look like? app-architecture Postgres Web /
API Application Server Load Balancer Periodic Job Application Server 3rd-party Service A 3rd-party Service B Monitored email account End users: native mobile app Admin users: desktop browsers
But strange things are afoot
the server sometimes gets really slow
the team has to manually restart the application server
incident response time is ~5 minutes
Yes, strange things are afoot
Pain, frustration, anger
Just the facts
sometimes, things get slow
all requests seem to be affected
the JVM stays up
restart the JVM and everything is fine
What could it be?
Demo
More facts, please!
constant full GCs
what’s in the heap
what application code was running
The right tools for the job
vmstat system-level: CPU, memory, disk, context switching
top per-process: CPU & memory
jps what’s our PID?
jstack status of all threads (right now-ish!)
jcmd what can’t it do?! jcmd [PID] help (sorry, JVM
6 users: see jinfo/jmap/jstack)
jstat GC classloader compiler
Mystery solved!
Now “just” fix it
idea 1: eliminate the leak
idea 2: eliminate the cache altogether?
idea 3: delete the feature
idea 4: full-text search engine
So we’re good now… until the next incident
Lessons
“it’s slow” could mean lots of things
“high CPU” could mean lots of things
collecting data is crucial in a crisis
reproducing the issue helps me sleep at night
Other “right tools for the job”
Heap analyzers
Profilers
Constant monitoring & alerting
Dynamic tracing
Learning more
Books
operators are standing by! man jstat man jstack jcmd [PID]
help [COMMAND] etc.
Thank you! Colin Jones @trptcolin 8th Light