Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Beyond top: Command-Line Monitoring on the JVM ...

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Colin Jones Colin Jones
October 29, 2015
Programming
1
690

Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)

A session from JavaOne 2015

Avatar for Colin Jones

Colin Jones

October 29, 2015
Tweet

More Decks by Colin Jones

See All by Colin Jones
A Bug's Life: What if `select` is Broken After All?
Avatar for Colin Jones trptcolin
0
170
Underestimated costs of microservice architectures
Avatar for Colin Jones trptcolin
3
1.6k
FP vs. OOP: Beyond the Bikeshed
Avatar for Colin Jones trptcolin
0
450
Diving into the Details with DTrace! (RubyConf 2016 edition)
Avatar for Colin Jones trptcolin
2
520
Diving into the Details with DTrace
Avatar for Colin Jones trptcolin
3
530
Adopting FP: the good, the familiar, and the unknown
Avatar for Colin Jones trptcolin
0
270
Finding out what's *really* going on, with DTrace!
Avatar for Colin Jones trptcolin
1
390
Beyond top: Command-Line Monitoring on the JVM (ClojureRemote)
Avatar for Colin Jones trptcolin
0
160
ZooKeeper: Wait-free coordination for Internet-scale systems
Avatar for Colin Jones trptcolin
2
210

Other Decks in Programming

See All in Programming
SourceGeneratorのマーカー属性問題について
Avatar for tkym htkym
0
180
ポーリング処理廃止によるイベント駆動アーキテクチャへの移行
Avatar for Seitaro Fujigaki seitarof
3
1k
AI Assistants for Your Angular Solutions
Avatar for Manfred Steyer manfredsteyer
PRO
0
130
API Platformを活用したPHPによる本格的なWeb API開発 / api-platform-book-intro
Avatar for Takashi Kanemoto ttskch
1
130
The Ralph Wiggum Loop: First Principles of Autonomous Development
Avatar for Ryuichiro Semba sembayui
0
3.7k
15年目のiOSアプリを1から作り直す技術
Avatar for TAKEDA Yuki teakun
1
620
new(1.26) ← これすき / kamakura.go #8
Avatar for utagawa kiki utgwkk
0
2.2k
Ruby x Terminal
Avatar for Akira Matsuda a_matsuda
7
590
社内規程RAGの精度を73.3% → 100%に改善した話
Avatar for oharu121 oharu121
13
7.9k
Rで始めるML・LLM活用入門
Avatar for wakama1994 wakamatsu_takumu
0
170
最初からAWS CDKで技術検証してもいいんじゃない？
Avatar for アキキー | Akihisa Ikeda akihisaikeda
4
130
メタプログラミングで実現する「コードを仕様にする」仕組み/nikkei-tech-talk43
Avatar for 日本経済新聞社 エンジニア採用事務局 nikkei_engineer_recruiting
0
170

Featured

See All Featured
Building a Modern Day 
E-commerce SEO Strategy
Avatar for Aleyda Solis aleyda
45
8.8k
How to optimise 3,500 product descriptions for ecommerce in one day using ChatGPT
Avatar for Katarina Dahlin katarinadahlin
PRO
1
3.5k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
55
9.8k
Leading Effective Engineering Teams in the AI Era
Avatar for Addy Osmani addyosmani
9
1.7k
Thoughts on Productivity
Avatar for Jon Yablonski jonyablonski
75
5.1k
Hiding What from Whom? A Critical Review of the History of Programming languages for Music
Avatar for Tomoya Matsuura tomoyanonymous
2
540
Design and Strategy: How to Deal with People Who Don’t "Get" Design
Avatar for Morgane Peng morganepeng
133
19k
The agentic SEO stack - context over prompts
Avatar for schlessera schlessera
0
690
The SEO Collaboration Effect
Avatar for Kristina Bergwall kristinabergwall1
0
390
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
Avatar for Akihiro Kokubo akihiro_kokubo
PRO
67
37k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
Avatar for mongodb mongodb
49
9.9k
Crafting Experiences
Avatar for Bethany Jepchumba bethany
1
82

Transcript

  1. Beyond top(1) Command-Line Monitoring on the JVM Colin Jones @trptcolin

    8th Light
  2. None

  3. What to expect

  4. command-line tooling

  5. on the JVM

  6. introspection & serviceability

  7. --all-flags=false

  8. war stories

  9. real-life usage (well, re-enacted anyway)

  10. A long time ago in a software shop far, far

    away…

  11. Things are going pretty well

  12. What does this thing look like? app-architecture Postgres Web /

    API Application Server Load Balancer Periodic Job Application Server 3rd-party Service A 3rd-party Service B Monitored email account End users: native mobile app Admin users: desktop browsers

  13. But strange things are afoot

  14. the server sometimes gets really slow

  15. the team has to manually restart the application server

  16. incident response time is ~5 minutes

  17. Yes, strange things are afoot

  18. Pain, frustration, anger

  19. Just the facts

  20. sometimes, things get slow

  21. all requests seem to be affected

  22. the JVM stays up

  23. restart the JVM and everything is fine

  24. What could it be?

  25. Demo

  26. More facts, please!

  27. constant full GCs

  28. what’s in the heap

  29. what application code was running

  30. The right tools for the job

  31. vmstat system-level: CPU, memory, disk, context switching

  32. top per-process: CPU & memory

  33. jps what’s our PID?

  34. jstack status of all threads (right now-ish!)

  35. jcmd what can’t it do?! jcmd [PID] help (sorry, JVM

    6 users: see jinfo/jmap/jstack)

  36. jstat GC classloader compiler

  37. Mystery solved!

  38. Now “just” fix it

  39. idea 1: eliminate the leak

  40. idea 2: eliminate the cache altogether?

  41. idea 3: delete the feature

  42. idea 4: full-text search engine

  43. So we’re good now… until the next incident

  44. Lessons

  45. “it’s slow” could mean lots of things

  46. “high CPU” could mean lots of things

  47. collecting data is crucial in a crisis

  48. reproducing the issue helps me sleep at night

  49. Other “right tools for the job”

  50. Heap analyzers

  51. Profilers

  52. Constant monitoring & alerting

  53. Dynamic tracing

  54. Learning more

  55. Books

  56. operators are standing by! man jstat man jstack jcmd [PID]

    help [COMMAND] etc.

  57. Thank you! Colin Jones @trptcolin 8th Light