Upgrade to Pro — share decks privately, control downloads, hide ads and more …

なにもわからないところから始めるJVMモニタリング

 なにもわからないところから始めるJVMモニタリング

JVM Operation Casual Talks

Yuuki Tsubouchi (yuuk1)

April 07, 2014
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Technology

Transcript

  1. ͳʹ΋Θ͔Βͳ͍ͱ͜Ζ͔Β
    ࢝ΊΔ+7.ϞχλϦϯά

    View Slide

  2. id:y_uuki / @y_uuk1

    View Slide

  3. View Slide

  4. View Slide


  5. View Slide


  6. View Slide

  7. PSGI/Plack
    Devel::NYTProf
    Devel::KYTProf
    Starlet
    Plack::Middleware::ServerStatus::Lite

    View Slide

  8. PSGI/Plack
    Devel::NYTProf
    Devel::KYTProf
    Starlet
    Plack::Middleware::ServerStatus::Lite

    View Slide

  9. ΠϯλʔωοτΈΔ
    • http://gihyo.jp/dev/serial/01/jvm-arc
    !
    • http://www.slideshare.net/ChihiroIto1/java-16802525
    Java͸ͲͷΑ͏ʹಈ͘ͷ͔ʙਤղͰΘ͔ΔJVMͷ࢓૊Έ
    Java͸ͲͷΑ͏ʹಈ͘ͷ͔ʙεϥΠυͰΘ͔ΔJVMͷ࢓૊Έ

    View Slide

  10. ϝϞϦߏ଄ / GCͷछྨ
    • Eden, Survivor, Tenured, Permanent…
    • γϦΞϧGCɺύϥϨϧGCɺίϯΧϨϯτGC

    View Slide

  11. ϩάΛు͘
    • http://n-agetsuma.hatenablog.com/entry/2014/03/30/175329
    τϥϒϧʹඋ͑ΔJVMΦϓγϣϯ
    java -Xms?m -Xmx?m -XX:MaxPermSize=?m
    -Xloggc:/var/log/app/gc.log
    -XX:+PrintGCDetails
    -XX:+PrintGCDateStamps
    -XX:+UseGCLogFileRotation
    -XX:NumberOfGCLogFiles=5
    -XX:GCLogFileSize=10M
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:HeapDumpPath=/var/log/app
    -XX:ErrorFile=/var/log/app/jvm_err_pid%p.log

    View Slide

  12. ౷ܭ৘ใΈΔπʔϧ
    • jstat
    • jvmtop
    • Jolokia (JMX)
    • much more…

    View Slide

  13. jstat
    [y_uuki@hoge]$ sudo jstat -gc 30545 1000
    S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
    3072.0 3072.0 0.0 2400.0 343552.0 181900.4 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 189558.2 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 291417.5 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 291423.7 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 300260.1 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 300266.3 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 338223.4 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 2513.7 0.0 343552.0 5373.4 699392.0 125337.0 55808.0 55592.9 720 8.041 0 0.000 8.041
    3072.0 3072.0 2513.7 0.0 343552.0 14378.4 699392.0 125337.0 55808.0 55592.9 720 8.041 0 0.000 8.041
    3072.0 3072.0 2513.7 0.0 343552.0 14381.3 699392.0 125337.0 55808.0 55592.9 720 8.041 0 0.000 8.041

    View Slide

  14. jvmtop
    $ jvmtop.sh
    JvmTop 0.8.0 alpha - 21:48:03, amd64, 4 cpus, Linux 3.10.23, load avg 0.30
    http://code.google.com/p/jvmtop
    !
    PID 30545: play.core.server.NettyServer
    ARGS:
    VMARGS: -Xms1024m -Xmx1024m -XX:MaxPermSize=256m -XX:ReservedCodeCacheSiz[...]
    VM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.7.0_51
    UP: 7:49m #THR: 41 #THRPEAK: 44 #THRCREATED: 5867 USER: apache
    GC-Time: 0: 0m #GC-Runs: 892 #TotalLoadedClasses: 10033
    CPU: 2.21% GC: 0.00% HEAP: 454m /1019m NONHEAP: 64m / 384m
    !
    TID NAME STATE CPU TOTALCPU BLOCKEDBY
    5862 RMI TCP Connection(3)-127.0.0. RUNNABLE 9.20% 0.29%
    31 play-scheduler-1 TIMED_WAITING 0.26% 4.43%
    5867 JMX server connection timeout TIMED_WAITING 0.24% 0.01%
    27 New I/O worker #8 RUNNABLE 0.01% 0.71%
    21 New I/O worker #2 RUNNABLE 0.01% 0.75%
    22 New I/O worker #3 RUNNABLE 0.01% 0.72%
    20 New I/O worker #1 RUNNABLE 0.01% 0.72%
    24 New I/O worker #5 RUNNABLE 0.00% 0.72%
    26 New I/O worker #7 RUNNABLE 0.00% 0.72%
    23 New I/O worker #4 RUNNABLE 0.00% 0.72%
    Note: Only top 10 threads (according cpu load) are shown!

    View Slide

  15. Jolokia
    $ curl http://localhost:8778/jolokia/read/java.lang:type=Memory/
    NonHeapMemoryUsage
    {"timestamp":1396794086,"status":200,"request":
    {"mbean":"java.lang:type=Memory","attribute":"NonHeapMemoryUsage","type":"re
    ad"},"value":{"max":402653184,"committed":59179008,"init":24576000,"used":
    58814424}}
    -javaagent:/path/to/jolokia-jvm-1.2.0-agent.jar

    View Slide

  16. ՄࢹԽ
    • Graphite
    • Sensu
    • Fluentd
    • Mackerel
    • NewRelic

    View Slide

  17. metrics
    ࣌ܥྻDB
    αʔό؅ཧπʔϧ

    View Slide

  18. • sensu / sensu-community-plugins
    • java/jstat-metrics.py

    View Slide

  19. • fluent-plugin-jstat
    • fluent-plugin-jolokia

    View Slide

  20. • sensu tcp handler
    • fluent-plugin-graphite

    View Slide

  21. View Slide

  22. NewRelic
    -javaagent:/path/to/newrelic.jar

    View Slide

  23. View Slide

  24. Stop The World ʹඋ͑ͯ
    • ϞχλϦϯάͷ࢓૊ΈΛ੔͔͑ͯΒͷJVM ύϥ
    ϝʔλνϡʔχϯά

    View Slide

  25. ͓·͚

    View Slide

  26. ॻ੶
    2010/03 2011/10 2014/05

    View Slide

  27. Πϯλʔωοτ
    σϑΥϧτͷΧςΰϦશ෦ফͯ͠
    Linux, Kernel, Docker, JVM, Perl,
    Scala, Graphite, Fluentd, Sensu, …
    ͦΕͬΆ͍λάΛొ࿥͠·͘Δ

    View Slide

  28. View Slide

  29. IUUQTNBDLFSFMJP

    View Slide