なにもわからないところから始めるJVMモニタリング

Transcript

1. 1.

   ͳʹ΋Θ͔Βͳ͍ͱ͜Ζ͔Β ࢝ΊΔ
   +7.
   ϞχλϦϯά

2. 2.

   id:y_uuki / @y_uuk1

3. 3.

   None
4. 4.

   None
5. 5.

   ✘

6. 6.

   ✘

7. 7.

   PSGI/Plack Devel::NYTProf Devel::KYTProf Starlet Plack::Middleware::ServerStatus::Lite

8. 8.

   PSGI/Plack Devel::NYTProf Devel::KYTProf Starlet Plack::Middleware::ServerStatus::Lite

9. 9.

   ΠϯλʔωοτΈΔ • http://gihyo.jp/dev/serial/01/jvm-arc ! • http://www.slideshare.net/ChihiroIto1/java-16802525 Java͸ͲͷΑ͏ʹಈ͘ͷ͔ʙਤղͰΘ͔ΔJVMͷ࢓૊Έ Java͸ͲͷΑ͏ʹಈ͘ͷ͔ʙεϥΠυͰΘ͔ΔJVMͷ࢓૊Έ

10. 10.

    ϝϞϦߏ଄ / GCͷछྨ • Eden, Survivor, Tenured, Permanent… • γϦΞϧGCɺύϥϨϧGCɺίϯΧϨϯτGC

11. 11.

    ϩάΛు͘ • http://n-agetsuma.hatenablog.com/entry/2014/03/30/175329 τϥϒϧʹඋ͑ΔJVMΦϓγϣϯ java -Xms?m -Xmx?m -XX:MaxPermSize=?m -Xloggc:/var/log/app/gc.log -XX:+PrintGCDetails

    -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=10M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/app -XX:ErrorFile=/var/log/app/jvm_err_pid%p.log
12. 12.

    ౷ܭ৘ใΈΔπʔϧ • jstat • jvmtop • Jolokia (JMX) • much

    more…
13. 13.

    jstat [y_uuki@hoge]$ sudo jstat -gc 30545 1000 S0C S1C S0U

    S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT 3072.0 3072.0 0.0 2400.0 343552.0 181900.4 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034 3072.0 3072.0 0.0 2400.0 343552.0 189558.2 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034 3072.0 3072.0 0.0 2400.0 343552.0 291417.5 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034 3072.0 3072.0 0.0 2400.0 343552.0 291423.7 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034 3072.0 3072.0 0.0 2400.0 343552.0 300260.1 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034 3072.0 3072.0 0.0 2400.0 343552.0 300266.3 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034 3072.0 3072.0 0.0 2400.0 343552.0 338223.4 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034 3072.0 3072.0 2513.7 0.0 343552.0 5373.4 699392.0 125337.0 55808.0 55592.9 720 8.041 0 0.000 8.041 3072.0 3072.0 2513.7 0.0 343552.0 14378.4 699392.0 125337.0 55808.0 55592.9 720 8.041 0 0.000 8.041 3072.0 3072.0 2513.7 0.0 343552.0 14381.3 699392.0 125337.0 55808.0 55592.9 720 8.041 0 0.000 8.041 …
14. 14.

    jvmtop $ jvmtop.sh <pid> JvmTop 0.8.0 alpha - 21:48:03, amd64,

    4 cpus, Linux 3.10.23, load avg 0.30 http://code.google.com/p/jvmtop ! PID 30545: play.core.server.NettyServer ARGS: VMARGS: -Xms1024m -Xmx1024m -XX:MaxPermSize=256m -XX:ReservedCodeCacheSiz[...] VM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.7.0_51 UP: 7:49m #THR: 41 #THRPEAK: 44 #THRCREATED: 5867 USER: apache GC-Time: 0: 0m #GC-Runs: 892 #TotalLoadedClasses: 10033 CPU: 2.21% GC: 0.00% HEAP: 454m /1019m NONHEAP: 64m / 384m ! TID NAME STATE CPU TOTALCPU BLOCKEDBY 5862 RMI TCP Connection(3)-127.0.0. RUNNABLE 9.20% 0.29% 31 play-scheduler-1 TIMED_WAITING 0.26% 4.43% 5867 JMX server connection timeout TIMED_WAITING 0.24% 0.01% 27 New I/O worker #8 RUNNABLE 0.01% 0.71% 21 New I/O worker #2 RUNNABLE 0.01% 0.75% 22 New I/O worker #3 RUNNABLE 0.01% 0.72% 20 New I/O worker #1 RUNNABLE 0.01% 0.72% 24 New I/O worker #5 RUNNABLE 0.00% 0.72% 26 New I/O worker #7 RUNNABLE 0.00% 0.72% 23 New I/O worker #4 RUNNABLE 0.00% 0.72% Note: Only top 10 threads (according cpu load) are shown!
15. 15.

    Jolokia $ curl http://localhost:8778/jolokia/read/java.lang:type=Memory/ NonHeapMemoryUsage {"timestamp":1396794086,"status":200,"request": {"mbean":"java.lang:type=Memory","attribute":"NonHeapMemoryUsage","type":"re ad"},"value":{"max":402653184,"committed":59179008,"init":24576000,"used": 58814424}} -javaagent:/path/to/jolokia-jvm-1.2.0-agent.jar

16. 16.

    ՄࢹԽ • Graphite • Sensu • Fluentd • Mackerel •

    NewRelic
17. 17.

    metrics ࣌ܥྻDB αʔό؅ཧπʔϧ

18. 18.

    • sensu / sensu-community-plugins • java/jstat-metrics.py

19. 19.

    • fluent-plugin-jstat • fluent-plugin-jolokia

20. 20.

    • sensu tcp handler • fluent-plugin-graphite

21. 21.

    None
22. 22.

    NewRelic -javaagent:/path/to/newrelic.jar

23. 23.

    None
24. 24.

    Stop The World ʹඋ͑ͯ • ϞχλϦϯάͷ࢓૊ΈΛ੔͔͑ͯΒͷJVM ύϥ ϝʔλνϡʔχϯά

25. 25.

    ͓·͚

26. 26.

    ॻ੶ 2010/03 2011/10 2014/05

27. 27.

    Πϯλʔωοτ σϑΥϧτͷΧςΰϦશ෦ফͯ͠ Linux, Kernel, Docker, JVM, Perl, Scala, Graphite, Fluentd,

    Sensu, … ͦΕͬΆ͍λάΛొ࿥͠·͘Δ
28. 28.

    None
29. 29.

    IUUQTNBDLFSFMJP