なにもわからないところから始めるJVMモニタリング

JVM Operation Casual Talks

Transcript

1. ͳʹ΋Θ͔Βͳ͍ͱ͜Ζ͔Β
   ࢝ΊΔ+7.ϞχλϦϯά
   

   View Slide

2. id:y_uuki / @y_uuk1
   

   View Slide

3. View Slide

4. View Slide

5. ✘
   

   View Slide

6. ✘
   

   View Slide

7. PSGI/Plack
   Devel::NYTProf
   Devel::KYTProf
   Starlet
   Plack::Middleware::ServerStatus::Lite
   

   View Slide

8. PSGI/Plack
   Devel::NYTProf
   Devel::KYTProf
   Starlet
   Plack::Middleware::ServerStatus::Lite
   

   View Slide

9. ΠϯλʔωοτΈΔ
   • http://gihyo.jp/dev/serial/01/jvm-arc
   !
   • http://www.slideshare.net/ChihiroIto1/java-16802525
   Java͸ͲͷΑ͏ʹಈ͘ͷ͔ʙਤղͰΘ͔ΔJVMͷ࢓૊Έ
   Java͸ͲͷΑ͏ʹಈ͘ͷ͔ʙεϥΠυͰΘ͔ΔJVMͷ࢓૊Έ
   

   View Slide

10. ϝϞϦߏ଄ / GCͷछྨ
    • Eden, Survivor, Tenured, Permanent…
    • γϦΞϧGCɺύϥϨϧGCɺίϯΧϨϯτGC
    

    View Slide

11. ϩάΛు͘
    • http://n-agetsuma.hatenablog.com/entry/2014/03/30/175329
    τϥϒϧʹඋ͑ΔJVMΦϓγϣϯ
    java -Xms?m -Xmx?m -XX:MaxPermSize=?m
    -Xloggc:/var/log/app/gc.log
    -XX:+PrintGCDetails
    -XX:+PrintGCDateStamps
    -XX:+UseGCLogFileRotation
    -XX:NumberOfGCLogFiles=5
    -XX:GCLogFileSize=10M
    -XX:+HeapDumpOnOutOfMemoryError
    -XX:HeapDumpPath=/var/log/app
    -XX:ErrorFile=/var/log/app/jvm_err_pid%p.log
    

    View Slide

12. ౷ܭ৘ใΈΔπʔϧ
    • jstat
    • jvmtop
    • Jolokia (JMX)
    • much more…
    

    View Slide

13. jstat
    [y_uuki@hoge]$ sudo jstat -gc 30545 1000
    S0C S1C S0U S1U EC EU OC OU PC PU YGC YGCT FGC FGCT GCT
    3072.0 3072.0 0.0 2400.0 343552.0 181900.4 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 189558.2 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 291417.5 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 291423.7 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 300260.1 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 300266.3 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 0.0 2400.0 343552.0 338223.4 699392.0 125193.0 55808.0 55592.9 719 8.034 0 0.000 8.034
    3072.0 3072.0 2513.7 0.0 343552.0 5373.4 699392.0 125337.0 55808.0 55592.9 720 8.041 0 0.000 8.041
    3072.0 3072.0 2513.7 0.0 343552.0 14378.4 699392.0 125337.0 55808.0 55592.9 720 8.041 0 0.000 8.041
    3072.0 3072.0 2513.7 0.0 343552.0 14381.3 699392.0 125337.0 55808.0 55592.9 720 8.041 0 0.000 8.041
    …
    

    View Slide

14. jvmtop
    $ jvmtop.sh
    JvmTop 0.8.0 alpha - 21:48:03, amd64, 4 cpus, Linux 3.10.23, load avg 0.30
    http://code.google.com/p/jvmtop
    !
    PID 30545: play.core.server.NettyServer
    ARGS:
    VMARGS: -Xms1024m -Xmx1024m -XX:MaxPermSize=256m -XX:ReservedCodeCacheSiz[...]
    VM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.7.0_51
    UP: 7:49m #THR: 41 #THRPEAK: 44 #THRCREATED: 5867 USER: apache
    GC-Time: 0: 0m #GC-Runs: 892 #TotalLoadedClasses: 10033
    CPU: 2.21% GC: 0.00% HEAP: 454m /1019m NONHEAP: 64m / 384m
    !
    TID NAME STATE CPU TOTALCPU BLOCKEDBY
    5862 RMI TCP Connection(3)-127.0.0. RUNNABLE 9.20% 0.29%
    31 play-scheduler-1 TIMED_WAITING 0.26% 4.43%
    5867 JMX server connection timeout TIMED_WAITING 0.24% 0.01%
    27 New I/O worker #8 RUNNABLE 0.01% 0.71%
    21 New I/O worker #2 RUNNABLE 0.01% 0.75%
    22 New I/O worker #3 RUNNABLE 0.01% 0.72%
    20 New I/O worker #1 RUNNABLE 0.01% 0.72%
    24 New I/O worker #5 RUNNABLE 0.00% 0.72%
    26 New I/O worker #7 RUNNABLE 0.00% 0.72%
    23 New I/O worker #4 RUNNABLE 0.00% 0.72%
    Note: Only top 10 threads (according cpu load) are shown!
    

    View Slide

15. Jolokia
    $ curl http://localhost:8778/jolokia/read/java.lang:type=Memory/
    NonHeapMemoryUsage
    {"timestamp":1396794086,"status":200,"request":
    {"mbean":"java.lang:type=Memory","attribute":"NonHeapMemoryUsage","type":"re
    ad"},"value":{"max":402653184,"committed":59179008,"init":24576000,"used":
    58814424}}
    -javaagent:/path/to/jolokia-jvm-1.2.0-agent.jar
    

    View Slide

16. ՄࢹԽ
    • Graphite
    • Sensu
    • Fluentd
    • Mackerel
    • NewRelic
    

    View Slide

17. metrics
    ࣌ܥྻDB
    αʔό؅ཧπʔϧ
    

    View Slide

18. • sensu / sensu-community-plugins
    • java/jstat-metrics.py
    

    View Slide

19. • fluent-plugin-jstat
    • fluent-plugin-jolokia
    

    View Slide

20. • sensu tcp handler
    • fluent-plugin-graphite
    

    View Slide

21. View Slide

22. NewRelic
    -javaagent:/path/to/newrelic.jar
    

    View Slide

23. View Slide

24. Stop The World ʹඋ͑ͯ
    • ϞχλϦϯάͷ࢓૊ΈΛ੔͔͑ͯΒͷJVM ύϥ
    ϝʔλνϡʔχϯά
    

    View Slide

25. ͓·͚
    

    View Slide

26. ॻ੶
    2010/03 2011/10 2014/05
    

    View Slide

27. Πϯλʔωοτ
    σϑΥϧτͷΧςΰϦશ෦ফͯ͠
    Linux, Kernel, Docker, JVM, Perl,
    Scala, Graphite, Fluentd, Sensu, …
    ͦΕͬΆ͍λάΛొ࿥͠·͘Δ
    

    View Slide

28. View Slide

29. IUUQTNBDLFSFMJP
    

    View Slide