Embrace the JVM

This talk highlights various debugging and maintenance tools that help you with writing Clojure code and managing it in production.

Transcript

1. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 1/43 Embrace the JVM Embrace the JVM Alexander Yakushev Alexander Yakushev Dutch Clojure Days 2018 1
2. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 2/43 Bio Bio Backend Engineer @ Grammarly Clojure, Common Lisp, Java We use Clojure for: NLP projects ETL pipelines Infrastructure management The company actively uses Java and Scala 2 . 1
3. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 3/43 Clojure is built on JVM Clojure is built on JVM State-of-the-art platform (cross-platform runtime, GC, JIT) Exhaustive ecosystem (libraries, frameworks) Battle-tested deployment story (application containers, Docker) Debugging and maintainance tools 3 . 1
4. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 4/43 Maintainance is important Maintainance is important Releasing the program is only the beginning. "Release It! 2nd ed." by Michael Nygard We also need to: Know what and how program is doing Treat and understand crashes Dig into lower levels when necessary 4 . 1
5. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 5/43 Observability Observability 5 . 1
6. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 6/43 Find running Java processes Find running Java processes ps aux | grep java jcmd 6 . 1
7. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 7/43 jcmd jcmd 7 . 1
8. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 8/43 jinfo jinfo Show various information about JVM process: jinfo <PID> VM flags All Java properties Classpath 8 . 1
9. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 9/43 jinfo jinfo Enable GC logging: jinfo -flag +PrintGC <PID> 9 . 1
10. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 10/43 jstat jstat Continuously monitor GC and JIT: jstat -gcutil -h10 <PID> 2s jstat -gc -h10 <PID> 2s jstat -compiler -h10 <PID> 2s 10 . 1
11. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 11/43 jstack jstack Print stacktraces of all threads: jstack <PID>
12. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 12/43 11 . 1
13. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 13/43 jstack jstack Print additional information about locks: jstack -l <PID> 12 . 1
14. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 14/43 Performance profiling Performance profiling 13 . 1
15. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 15/43 VisualVM VisualVM
16. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 16/43 14 . 1
17. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 17/43 VisualVM VisualVM
18. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 18/43 15 . 1
19. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 19/43 VisualVM VisualVM
20. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 20/43 16 . 1
21. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 21/43 VisualVM VisualVM 17 . 1
22. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 22/43 VisualVM sampling profiler VisualVM sampling profiler (def json (slurp "sample.json")) (def edn (slurp "sample.edn")) (defn parse-json [] (cheshire.core/parse-string json)) (defn parse-edn [] (clojure.edn/read-string edn)) (defn run [] (while true (parse-json) (parse-edn))) (run) 18 . 1
23. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 23/43 VisualVM sampling profiler VisualVM sampling profiler 19 . 1
24. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 24/43 VisualVM sampling profiler VisualVM sampling profiler 20 . 1
25. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 25/43 VisualVM sampling profiler VisualVM sampling profiler 21 . 1
26. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 26/43 clj-async-profiler clj-async-profiler Embedded profiling with flamegraphs. (require '[clj-async-profiler.core :as prof]) (prof/profiler-for 10 {}) (prof/serve-files 8080) 22 . 1
27. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 27/43 Flame Graph Search clojure.. java/lang/Thread.run cl.. clojure/lang/EdnReader.read cheshire/core$parse_string.invoke clojure/lang/EdnReader.readDelimitedList cloj.. cheshire/parse$.. clojure/lang/EdnReade.. jav.. cheshire/parse.. clojure/lang/EdnReader$MapReader.invoke cloj.. clojure/lang/EdnReader$MapReader.invoke boot/user$eval12788.invokeStatic cheshire/parse.. java/ut.. cheshire/parse$pa.. clojure/lang/AFn.run cl.. clojure/lang/EdnReader.readDelimitedList com/fasterxml/.. com/fasterxml/.. com/fasterxml/.. boot/user$parse_edn.invoke j.. cheshire/parse$p.. com/fasterxml.. j.. java/util/concurrent/ThreadPoolExecutor$Worker.run java/ut.. java.. com/fasterxml.. cheshire/core$parse_string.invokeSta.. ja.. boot/user$parse_json.invokeStatic cloj.. clojure/lang/EdnRead.. cloju.. cloj.. jav.. j.. clojure/edn$read_string.invokeStatic cl.. clojure/lang/EdnReader.readString clojure/lang/EdnReader$MapReader.invoke cheshire/parse$.. clojure/lang/EdnReader.readDelimitedList clojure/edn$read_string.invokeStatic c.. c.. j.. ja.. boot/user$parse_json.invoke boot/user$run.invoke clojure/edn$read_string.invoke ja.. clojur.. cloj.. clojure/lang/EdnReader... java/uti.. java/util/concurrent/ThreadPoolExecutor.runWorker java/ut.. chesh.. boot/user$parse_edn.invokeStatic boot/user$eval12788.invoke cheshire/core$parse_string.invokeSta.. clojure/lang/EdnReader.readDelimitedList chesh.. clojure/lang/EdnReader$MapReader.invoke j.. cheshire/core$parse_string.invoke cloj.. clojure/lang/EdnReader... cheshire/parse$pa.. com/fasterxml/ja.. cheshire/pa.. clojure/lang/EdnReader.readDelimitedList cheshire/pa.. clojure.. java/uti.. clojure.. cheshire/par.. cloj.. cl.. cheshire/par.. cloj.. com/fasterxml/j.. java/.. clojure/lang/EdnReader.read c.. cl.. clojure/lang/EdnReader.read cheshire/parse$p.. cloj.. clo.. clo.. clojure/lang/EdnReader$MapReader.invoke boot/user$run.invokeStatic 23 . 1
28. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 28/43 clj-async-profiler clj-async-profiler Flamegraphs are a great representation. Intuitive Portable Profiler is controllable from your code. Profiling data is in textual format. Malleable 24 . 1
29. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 29/43 Memory inspection Memory inspection 25 . 1
30. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 30/43 jmap jmap Dump the heap contents into a file: jmap -dump:format=b,file=<file> <PID> 26 . 1
31. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 31/43 Eclipse Memory Analyzer Eclipse Memory Analyzer
32. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 32/43 27 . 1
33. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 33/43 clj-memory-meter clj-memory-meter Measure how much objects occupy in memory. Useful for optimizing sizes of data structures. 28 . 1
34. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 34/43 Java the language Java the language 29 . 1
35. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 35/43 clj-java-decompiler clj-java-decompiler Decompile Clojure-produced bytecode into Java Useful to understand how Clojure compiler works Also, to double-check tight performance spots 30 . 1
36. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 36/43 clj-async-profiler clj-async-profiler (require '[clj-java-decompiler.core :refer [decompile]]) (decompile (dotimes [i 100] (loop [j 0, sum 0] (if (< j 200) (recur (inc j) (+ sum j)) (println sum))))) public static Object invokeStatic() { for (long n__5742__auto__209 = 100L, i = 0L; i < n__5742__auto__209; ++i) { long j = 0L; long sum = 0L; while (j < 200L) { final long inc = Numbers.inc(j); sum = Numbers.add(sum, j); j = inc; } core$println.invokeStatic(ArraySeq.create(Numbers.num(sum))); } return null; } 31 . 1
37. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 37/43 Writing Java in Clojure projects Writing Java in Clojure projects Convenient for low-level logic: imperative loops, mutable fields, arrays Useful for writing typed facades around Clojure code 32 . 1
38. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 38/43 Virgil Virgil Recompile Java automatically withour restarting the REPL Like Figwheel for Java 33 . 1
39. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 39/43 The Multiplier Language The Multiplier Language x: input (experience, effort, tools, community) y: what the language gives back to you y = kx + b 34 . 1
40. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 40/43 The Multiplier Language The Multiplier Language Regular languages: k = 1, b > 0 Bad languages: k < 1 Clojure: k >> 1, b = could-be-more y = kx + b 35 . 1
41. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 41/43 Conclusions Conclusions Embrace and cherish your heritage. Look for things that make you more efficient. Clojure makes the whole greater than the sum of its parts. 36 . 1
42. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 42/43 Links Links https://visualvm.github.io/ https://github.com/clojure-goes-fast/clj-async-profiler https://www.eclipse.org/mat/ https://github.com/clojure-goes-fast/clj-memory-meter https://github.com/clojure-goes-fast/clj-java-decompiler https://github.com/ztellman/virgil http://clojure-goes-fast.com @unlog1c 37 . 1
43. 4/21/2018 Embrace the JVM file:///Users/alex/projects/talks/clojuredays-2018/index.html?print-pdf 43/43 (go-loop [] (when-let [q (<! questions)] (answer q) (recur))) 38 . 1