Avoiding Benchmarking Pitfalls on the JVM (BBL 2014-09-12)

Slides supporting a BBL talk that I recently did on the topic of benchmarking on the JVM. People don't expect benchmarking to be that tricky to get right.

Transcript

1. Avoiding Benchmarking Pitfalls on the JVM Julien Ponge
2. @jponge ! Maître de Conférences INSA/Telecom ! CITI-INRIA Laboratory DynaMid team Associate Director (industry) ! Golo, IzPack, Maven Mojo, GlassFish, Devoxx, etc ! Oracle Java Magazine
3. Why are we here today?
5. Simple language Simple to use & hack
7. Function/closures calls Method dispatch (hard!) Arithmetic operations (…)
8. Runtime Bytecode JVM
9. “Do stuff” start stop “repeat, rinse” Theory…
10. “It doesn’t make (any) sense at all…” Practice…
11. package main ! import ( "fmt" "time" ) ! func Fib(n int64) int64 { if (n <= 2) { return n } else { return Fib(n - 1) + Fib(n - 2) } } ! func main() { for i := 0; i < 10; i++ { start := time.Now() Fib(44) fmt.Println(time.Since(start)) } } 4.738337584s 4.477160663s 4.408593662s 4.398906292s 4.533431874s 4.57529716s 4.519544937s 4.673349279s 4.945866449s 4.886524219s “Looks ok”
12. object FibApp extends App { ! def fib(n: Long): Long = if (n <= 2) n else fib(n - 1) + fib(n - 2) ! def howfast[T](block: => T): T = { val start = System.currentTimeMillis() val res = block println((System.currentTimeMillis() - start) + "ms") res } ! for (i <- 1 to 10) { howfast { fib(44) } } } 2787ms 3068ms 2857ms 2832ms 2797ms 2797ms 2839ms 2778ms 2896ms 2752ms “Looks ok”
13. “Do stuff” start stop “repeat, rinse” You could be “ok”: I/O, threads, complex call graphs, …
14. Static compilation Guess tricks at compilation time (C, C++, Go, etc)
15. Interpretation Just interpret from bytecode / AST (Python, Ruby, etc)
16. Dynamic compilation Guess tricks during execution (Java, Scala, JavaScript/V8, etc) ! ! Hybrid interpreter + JIT (HotSpot) “Just JIT” (V8, JRockit)
17. Dynamic compilation …is a (micro)benchmark killer.
18. Pitfalls in action
19. static double distance(double x1, double y1, double x2, double y2) { double dx = x2 - x1; double dy = y2 - y1; return Math.sqrt((dx * dx) + (dy * dy)); } How fast?
20. Demo (a not so naïve framework)
21. Dead-code elimination Constant folding Loop unrolling On-stack replacement Inlining Locks removal (…) https://wiki.openjdk.java.net/display/HotSpot/PerformanceTechniques
22. Caliper Google ! JUnitBenchmarks CarrotLabs ! JMH OpenJDK ✔ ⨉ ⨉
23. JMH to the rescue
24. “JMH is a Java harness for building, running, and analysing nano/micro/milli/ macro benchmarks written in Java and other languages targeting the JVM.”
25. Helps defeating unwanted optimisations.
26. Demo (JMH in anger)
29. A few examples Micro-benchmarks from Golo
36. JMH is great, but: ! • doubt • peer review • beware HotSpot • results don’t compose • read all Aleksey’s samples
37. (the end)
38. Julien Ponge @jponge ! https://julien.ponge.org/ http://dynamid.citi-lab.fr/ http://golo-lang.org/