Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Avoiding Benchmarking Pitfalls on the JVM (BBL 2014-09-12)

50a17cd98aab2cc4d8e144741e11b1b7?s=47 Julien Ponge
September 12, 2014

Avoiding Benchmarking Pitfalls on the JVM (BBL 2014-09-12)

Slides supporting a BBL talk that I recently did on the topic of benchmarking on the JVM. People don't expect benchmarking to be that tricky to get right.

50a17cd98aab2cc4d8e144741e11b1b7?s=128

Julien Ponge

September 12, 2014
Tweet

Transcript

  1. Avoiding Benchmarking Pitfalls on the JVM Julien Ponge

  2. @jponge ! Maître de Conférences INSA/Telecom ! CITI-INRIA Laboratory DynaMid

    team Associate Director (industry) ! Golo, IzPack, Maven Mojo, GlassFish, Devoxx, etc ! Oracle Java Magazine
  3. Why are we here today?

  4. None
  5. Simple language Simple to use & hack

  6. None
  7. Function/closures calls Method dispatch (hard!) Arithmetic operations (…)

  8. Runtime Bytecode JVM

  9. “Do stuff” start stop “repeat, rinse” Theory…

  10. “It doesn’t make (any) sense at all…” Practice…

  11. package main ! import ( "fmt" "time" ) ! func

    Fib(n int64) int64 { if (n <= 2) { return n } else { return Fib(n - 1) + Fib(n - 2) } } ! func main() { for i := 0; i < 10; i++ { start := time.Now() Fib(44) fmt.Println(time.Since(start)) } } 4.738337584s 4.477160663s 4.408593662s 4.398906292s 4.533431874s 4.57529716s 4.519544937s 4.673349279s 4.945866449s 4.886524219s “Looks ok”
  12. object FibApp extends App { ! def fib(n: Long): Long

    = if (n <= 2) n else fib(n - 1) + fib(n - 2) ! def howfast[T](block: => T): T = { val start = System.currentTimeMillis() val res = block println((System.currentTimeMillis() - start) + "ms") res } ! for (i <- 1 to 10) { howfast { fib(44) } } } 2787ms 3068ms 2857ms 2832ms 2797ms 2797ms 2839ms 2778ms 2896ms 2752ms “Looks ok”
  13. “Do stuff” start stop “repeat, rinse” You could be “ok”:

    I/O, threads, complex call graphs, …
  14. Static compilation Guess tricks at compilation time (C, C++, Go,

    etc)
  15. Interpretation Just interpret from bytecode / AST (Python, Ruby, etc)

  16. Dynamic compilation Guess tricks during execution (Java, Scala, JavaScript/V8, etc)

    ! ! Hybrid interpreter + JIT (HotSpot) “Just JIT” (V8, JRockit)
  17. Dynamic compilation …is a (micro)benchmark killer.

  18. Pitfalls in action

  19. static double distance(double x1, double y1, double x2, double y2)

    { double dx = x2 - x1; double dy = y2 - y1; return Math.sqrt((dx * dx) + (dy * dy)); } How fast?
  20. Demo (a not so naïve framework)

  21. Dead-code elimination Constant folding Loop unrolling On-stack replacement Inlining Locks

    removal (…) https://wiki.openjdk.java.net/display/HotSpot/PerformanceTechniques
  22. Caliper Google ! JUnitBenchmarks CarrotLabs ! JMH OpenJDK ✔ ⨉

  23. JMH to the rescue

  24. “JMH is a Java harness for building, running, and analysing

    nano/micro/milli/ macro benchmarks written in Java and other languages targeting the JVM.”
  25. Helps defeating unwanted optimisations.

  26. Demo (JMH in anger)

  27. None
  28. None
  29. A few examples Micro-benchmarks from Golo

  30. None
  31. None
  32. None
  33. None
  34. None
  35. None
  36. JMH is great, but: ! • doubt • peer review

    • beware HotSpot • results don’t compose • read all Aleksey’s samples
  37. (the end)

  38. Julien Ponge @jponge ! https://julien.ponge.org/ http://dynamid.citi-lab.fr/ http://golo-lang.org/