Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Your benchmark may not guide real application performance

Your benchmark may not guide real application performance

JSConfJP 2019

9e35c2a485ae27857628ab78df10e407?s=128

Tetsuharu Ohzeki

December 01, 2019
Tweet

Transcript

  1. Your benchmark may not guide real application performance Tetsuharu OHZEKI

  2. Performance is always hot topic • We love fast software!

    • Low latency, high throughput, power efficient, fast response… • Would you like to use slow software? • Have you ever see that people saying “many features are important than speed” loves slow software actually? • No!
  3. Performance is important for • User Experience • One of

    the fundamental value of software • Marketing value • New comer sometimes beat down other products by performance • In 2008, Google Chrome beat down other browsers
  4. Make software performant • Random optimization does not contribute to

    actual user experience • “Don’t guess, measure” is always right • We can use benchmark to measure our software
  5. Benchmark • Score software performance as Quantitive value • i.e.

    normalize software performance by benchmark • Reproducibility is important • Keep our application faster from regression • We use benchmark to evaluate our application performance
  6. Questions • Does your benchmark is really make sense? •

    Does your benchmark scores real application scenario actually?
  7. Goals • Show that benchmarking by real scenario is
 important

    principle to make your application faster • Introduce (pitfall) case studies related to benchmarks
  8. Outline 1. Introduction 2. What should we focus? ⬅ 3.

    JS cost is difficult 4. Critical path may be hidden 5. How to improve performance? 6. Conclusion
  9. Work on video-streaming service… • In this case, performance key

    is when start to play video • What is meaningful metric? • First Meaningful Paint/First Content Paint is nice • Time to Interactive? • Is it meaningful for this service actually? • Think about the bad case that page is responsible but video is not started
  10. General metrics may not suite special case • General metrics

    is useful to measure performance of general web page • e.g. Startup time • But general metrics cannot catch up application specific performance • Measure real scenario for your application • What is your application doing? • What is purpose?
  11. Lesson • Performance Metrics is not simple • General Purpose

    • Application specific • We should think about what performance metrics is most suitable • Not only Lighthouse! • We should focus actual scenario that our application will do
  12. Outline 1. Introduction 2. What should we focus? 3. JS

    cost is difficult ⬅ 4. Critical path may be hidden 5. How to improve performance 6. Conclusion
  13. When you optimize your code… • I’d like to optimize

    my slow code! • But the running time/ops is pretty small… (e.g. ~0.1ms/ops) • I cannot find a difference! • I have nice idea. Run this code 10000 times • The result will be stretched! Easy to compare! • …Wait! Is this nice approach really?

  14. JSVM has multiple tiers • JSVM has multiple tier to

    optimize user code • e.g. JavaScriptCore has 4 tier (LLInt, Baseline, DFG, FTL) • JIT compiler change optimization level speculatively by how much you code run • Hot path (executed frequently) would be heavily optimized • Cold path (executed rarely) would be less optimized
  15. Hot loop may not be what your application do actualy

    • Typical micro-benchmark execute many iterations to stabilize results • But many iteration would make functions compiled with heavy optimizations by highest JIT tier • If your actual workload is run only several times, many iteration leads a different results from what you expected • Let’s see execution time changes of some cases from JetStream2
  16. Plot JetStream2/prepack-wtb execution times of each iteration (change to iteration=100)

    Running Time (ms) 150 300 450 600 Iteration Count 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58 61 64 67 70 73 76 79 82 85 88 91 94 97100 WebKit r252841 Chrome Canary 80.0.3976.0 Firefox 72.0a1 (20191128214853) Lower Tier Highest Tier
  17. Plot JetStream2/Air execution times of each iteration (iteration=120) Running Time

    (ms) 0 22.5 45 67.5 90 Iteration Count 1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 WebKit r252841 Chrome Canary 80.0.3976.0 Firefox 72.0a1 (20191128214853) Lower Tier Highest Tier
  18. Lesson • JSVM changes optimization levels by execution counts •

    Workload may changes your benchmark score • Be careful to profile on actual workload as possible • Invalid assumption mislead your optimization strategy • By misleading, your application might go wrong…
  19. Outline 1. Introduction 2. What should we focus? 3. JS

    cost is difficult 4. Critical path may be hidden⬅ 5. How to improve performance 6. Conclusion
  20. I tried to improve the page load time… • Add

    ‘defer’ attribute to <script/> to improve the overall page init speed • But it did not improve the first meaningful paint. Why? cite: https://docs.google.com/presentation/d/1MXlFGqFQFJByv8k6Ege0pt0GwJQqbjoh7GdIYia9UQg/
  21. Before After • Achieved to improve
 sub-resource loading • But

    no improvement for
 the critical path cite: https://docs.google.com/presentation/d/1MXlFGqFQFJByv8k6Ege0pt0GwJQqbjoh7GdIYia9UQg/
  22. Why? • The critical path depends on a “bootstrap” script

    which starts working on DOMContentLoaded • script[defer] does not change this behavior • This “bootstrap” script is small size and fast execution • The profiler does not show up it as a “bottleneck” point easily cite: https://docs.google.com/presentation/d/1MXlFGqFQFJByv8k6Ege0pt0GwJQqbjoh7GdIYia9UQg/
  23. (Unfortunately) Finding bottlenecks is hard •Using several tools is better

    for crosscutting analyzing bottleneck • But be careful, profiler sometimes shows unrelated values • It often requires domain specific knowledge • How your application works? • Is it a real bottleneck? • Performance Tracing for Tasks • Causal Profling [Curtsinger+, SOSP ‘15] (Virtual Speedup)
 cite: https://docs.google.com/presentation/d/1MXlFGqFQFJByv8k6Ege0pt0GwJQqbjoh7GdIYia9UQg/
  24. Benchmark Site for Networking • Firefox results slower than Chrome’s

    one on same devices • https://bugzilla.mozilla.org/show_bug.cgi?id=1556022 • https://bugzilla.mozilla.org/show_bug.cgi?id=1570313 • This means simply that “Firefox network stack is slow"? • We tend to think so. Really?
  25. What did this benchmark measure in Firefox? https://twitter.com/hsivonen/status/1179763669535805441

  26. • This benchmark caused many translation from utf8 -> utf16

    • This site use XMLHttpRequest but its responseType is text for download test • Why not use “.responseType=arraybuffer”? • In worst case, this waste 59% of overall processing time in paint phase • Fancy animation caused performance issue that is not related to networking! What did this benchmark measure in Firefox?
  27. • This benchmark caused many translation from utf8 -> utf16

    • This site use XMLHttpRequest but its responseType is text for download test • Why not use “.responseType=arraybuffer”? • In worst case, this waste 59% of overall processing time in paint phase • Fancy animation caused performance issue that is not related to networking! What did this benchmark measure in Firefox?
  28. Lesson • Critical path is important but they might be

    hidden • Profiler might not shown them • There are may be problem which you cannot control • Improve your application actually, insight for your application specific behavior is most important • Breakdown bottlenecks with various tools & knowledge
  29. Outline 1. Introduction 2. What should we focus? 3. JS

    cost is difficult 4. Critical path may be hidden 5. How to improve performance⬅ 6. Conclusion
  30. Use benchmark to keep your app faster • “The way

    to make a program faster is to never let get it slower” • https://webkit.org/performance/ • Let’s benchmark your application continuously, and plot results, per commit
  31. Use benchmark to keep your app faster • Focus long

    term Trend • Each of score may bit change randomly by others • Other OS’ services, other guests on hypervisor, and more • Reproducible Infrastructure is important to test again
  32. Outline 1. Introduction 2. What should we focus? 3. JS

    cost is difficult 4. Critical path may be hidden 5. How to improve performance 6. Conclusion ⬅
  33. Conclusions • Real scenario guide what you should improve performance

    • Analyze perf issues deeply with tools & your app specific knowledge • CI is nice to keep performance through iteration cycles • First step: Benchmark your application based on your story