Six Years of Ruby Performance: A History

Ruby keeps getting faster. And people keep asking, “but how fast is it for Rails?” Rails makes a great way to measure Ruby’s speed, and how Ruby has changed version-by-version. Let’s look at six years of performance for apps big and small.

How fast is 2.6.0? With JIT or not? How do I measure? How close is Ruby 3x3? Should I upgrade?

Transcript

1. Six Years of Ruby
   Performance History
   But How to Measure...?
   

   View Slide

2. Have You Seen the Toy Museum?
   Brighton's fun. If you
   haven't seen the toy
   and model museum,
   it's right under the
   train station.
   

   View Slide

3. I Love Questions - Seriously
   Am I speaking too fast? Tell me.
   Methodology question? Speak up.
   I can’t always see your hands, so just ask.
   Separately: AppFolio pays me to do Ruby stuff.

   Thank you, AppFolio!
   

   View Slide

4. Ruby 3x3
   In Japan, Matz
   spoke about Ruby
   3x3 at RubyKaigi.
   He said the speed is
   doing well.
   

   View Slide

5. Will JIT Fix It?
   With MJIT, Ruby 2.6 is 280% the speed of
   2.0.0. We’re nearly done!
   

   View Slide

6. Except...
   But JIT doesn’t help with Rails. With
   MJIT, Rails is slower than without it.
   MJIT will get a little faster, but...
   

   View Slide

7. Ruby or Rails?
   I’m going to look at using Rails to
   measure Ruby speed because… Rails is
   important code that is slow right now.
   

   View Slide

8. Old Way: Measure with RRB
   Rails Ruby Bench answers “how fast is
   Ruby for a real-world, production Rails
   app?”
   

   View Slide

9. View Slide

10. View Slide

11. View Slide

12. How Does RRB Measure?
    Rails Ruby Bench uses a highly
    concurrent simulated real world workload
    on a real world forum app called
    Discourse.
    

    View Slide

13. Throughput, Not Latency
    RRB uses 10 processes and 60 threads,
    based on measurements showing that to be
    the fastest configuration. We'll talk more
    about that later.
    

    View Slide

14. "Real World", "Production"
    Us: It’s about 172% faster. But the times
    are all mixed together. Which part is slow?
    RRB:
    

    View Slide

15. "Real World", "Production"
    Us: It’s about 172% faster. But the times
    are all mixed together. Which part is slow?
    RRB: ... "
    

    View Slide

16. Micro vs Macro Benchmarks
    Benchmarks come in many
    sizes.
    RRB is very large.
    

    View Slide

17. Micro/Macro
    Small benchmarks are specific.
    Large ones are representative.
    

    View Slide

18. Rails Ruby Bench is Macro
    RRB answers “how fast is this workload?”
    but not “which part is slow?”
    

    View Slide

19. This is a Job For...
    To explore specific questions, we'll use a
    smaller benchmark. What should that
    benchmark be like?
    

    View Slide

20. RSB == Rails Simpler Bench
    RSB should make it easy to time various
    operations. But not mixed into one single
    timing like RRB.
    

    View Slide

21. Routes for Exploration
    We’ll begin exploring with a trivial “hello,
    world” route that returns a static string.
    

    View Slide

22. Specific
    We’ll start with a single-process, single-
    thread concurrency model. Simple, with
    low latency.
    

    View Slide

23. Specificity
    So I built it. Let's see what it says about a
    few different questions.
    

    View Slide

24. First: Rails Overhead
    Rails takes some time to run for each
    request. A static “Hello, World” route is
    just benchmarking Rails overhead. What
    does it find?
    

    View Slide

25. View Slide

26. First Off... WHAT?
    Remember how with RRB, half the gains
    came from Ruby 2.1 and 2.2? This doesn’t
    look like that at all!
    

    View Slide

27. Wait, How Much?
    Also in this graph Ruby 2.6 gained only
    about 30% over 2.0.0p0, and not 70%+.
    What?
    

    View Slide

28. Oh. Wait.
    We’re not measuring just Ruby framework
    code yet. We’re also measuring the app
    server (Puma.)
    

    View Slide

29. Framework Time, App Time
    To measure Rails framework time versus
    the app server, I'll build a quick Rack app
    and measure it.
    

    View Slide

30. View Slide

31. And That's Why They Say...
    This is why Rails has a special API mode.
    Normal Rails has a lot of overhead
    compared to Rack.
    

    View Slide

32. And Now, the Final Total
    If we turn reqs/sec into secs/req and
    subtract, we get Rails' overhead per
    request.
    

    View Slide

33. View Slide

34. Ruby 3x3 Means Concurrency
    We've looked at framework overhead.
    What about concurrency? Is concurrency
    improving?
    

    View Slide

35. Exploration: Concurrency
    On a multicore machine we get better
    throughput with more processes and
    threads. But how many?
    

    View Slide

36. Maximum CRuby Throughput
    We’ll look for a configuration which will
    use a GIL-friendly number of processes
    and threads.
    

    View Slide

37. The Experiment
    I set up RSB to run with varying numbers
    of processes and threads on Ruby from 2.0
    to 2.6. For instance, 4 processes beats 4
    threads...
    

    View Slide

38. View Slide

39. Threads Bad?
    In fact, sometimes threads are clearly
    worse than no threads. Let's look at 4
    processes - with only 1 thread/process
    versus with 8.
    

    View Slide

40. View Slide

41. Sometimes?
    With more processes, threads help... But
    only a little. Here's 8 processes, with 1
    thread versus 4 threads.
    

    View Slide

42. View Slide

43. But Why?
    This is a workload difference between
    RRB and RSB. RRB has lots of non-Ruby
    work, and lots of context switches.
    

    View Slide

44. CRuby Threads, CRuby GIL
    CRuby only benefits from threads for
    non-Ruby time - database, I/O, C
    extensions, etc. RSB is nearly 100% Ruby
    work.
    

    View Slide

45. One More Observation
    On each previous
    graph, from left to
    right, each row does
    roughly the same
    thing...
    

    View Slide

46. So...
    That means each
    Ruby is handling
    concurrency the
    same way.
    

    View Slide

47. Is There a Win Here?
    One reminder: if we could run Ruby
    concurrently (with Guilds?) then we might
    be able to get far more speed from this
    benchmark.
    

    View Slide

48. Rails and MJIT
    We know MJIT isn't helping for Rails yet.
    But is it coming closer? Let's check.
    That means comparing 2.6 performance to
    prerelease Ruby 2.7 performance.
    

    View Slide

49. Rails w/ MJIT - Getting Closer
    Ruby 2.6 vs recent
    2.7 - with 2.6,
    MJIT is 83% the
    speed. With 2.7, it's
    94%.
    

    View Slide

50. Rack w/ MJIT - No Change
    Ruby 2.6, Rack w/
    MJIT is 97% the
    speed. With 2.7, it's
    also 97%.
    

    View Slide

51. What's Next?
    There’s a lot more I can benchmark with
    RSB, and I do. I post the results to
    engineering.appfolio.com. Future
    comparisons will include app servers,
    memory allocators and floating point.
    

    View Slide

52. Methodology
    RSB is pretty methodologically solid,
    though the code is new. This isn’t a 100%
    full description, but...
    

    View Slide

53. Methodology Misc
    Dedicated EC2 Linux Instance
    Localhost, not TCP
    Multiple consecutive batches,
    random order
    One route/concurrency config
    per batch
    Many requests/batch
    No Redis or cache
    No destructive routes
    Fail batch > 0.01% error
    Configurable warmups
    Low-overhead C load tester
    (wg/wrk)
    Rails 4.2 for Ruby 2.0 ->
    2.6+ compatibility
    

    View Slide

54. The Code, The Data
    RSB Code:
    https://github.com/noahgibbs/rsb
    Data for these graphs:
    https://github.com/noahgibbs/rrb_datavis
    

    View Slide

55. Questions?
    Comments? Objections?
    Recitations? Show
    Tunes?
    Should you discover an important question you
    missed, I can be reached as @codefolio on
    Twitter or the.codefolio.guy@gmail.com as well.
    I love talking Ruby!
    These slides: http://bit.ly/brighton2019-gibbs
    

    View Slide