Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Parallel testing with Ractors: putting CPUs to work

Parallel testing with Ractors: putting CPUs to work

Parallelizing tests is an opportune way of reducing the total runtime for a test suite. Rails achieves this by forking multiple separate workers that fetch tests from a queue. In Ruby 3, Ractors introduced new mechanisms for executing code in parallel. Can they be leveraged by a test framework? And how would that compare to current parallelization solutions?

Let’s find the answers to these questions by building a test framework built on Ractors, from scratch. We’ll compare the current solutions for parallelization and what advantages or limitations Ractors bring when used in this context.

8f5c3bc85a6a4ce29e0ec1801ba8e673?s=128

Vinicius Stock

September 20, 2021
Tweet

Transcript

  1. Vinicius Stock Parallel testing with Ractors Putting CPUs to work

  2. Vinicius “Vini” Stock Senior developer @ Shopify Ruby Types team

    Twitter & GitHub: @vinistock https://vinistock.com
  3. RubyConf 2020 Ractor demonstration

  4. Test execution strategies Introduction to Ractors Build a test framework

    Parallelize using Ractors Highlights
  5. How do we execute tests?

  6. Figure out which tests to run Separate tests into groups

    CPU 1 Run test group #1 CPU 2 Run test group #2 CPU 3 Run test group #3 CPU 4 Run test group #4 Execution strategy #1 - Groups Simple Doesn’t guarantee even distribution
  7. CPU 1 CPU 2 Execution strategy #1 Unit test #1

    Unit test #2 Unit test #3 Integration test #1 Unit test #4 Integration test #2 CPU 2 is idle! Unit test #5
  8. Figure out which tests to run Randomize and organize tests

    in a queue Test 1, test 2, test 3, test 4 CPU 1 CPU 2 CPU 3 CPU 4 Execution strategy #2 - Queue
  9. CPU 1 Queue Execution strategy #2 PostTest#test_title PostTest#test_author AuthorTest#test_name PostTest#test_score

    AuthorTest#test_bio CPU 2 PostTest#test_title PostTest#test_author AuthorTest#test_name PostTest#test_score
  10. Ruby did not have full support for parallelism How do

    we parallelize tests right now?
  11. Existing strategy for parallelizing tests https://github.com/rails/rails

  12. Rails Minitest plugin Queue object 
 Test 1, test 2,

    test 3, test 4 PROCESS 1 PROCESS 2 PROCESS 3 PROCESS 4 DRb
  13. Ruby 3.0 - Ractors How do they compare to processes?

  14. Ractors - Communication

  15. Ractors - Worker pool example

  16. Test frameworks Execution Utilities and syntax Reporting

  17. Test frameworks - execution • Regular classes • Keep track

    of them • Run tests on each one
  18. Test frameworks - utilities • Define the syntax • Assist

    execution flow • Delegate information • Provide helpers
  19. Test frameworks - reporting • Display progress • Display aggregated

    results
  20. Test frameworks Let’s implement it!

  21. Test frameworks - execution How do we keep track of

    classes?
  22. Test frameworks - execution How do we model the test

    queue?
  23. Test frameworks - execution

  24. Test frameworks - execution PostTest :test_author #<PostTest:0x0…> setup send(:test_author) teardown

  25. Test frameworks - execution Basic execution is complete

  26. Test frameworks - utilities & reporting assert(something) Passed continue Failed

    Skip to next item in the queue Save failure info Reporter
  27. Test frameworks - utilities & reporting Print failure F

  28. Test frameworks - utilities & reporting PostTest :test_author #<PostTest:0x0…> Increment

    total tests setup send(:test_author) teardown Increment successful tests Print success dot
  29. Test frameworks - execution steps We got the foundation Let’s

    run it
  30. Test frameworks - execution

  31. Test frameworks - execution steps We have our basic test

    framework
  32. Parallelizing our framework

  33. Parallelizing our framework We can’t use a singleton pattern for

    the reporter
  34. Parallelizing our framework

  35. Parallelizing our framework We parallelized our framework!

  36. https://github.com/vinistock/loupe

  37. Ractors vs Processes

  38. Highlights Can only test Ractor compatible code Can’t make broad

    usage of class variables Can’t make broad usage of meta-programming
  39. Highlights Can test code that spawns more Ractors Can run

    our tests in parallel Doesn’t pay initial cost of forking processes
  40. Seg fault on GC sweep + yield https://bugs.ruby-lang.org/issues/18117 Seg fault

    when freeing classes created inside Ractors https://bugs.ruby-lang.org/issues/18119 Dead lock when using autoload https://bugs.ruby-lang.org/issues/18120 Bug hunt! Seg fault on GC sweep + yield https://bugs.ruby-lang.org/issues/18117
  41. Ractors will continue to evolve

  42. Thank you! Twitter & GitHub: @vinistock https://vinistock.com