RubyConf - Parallel testing with Ractors

Parallelizing tests is an opportune way of reducing the total runtime for a test suite. Rails achieves this by forking multiple separate workers that fetch tests from a queue. In Ruby 3, Ractors introduced new mechanisms for executing code in parallel. Can they be leveraged by a test framework? And how would that compare to current parallelization solutions?

Let’s find the answers to these questions by building a test framework built on Ractors, from scratch. We’ll compare the current solutions for parallelization and what advantages or limitations Ractors bring when used in this context.

Transcript

1. Vinicius Stock Parallel testing with Ractors Putting CPUs to work
2. Vinicius “Vini” Stock Senior developer @ Shopify Ruby Types team Twitter & GitHub: @vinistock https://vinistock.com
3. RubyConf 2020 Ractor demonstration
4. Test execution strategies Introduction to Ractors Build a test framework Parallelize using Ractors Highlights
5. How do we execute tests?
6. Figure out which tests to run Separate tests into groups CPU 1 Run test group #1 CPU 2 Run test group #2 CPU 3 Run test group #3 CPU 4 Run test group #4 Execution strategy #1 - Groups Simple Doesn’t guarantee even distribution
7. CPU 1 CPU 2 Execution strategy #1 Unit test #1 Unit test #2 Unit test #3 Integration test #1 Unit test #4 Integration test #2 CPU 2 is idle! Unit test #5
8. Figure out which tests to run Randomize and organize tests in a queue Test 1, test 2, test 3, test 4 CPU 1 CPU 2 CPU 3 CPU 4 Execution strategy #2 - Queue
9. CPU 1 Queue Execution strategy #2 PostTest#test_title PostTest#test_author AuthorTest#test_name PostTest#test_score AuthorTest#test_bio CPU 2 PostTest#test_title PostTest#test_author AuthorTest#test_name PostTest#test_score
10. Figure out which tests to run Separate tests into groups CPU 1 Run test group #1 CPU 2 Run test group #2 CPU 3 Run test group #3 CPU 4 Run test group #4 Execution strategy #3 - Work stealing https://en.wikipedia.org/wiki/Work_stealing
11. CPU 1 CPU 2 Execution strategy #3 Unit test #1 Unit test #2 Unit test #3 Integration test #1 Unit test #4 Integration test #2 Unit test #5 Reduces communication overhead
12. Ruby did not have full support for parallelism How do we parallelize tests right now?
13. Existing strategy for parallelizing tests https://github.com/rails/rails
14. Rails Minitest plugin Queue object 
 Test 1, test 2, test 3, test 4 PROCESS 1 PROCESS 2 PROCESS 3 PROCESS 4 DRb
15. Ruby 3.0 - Ractors How do they compare to processes?
16. Ractors - Communication
17. Ractors - Worker pool example
18. Test frameworks Execution Utilities and syntax Reporting
19. Test frameworks - execution • Regular classes • Keep track of them • Run tests on each one
20. Test frameworks - utilities • Define the syntax • Assist execution flow • Delegate information • Provide helpers
21. Test frameworks - reporting • Display progress • Display aggregated results
22. Test frameworks Let’s implement it!
23. Test frameworks - execution How do we keep track of classes?
24. Test frameworks - execution How do we model the test queue?
25. Test frameworks - execution
26. Test frameworks - execution PostTest :test_author #<PostTest:0x0…> setup send(:test_author) teardown
27. Test frameworks - execution Basic execution is complete
28. Test frameworks - utilities & reporting assert(something) Passed continue Failed Skip to next item in the queue Save failure info Reporter
29. Test frameworks - utilities & reporting Print failure F
30. Test frameworks - utilities & reporting PostTest :test_author #<PostTest:0x0…> Increment total tests setup send(:test_author) teardown Increment successful tests Print success dot
31. Test frameworks - execution steps We got the foundation Let’s run it
32. Test frameworks - execution
33. Test frameworks - execution steps We have our basic test framework
34. Parallelizing our framework
35. Parallelizing our framework We can’t use a singleton pattern for the reporter
36. Parallelizing our framework
37. Parallelizing our framework We parallelized our framework!
38. https://github.com/vinistock/loupe
39. Ractors vs Processes
40. Highlights Can only test Ractor compatible code Can’t make broad usage of class variables Can’t make broad usage of meta-programming
41. Highlights Can test code that spawns more Ractors Can run our tests in parallel Doesn’t pay initial cost of forking processes
42. Seg fault on GC sweep + yield https://bugs.ruby-lang.org/issues/18117 Seg fault when freeing classes created inside Ractors https://bugs.ruby-lang.org/issues/18119 Dead lock when using autoload https://bugs.ruby-lang.org/issues/18120 Bug hunt! Seg fault on GC sweep + yield https://bugs.ruby-lang.org/issues/18117
43. Ractors will continue to evolve
44. Thank you! Twitter & GitHub: @vinistock https://vinistock.com