Parallel testing with Ractors: putting CPUs to work

Parallelizing tests is an opportune way of reducing the total runtime for a test suite. Rails achieves this by forking multiple separate workers that fetch tests from a queue. In Ruby 3, Ractors introduced new mechanisms for executing code in parallel. Can they be leveraged by a test framework? And how would that compare to current parallelization solutions?

Let’s find the answers to these questions by building a test framework built on Ractors, from scratch. We’ll compare the current solutions for parallelization and what advantages or limitations Ractors bring when used in this context.

Transcript

1. Vinicius Stock Parallel testing with Ractors Putting CPUs to work
2. Vinicius “Vini” Stock Senior developer @ Shopify Ruby Types team Twitter & GitHub: @vinistock https://vinistock.com
3. RubyConf 2020 Ractor demonstration
4. Test execution strategies Introduction to Ractors Build a test framework Parallelize using Ractors Highlights
5. How do we execute tests?
6. Figure out which tests to run Separate tests into groups CPU 1 Run test group #1 CPU 2 Run test group #2 CPU 3 Run test group #3 CPU 4 Run test group #4 Execution strategy #1 - Groups Simple Doesn’t guarantee even distribution
7. CPU 1 CPU 2 Execution strategy #1 Unit test #1 Unit test #2 Unit test #3 Integration test #1 Unit test #4 Integration test #2 CPU 2 is idle! Unit test #5
8. Figure out which tests to run Randomize and organize tests in a queue Test 1, test 2, test 3, test 4 CPU 1 CPU 2 CPU 3 CPU 4 Execution strategy #2 - Queue
9. CPU 1 Queue Execution strategy #2 PostTest#test_title PostTest#test_author AuthorTest#test_name PostTest#test_score AuthorTest#test_bio CPU 2 PostTest#test_title PostTest#test_author AuthorTest#test_name PostTest#test_score
10. Ruby did not have full support for parallelism How do we parallelize tests right now?
11. Existing strategy for parallelizing tests https://github.com/rails/rails
12. Rails Minitest plugin Queue object 
 Test 1, test 2, test 3, test 4 PROCESS 1 PROCESS 2 PROCESS 3 PROCESS 4 DRb
13. Ruby 3.0 - Ractors How do they compare to processes?
14. Ractors - Communication
15. Ractors - Worker pool example
16. Test frameworks Execution Utilities and syntax Reporting
17. Test frameworks - execution • Regular classes • Keep track of them • Run tests on each one
18. Test frameworks - utilities • Define the syntax • Assist execution flow • Delegate information • Provide helpers
19. Test frameworks - reporting • Display progress • Display aggregated results
20. Test frameworks Let’s implement it!
21. Test frameworks - execution How do we keep track of classes?
22. Test frameworks - execution How do we model the test queue?
23. Test frameworks - execution
24. Test frameworks - execution PostTest :test_author #<PostTest:0x0…> setup send(:test_author) teardown
25. Test frameworks - execution Basic execution is complete
26. Test frameworks - utilities & reporting assert(something) Passed continue Failed Skip to next item in the queue Save failure info Reporter
27. Test frameworks - utilities & reporting Print failure F
28. Test frameworks - utilities & reporting PostTest :test_author #<PostTest:0x0…> Increment total tests setup send(:test_author) teardown Increment successful tests Print success dot
29. Test frameworks - execution steps We got the foundation Let’s run it
30. Test frameworks - execution
31. Test frameworks - execution steps We have our basic test framework
32. Parallelizing our framework
33. Parallelizing our framework We can’t use a singleton pattern for the reporter
34. Parallelizing our framework
35. Parallelizing our framework We parallelized our framework!
36. https://github.com/vinistock/loupe
37. Ractors vs Processes
38. Highlights Can only test Ractor compatible code Can’t make broad usage of class variables Can’t make broad usage of meta-programming
39. Highlights Can test code that spawns more Ractors Can run our tests in parallel Doesn’t pay initial cost of forking processes
40. Seg fault on GC sweep + yield https://bugs.ruby-lang.org/issues/18117 Seg fault when freeing classes created inside Ractors https://bugs.ruby-lang.org/issues/18119 Dead lock when using autoload https://bugs.ruby-lang.org/issues/18120 Bug hunt! Seg fault on GC sweep + yield https://bugs.ruby-lang.org/issues/18117
41. Ractors will continue to evolve
42. Thank you! Twitter & GitHub: @vinistock https://vinistock.com