Parallel testing world

by Shota Fukumori at the RubyConf2011 second day, 14:10-

Transcript

1. Parallel Testing World Shota Fukumori (sora_h) at RubyConf 2011
2. Hi • I’m from Japan.
3. I’m Shota Fukumori A ruby committer.
4. Also known as sora_h
5. Committer ranking as of today
6. Committer ranking as of today me
7. Committer ranking as of today me The patch monster (also known as nobu)
8. Committer ranking as of today me The patch monster (also known as nobu) Where is matz?
9. Committer ranking as of today me The patch monster (also known as nobu) No matz.
10. Twitter @sora_h
11. Sarah Mei (not me)
12. http://sorah.jp/
13. DISCLAIMER • I’ll talk about parallel testing. • My English skill is unknown. • If you want to listen to general news around ruby 1.9.3, I strongly recommend you to move to Room 2. • Room 2 14:10: “Implementation of Ruby 1.9.3 and later” by ko1
14. DISCLAIMER • All “Ruby” without any notes in this presentation mean “CRuby” (a.k.a. MRI).
15. I missed a schedule of talking at RubyKaigi • 15 minutes left when I ended a talk
16. I missed a preparing schedule today • I finished writing today’s presentation file yesterday’s night (10pm).
17. Agenda • Why Parallel Testing • Multi-Thread or Multi-Process • test/unit parallelization • How it works • Performance
18. Agenda • Why Parallel Testing • Multi-Thread or Multi-Process • test/unit parallelization • How it works • Performance
19. Benefits • In most case, tests can run more faster • Faster test → Fast development cycle • In TDD, BDD development • At Ruby, committers have to run `make test-all` before commit to repository. • Faster test make developers happy :-)
20. Develop Test Push Deploy With slow test: Implemented!
21. Develop Test Push Deploy With slow test: Umm Slow tests....
22. Develop Test Push Deploy With slow test: --Break time--
23. Develop Test Push Deploy With slow test: Ugh test failed...
24. Develop Test Push Deploy With slow test: Fixed!
25. Develop Test Push Deploy With slow test: Umm Slow tests....
26. Develop Test Push Deploy With slow test: --Break time--
27. lol
28. :(
29. Develop Test Push Deploy With fast test: Implemented!
30. Develop Test Push Deploy With fast test: Um, some test failed...
31. Develop Test Push Deploy Fixed! With fast test:
32. Develop Test Push Deploy Tests passed, yay! With fast test:
33. Benefits • You can fix failure with short time
34. Benefits • Q. You should run only few tests (like around the fixed point) at once. • A. Some changes make some failures at another point. In ruby, running all test at each commits is recommended.
35. Example • Cookpad: http://cookpad.com/ • The most popular cooking recipe sharing site in Japan • 1.8.7 + Rails 2.3
36. Example • https://github.com/grosser/parallel_tests • Parallel Testing for RSpec
37. Example • Before (without parallel_tests): 16 minutes • After (with parallel_tests): 3.5 minutes
38. Example • Cookpad wrote a in-house library • rake cookpad:spec:remote
39. Example • rake cookpad:spec:remote • 4 servers: • Core i7 (8 threads) + 16GB RAM + SSD • = 32 threads (!)
40. Example • rake cookpad:spec:remote • Send local code to remote servers for running test with `rsync` • Run test via ssh. Outputs will be merged
41. Example •parallel_tests (at local machine): 3.5 minutes •cookpad:spec:remote : 50 seconds •19.2x faster
42. Example • This example is from: • http://www.slideshare.net/hotchpotch/ ruby01 Video: http://vimeo.com/22188723 • Sorry It’s Japanese...
43. Benefits • Faster test, Faster development, Faster deployment.
44. Benefits • So, How we can run test faster?
45. Benefits • Use high power PC • Parallel running
46. Benefits • Use high power PC • Parallel running
47. Why parallel testing • Modern machines have a multi (core| thread) • Use multi (core|thread) is the easiest way to run test faster!
48. Agenda • Why Parallel Testing • Multi-Thread or Multi-Process • test/unit parallelization • How it works • Performance
49. Multi-(thread|process) • Currently Ruby’s `Thread` doesn’t run parallel. • So, running multiple tests using `Thread` class in Ruby doesn’t make sense.
50. Multi-`Thread` ? • Ruby’s Thread can’t run multiple threads at once... Time Thread 1 Thread 2 Thread 3 ŋŋŋ Running Thread
51. Multi-Process ? • Multi process is good parallelization method at ruby. Time Test 2 Test 1 ŋŋŋ Core A Core B Test 1 Other 1 Other 2 Test 2 ↓Running process on each core Other: other process Test: testing process
52. Agenda • Why Parallel Testing • Multi-Thread or Multi-Process • test/unit parallelization • How it works • Performance
53. What’s test/unit • Unit testing library that shipped with Ruby • Before 1.9: test/unit doesn’t have any dependencies • After 1.9: test/unit is wrapper of minitest because of compatibility. • test/unit is used for Ruby’s unit testing.
54. test/unit parallelization • I wrote a patch that allows test/unit runs multiple tests at once • Because I wanted to make Ruby’s `make test-all` (make target to test ruby) more more faster! :p • Fast is important™
55. How it works • I’ll use the following words to describe how it works: • Master: a process that is started first. This Process sends a instruction to workers. • Example: a ruby process that started by `make test-all` • Worker: a process that is started by Master process. This process runs tests.
56. How it works 1. Start master process (by like `make test- all`) 2. Master process starts worker processes 3. Master process sends a test file names to worker processes 4. Worker processes read a file that named passed by master process
57. How it works 5. Worker processes run a test. 6. Worker processes return a test result to master process. 7. Master process send another test file name to worker process 8. Do from step 4 again.
58. How it works • this feature uses stdin/stdout for communication between worker process and master process. • and this feature parallelize each test file. • so you have to separate tests and TestCase if you use this feature.
59. How it works Master Worker Worker Test files test_a.rb test_b.rb test_c.rb
60. How it works • User starts master process Master Test files test_a.rb test_b.rb test_c.rb
61. How it works • Master process starts worker processes using IO.popen Master Worker Worker Test files test_a.rb test_b.rb test_c.rb IO.popen
62. How it works • Master process sends file names of tests to worker process Master Worker Worker Test files test_a.rb test_b.rb test_c.rb test_a.rb test_b.rb
63. How it works • Worker process reads test file from file system. Master Worker Worker Test files test_a.rb test_b.rb test_c.rb
64. How it works • Worker process runs test Master Worker Worker Test files test_c.rb result result
65. How it works • Worker process sends result to Master process. master saves results from workers. Master Worker Worker Test files test_c.rb result result
66. How it works • Master sends remaining test file name. Master Worker Worker Test files test_c.rb result result test_c.rb
67. How it works • Worker process runs next test Master Worker Worker Test files test_c.rb result result
68. How it works • If there are no remaining test files, send quit instruction to worker Master Worker Worker Test files result result result Quit
69. How it works • If there are no remaining test files, send quit instruction to worker Master Worker Worker Test files result result result Bye
70. How it works • If there are no remaining test files, send quit instruction to worker Master Test files result result result
71. How it works • If there are any results that contains more than one failure, master will retry in default Master result result result I have a failure!
72. How it works • If there are any results that contains more than one failure, it’ll retry by master process in default Master result result test_b.rb Retry
73. How it works • If there are any results that contains more than one failure, it’ll retry by master process in default Master result result result There are no failures.
74. How it works • Because some tests fail because of parallelization. Master result result result There are no failures.
75. How it works • Then merge the results and show to user Master result result result User via STDOUT
76. This is • all of a mechanism of this this. • it works well, but...
77. In Ruby • some tests don’t consider a parallelization • it can guess (I guessed that) • The issues is: • Tests around network are using same port • Tests are using same directory
78. Real World Issues • test/ruby/test_signal.rb • test/ruby/test_process.rb • test/net/test_{http,https}.rb • test/csv/* • tests of rubygems
79. test_signal.rb • test around signal handling of ruby. • this test run Kernel#fork to check fork is implemented. • test/unit’s at_exit called each fork. • I fixed test/unit to make be runnable this test. • Patch description: Add a flag to test/unit and interrupt if a flag is true in at_exit.
80. test_process.rb • This test modifies STDIN/OUT. • this feature uses STDIN/OUT, so I modified to duplicate STDIN/OUT before run test in worker process.
81. test_{http,https}.rb • These tests use same port number in test. • I fixed port number; but a fix is committed before I commit by Yui (nurse).
82. test/csv/* • Tests included in this directory use same temporally directory name • All testcases use same directory name and they deletes temporaly directory in `teardown` phase. • So...
83. test/csv/* Time test_csv_b.rb test_csv_a.rb Running Tests Test finished. Deleting the dir. Still Running...
84. test/csv/* Time test_csv_b.rb test_csv_a.rb Running Tests Deleted! “No such files or directory”
85. test/csv/* • Renamed temporary directory names to be unique each tests in test/csv/*.
86. test/rubygems/* • Recently they forget writing dependencies of test. • It makes problem at parallel_test. • because... ok, let’s show a example. Note: “dependency” is a file should be required.
87. test/rubygems/* • if there are test_a.rb, test_b.rb, and foo.rb. • also test_a.rb and test_b.rb depends on foo.rb. • test_a.rb includes `require ‘foo’`, •but test_b.rb doesn’t.
88. test/rubygems/* • When not using parallel testing: Time test_a.rb test_b.rb Running Test foo.rb is required. no errors. require ‘foo’
89. test/rubygems/* • When parallel testing: Time test_a.rb test_b.rb Running Tests require ‘foo’ Ugh, foo.rb is not required... Exception ocucred...
90. test/rubygems/* • I’m fixing that if I found. • This case can be found in any tests, but I founded only in the rubygems test. • I don’t know why. :p
91. test/rubygems/* @rubygems_developers: please merge r33232 (in ruby-repo) https://github.com/rubygems/rubygems/issues/180 or http://bit.ly/rubyconf2011_rubygems_issue
92. Agenda • Why Parallel Testing • Multi-Thread or Multi-Process • test/unit parallelization • How it works • Performance
93. Performance • Kenta Murata a.k.a. @mrkn measured and created a performance graphs for me. • He’ll talk at the last session of room 2, about the ruby’s number system. • Thanks!
94. Performance • Measured at the following environment: • OS: Mac OS X 10.6.6 • CPU: Intel Core i7 2.66 GHz • (2 cores 4 threads) • Memory: 8GB 1067 MHz DDR3 • Test files is from ruby’s repository
95. Performance 0 32.5 65.0 97.5 130.0 single 1 2 3 5 8 13 Elapsed time Seconds worker(s)
96. Performance • Without parallelization (“no -j”): 121.49 seconds • Enabled parallelization with 5 workers: 43.41 seconds
97. Performance • Without parallelization (“no -j”): 121.49 seconds • Enabled parallelization with 5 workers: 43.41 seconds 2.79x Faster!
98. How to use • Separate TestCase-s to multiple file. • Write script to run multiple test file using Test::Unit::AutoRunner. • good example is ruby’s test/runner.rb in ruby’s repository. • Run the script with argument -j N. • Running the script with --help provides more information.
99. The patch made me • A committer! • I was very happy then.
100. That’s all. Thank you!
101. Announcement • Ruby 1.9.3 RC1 has been released! • You can use parallelization by installing this! • Let’s try it and tell us if you found a bug. • More about: http://www.ruby-lang.org/
102. Any Questions? If I can’t answer to your question because of my English skill, please send tweet to @sora_h or mail me: sorah@tubusu.net.