Parallel testing world - Speaker Deck

Slide 1

Slide 1 text

Parallel Testing World Shota Fukumori (sora_h) at RubyConf 2011

Slide 2

Slide 2 text

Hi • I’m from Japan.

Slide 3

Slide 3 text

I’m Shota Fukumori A ruby committer.

Slide 4

Slide 4 text

Also known as sora_h

Slide 5

Slide 5 text

Committer ranking as of today

Slide 6

Slide 6 text

Committer ranking as of today me

Slide 7

Slide 7 text

Committer ranking as of today me The patch monster (also known as nobu)

Slide 8

Slide 8 text

Committer ranking as of today me The patch monster (also known as nobu) Where is matz?

Slide 9

Slide 9 text

Committer ranking as of today me The patch monster (also known as nobu) No matz.

Slide 10

Slide 10 text

Twitter @sora_h

Slide 11

Slide 11 text

Sarah Mei (not me)

Slide 12

Slide 12 text

http://sorah.jp/

Slide 13

Slide 13 text

DISCLAIMER • I’ll talk about parallel testing. • My English skill is unknown. • If you want to listen to general news around ruby 1.9.3, I strongly recommend you to move to Room 2. • Room 2 14:10: “Implementation of Ruby 1.9.3 and later” by ko1

Slide 14

Slide 14 text

DISCLAIMER • All “Ruby” without any notes in this presentation mean “CRuby” (a.k.a. MRI).

Slide 15

Slide 15 text

I missed a schedule of talking at RubyKaigi • 15 minutes left when I ended a talk

Slide 16

Slide 16 text

I missed a preparing schedule today • I ﬁnished writing today’s presentation ﬁle yesterday’s night (10pm).

Slide 17

Slide 17 text

Agenda • Why Parallel Testing • Multi-Thread or Multi-Process • test/unit parallelization • How it works • Performance

Slide 18

Slide 18 text

Agenda • Why Parallel Testing • Multi-Thread or Multi-Process • test/unit parallelization • How it works • Performance

Slide 19

Slide 19 text

Beneﬁts • In most case, tests can run more faster • Faster test → Fast development cycle • In TDD, BDD development • At Ruby, committers have to run `make test-all` before commit to repository. • Faster test make developers happy :-)

Slide 20

Slide 20 text

Develop Test Push Deploy With slow test: Implemented!

Slide 21

Slide 21 text

Develop Test Push Deploy With slow test: Umm Slow tests....

Slide 22

Slide 22 text

Develop Test Push Deploy With slow test: --Break time--

Slide 23

Slide 23 text

Develop Test Push Deploy With slow test: Ugh test failed...

Slide 24

Slide 24 text

Develop Test Push Deploy With slow test: Fixed!

Slide 25

Slide 25 text

Develop Test Push Deploy With slow test: Umm Slow tests....

Slide 26

Slide 26 text

Develop Test Push Deploy With slow test: --Break time--

Slide 27

Slide 27 text

lol

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Develop Test Push Deploy With fast test: Implemented!

Slide 30

Slide 30 text

Develop Test Push Deploy With fast test: Um, some test failed...

Slide 31

Slide 31 text

Develop Test Push Deploy Fixed! With fast test:

Slide 32

Slide 32 text

Develop Test Push Deploy Tests passed, yay! With fast test:

Slide 33

Slide 33 text

Beneﬁts • You can ﬁx failure with short time

Slide 34

Slide 34 text

Beneﬁts • Q. You should run only few tests (like around the ﬁxed point) at once. • A. Some changes make some failures at another point. In ruby, running all test at each commits is recommended.

Slide 35

Slide 35 text

Example • Cookpad: http://cookpad.com/ • The most popular cooking recipe sharing site in Japan • 1.8.7 + Rails 2.3

Slide 36

Slide 36 text

Example • https://github.com/grosser/parallel_tests • Parallel Testing for RSpec

Slide 37

Slide 37 text

Example • Before (without parallel_tests): 16 minutes • After (with parallel_tests): 3.5 minutes

Slide 38

Slide 38 text

Example • Cookpad wrote a in-house library • rake cookpad:spec:remote

Slide 39

Slide 39 text

Example • rake cookpad:spec:remote • 4 servers: • Core i7 (8 threads) + 16GB RAM + SSD • = 32 threads (!)

Slide 40

Slide 40 text

Example • rake cookpad:spec:remote • Send local code to remote servers for running test with `rsync` • Run test via ssh. Outputs will be merged

Slide 41

Slide 41 text

Example •parallel_tests (at local machine): 3.5 minutes •cookpad:spec:remote : 50 seconds •19.2x faster

Slide 42

Slide 42 text

Example • This example is from: • http://www.slideshare.net/hotchpotch/ ruby01 Video: http://vimeo.com/22188723 • Sorry It’s Japanese...

Slide 43

Slide 43 text

Beneﬁts • Faster test, Faster development, Faster deployment.

Slide 44

Slide 44 text

Beneﬁts • So, How we can run test faster?

Slide 45

Slide 45 text

Beneﬁts • Use high power PC • Parallel running

Slide 46

Slide 46 text

Beneﬁts • Use high power PC • Parallel running

Slide 47

Slide 47 text

Why parallel testing • Modern machines have a multi (core| thread) • Use multi (core|thread) is the easiest way to run test faster!

Slide 48

Slide 48 text

Agenda • Why Parallel Testing • Multi-Thread or Multi-Process • test/unit parallelization • How it works • Performance

Slide 49

Slide 49 text

Multi-(thread|process) • Currently Ruby’s `Thread` doesn’t run parallel. • So, running multiple tests using `Thread` class in Ruby doesn’t make sense.

Slide 50

Slide 50 text

Multi-`Thread` ? • Ruby’s Thread can’t run multiple threads at once... Time Thread 1 Thread 2 Thread 3 ŋŋŋ Running Thread

Slide 51

Slide 51 text

Multi-Process ? • Multi process is good parallelization method at ruby. Time Test 2 Test 1 ŋŋŋ Core A Core B Test 1 Other 1 Other 2 Test 2 ↓Running process on each core Other: other process Test: testing process

Slide 52

Slide 52 text

Agenda • Why Parallel Testing • Multi-Thread or Multi-Process • test/unit parallelization • How it works • Performance

Slide 53

Slide 53 text

What’s test/unit • Unit testing library that shipped with Ruby • Before 1.9: test/unit doesn’t have any dependencies • After 1.9: test/unit is wrapper of minitest because of compatibility. • test/unit is used for Ruby’s unit testing.

Slide 54

Slide 54 text

test/unit parallelization • I wrote a patch that allows test/unit runs multiple tests at once • Because I wanted to make Ruby’s `make test-all` (make target to test ruby) more more faster! :p • Fast is important™

Slide 55

Slide 55 text

How it works • I’ll use the following words to describe how it works: • Master: a process that is started ﬁrst. This Process sends a instruction to workers. • Example: a ruby process that started by `make test-all` • Worker: a process that is started by Master process. This process runs tests.

Slide 56

Slide 56 text

How it works 1. Start master process (by like `make test- all`) 2. Master process starts worker processes 3. Master process sends a test ﬁle names to worker processes 4. Worker processes read a ﬁle that named passed by master process

Slide 57

Slide 57 text

How it works 5. Worker processes run a test. 6. Worker processes return a test result to master process. 7. Master process send another test ﬁle name to worker process 8. Do from step 4 again.

Slide 58

Slide 58 text

How it works • this feature uses stdin/stdout for communication between worker process and master process. • and this feature parallelize each test ﬁle. • so you have to separate tests and TestCase if you use this feature.

Slide 59

Slide 59 text

How it works Master Worker Worker Test ﬁles test_a.rb test_b.rb test_c.rb

Slide 60

Slide 60 text

How it works • User starts master process Master Test ﬁles test_a.rb test_b.rb test_c.rb

Slide 61

Slide 61 text

How it works • Master process starts worker processes using IO.popen Master Worker Worker Test ﬁles test_a.rb test_b.rb test_c.rb IO.popen

Slide 62

Slide 62 text

How it works • Master process sends ﬁle names of tests to worker process Master Worker Worker Test ﬁles test_a.rb test_b.rb test_c.rb test_a.rb test_b.rb

Slide 63

Slide 63 text

How it works • Worker process reads test file from file system. Master Worker Worker Test files test_a.rb test_b.rb test_c.rb

Slide 64

Slide 64 text

How it works • Worker process runs test Master Worker Worker Test ﬁles test_c.rb result result

Slide 65

Slide 65 text

How it works • Worker process sends result to Master process. master saves results from workers. Master Worker Worker Test ﬁles test_c.rb result result

Slide 66

Slide 66 text

How it works • Master sends remaining test ﬁle name. Master Worker Worker Test ﬁles test_c.rb result result test_c.rb

Slide 67

Slide 67 text

How it works • Worker process runs next test Master Worker Worker Test ﬁles test_c.rb result result

Slide 68

Slide 68 text

How it works • If there are no remaining test ﬁles, send quit instruction to worker Master Worker Worker Test ﬁles result result result Quit

Slide 69

Slide 69 text

How it works • If there are no remaining test ﬁles, send quit instruction to worker Master Worker Worker Test ﬁles result result result Bye

Slide 70

Slide 70 text

How it works • If there are no remaining test ﬁles, send quit instruction to worker Master Test ﬁles result result result

Slide 71

Slide 71 text

How it works • If there are any results that contains more than one failure, master will retry in default Master result result result I have a failure!

Slide 72

Slide 72 text

How it works • If there are any results that contains more than one failure, it’ll retry by master process in default Master result result test_b.rb Retry

Slide 73

Slide 73 text

How it works • If there are any results that contains more than one failure, it’ll retry by master process in default Master result result result There are no failures.

Slide 74

Slide 74 text

How it works • Because some tests fail because of parallelization. Master result result result There are no failures.

Slide 75

Slide 75 text

How it works • Then merge the results and show to user Master result result result User via STDOUT

Slide 76

Slide 76 text

This is • all of a mechanism of this this. • it works well, but...

Slide 77

Slide 77 text

In Ruby • some tests don’t consider a parallelization • it can guess (I guessed that) • The issues is: • Tests around network are using same port • Tests are using same directory

Slide 78

Slide 78 text

Real World Issues • test/ruby/test_signal.rb • test/ruby/test_process.rb • test/net/test_{http,https}.rb • test/csv/* • tests of rubygems

Slide 79

Slide 79 text

test_signal.rb • test around signal handling of ruby. • this test run Kernel#fork to check fork is implemented. • test/unit’s at_exit called each fork. • I fixed test/unit to make be runnable this test. • Patch description: Add a flag to test/unit and interrupt if a flag is true in at_exit.

Slide 80

Slide 80 text

test_process.rb • This test modiﬁes STDIN/OUT. • this feature uses STDIN/OUT, so I modiﬁed to duplicate STDIN/OUT before run test in worker process.

Slide 81

Slide 81 text

test_{http,https}.rb • These tests use same port number in test. • I ﬁxed port number; but a ﬁx is committed before I commit by Yui (nurse).

Slide 82

Slide 82 text

test/csv/* • Tests included in this directory use same temporally directory name • All testcases use same directory name and they deletes temporaly directory in `teardown` phase. • So...

Slide 83

Slide 83 text

test/csv/* Time test_csv_b.rb test_csv_a.rb Running Tests Test ﬁnished. Deleting the dir. Still Running...

Slide 84

Slide 84 text

test/csv/* Time test_csv_b.rb test_csv_a.rb Running Tests Deleted! “No such ﬁles or directory”

Slide 85

Slide 85 text

test/csv/* • Renamed temporary directory names to be unique each tests in test/csv/*.

Slide 86

Slide 86 text

test/rubygems/* • Recently they forget writing dependencies of test. • It makes problem at parallel_test. • because... ok, let’s show a example. Note: “dependency” is a ﬁle should be required.

Slide 87

Slide 87 text

test/rubygems/* • if there are test_a.rb, test_b.rb, and foo.rb. • also test_a.rb and test_b.rb depends on foo.rb. • test_a.rb includes `require ‘foo’`, •but test_b.rb doesn’t.

Slide 88

Slide 88 text

test/rubygems/* • When not using parallel testing: Time test_a.rb test_b.rb Running Test foo.rb is required. no errors. require ‘foo’

Slide 89

Slide 89 text

test/rubygems/* • When parallel testing: Time test_a.rb test_b.rb Running Tests require ‘foo’ Ugh, foo.rb is not required... Exception ocucred...

Slide 90

Slide 90 text

test/rubygems/* • I’m ﬁxing that if I found. • This case can be found in any tests, but I founded only in the rubygems test. • I don’t know why. :p

Slide 91

Slide 91 text

test/rubygems/* @rubygems_developers: please merge r33232 (in ruby-repo) https://github.com/rubygems/rubygems/issues/180 or http://bit.ly/rubyconf2011_rubygems_issue

Slide 92

Slide 92 text

Agenda • Why Parallel Testing • Multi-Thread or Multi-Process • test/unit parallelization • How it works • Performance

Slide 93

Slide 93 text

Performance • Kenta Murata a.k.a. @mrkn measured and created a performance graphs for me. • He’ll talk at the last session of room 2, about the ruby’s number system. • Thanks!

Slide 94

Slide 94 text

Performance • Measured at the following environment: • OS: Mac OS X 10.6.6 • CPU: Intel Core i7 2.66 GHz • (2 cores 4 threads) • Memory: 8GB 1067 MHz DDR3 • Test ﬁles is from ruby’s repository

Slide 95

Slide 95 text

Performance 0 32.5 65.0 97.5 130.0 single 1 2 3 5 8 13 Elapsed time Seconds worker(s)

Slide 96

Slide 96 text

Performance • Without parallelization (“no -j”): 121.49 seconds • Enabled parallelization with 5 workers: 43.41 seconds

Slide 97

Slide 97 text

Performance • Without parallelization (“no -j”): 121.49 seconds • Enabled parallelization with 5 workers: 43.41 seconds 2.79x Faster!

Slide 98

Slide 98 text

How to use • Separate TestCase-s to multiple ﬁle. • Write script to run multiple test ﬁle using Test::Unit::AutoRunner. • good example is ruby’s test/runner.rb in ruby’s repository. • Run the script with argument -j N. • Running the script with --help provides more information.

Slide 99

Slide 99 text

The patch made me • A committer! • I was very happy then.

Slide 100

Slide 100 text

That’s all. Thank you!

Slide 101

Slide 101 text

Announcement • Ruby 1.9.3 RC1 has been released! • You can use parallelization by installing this! • Let’s try it and tell us if you found a bug. • More about: http://www.ruby-lang.org/

Slide 102

Slide 102 text

Any Questions? If I can’t answer to your question because of my English skill, please send tweet to @sora_h or mail me: [email protected].