$30 off During Our Annual Pro Sale. View Details »

Parallel testing world

Parallel testing world

by Shota Fukumori at the RubyConf2011 second day, 14:10-

Sorah Fukumori

October 02, 2011
Tweet

More Decks by Sorah Fukumori

Other Decks in Technology

Transcript

  1. Parallel Testing World
    Shota Fukumori (sora_h)
    at RubyConf 2011

    View Slide

  2. Hi
    • I’m from Japan.

    View Slide

  3. I’m Shota Fukumori
    A ruby committer.

    View Slide

  4. Also known as
    sora_h

    View Slide

  5. Committer ranking as of today

    View Slide

  6. Committer ranking as of today
    me

    View Slide

  7. Committer ranking as of today
    me
    The patch monster
    (also known as nobu)

    View Slide

  8. Committer ranking as of today
    me
    The patch monster
    (also known as nobu)
    Where is matz?

    View Slide

  9. Committer ranking as of today
    me
    The patch monster
    (also known as nobu)
    No matz.

    View Slide

  10. Twitter @sora_h

    View Slide

  11. Sarah
    Mei
    (not me)

    View Slide

  12. http://sorah.jp/

    View Slide

  13. DISCLAIMER
    • I’ll talk about parallel testing.
    • My English skill is unknown.
    • If you want to listen to general news
    around ruby 1.9.3, I strongly recommend
    you to move to Room 2.
    • Room 2 14:10:
    “Implementation of Ruby 1.9.3 and later” by ko1

    View Slide

  14. DISCLAIMER
    • All “Ruby” without any notes in this
    presentation mean “CRuby” (a.k.a. MRI).

    View Slide

  15. I missed a schedule of
    talking at RubyKaigi
    • 15 minutes left when I ended a talk

    View Slide

  16. I missed a preparing
    schedule today
    • I finished writing today’s presentation file
    yesterday’s night (10pm).

    View Slide

  17. Agenda
    • Why Parallel Testing
    • Multi-Thread or Multi-Process
    • test/unit parallelization
    • How it works
    • Performance

    View Slide

  18. Agenda
    • Why Parallel Testing
    • Multi-Thread or Multi-Process
    • test/unit parallelization
    • How it works
    • Performance

    View Slide

  19. Benefits
    • In most case, tests can run more faster
    • Faster test → Fast development cycle
    • In TDD, BDD development
    • At Ruby, committers have to run `make
    test-all` before commit to repository.
    • Faster test make developers happy :-)

    View Slide

  20. Develop
    Test
    Push
    Deploy
    With slow test:
    Implemented!

    View Slide

  21. Develop
    Test
    Push
    Deploy
    With slow test:
    Umm Slow
    tests....

    View Slide

  22. Develop
    Test
    Push
    Deploy
    With slow test:
    --Break time--

    View Slide

  23. Develop
    Test
    Push
    Deploy
    With slow test:
    Ugh test failed...

    View Slide

  24. Develop
    Test
    Push
    Deploy
    With slow test:
    Fixed!

    View Slide

  25. Develop
    Test
    Push
    Deploy
    With slow test:
    Umm Slow
    tests....

    View Slide

  26. Develop
    Test
    Push
    Deploy
    With slow test:
    --Break time--

    View Slide

  27. lol

    View Slide

  28. :(

    View Slide

  29. Develop
    Test
    Push
    Deploy
    With fast test:
    Implemented!

    View Slide

  30. Develop
    Test
    Push
    Deploy
    With fast test:
    Um, some test
    failed...

    View Slide

  31. Develop
    Test
    Push
    Deploy
    Fixed!
    With fast test:

    View Slide

  32. Develop
    Test
    Push
    Deploy
    Tests passed,
    yay!
    With fast test:

    View Slide

  33. Benefits
    • You can fix failure with short time

    View Slide

  34. Benefits
    • Q. You should run only few tests (like
    around the fixed point) at once.
    • A. Some changes make some failures at
    another point.
    In ruby, running all test at each commits is
    recommended.

    View Slide

  35. Example
    • Cookpad: http://cookpad.com/
    • The most popular cooking recipe sharing
    site in Japan
    • 1.8.7 + Rails 2.3

    View Slide

  36. Example
    • https://github.com/grosser/parallel_tests
    • Parallel Testing for RSpec

    View Slide

  37. Example
    • Before (without parallel_tests):
    16 minutes
    • After (with parallel_tests):
    3.5 minutes

    View Slide

  38. Example
    • Cookpad wrote a in-house library
    • rake cookpad:spec:remote

    View Slide

  39. Example
    • rake cookpad:spec:remote
    • 4 servers:
    • Core i7 (8 threads) + 16GB RAM + SSD
    • = 32 threads (!)

    View Slide

  40. Example
    • rake cookpad:spec:remote
    • Send local code to remote servers for
    running test with `rsync`
    • Run test via ssh. Outputs will be merged

    View Slide

  41. Example
    •parallel_tests (at local machine):
    3.5 minutes
    •cookpad:spec:remote :
    50 seconds
    •19.2x faster

    View Slide

  42. Example
    • This example is from:
    • http://www.slideshare.net/hotchpotch/
    ruby01
    Video: http://vimeo.com/22188723
    • Sorry It’s Japanese...

    View Slide

  43. Benefits
    • Faster test,
    Faster development,
    Faster deployment.

    View Slide

  44. Benefits
    • So, How we can run test faster?

    View Slide

  45. Benefits
    • Use high power PC
    • Parallel running

    View Slide

  46. Benefits
    • Use high power PC
    • Parallel running

    View Slide

  47. Why parallel testing
    • Modern machines have a multi (core|
    thread)
    • Use multi (core|thread) is the easiest way
    to run test faster!

    View Slide

  48. Agenda
    • Why Parallel Testing
    • Multi-Thread or Multi-Process
    • test/unit parallelization
    • How it works
    • Performance

    View Slide

  49. Multi-(thread|process)
    • Currently Ruby’s `Thread` doesn’t run
    parallel.
    • So, running multiple tests using `Thread`
    class in Ruby doesn’t make sense.

    View Slide

  50. Multi-`Thread` ?
    • Ruby’s Thread can’t run multiple threads at
    once...
    Time
    Thread 1 Thread 2 Thread 3 ŋŋŋ
    Running Thread

    View Slide

  51. Multi-Process ?
    • Multi process is good parallelization
    method at ruby.
    Time
    Test 2
    Test 1
    ŋŋŋ
    Core A
    Core B Test 1
    Other 1 Other 2
    Test 2
    ↓Running process on each core
    Other: other process
    Test: testing process

    View Slide

  52. Agenda
    • Why Parallel Testing
    • Multi-Thread or Multi-Process
    • test/unit parallelization
    • How it works
    • Performance

    View Slide

  53. What’s test/unit
    • Unit testing library that shipped with Ruby
    • Before 1.9: test/unit doesn’t have any
    dependencies
    • After 1.9: test/unit is wrapper of minitest
    because of compatibility.
    • test/unit is used for Ruby’s unit testing.

    View Slide

  54. test/unit parallelization
    • I wrote a patch that allows test/unit runs
    multiple tests at once
    • Because I wanted to make Ruby’s `make
    test-all` (make target to test ruby) more
    more faster! :p
    • Fast is important™

    View Slide

  55. How it works
    • I’ll use the following words to describe how
    it works:
    • Master: a process that is started first. This
    Process sends a instruction to workers.
    • Example: a ruby process that started by `make test-all`
    • Worker: a process that is started by Master
    process. This process runs tests.

    View Slide

  56. How it works
    1. Start master process (by like `make test-
    all`)
    2. Master process starts worker processes
    3. Master process sends a test file names to
    worker processes
    4. Worker processes read a file that named
    passed by master process

    View Slide

  57. How it works
    5. Worker processes run a test.
    6. Worker processes return a test result to
    master process.
    7. Master process send another test file name
    to worker process
    8. Do from step 4 again.

    View Slide

  58. How it works
    • this feature uses stdin/stdout for
    communication between worker process
    and master process.
    • and this feature parallelize each test file.
    • so you have to separate tests and TestCase
    if you use this feature.

    View Slide

  59. How it works
    Master
    Worker
    Worker
    Test files
    test_a.rb
    test_b.rb
    test_c.rb

    View Slide

  60. How it works
    • User starts master process
    Master
    Test files
    test_a.rb
    test_b.rb
    test_c.rb

    View Slide

  61. How it works
    • Master process starts worker processes
    using IO.popen
    Master
    Worker
    Worker
    Test files
    test_a.rb
    test_b.rb
    test_c.rb
    IO.popen

    View Slide

  62. How it works
    • Master process sends file names of tests to
    worker process
    Master
    Worker
    Worker
    Test files
    test_a.rb
    test_b.rb
    test_c.rb
    test_a.rb
    test_b.rb

    View Slide

  63. How it works
    • Worker process reads test file from file
    system.
    Master
    Worker
    Worker
    Test files
    test_a.rb
    test_b.rb
    test_c.rb

    View Slide

  64. How it works
    • Worker process runs test
    Master
    Worker
    Worker
    Test files
    test_c.rb
    result
    result

    View Slide

  65. How it works
    • Worker process sends result to Master
    process. master saves results from workers.
    Master
    Worker
    Worker
    Test files
    test_c.rb
    result
    result

    View Slide

  66. How it works
    • Master sends remaining test file name.
    Master
    Worker
    Worker
    Test files
    test_c.rb
    result
    result
    test_c.rb

    View Slide

  67. How it works
    • Worker process runs next test
    Master
    Worker
    Worker
    Test files
    test_c.rb
    result
    result

    View Slide

  68. How it works
    • If there are no remaining test files, send
    quit instruction to worker
    Master
    Worker
    Worker
    Test files
    result
    result
    result
    Quit

    View Slide

  69. How it works
    • If there are no remaining test files, send
    quit instruction to worker
    Master
    Worker
    Worker
    Test files
    result
    result
    result
    Bye

    View Slide

  70. How it works
    • If there are no remaining test files, send
    quit instruction to worker
    Master
    Test files
    result
    result
    result

    View Slide

  71. How it works
    • If there are any results that contains more
    than one failure, master will retry in default
    Master
    result
    result
    result
    I have a failure!

    View Slide

  72. How it works
    • If there are any results that contains more
    than one failure, it’ll retry by master
    process in default
    Master
    result
    result
    test_b.rb
    Retry

    View Slide

  73. How it works
    • If there are any results that contains more
    than one failure, it’ll retry by master
    process in default
    Master
    result
    result
    result
    There are no
    failures.

    View Slide

  74. How it works
    • Because some tests fail because of
    parallelization.
    Master
    result
    result
    result
    There are no
    failures.

    View Slide

  75. How it works
    • Then merge the results and show to user
    Master
    result
    result
    result
    User
    via STDOUT

    View Slide

  76. This is
    • all of a mechanism of this this.
    • it works well, but...

    View Slide

  77. In Ruby
    • some tests don’t consider a parallelization
    • it can guess (I guessed that)
    • The issues is:
    • Tests around network are using same
    port
    • Tests are using same directory

    View Slide

  78. Real World Issues
    • test/ruby/test_signal.rb
    • test/ruby/test_process.rb
    • test/net/test_{http,https}.rb
    • test/csv/*
    • tests of rubygems

    View Slide

  79. test_signal.rb
    • test around signal handling of ruby.
    • this test run Kernel#fork to check fork is
    implemented.
    • test/unit’s at_exit called each fork.
    • I fixed test/unit to make be runnable this
    test.
    • Patch description: Add a flag to test/unit
    and interrupt if a flag is true in at_exit.

    View Slide

  80. test_process.rb
    • This test modifies STDIN/OUT.
    • this feature uses STDIN/OUT, so I modified
    to duplicate STDIN/OUT before run test in
    worker process.

    View Slide

  81. test_{http,https}.rb
    • These tests use same port number in test.
    • I fixed port number; but a fix is committed
    before I commit by Yui (nurse).

    View Slide

  82. test/csv/*
    • Tests included in this directory use same
    temporally directory name
    • All testcases use same directory name and
    they deletes temporaly directory in
    `teardown` phase.
    • So...

    View Slide

  83. test/csv/*
    Time
    test_csv_b.rb
    test_csv_a.rb
    Running Tests
    Test finished.
    Deleting the dir.
    Still Running...

    View Slide

  84. test/csv/*
    Time
    test_csv_b.rb
    test_csv_a.rb
    Running Tests
    Deleted!
    “No such files or
    directory”

    View Slide

  85. test/csv/*
    • Renamed temporary directory names to be
    unique each tests in test/csv/*.

    View Slide

  86. test/rubygems/*
    • Recently they forget writing dependencies
    of test.
    • It makes problem at parallel_test.
    • because... ok, let’s show a example.
    Note: “dependency” is a file should be required.

    View Slide

  87. test/rubygems/*
    • if there are test_a.rb, test_b.rb, and foo.rb.
    • also test_a.rb and test_b.rb depends on
    foo.rb.
    • test_a.rb includes `require ‘foo’`,
    •but test_b.rb doesn’t.

    View Slide

  88. test/rubygems/*
    • When not using parallel testing:
    Time
    test_a.rb test_b.rb
    Running Test
    foo.rb is
    required. no errors.
    require ‘foo’

    View Slide

  89. test/rubygems/*
    • When parallel testing:
    Time
    test_a.rb
    test_b.rb
    Running Tests require ‘foo’
    Ugh, foo.rb is not
    required...
    Exception ocucred...

    View Slide

  90. test/rubygems/*
    • I’m fixing that if I found.
    • This case can be found in any tests, but I
    founded only in the rubygems test.
    • I don’t know why. :p

    View Slide

  91. test/rubygems/*
    @rubygems_developers:
    please merge r33232 (in ruby-repo)
    https://github.com/rubygems/rubygems/issues/180
    or
    http://bit.ly/rubyconf2011_rubygems_issue

    View Slide

  92. Agenda
    • Why Parallel Testing
    • Multi-Thread or Multi-Process
    • test/unit parallelization
    • How it works
    • Performance

    View Slide

  93. Performance
    • Kenta Murata a.k.a. @mrkn measured and
    created a performance graphs for me.
    • He’ll talk at the last session of room 2,
    about the ruby’s number system.
    • Thanks!

    View Slide

  94. Performance
    • Measured at the following environment:
    • OS: Mac OS X 10.6.6
    • CPU: Intel Core i7 2.66 GHz
    • (2 cores 4 threads)
    • Memory: 8GB 1067 MHz DDR3
    • Test files is from ruby’s repository

    View Slide

  95. Performance
    0
    32.5
    65.0
    97.5
    130.0
    single 1 2 3 5 8 13
    Elapsed time
    Seconds
    worker(s)

    View Slide

  96. Performance
    • Without parallelization (“no -j”):
    121.49 seconds
    • Enabled parallelization with 5 workers:
    43.41 seconds

    View Slide

  97. Performance
    • Without parallelization (“no -j”):
    121.49 seconds
    • Enabled parallelization with 5 workers:
    43.41 seconds
    2.79x
    Faster!

    View Slide

  98. How to use
    • Separate TestCase-s to multiple file.
    • Write script to run multiple test file using
    Test::Unit::AutoRunner.
    • good example is ruby’s test/runner.rb in
    ruby’s repository.
    • Run the script with argument -j N.
    • Running the script with --help provides
    more information.

    View Slide

  99. The patch made me
    • A committer!
    • I was very happy then.

    View Slide

  100. That’s all.
    Thank you!

    View Slide

  101. Announcement
    • Ruby 1.9.3 RC1 has been released!
    • You can use parallelization by installing this!
    • Let’s try it and tell us if you found a bug.
    • More about: http://www.ruby-lang.org/

    View Slide

  102. Any Questions?
    If I can’t answer to your question because of
    my English skill,
    please send tweet to @sora_h
    or
    mail me: [email protected].

    View Slide