Photo by Masahiro Ihara Kenta Murata @mrkn COOKPAD Inc. Development Infrastructure Engineer CRuby committer bigdecimal support Asakusa.rb RubySapporo RailsGirlsTokyo (coach) Sunday, September 16, 12
Chanko How COOKPAD safely releases multiple feature prototypes - in production - for test segments of their 15 million engaged users @mrkn (Kenta Murata), @shingo (Shingo Morita) http://www.flickr.com/photos/june29/3396011694/ http://www.confreaks.com/events/railsconf2012 https://speakerdeck.com/u/mrkn/p/chanko Sunday, September 16, 12
After Chanko IDEA BUILD Limited release IDEA MEASURE Public release RELEASE http://photozou.jp/photo/show/606813/95100763 http://photozou.jp/photo/show/276167/58451178 BUILD LEARN https://speakerdeck.com/u/mrkn/p/chanko?slide=20 Sunday, September 16, 12
After Chanko IDEA BUILD Limited release IDEA MEASURE Public release RELEASE http://photozou.jp/photo/show/606813/95100763 http://photozou.jp/photo/show/276167/58451178 BUILD LEARN https://speakerdeck.com/u/mrkn/p/chanko?slide=20 Sunday, September 16, 12
The maximum CI waiting minutes 2N min Prescribed working time 8 hours == 480 min The maximum count of deploy 480 / 2N == 240 / N N : The CI waiting time to deploy 40 times 240 / 40 == 6 min Sunday, September 16, 12
The maximum CI waiting minutes 2N min Prescribed working time 8 hours == 480 min The maximum count of deploy 480 / 2N == 240 / N N : The CI waiting time to deploy 40 times 240 / 40 == 6 min Sunday, September 16, 12
The maximum CI waiting minutes 2N min Prescribed working time 8 hours == 480 min The maximum count of deploy 480 / 2N == 240 / N N : The CI waiting time to deploy 40 times 240 / 40 == 6 min Sunday, September 16, 12
The maximum CI waiting minutes 2N min Prescribed working time 8 hours == 480 min The maximum count of deploy 480 / 2N == 240 / N N : The CI waiting time to deploy 40 times 240 / 40 == 6 min Sunday, September 16, 12
The test system for the developers should provide an environment which isn’t broken by simultaneous test executions. Necessary condition Sunday, September 16, 12
Three conditions • CI should be fast • Developer-testing environment should be available for simultaneous executions • Developer-testing should be fast Sunday, September 16, 12
ThreeFour conditions • CI should be fast • Developer-testing environment should be available for simultaneous executions • Developer-testing should be fast • The system should be able to scale out for increasing tests Sunday, September 16, 12
Big Procedural Flow 1. Rsync the application’s code tree 2. Prepare workers’ environment • MySQL and Solr 3. Run RSpec remotely Sunday, September 16, 12
1. Grouping spec les so that each the group has almost same execution time based on the last execution time log 2. Executing RSpec for each the group on the remote workers via ssh 3. Results of RSpec is collected via pipe, and displayed so that the user can easily distinguish which worker makes the results. How to distribute Sunday, September 16, 12
spec_execution_times.log • Execution times are recorded for each spec le. • An execution time of a spec le is de ned as one of the top-level example group in the spec le. Sunday, September 16, 12
Condition Execution time single rspec too long; unmeasurable 8-core parallel_tests over 1 hour 5 workers, each have 6-cores about 15 min Sunday, September 16, 12
Wrong balancing • The execution time on each worker was varied. • Runtime conditions of servers affected the execution times. • I retired the way to divide spec les into groups before executing RSpec. Sunday, September 16, 12
Dynamic distributing • I need the method to distribute spec les dynamically according to runtime conditions of workers • The answer is Producer-Consumer pattern Sunday, September 16, 12
Producer-Consumer pattern • The basic way to implement parallel task processing. • Producers produce tasks and enqueue them into a queue. • Consumers dequeue tasks from the queue, and processes them. Sunday, September 16, 12
By this pattern • Slow workers process fewer tasks than the others. • Fast workers process much more tasks than the others. • As a result the execution times of workers are almost same. Sunday, September 16, 12
I implemented it • Using dRuby and Rinda::TupleSpace • Producer and consumers communicate with each other via a queue represented by Rinda::TupleSpace • On workers, dRuby gateway is executed via ssh, and the gateway process launches each consumer DRbObject. Sunday, September 16, 12
Consumer process • A consumer takes a spec le from a queue, a TupleSpace, and execute it. • I want to avoid spending time for generating a new process and loading and initializing Rails environment by executing “rspec” command. • I need to use rspec abnormally. Sunday, September 16, 12
RSpec::Core::CommandLine#run def run ... @configuration.load_spec_files @world.announce_filters @configuration.reporter.report(...) do |reporter| begin @configuration.run_hook(:before, :suite) @world.example_groups.ordered.map {|g| g.run(reporter) }.all? ? 0 : @configuration.failure_exit_code ensure @configuration.run_hook(:after, :suite) end end end Sunday, September 16, 12
RSpec::Core::CommandLine#run def run ... @configuration.load_spec_files @world.announce_filters @configuration.reporter.report(...) do |reporter| begin @configuration.run_hook(:before, :suite) @world.example_groups.ordered.map {|g| g.run(reporter) }.all? ? 0 : @configuration.failure_exit_code ensure @configuration.run_hook(:after, :suite) end end end Sunday, September 16, 12
RSpec::Core::CommandLine#run def run ... @configuration.load_spec_files @world.announce_filters @configuration.reporter.report(...) do |reporter| begin @configuration.run_hook(:before, :suite) @world.example_groups.ordered.map {|g| g.run(reporter) }.all? ? 0 : @configuration.failure_exit_code ensure @configuration.run_hook(:after, :suite) end end end Sunday, September 16, 12
Normally RSpec • Example groups are de ned in global RSpec::Core::World object • Running example groups in the global RSpec::Core::World object is running examples. Sunday, September 16, 12
Abnormally RSpec • Need to load a single spec le, run example groups only de ned in the spec le, and clear the all example groups from the global RSpec environment. • RSpec::Core::World#clear does it. Sunday, September 16, 12
Consumer’s main loop def main loop do _, spec_file = @tuple_space.take [:spec_file, String] load File.expand_path(spec_file) begin @configuration.run_hook(:before, :suite) @world.example_groups.ordered.map {|g| g.run(reporter) }.all? ? 0 : @configuration.failure_exit_code ensure @configuration.run_hook(:after, :suite) @world.clear end end end Sunday, September 16, 12
Conclusion • Testing facilities in COOKPAD are introduced • The mechanisms of distributed RSpec in COOKPAD are introduced • My current attempts don’t get in time. • remote_spec.gem is coming soon. Sunday, September 16, 12