The Resurrection of the Fast Parallel Test Runner

RubyKaigi 2023
https://rubykaigi.org/2023/presentations/koic.html#may12

Transcript

1. Koichi ITO / ESM, Inc.
   RubyKaigi 2023
   The Resurrection of
   the Fast Parallel Test Runner
   May, 12th, 2023
   Matsumoto Performing Arts Centre
   The Force Awakens
   

   View Slide

2. !LPJD
   w 044QSPHSBNNFS
   w 3VCP$PQDPSFUFBN
   w &OHJOFFSJOH.BOBHFSBOE
   %JTUJOHVJTIFE&OHJOFFSPG&4. *OD
   w 3VCZ,BJHJTQFBLFSBU
   -5 5BLFPVU
   BOE
   

   View Slide

3. .BJOUBJO044&WFSZ%BZ
   IUUQTHJUIVCDPNLPJD
   

   View Slide

4. &4. *OD
   

   View Slide

5. $P⒎FFIPVTF4QPOTPS
   w"CF ᘖᘣඒֶΞϕ
   
   w0LJOBEP ͓͖ͳಊ
   
   w:BNBHB ٤஡ࢁխ
   
   w"OENPSF
   ☕
   

   View Slide

6. /PWFMUZ
   

   View Slide

7. 4
   Q
   FBLFS
   4
   Q
   FBLFS
   JOQ
   FSTPO
   JOQ
   FSTPO
   JOQ
   FSTPO
   JOQ
   FSTPO
   JOQ
   FSTPO
   JOQ
   FSTPO
   POMJOF
   POMJOF
   POMJOF
   POMJOF
   POMJOF
   POMJOF
   POMJOF
   4QFBLFSTBOE"UUFOEFFT
   PSHBOJ[FS
   PSHBOJ[FS
   "EWJTPS "EWJTPS
   

   View Slide

8. Support OSS community
   

   View Slide

9. +PJOVT
   🗾w-FUTHSPXUPHFUIFS
   w:PVDBOXPSL
   SFNPUFMZGSPN
   BOZXIFSFJO+BQBO
   w4FFBHJMFFTNDPKQ
   

   View Slide

10. View Slide

11. View Slide

12. $POUFOUT
    8IBUTQBSBMMFMUFTUJOH
    1BSBMMFMUFTUSVOOFST
    3FTVSSFDUJPOPGUFTURVFVF
    

    View Slide

13. 8IBUTQBSBMMFMUFTUJOH
    
    

    View Slide

14. 5PNF MFHBDZDPEF
    JTTJNQMZDPEF
    XJUIPVUUFTUT
    803,*/(&''&$5*7&-:8*5)-&("$:$0%&
    
    l
    "CPPLJOUIF6ODMF#PCTFSJFTTBZT
    

    View Slide

15. w 4PGUXBSFDPEFLFFQTHSPXJOHMBSHFS
    w #VTJOFTTHSPXUIBOETPGUXBSFHSPXUI
    HPIBOEJOIBOE
    w $POUJOVPVTMZQFSGPSN5%% #%% 5"%
    BOESFHSFTTJPOUFTUT
    5FTUDPEFJTJNQPSUBOU
    

    View Slide

16. )PXMPOHEPZPV
    XBJUGPSZPVSCVJME
    

    View Slide

17. ⏳
    "JNGPS
    NJOVUFTPSMFTT
    

    View Slide

18. w FYBNQMFT 6TJOH34QFD
    
    w "SFBMXPSMEBQQMJDBUJPOUBLFPWFSBO
    IPVS
    "SFBMXPSMEBQQMJDBUJPO
    $ bin/rails stats
    (snip)
    Code LOC: 12796 Test LOC: 23518 Code to Test Ratio: 1:1.8
    

    View Slide

19. "OFXQSPCMFN
    4MPXUFTU
    

    View Slide

20. w $BOOPUSVOUFTUTPOQVMMSFRVFTUTJO
    BEBZ
    w 5IFCVJMETBSFOUpOJTIFEFWFOCZ
    UIFOFYUNPSOJOH
    *UUBLFTBOIPVSUPDPNQMFUFBCVJME
    

    View Slide

21. 
    

    View Slide

22. w TFDPOETJOBEBZ
    w 'PSFYBNQMF TMPXUFTUTUBLJOH
    TFDPOETFBDIXJMMSFRVJSFNJOVUFT
    w )PXNBOZDPNNJUTEPFTZPVSUFBN
    NBLFFWFSZEBZ
    5JNFJTJNQPSUBOU
    

    View Slide

23. w 1SPpMJOHCBTFEUVOJOH%POUHVFTT NFBTVSF
    • FH
    w .BZCFTQFEVQXJUIBDMFBOJOHTUSBUFHZ
    w FH%BUBCBTF$MFBOFSP⒎FSTEBUBCBTFDMFBOJOH
    TUSBUFHJFTTVDIBTtransaction deletion BOE
    truncation
    JUTBHPPEJEFBUPDIPPTFUIFPQUJNBMPOF
    "EESFTTJOH4MPX5FTU#PUUMFOFDLT
    $ rspec -p 30 spec/acceptance
    

    View Slide

24. .ZSFDPNNFOEBUJPO
    w 'JSTU QSFQBSFNPOFZ💰JGZPVBSF
    CVJMEJOHCVTJOFTTTPGUXBSF
    w /FYU QSFQBSFFOPVHI$16TBOENFNPSZ
    w 'JOBMMZ 1BSBMMFMJ[FZPVSUFTUT
    w 5IBUTJU"'"*, JUTUIFNPTUF⒎FDUJWF
    

    View Slide

25. 1BSBMMFM5FTUJOH
    

    View Slide

26. 4FSJBM5FTUJOH
    4JOHMFXPSLFS
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    w 8PSLFS5FTUFYFDVUJPOQSPDFTT
    w 4VJUF5FTUFYFDVUJPOVOJU UPQMFWFM
    TestDMBTT describe %4- pMF FUD
    
    QSPDFTTJOHUJNF
    $ bundle exec ruby -Itest path # or rspec path
    

    View Slide

27. 1BSBMMFM5FTUJOH
    1BSBMMFMXPSLFS
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    1BSBMMFMXPSLFS
    1BSBMMFMXPSLFS
    QSPDFTTJOHUJNF
    

    View Slide

28. 4FSJBMWT1BSBMMFM
    1BSBMMFMXPSLFS
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    1BSBMMFMXPSLFS
    1BSBMMFMXPSLFS
    4JOHMFXPSLFS
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    /PXBJUUJNF
    QSPDFTTJOHUJNF
    

    View Slide

29. 1BSBMMFMJ[BUJPOJTHSFBU
    5JNFJTNPSFWBMVBCMF
    UIBOBOZUIJOH
    

    View Slide

30. 
    1BSBMMFMUFTUSVOOFST
    

    View Slide

31. w .JOJUFTU
    w 3BJMT
    w QBSBMMFM@UFTUT
    w UFTURVFVF
    5PVSPG1BSBMMFM5FTUJOH5PPMT
    

    View Slide

32. NJOJUFTUNJOJUFTU
    NJOJUFTUQSPWJEFTBDPNQMFUFTVJUFPGUFTUJOHGBDJMJUJFT
    TVQQPSUJOH5%% #%% NPDLJOH BOECFODINBSLJOH
    

    View Slide

33. w #BDLFOEPGActiveSupport::TestCaseJO3BJMT
    w 3BOEPNJ[FETFSJBMUFTUJTUIFEFGBVMU
    w parallelize_me! QSPWJEFTQBSBMMFMUFTUJOHGPSUIF
    RVFVFNPEFM*UVTFTNVMUJUISFBEJOHCZEFGBVMU
    w :PVDBODIBOHFUIFQBSBMMFMFYFDVUJPONPEVMFCZ
    Minitest.parallel_executor
    .JOJUFTU
    

    View Slide

34. &YUFOTJPOQPJOUGPSQBSBMMFMUFTUJOH
    .JOJUFTU
    QBSBMMFM@FYFDVUPS
    JOUFSGBDF
    w 5ISFBECZEFGBVMUGPSUIFQBSBMMFMUFTU
    w *.0 UFTUTUIBUBSFDPNQMFUFEJONFNPSZBSFOPUGBTU
    &⒎FDUJWFGPSUFTUTUIBUVTF*0
    *OUFSGBDFGPSQBSBMMFMUFTUJOH
    .JOJUFTU
    1BSBMMFM
    &YFDVUPS
    !RVFVF
    "DMBTTUIBUJNQMFNFOUT
    NFUIPETTVDIBTstart
    << size BOEshutdown
    BVUPSVO
    

    View Slide

35. EFGTUBSU
    !QPPMTJ[FUJNFTNBQ\
    5ISFBEOFX !RVFVF
    EPcRVFVFc
    5ISFBEDVSSFOUBCPSU@PO@FYDFQUJPOUSVF
    XIJMF KPCRVFVFQPQ
    
    LMBTT NFUIPE SFQPSUFSKPC
    SFQPSUFSTZODISPOJ[F\SFQPSUFSQSFSFDPSELMBTT NFUIPE^
    SFTVMU.JOJUFTUSVO@POF@NFUIPELMBTT NFUIPE
    SFQPSUFSTZODISPOJ[F\SFQPSUFSSFDPSESFTVMU^
    FOE
    FOE
    ^
    FOE
    EFGXPSL!RVFVFXPSLFOE
    .JOJUFTU1BSBMMFM&YFDVUPS
    $SFBUFsizeOVNPGOFXUISFBEXJUI@queue
    -PPQXJUIqueue.pop
    $BMMBUFTUNFUIPE
    .JOJUFTU1BSBMMFM&YFDVUPS XPSL
    
    IUUQTHJUIVCDPNNJOJUFTUNJOJUFTUCMPCWMJCNJOJUFTUQBSBMMFMSC--
    /05&@queueDBOCF
    TIBSFEXJUIBOJOTUBODF
    WBSJBCMFCFDBVTFJUJTB
    UISFBENPEFM
    

    View Slide

36. .JOJUFTU5FTU
    NPEVMF5FTUOPEPD
    EFG@TZODISPOJ[F.JOJUFTU5FTUJP@MPDLTZODISPOJ[F\ZJFME^
    FOE
    NPEVMF$MBTT.FUIPETOPEPD
    EFGSVO@POF@NFUIPELMBTT NFUIPE@OBNF SFQPSUFS
    .JOJUFTUQBSBMMFM@FYFDVUPS
    FOE
    EFGUFTU@PSEFS
    QBSBMMFM
    FOOOE
    .JOJUFTU
    3VOOBCMF
    .JOJUFTU
    5FTU
    IUUQTHJUIVCDPNNJOJUFTUNJOJUFTUCMPCWMJCNJOJUFTUQBSBMMFMSC--
    FORVFVFUPQBSBMMFM@FYFDVUPS
    .JOJUFTU
    1BSBMMFM5FTU
    $BTT.FUIPET
    NJYJO
    DMBTT.JOJUFTU5FTU.JOJUFTU3VOOBCMF
    FYUFOE.JOJUFTU1BSBMMFM5FTU$MBTT.FUIPET
    FOE
    

    View Slide

37. <"QQFOEJY>IFMMSC
    IUUQTHJUIVCDPNNJOJUFTUNJOJUFTUCMPCWMJCNJOJUFTUIFMMSC
    w %FpOJOHUIFFOWJSPONFOU
    WBSJBCMFMT_HELLBDUJWBUFT
    NJOJUFTUIFMMSC
    w "DUJWBUFQBSBMMFMBOEQSPWFJU
    

    View Slide

38. SBJMTSBJMT
    3BJMTJTBXFCBQQMJDBUJPOGSBNFXPSLUIBUJODMVEFTFWFSZUIJOH
    OFFEFEUPDSFBUFEBUBCBTFCBDLFEXFCBQQMJDBUJPOT
    BDDPSEJOHUPUIF.PEFM7JFX$POUSPMMFS .7$
    QBUUFSO
    

    View Slide

39. w 1BSBMMFMUFTUJOHCZEFGBVMU 4JODF3BJMT
    
    w %FGBVMUUISFTIPMEJT "WPJEJOHUIFPWFSIFBEPGGPSLJOH
    
    3BJMTAS::TestCase
    $ bin/rails test
    Running 60 tests in parallel using 8 processes
    Run options: --seed 21371
    # Running:
    .....................................................
    .......
    Finished in 0.205488s, 291.9879 runs/s, 291.9879
    assertions/s.
    60 runs, 60 assertions, 0 failures, 0 errors, 0 skips
    MFTTUIBOUFTUT UFTUT
    $ bin/rails test
    Running 10 tests in a single process (parallelization
    threshold is 50)
    Run options: --seed 1398
    # Running:
    ..........
    Finished in 0.016447s, 608.0136 runs/s, 608.0136
    assertions/s.
    10 runs, 10 assertions, 0 failures, 0 errors, 0 skips
    

    View Slide

40. ENV["RAILS_ENV"] ||= "test"
    require_relative "../config/environment"
    require "rails/test_help"
    class ActiveSupport::TestCase
    # Run tests in parallel with specified workers
    parallelize(workers: :number_of_processors)
    # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
    fixtures :all
    # Add more helper methods to be used by all tests here...
    end
    UFTUUFTU@IFMQFSSC
    QBSBMMFMJ[BUJPOCZEFGBVMU
    PQFODMBTT
    

    View Slide

41. module ActiveSupport
    class TestCase < ::Minitest::Test
    class << self
    def parallelize(workers: :number_of_processors, with: :processes,
    threshold: ActiveSupport.test_parallelization_threshold)
    workers = Concurrent.physical_processor_count if workers
    == :number_of_processors
    workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"]
    return if workers <= 1
    Minitest.parallel_executor =
    ActiveSupport::Testing::ParallelizeExecutor.new(size: workers, with:
    with, threshold: threshold)
    ennnnnd
    *OIFSJUMinitest::Test
    $IBOHFUPQBSBMMFMFYFDVUPSPG3BJMT
    "DUJWF4VQQPSU5FTU$BTF
    

    View Slide

42. &YUFOTJPOQPJOUGPSQBSBMMFMUFTUJOH
    .JOJUFTU
    QBSBMMFM@FYFDVUPS
    JOUFSGBDF
    w 5ISFBECZEFGBVMUGPSUIFQBSBMMFMUFTU
    w *.0 UFTUTUIBUBSFDPNQMFUFEJONFNPSZBSFOPUGBTU
    &⒎FDUJWFGPSUFTUTUIBUVTF*0
    *OUFSGBDFGPSQBSBMMFMUFTUJOH
    .JOJUFTU
    1BSBMMFM
    &YFDVUPS
    !RVFVF
    "DMBTTUIBUJNQMFNFOUT
    NFUIPETTVDIBTstart
    << size BOEshutdown
    BVUPSVO
    

    View Slide

43. "45FTUJOH1BSBMMFM&YFDVUPS
    .JOJUFTU
    QBSBMMFM@FYFDVUPS
    JOUFSGBDF
    w 3FQMBDFXJUIUIF.JOJUFTUFYUFOTJPOQPJOU
    w 4XJUDIGSPNUISFBEUPQSPDFTTQBSBMMFMJ[BUJPO
    "OJOUFSGBDFGPSUIFQBSBMMFMUFTU
    .JOJUFTU
    1BSBMMFM
    &YFDVUPS
    !RVFVF
    BVUPSVO
    "DUJWF4VQQPSU
    5FTUJOH
    1BSBMMFMJ[F&YFDVUPS
    !QBSBMMFM@FYFDVUPS
    "DUJWF4VQQPSU
    5FTUJOH
    1BSBMMFMJ[BUJPO
    

    View Slide

44. "45FTUJOH1BSBMMFMJ[BUJPO
    %3C'SPOU
    w 4FSWFS8PSLFSNPEFM NVMUJQSPDFTTJOH
    
    w *OUFSQSPDFTTDPNNVOJDBUJPOVTJOHE3VCZ
    1BSBMMFMJ[BUJPO
    4FSWFS
    RVFVF
    1BSBMMFMJ[BUJPO
    8PSLFS
    FORVFVF
    GPSLFEQSPDFTT
    ESCVOJY
    
    O
    NBJOQSPDFTT
    1BSBMMFMJ[BUJPO
    8PSLFS
    EFRVFVF XIJMFKPC!RVFVFQPQ
    QFSGPSN@KPC KPC
    
    FOE
    $BMMMinitest.run_one_method
    1BSBMMFMJ[BUJPO
    8PSLFS
    1BSBMMFMJ[BUJPO
    

    View Slide

45. w AS::TestCase.parallelizeCZEFGBVMU
    w 3BJMTQSPWJEFTUIFCFOFpUTPGNVMUJDPSF
    QSPDFTTJOHUISPVHIUIFVTFPGQSPDFTTFT
    w 3VOTQBSBMMFMJTNCZEFGBVMUXJUI
    Concurrent.physical_processor_count
    w 3VOUJNF$16TBOENFNPSZBSFJNQPSUBOU
    3BJMTJTQSBHNBUJD
    

    View Slide

46. 3BJMTEPFTOU
    TVQQPSU34QFDCZ
    EFGBVMU🤔
    

    View Slide

47. w HSPTTFSQBSBMMFM@UFTUT
    w UNNUFTURVFVF
    1BSBMMFMUFTUSVOOFS
    

    View Slide

48. HSPTTFSQBSBMMFM@UFTUT
    4QFFEVQ5FTU6OJU34QFD$VDVNCFS
    4QJOBDICZSVOOJOHQBSBMMFMPONVMUJQMF$16DPSFT
    

    View Slide

49. w %FQFOETPOUIFQBSBMMFMHFN
    • Parallel.each { ... }
    • Parallel.map { ... }
    w 3VOQSFTQMJUUJOHUFTUTJOQBSBMMFMGPSBMMUFTUpMFT
    w *UDBOVTFQBSBMMFM@SVOUJNF@STQFDMPHBGUFSSVOOJOH
    w 7FSZTJNQMF😀
    QBSBMMFM@UFTUT
    

    View Slide

50. w 5IFSFJTBUJNFEJ⒎FSFODFCFUXFFOUIFXPSLFS
    UIBUpOJTIFTpSTUBOEUIFXPSLFSUIBUpOJTIFT
    MBTU
    w 'JOJTIFEXPSLFSTIBWFXBTUFEXBJUUJNF
    w /PUFUIBUUIFFYFDVUJPOSFTVMUNBUSJYpMFJT
    VTFEUPNJOJNJ[FUIFUJNFEJ⒎FSFODFCFUXFFO
    UFTUTBTNVDIBTQPTTJCMF
    DPOTPGQSFTQMJUUJOH
    

    View Slide

51. UNNUFTURVFVF
    :FUBOPUIFSQBSBMMFMUFTUSVOOFS CVJMUVTJOHB
    DFOUSBMJ[FERVFVFUPFOTVSFPQUJNBMEJTUSJCVUJPOPG
    UFTUTCFUXFFOXPSLFST
    

    View Slide

52. w 5IFBVUIPSJT"NBO(VQUB,BSNBOJ !UNN
    
    w 3FBEZXPSLFSTEFRVFVFBOESVOUFTUTGSPNUIF
    RVFVFXBJUJOHUPCFSVO
    w 'FXFSXBJUJOHXPSLFSTBOEMFTTXBTUF
    w 4BNFBTQBSBMMFM@UFTUT VTJOHUFTU@RVFVF@TUBUT
    FYFDVUJPOSFTVMUNBUSJYBOESVOOJOHTMPXUFTUT
    pSTU
    UFTURVFVF
    

    View Slide

53. 6/*94PDLFUPS5$14PDLFU
    5ISFFQSPDFTTUZQFT
    6/*94PDLFUPS5$14PDLFU
    6/*94FSWFSPS5$14FSWFS
    UFTURVFVF
    NBTUFSQSPDFTT
    UFTURVFVFTVJUF
    EJTDPWFSZQSPDFTT
    
    UFTURVFVF
    XPSLFSQSPDFTT
    
    
    
    %JTUSJCVUJOHUFTUTVJUF
    GSPNRVFVFBOE
    TVNNBSJ[FUIFSFTVMUT
    %FRVFVFBOESVOUFTUTVJUFT
    1BSTFUFTUTVJUFTBOEFORVFVFJU
    1⃣ 2⃣
    3⃣
    

    View Slide

54. class Runner
    def execute_internal
    start_master
    prepare(@concurrency)
    @prepared_time = Time.now
    start_relay if relay?
    discover_suites
    spawn_workers
    distribute_queue
    ensure
    stop_master
    kill_subprocesses
    ennd
    3VOOFSTUBSU@NBTUFS
    def start_master
    if !relay?
    if @socket =~ /\A(?:(.+):)?(\d+)\z/
    address = $1 || '0.0.0.0'
    port = $2.to_i
    @socket = "#{$1}:{#$2}"
    @server = TCPServer.new(address, port)
    else
    FileUtils.rm_f(@socket)
    @server = UNIXServer.new(@socket)
    end
    ennd
    1⃣
    .BTUFS1SPDFTT
    .BTUFS1SPDFTT
    

    View Slide

55. class Runner
    def execute_internal
    start_master
    prepare(@concurrency)
    @prepared_time = Time.now
    start_relay if relay?
    discover_suites
    spawn_workers
    distribute_queue
    ensure
    stop_master
    kill_subprocesses
    ennd
    3VOOFSEJTDPWFS@TVJUFT
    def discover_suites
    return if relay?
    return if @allowlist.any? && @awaited_suites.empty?
    @discovering_suites_pid = fork do
    terminate = false
    Signal.trap("INT") { terminate = true }
    $0 = 'test-queue suite discovery process'
    @test_framework.all_suite_files.each do |path|
    @test_framework.suites_from_file(path).each do |
    suite_name, suite|
    Kernel.exit!(0) if terminate
    @server.connect_address.connect do |sock|
    sock.puts("TOKEN=#{@run_token}")
    sock.puts("NEW SUITE #{Marshal.dump([suite_name,
    path])}")
    ennd
    2⃣
    %JTDPWFSZ1SPDFTT
    .BTUFS1SPDFTT
    

    View Slide

56. class Runner
    def execute_internal
    start_master
    prepare(@concurrency)
    @prepared_time = Time.now
    start_relay if relay?
    discover_suites
    spawn_workers
    distribute_queue
    ensure
    stop_master
    kill_subprocesses
    ennd
    3VOOFSTQBXO@XPSLFST
    def spawn_workers
    @concurrency.times do |i|
    num = i + 1
    pid = fork do
    @server.close if @server
    iterator = Iterator.new(@test_framework, relay? ?
    @relay : @socket, method(:around_filter),
    early_failure_limit: @early_failure_limit, run_token:
    @run_token)
    after_fork_internal(num, iterator)
    ret = run_worker(iterator) || 0
    cleanup_worker
    Kernel.exit! ret
    end
    @workers[pid] = Worker.new(pid, num)
    ennd
    3⃣
    3VOXPSLFS🏃
    8PSLFS1SPDFTT
    .BTUFS1SPDFTT
    

    View Slide

57. 6/*94PDLFUPS5$14PDLFU
    6/*94PDLFUPS5$14PDLFU
    6/*94FSWFSPS5$14FSWFS
    *OUFSBDUJPOPGQSPDFTTFT
    UFTURVFVF
    NBTUFSQSPDFTT
    UFTURVFVFTVJUF
    EJTDPWFSZQSPDFTT
    UFTURVFVF
    XPSLFSQSPDFTT
    %FRVFVFBOESVOUFTUTVJUFT
    1BSTFUFTUTVJUFTBOE
    FORVFVFJU
    %JTUSJCVUJOHUFTUTVJUF
    GSPNRVFVFBOE
    TVNNBSJ[FUIFSFTVMUT
    GPSL
    /&846*5&
    GPSL
    .BSTIBMEVNQ
    .BSTIBMMPBE
    5FTU'SBNFXPSL
    3VOOFS
    101
    

    View Slide

58. $PNNBOEMJTU
    $PNNBOE %JSFDUJPO %FTDSJQUJPO
    /&846*5& EJTDPWFSZˠNBTUFS &ORVFVFBUFTUTVJUF
    101 XPSLFSˠNBTUFS %FRVFVFBUFTUTVJUF
    8"*5 NBTUFSˠXPSLFS "XBJUBUFTUTVJUFXIFOEFRVFVF
    ,"#00. XPSLFSˠNBTUFS *NNFEJBUFMZBCPSU
    3&.05&."45&3 XPSLFSˠNBTUFS 3FHJTUFSBSFNPUFXPSLFSUPUIFSFNPUFNBTUFSJGSFMBZ
    0, NBTUFSˠXPSLFS 3FQMZSFHJTUFSTVDDFTTUPUIFSFNPUFXPSLFSJGSFMBZ
    803,&3 XPSLFSˠNBTUFS $PNQMFUFPSBCPSUBSFNPUFXPSLFSJGSFMBZ
    "MXBZTNBUDIBVOJRVFTOKENJTTVFEBOEVTFEBTBQSPUPDPMXJUIUIFNBTUFSQSPDFTT
    

    View Slide

59. class Runner
    def execute_internal
    start_master
    prepare(@concurrency)
    @prepared_time = Time.now
    start_relay if relay?
    discover_suites
    spawn_workers
    distribute_queue
    ensure
    stop_master
    kill_subprocesses
    ennd
    3VOOFSEJTUSJCVUF@RVFVF
    def distribute_queue
    # snip
    until !awaiting_suites? && @queue.empty? && remote_workers == 0
    # snip
    case cmd
    when /\APOP (\S+) (\d+)/
    # snip
    when /\AREMOTE MASTER (\d+) ([\w\.-]+)(?: (.+))?/
    # snip
    when /\AWORKER (\d+)/
    # snip
    when /\ANEW SUITE (.+)/
    # snip
    when /\AKABOOM/
    break
    else
    warn("Ignoring unrecognized command: \"#{cmd}\"")
    end
    sock.close
    ennd .BTUFS1SPDFTT
    .BTUFS1SPDFTT
    

    View Slide

60. def each
    # snip
    loop do
    if @early_failure_limit && @failures >= @early_failure_limit
    connect_to_master('KABOOM'); break
    else
    client = connect_to_master("POP #{Socket.gethostname} #{Process.pid}")
    end
    break if client.nil?
    _r, _w, e = IO.select([client], nil, [client], nil)
    break unless e.empty?
    if data = client.read(65536)
    client.close
    item = Marshal.load(data)
    # snip
    suite_name, path = item
    suite = load_suite(suite_name, path)
    # snip
    yield suite
    *UFSBUPSFBDI
    8PSLFS1SPDFTT
    class Iterator
    include Enumerable
    

    View Slide

61. 2VFVF3VOOFS
    SVO@TQFDT JUFSBUPS
    
    4UBUT
    3VOOFS34QFD
    5FTU'SBNFXPSL
    34QFD
    4UBUT4VJUF
    OFX
    3VOOFS
    34QFD
    3VOOFS
    5FTU'SBNFXPSL
    *UFSBUPS
    SVO@TQFDT FYBNQMF@HSPVQT
    
    UFTURVFVFSVOUJNF
    UFTUJOHGSBNFXPSL
    BEBQUFSDMBTT
    FYFDVUF
    OFX
    BMM@TVJUF@pMFT
    TVJUFT@GSPN@pMF
    OFX
    &OVNFSBCMF
    FBDI
    OFX
    8PSLFS
    OFX
    !RVFVF
    TUBUVT
    🏃💨
    TestQueue::Runner::
    RSpec.new.execute
    FYFSTQFDRVFVF
    SVO@XPSLFS
    BMM@TVJUF@pMFT
    SVO@XPSLFS
    BMM@TVJUFT
    SVO@FBDI
    TVJUFT@GSPN@pMF
    %FTJHO34QFD
    

    View Slide

62. w UFTURVFVFTVQQPSUT34QFD 34QFD
    .JOJUFTU .JOJUFTU UFTUVOJU $VDVNCFS
    w UFTURVFVFQSPWJEFTUIFDPPSEJOBUJPOPG
    QSPDFTTFTJODPNNPOQBSBMMFMQSPDFTTJOH
    w 5FTUTVJUFBOEUFTUSVOOFS"1*TBSFUFTUJOH
    GSBNFXPSLTQFDJpD
    1MVHHBCMF%FTJHO
    

    View Slide

63. 2VFVF3VOOFS
    SVO@TQFDT JUFSBUPS
    
    4UBUT
    3VOOFS34QFD
    5FTU'SBNFXPSL
    34QFD
    4UBUT4VJUF
    OFX
    3VOOFS
    34QFD
    3VOOFS
    5FTU'SBNFXPSL
    *UFSBUPS
    SVO@TQFDT FYBNQMF@HSPVQT
    
    UFTURVFVFSVOUJNF
    UFTUJOHGSBNFXPSL
    BEBQUFSDMBTT
    FYFDVUF
    OFX
    BMM@TVJUF@pMFT
    TVJUFT@GSPN@pMF
    OFX
    &OVNFSBCMF
    FBDI
    OFX
    8PSLFS
    OFX
    !RVFVF
    TUBUVT
    🏃💨
    TestQueue::Runner::
    RSpec.new.execute
    FYFSTQFDRVFVF
    SVO@XPSLFS
    BMM@TVJUF@pMFT
    SVO@XPSLFS
    BMM@TVJUFT
    SVO@FBDI
    TVJUFT@GSPN@pMF
    %FTJHO34QFD
    

    View Slide

64. 2VFVF3VOOFS
    SVO@TQFDT JUFSBUPS
    
    4UBUT
    3VOOFS34QFD
    5FTU'SBNFXPSL
    34QFD
    4UBUT4VJUF
    OFX
    3VOOFS
    34QFD
    3VOOFS
    5FTU'SBNFXPSL
    *UFSBUPS
    SVO@TQFDT FYBNQMF@HSPVQT
    
    UFTURVFVFSVOUJNF
    UFTUJOHGSBNFXPSL
    BEBQUFSDMBTT
    FYFDVUF
    OFX
    BMM@TVJUF@pMFT
    TVJUFT@GSPN@pMF
    OFX
    &OVNFSBCMF
    FBDI
    OFX
    8PSLFS
    OFX
    !RVFVF
    TUBUVT
    3VOOFS
    🏃💨
    TestQueue::Runner::
    RSpec.new.execute
    FYFSTQFDRVFVF
    SVO@XPSLFS
    BMM@TVJUF@pMFT
    SVO@XPSLFS
    BMM@TVJUFT
    SVO@FBDI
    TVJUFT@GSPN@pMF
    &YUFOTJPOQPJOU
    

    View Slide

65. 2VFVF3VOOFS
    SVO@TQFDT JUFSBUPS
    
    4UBUT
    3VOOFS34QFD
    5FTU'SBNFXPSL
    34QFD
    4UBUT4VJUF
    OFX
    3VOOFS
    34QFD
    3VOOFS
    5FTU'SBNFXPSL
    *UFSBUPS
    SVO@TQFDT FYBNQMF@HSPVQT
    
    UFTURVFVFSVOUJNF
    UFTUJOHGSBNFXPSL
    BEBQUFSDMBTT
    FYFDVUF
    OFX
    BMM@TVJUF@pMFT
    TVJUFT@GSPN@pMF
    OFX
    &OVNFSBCMF
    FBDI
    OFX
    8PSLFS
    OFX
    !RVFVF
    TUBUVT
    5FTU'SBNFXPSL
    🏃💨
    TestQueue::Runner::
    RSpec.new.execute
    FYFSTQFDRVFVF
    SVO@XPSLFS
    BMM@TVJUF@pMFT
    SVO@XPSLFS
    BMM@TVJUFT
    SVO@FBDI
    TVJUFT@GSPN@pMF
    &YUFOTJPOQPJOU
    

    View Slide

66. UFTURVFVFMJCUFTU@RVFVFSVOOFSSTQFDSC
    UFTURVFVFMJCUFTU@RVFVFSVOOFSSTQFDSC
    3VOOFS34QFDSVO@XPSLFS
    module QueueRunner < Runner
    def run_specs(iterator)
    @configuration.reporter.report(0) do |reporter|
    @configuration.with_suite_hooks do
    iterator.map { |g|
    start = Time.now
    if g.is_a? ::RSpec::Core::Example
    print " #{g.full_description}: "
    example = g
    g = example.example_group
    ::RSpec.world.filtered_examples.clear
    ::RSpec.world.filtered_examples[g] = [example]
    else
    print " #{g.description}: "
    end
    ret = g.run(reporter)
    # snip
    end
    alias_method :run_each, :run_specs
    module TestQueue
    class Runner
    class RSpec < Runner
    def run_worker(iterator)
    rspec = ::RSpec::Core::QueueRunner.new
    rspec.run_each(iterator).to_i
    ennnnd
    3VOTVJUF
    Iterator#each☝
    /BJWFJNQMFNFOUBUJPO
    UIBUEFQFOETPO34QFDT
    JOUFSOBM"1*
    

    View Slide

67. 2VFVF3VOOFS
    SVO@TQFDT JUFSBUPS
    
    4UBUT
    3VOOFS34QFD
    5FTU'SBNFXPSL
    34QFD
    4UBUT4VJUF
    OFX
    3VOOFS
    34QFD
    3VOOFS
    5FTU'SBNFXPSL
    *UFSBUPS
    SVO@TQFDT FYBNQMF@HSPVQT
    
    UFTURVFVFSVOUJNF
    UFTUJOHGSBNFXPSL
    BEBQUFSDMBTT
    FYFDVUF
    OFX
    BMM@TVJUF@pMFT
    TVJUFT@GSPN@pMF
    OFX
    &OVNFSBCMF
    FBDI
    OFX
    8PSLFS
    OFX
    !RVFVF
    TUBUVT
    🏃💨
    TestQueue::Runner::
    RSpec.new.execute
    FYFSTQFDRVFVF
    SVO@XPSLFS
    BMM@TVJUF@pMFT
    SVO@XPSLFS
    BMM@TVJUFT
    SVO@FBDI
    TVJUFT@GSPN@pMF
    %FTJHO34QFD
    

    View Slide

68. 4UBUT
    3VOOFS.JOJUFTU
    5FTU'SBNFXPSL
    .JOJUFTU
    4UBUT4VJUF
    .JOJUFTU
    .JOJUFTU
    3VOOFS
    5FTU'SBNFXPSL
    *UFSBUPS
    SVOOBCMFT JUFSBUPS
    
    UFTURVFVFSVOUJNF
    UFTUJOHGSBNFXPSL
    BEBQUFSDMBTT
    FYFDVUF
    OFX
    BMM@TVJUF@pMFT
    TVJUFT@GSPN@pMF
    BMM@TVJUF@pMFT
    &OVNFSBCMF
    SVO@XPSLFS
    BMM@TVJUFT
    OFX
    8PSLFS
    OFX
    !RVFVF
    TUBUVT
    FYFNJOJUFTURVFVF
    %FTJHO.JOJUFTU
    SVO
    🏃💨
    OFX
    TestQueue::Runner::
    Minitest.new.execute
    SVO@XPSLFS
    FBDI
    OFX
    TVJUFT@GSPN@pMF
    

    View Slide

69. NJOJUFTUMJCNJOJUFTUSC
    module Minitest
    def self.run args = []
    # snip
    begin
    __run reporter, options
    rescue Interrupt
    warn 'Interrupted. Exiting...'
    UFTURVFVFMJCUFTU@RVFVFSVOOFSNJOJUFTUSC
    module Minitest
    def self.__run reporter, options
    suites = Runnable.runnables
    suites.map { |suite| suite.run reporter, options }
    ennd
    module TestQueue
    class Runner
    class Minitest < Runner
    def run_worker(iterator)
    ::Minitest::Test.runnables = iterator
    ::Minitest.run ? 0 : 1
    ennnnd
    3VOOFS.JOJUFTUSVO@XPSLFS
    $BMM__run
    3VOTVJUF
    Iterator#each☝
    /BJWFJNQMFNFOUBUJPO
    UIBUEFQFOETPOPQFO
    DMBTTBOEJOUFSOBM"1*
    0QFODMBTT
    

    View Slide

70. 4VQQPSUUFTUJOHGSBNFXPSLT
    IUUQTHJUIVCDPNUNNUFTURVFVFUSFFNBTUFSMJCUFTU@RVFVFSVOOFS
    

    View Slide

71. 1BSBMMFM5FTUJOH "HBJO
    
    1BSBMMFMXPSLFS
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    4VJUF
    
    1BSBMMFMXPSLFS
    1BSBMMFMXPSLFS
    QSPDFTTJOHUJNF
    

    View Slide

72. DPSF$16T)ZQFS5ISFBEJOH
    TFD
    
    
    
    
    
    TFSJBM QBSBMMFM
    w8IFOUFTURVFVF
    SVOTBMPUPGUFTU
    TVJUFT JUXJMMCF
    FGGFDUJWF
    wFHTFDTFD
    JO3VCP$PQSFQP
    

    View Slide

73. % bundle exec rake spec
    Starting test-queue master (/tmp/test_queue_40616_5060.sock)
    ==> Summary (16 workers in 51.9441s)
    [ 1] 9 examples, 0 failures 1 suites in 34.0151s (pid 40649 exit 0 )
    [ 2] 109 examples, 0 failures 1 suites in 34.0151s (pid 40650 exit 0 )
    [ 3] 712 examples, 0 failures 46 suites in 34.0149s (pid 40651 exit 0 )
    [ 4] 2449 examples, 0 failures 61 suites in 34.0151s (pid 40652 exit 0 )
    [ 5] 129 examples, 0 failures 13 suites in 34.0151s (pid 40653 exit 0 )
    [ 6] 1628 examples, 0 failures 57 suites in 34.0150s (pid 40654 exit 0 )
    [ 7] 107 examples, 0 failures 1 suites in 34.0151s (pid 40655 exit 0 )
    [ 8] 1647 examples, 0 failures 61 suites in 34.0147s (pid 40656 exit 0 )
    [ 9] 2270 examples, 0 failures 59 suites in 34.0145s (pid 40657 exit 0 )
    [10] 1660 examples, 1 pending, 0 failures 57 suites in 34.0145s (pid 40658 exit 0 )
    [11] 132 examples, 0 failures 1 suites in 51.9309s (pid 40659 exit 0 )
    [12] 1600 examples, 0 failures 53 suites in 51.9306s (pid 40660 exit 0 )
    [13] 1924 examples, 0 failures 60 suites in 51.9295s (pid 40661 exit 0 )
    [14] 1993 examples, 0 failures 59 suites in 51.9281s (pid 40662 exit 0 )
    [15] 1967 examples, 0 failures 60 suites in 51.9271s (pid 40663 exit 0 )
    [16] 2303 examples, 0 failures 57 suites in 51.9237s (pid 40664 exit 0 )
    4MPXFTUBOEGBTUFTU
    4MPXFTU
    'BTUFTU
    

    View Slide

74. )PXTIPVMEUIF
    SVOOJOHPGUFTUT
    B⒎FDUPOFBOPUIFS
    /PUBUBMM
    5&45%3*7&/%&7&-01.&/5#:&9".1-&
    
    l
    *TPMBUFE5FTUJTJNQPSUBOU
    

    View Slide

75. w *TPMBUFUFTUTGSPNFBDIPUIFS
    w %BUBTUPSBHFBDDFTTFECZQBSBMMFM
    XPSLFSTTVDIBT3%#.4TIPVMECFTFU
    VQJOBQBSBMMFMNBOOFS
    w FH8PSLFS#SFBETUIFVQEBUFEEBUBPG
    XPSLFS"BOEUIFUFTUNBZGBJM
    4FUVQGPSQBSBMMFMUFTUJOH
    

    View Slide

76. w 1SPEVDUDPEFBOEUFTUDPEFGPSNBUPUFTUJOHQBJS
    w 5IFVOJUPGQBSBMMFMUBTLTJOQBSBMMFMUFTUTJTVTVBMMZ
    UIFpMFMFWFM
    w 1BSBMMFMJ[JOHUFTUTVJUFTJOTNBMMFSVOJUTDBOSFEVDF
    UIFMPDLJOHQFSJPEGPSXPSLFST
    w *OPUIFSXPSET GPMMPXJOH431MFBETUPGBTUFS
    QBSBMMFMUFTUT
    1BSBMMFMUFTUBOE431
    4JOHMF3FTQPOTJCJMJUZ1SJODJQMF
    

    View Slide

77. Parallel test runners
    Using RSpec
    Using Rails 6+ or AS
    Using Vanilla Ruby
    No
    Yes
    Yes
    No
    Yes
    No
    Start
    AS::
    TestCase
    parallel_tests
    or
    test-queue
    

    View Slide

78. 8IJDIPOFJTCFUUFSUPVTF
    QBSBMMFM@UFTUTPSUFTURVFVF
    

    View Slide

79. w /VNCFSPGEPXOMPBETPOSVCZHFNTPSH
    w /VNCFSPGTUBSTPO(JU)VC
    w 'SFRVFODZPGNBJOUFOBODF
    w QBSBMMFM@UFTUTJTNBJOUBJOFECVUUFTURVFVF
    XBTVONBJOUBJOFE
    w *OUFSFTUJOHEFTJHOCVUVOEFSVUJMJ[FE
    $SJUFSJBGPSHFNTFMFDUJPO
    

    View Slide

80. 3JTFVQ
    

    View Slide

81. 3FTVSSFDUJPOPGUFTURVFVF
    
    

    View Slide

82. w *OFFEUPSFMFBTFBOFXWFSTJPO
    w !UNNBTLFENFUPCFBNBJOUBJOFSBOE
    *BDDFQUFE
    w .BJOUFOBODFJTSFTVNFEBGUFSPS
    ZFBST
    'JSTUUIJOHT*EJEBTBNBJOUBJOFS
    

    View Slide

83. *GMFGUVOBUUFOEFE
    TPGUXBSFXJMMCSFBL
    CZ!ITCU
    XBTSFMFBTFE CVUTUJMM$*JTCSPLFO
    

    View Slide

84. /P$* OPNFSHF
    $*XBTCSPLFO
    

    View Slide

85. w *GPSHPUIPXUPVTF5SBWJT$*
    w *NOPUVQUPEBUFPOUIFSFDFOU
    EFWFMPQNFOUTPG5SBWJT$*
    w 8JUI(JU)VC"DUJPOT NBJOUFOBODFDBO
    CFEPOFXJUIPVUNF
    'SPN5SBWJT$*UP(JU)VC"DUJPOT
    

    View Slide

86. w #BUT#BTI"VUPNBUFE5FTUJOH4ZTUFN
    w !TTUFQIFOTPOQSPEVDU
    w 5IFSFQPTJUPSZIBTCFFOBSDIJWFECZUIF
    PXOFSPO"QS 😢
    w *UTIBSEUPSFQMBDFBUFTUJOHGSBNFXPSL
    w *UTOFHBUJWF CVU*NMFBWJOHJUBTJUJT
    &&UFTUJOHPGUFTURVFVF
    

    View Slide

87. w 5FTUOPUQBTTJOHXJUIMBUFTU
    WFSTJPOPG.JOJUFTU
    w 5IFUFTUDPEFBSPVOEUIF
    sleepNFUIPETFFNT
    TVTQJDJPVT
    w 'JYGBJMJOH(FNpMFGPS.JOJUFTU
    CZWFSTJPOQJOOJOH
    $VSSFOUTUBUVTPG&&UFTUJOH
    

    View Slide

88. $*IBTSFDPWFSFE
    

    View Slide

89. %FWFMPQNFOU
    FOWJSPONFOUJTSFBEZ
    *GUIF$*QBTTFT
    NBZCF*DBONFSHFJU
    

    View Slide

90. #FZPOE
    UIF3FTVSSFDUJPO
    

    View Slide

91. 4VQQPSUTUBUVTPGUFTUJOHGSBNFXPSLT
    5FTUJOH'SBNFXPSL $* /PUF
    34QFD ✅ 🤔5PPPME34QFDEPFTOUXPSLXJUI3VCZ
    34QFD ✅ ✅6TFEJO3VCP$PQ
    .JOJUFTU ✅ 🤔5PPPME
    .JOJUFTU ✅ CVU
    ✅6TFEJO3VCP$PQ.JOJUFTU
    $VDVNCFS ✅ CVU
    ❌0VUEBUFEUFTUDPEFFSSPSTJOUIFMBUFTUWFSTJPO
    UFTUVOJU ✅ 🤔/PUXPSLJOHXJUI3BJMT UNNUFTURVFVF
    
    34QFD EFW
    ✅ ✅6TFEJO3VCP$PQ
    5VSOJQ ✅ ✅/PUSBDLSFDPSEBTJUXBTKVTUNFSHFE
    

    View Slide

92. w "QVMMSFRVFTUIBTCFFOPQFOFECZB34QFD
    DPSFUFBNNFNCFS
    w 34QFDJTBEFWFMPQNFOUTUBUVT #VU
    *EFDJEFEUPQSPWJEFBGFBUVSFPGUFTURVFVF
    w 050) JUTIBSEGPSUFTURVFVFNBJOUBJOFST
    UPLOPX34QFDTJOUFSOBMJNQMFNFOUBUJPO
    4VQQPSU34QFD EFW
    
    

    View Slide

93. w 8PVMEOUJUCFDPOWFOJFOUJG34QFDIBEB
    QBSBMMFMUFTUSVOOFSGFBUVSFMJLF.JOJUFTU
    w 5IFTBNFDBOCFTBJEGPSUFTUVOJUBOE
    PUIFST
    *OUSPEVDFQBSBMMFMUFTUSVOOFSUP34QFD
    0QQPSUVOJUZGPSDPOUSJCVUJPO
    

    View Slide

94. 0OFNPSFUIJOH
    

    View Slide

95. w "MSFBEZJODMVEFTPuppetLintBTB
    QBSBMMFMSVOOFS
    w 8IJMFUIFQBSBMMFMHFNJTVTFEGPSQBSBMMFM
    FYFDVUJPOPG3VCP$PQ VTJOHB
    NFDIBOJTNMJLFUFTURVFVFNBZQSPWJEF
    FWFOGBTUFSSFTVMUT
    2VFVFTZTUFNOPUKVTUGPSUFTUJOH
    

    View Slide

96. w 1SPHSBNNJOHGPSJOUFSQSPDFTT
    DPNNVOJDBUJPOJTBCBTJDGPVOEBUJPOGPS
    MFWFSBHJOHNVMUJDPSF
    'VOEBNFOUBMTPGQBSBMMFMQSPHSBNNJOH
    

    View Slide

97. $PODMVTJPO
    

    View Slide

98. w UFTURVFVFEFQFOETPOJOUFSOBM"1*PGFBDI
    UFTUJOHGSBNFXPSL
    w *XBOUJUUPCFUFNQPSBSZUSBOTJFOU
    w .BZOPUSFBDIUFTURVFVF
    w )BWJOHBQMVHHBCMFQBSBMMFMUFTUNFDIBOJTNMJLF
    .JOJUFTUDPVMECFDPOWFOJFOU FH34QFDUPP
    
    )PQFGPSQBSBMMFMUFTUSVOOFS
    

    View Slide

99. w 1BSBMMFMUFTUJOHSFRVJSFTNPOFZ💰
    w %FWFMPQFSTBSFUIFNPTUWBMVBCMFSFTPVSDF
    JOUIFQSPKFDU
    w *UTFBTJFSUPHFUNPSFNFNPSZBOE$16T
    XJUINPOFZUIBOUPIJSFTFOJPSFOHJOFFST
    w %PZPVSCFTUUPTFDVSFUIFCVEHFU
    5IFSFJTBHPMECVMMFU
    

    View Slide

100. 1BSBMMFM5FTUJOH
     .BLFT
     :PV)BQQZ
     (JU)VC!LPJD
     

     View Slide