Upgrade to Pro — share decks privately, control downloads, hide ads and more …

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

文字起こし
https://github.com/operando/Notes/tree/master/%E9%96%8B%E7%99%BA%C3%97%E3%83%86%E3%82%B9%E3%83%88%20LT%E4%BC%9A%20-%20vol.2%20%23devtestlt

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにするサンプルプロジェクト
https://github.com/operando/dart-test-CircleCI-timing-data

開発×テスト LT会 - vol.2 #devtestlt
https://rakus.connpass.com/event/237602/

E77b6a5f919f7366d94f21eee9a014f5?s=128

operandoOS

March 15, 2022
Tweet

More Decks by operandoOS

Other Decks in Technology

Transcript

  1. CircleCIͰtiming dataʹ 
 ج͍ͮͨςετ෼ׂΛ 
 DartͰར༻Ͱ͖ΔΑ͏ʹͨ͠࿩ ։ൃ×ςετ LTձ - vol.2

    #devtestlt
  2. ࣗݾ঺հ • Shinobu Okano • @operandoOS / shinobu.dart • 10X,

    Inc. / Software Engineer • Flutter / Dart / Android / Automation
  3. CircleCIͰςετ 
 ճͯ͠·͔͢ʁ

  4. timing dataͷ࿩Λ ͢Δ্ͷͰલఏ஌ࣝ

  5. CircleCIͷςετϝλσʔλ • ςετ݁Ռ͕هࡌ͞ΕͨϑΝΠϧͷ͜ͱʢࡶʣ • ςετϝλσʔλΛ׆༻͢ΔͱTest InsightsͳͲ͕ར༻Ͱ͖Δ • timing data͸ςετϝλσʔλ͔Βੜ੒͞ΕΔ

  6. CircleCIͷςετϝλσʔλ Collecting test data https://circleci.com/docs/2.0/collect-test-data/

  7. CircleCIͰςετϝλσʔλΛอଘ͢Δ • store_test_resultsεςοϓͰςετ݁Ռ 
 ʢςετϝλσʔλʣΛอଘ͢Δ • https://circleci.com/docs/ja/2.0/con fi guration-reference/ #storetestresults

  8. CircleCIͰςετϝλσʔλΛอଘ͢Δ - store_test_results: path: test-results

  9. CircleCIͷςετϝλσʔλͷϑΥʔϚοτ • CircleCIͰςετ݁Ռ͕ςετϝλσʔλͱೝࣝ͞ΕΔϑΥʔϚοτ͸ҎԼ • JUnit XML • Cucumber JSON •

    ଟ͘ͷݴޠϥΠϒϥϦͰαϙʔτ͍ͯ͠Δͷ͸JUnit XML • ຊࢿྉͰ΋JUnit XMLΛѻ͍·͢
  10. JUnit XML Schema • https://github.com/windyroad/JUnit-Schema • https://github.com/sj26/rspec_junit_formatter/blob/ main/lib/rspec_junit_formatter.rb#L9-L11 • http://windyroad.org/dl/Open%20Source/JUnit.xsd

  11. CircleCIͷςετ෼ׂͱฒྻ࣮ߦ • 1ͭͷ࣮ߦ؀ڥʢExecutorʣͰશςετΛ࣮ߦ͢ΔͱCIͷ࣮ߦ͕࣌ؒ௕͘ͳΔ • CircleCIʹ͸ςετΛෳ਺ͷExecutorʹ෼ׂͤ͞ΔͨΊͷίϚϯυ͕͋Δ • ίϚϯυͰ෼ׂͨ͠ςετΛෳ਺ͷExecutorͰฒྻ࣮ߦͤ͞Δ͜ͱʹΑΓ 
 શςετ࣮ߦ͕ૣ͘ͳΓɺCIͷ࣮ߦ࣌ؒ΋୹͘ͳΔ

  12. CircleCIͷςετ෼ׂͱฒྻ࣮ߦ Running Tests in Parallel https://circleci.com/docs/2.0/parallelism-faster-jobs/

  13. CircleCIͷςετ෼ׂͱ ฒྻ࣮ߦ https://circleci.com/docs/ja/2.0/parallelism-faster-jobs/ • ެࣜυΩϡϝϯτͷਤΛ͓आΓͯ͠ 
 આ໌͢Δͱ͜Μͳײ͡ • Ұ൪Լ͕timing dataʹج͍ͮͨ

    
 ύλʔϯͷઆ໌
  14. ςετ෼ׂ • ෼ׂͷ࢓ํʹΑͬͯશମͷςετ࣮ߦ࣌ؒʹӨڹ͢Δ • ద౰ʹ෼ׂͯ͠ฒྻ࣮ߦͤ͞Δͱ؀ڥ͝ͱʹ࣮ߦ࣌ؒʹ 
 ͹Β͖͕ͭग़Δ • Մೳͳ͔͗Γ؀ڥ͝ͱͷ࣮ߦ࣌ؒͷ͹Β͖͕ͭ 


    খ͘͞ͳΔΑ͏ʹ෼ׂ͢Δͷ͕Α͍✨
  15. ࠷దͳςετ෼ׂ • ֤ςετέʔεͷ࣮ߦʹ͔͔Δ࣌ؒΛܭଌ͢Δ • ܭଌͨ࣌ؒ͠Λར༻͠શମͷςετ͕࣌ؒ୹͘ͳΔΑ͏ 
 ͍͍ײ͡ͷΞϧΰϦζϜͰ͍͍ײ͡ʹ෼ׂ͢Δ • ʮ͍͍ײ͡ʯ͕େࣄ😊

  16. CircleCIͷςετϝλσʔλ circleci tests split --split-by=timingsʹ͍ͭͯ 
 ௐ΂ͨɾࣗ࡞ͯ͠Έͨ htthttps://hoshinotsuyoshi.com/post/circleci_tests_split/

  17. CircleCIͷςετ෼ׂ • ҎԼͷ෼ׂํ๏Λαϙʔτ͍ͯ͠Δ • timing data • ϑΝΠϧ໊ɾΫϥε໊ • ϑΝΠϧαΠζ

    • CircleCI͸timing dataʹΑΔ෼ׂ͕࠷ྑͷํ๏ͱͯ͠Δ
  18. CircleCIͷtiming data • ֤ςετέʔεͷ࣮ߦ͕࣌ؒهࡌ͞Ε͍ͯΔJSONϑΝΠϧ • ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ੒͢Δ • timing dataͷ಺༰͸֬ೝͰ͖Δ

  19. CircleCIͷtiming data • ֤ςετέʔεͷ݁Ռʹ͸ɺΫϥε໊ͱϑΝΠϧ໊ͳͲ͕ 
 هࡌ͞Ε͍ͯΔ͜ͱΛCircleCI͸ظ଴͍ͯ͠Δ • ֤ςετϑΝΠϧͷςετ࣮ߦʹ͔͔Δ͕࣌ؒΘ͔Δσʔλߏ଄ • σϑΥϧτͷ෼ׂํ๏͕ɺϑΝΠϧ໊ʹج͍ͮͨ෼ׂͳͷͰ্ه͕ඞཁ

    • ςετϝλσʔλʹͦΕΒͷ৘ใ͕هࡌ͞Ε͍ͯͳ͍ͱtiming dataΛ 
 ར༻ͯ͠΋ࢥͬͨͱ͓Γʹςετ෼ׂ͕࣮ߦ͞Εͳ͍
  20. timing dataΛར༻ͯ͠ 
 ςετ෼ׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ circleci tests glob "test/**/*.rb" | circleci

    tests split --split-by=timings > /tmp/tests-to-run bundle exec rspec $(cat /tmp/tests-to-run)
  21. • circleci tests splitͰςετϑΝΠϧͷҰཡऔಘ • --split-by=timingsͰtiming dataΛར༻࣮͠ߦ؀ڥ͝ͱʹ 
 ςετ෼ׂΛߦ͏Α͏ʹࢦఆ •

    bundle exec rspec΁࣮ߦ؀ڥ͝ͱʹ෼ׂ͞ΕͨςετΛ 
 ࣮ߦ͢ΔΑ͏ʹࢦఆ timing dataΛར༻ͯ͠ 
 ςετ෼ׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ
  22. Dartͷtestʹ͍ͭͯ

  23. Why Dart test? • ࢲ͕ॴଐ͢Δձࣾ 10XͰ࡞͍ͬͯΔαʔϏε StailerͰ͸ 
 API࣮૷ʹαʔόʔαΠυDartΛ࢖༻ •

    αʔόʔαΠυDartͰ͸CIʹCircleCIΛར༻͍ͯ͠Δ • DartͰςετΛॻ͘৔߹ʮtestʯύοέʔδ͕ϝδϟʔ • https://pub.dev/packages/test
  24. Dart sharding tests • testύοέʔδʹ͸ςετ෼ׂ͢Δػೳ͕͋Δʂʂ • ݱࡏ 10XͰ͸testͷػೳΛ࢖ͬͯCircleCI্ͰςετΛ෼ׂɾฒྻ࣮ߦͯ͠ ͍Δ •

    ͔͠͠ɺ͜ͷ෼ׂํ๏͸֤ςετέʔεͷ࣮ߦ࣌ؒͳͲΛݩʹ 
 ςετ෼ׂ͞ΕΔΘ͚Ͱ͸ͳ͍ • ͦͷͨΊExecutor͝ͱʹ࣮ߦ࣌ؒͷ͹Β͖͕ͭେ͖͍ͷ͕՝୊
  25. Dart sharding tests dart test --total-shards 3 --shard-index 0 path/to/test.dart

    dart test --total-shards 3 --shard-index 1 path/to/test.dart dart test --total-shards 3 --shard-index 2 path/to/test.dart
  26. Dart sharding tests testύοέʔδͷςετ෼ׂͷ࣮૷ https://github.com/dart-lang/test/blob/2afbff8cc059b37b4b1a69fc2ddffc53290e4780/ pkgs/test_core/lib/src/runner.dart#L428-L443

  27. Dart testsͰCircleCIͷtiming dataΛར༻ͯ͠ 
 ΋ͬͱ͍͍ײ͡ʹςετ෼ׂɾฒྻ࣮ߦ͍ͨ͠ʂ ͱࢥͬͨΘ͚Ͱ…

  28. Dart testͷ݁ՌΛJunit XMLͰग़ྗ͢Δ • Dart test͸ςετ݁ՌΛJSONग़ྗ͢Δػೳ͕͋Δ • ͦͷJSONग़ྗ͞Εͨ݁ՌΛJunit XMLʹม׵͢Δπʔϧ͕͋Δ •

    https://github.com/TOPdesk/dart-junitreport • ͜ΕΛ࢖ͬͯDart testͷςετ݁ՌΛJunit XMLͰग़ྗ͢Δ
  29. ग़ྗͨ͠Junit XMLΛςετϝλσʔλͱͯ͠ CircleCIʹೝࣝͤ͞Δ • store_test_resultsεςοϓʹJunit XMLΛग़ྗͨ͠σΟϨΫτϦΛ ࢦఆ͢Δ • ͜ΕͰςετ݁ՌͷJunit XML͕CircleCIʹςετϝλσʔλ

    
 ͱͯ͠ೝࣝ͞ΕΔ • ೝࣝͨ͠ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ੒͢Δ
  30. ςετ࣮ߦͷίϚϯυΛॻ͖׵͑Δ • timing dataΛར༻ͯ͠ςετ෼ׂΛߦ͍ɺ෼ׂͨ͠ϑΝΠϧ ͷΈͰςετ࣮ߦ͢ΔΑ͏ʹίϚϯυΛॻ͖׵͑Δ TESTFILES=$(circleci tests glob "test/**/*_test.dart" |

    circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded
  31. Α͠ɺ͜ΕͰ׬ᘳ✨✨✨✨

  32. Ͱ͸...ͳ͔ͬͨ...💥💥💥💥

  33. Կ͕ى͖͔ͨ...😨 • "No timing found for <ϑΝΠϧ໊>"ͱ͍͏ϝοηʔδ͕ग़ͨ • ςετϑΝΠϧʹର͢Δtiming data͕ଘࡏ͠ͳ͍ͥʂͬͯݴͬͯΔ

    • timing dataͷϑΝΠϧ͸ੜ੒͞Ε͍͕ͯͨɺԿ͔͠ΒͷݪҼͰ 
 ͏·͘ར༻͞Ε͍ͯͳ͍ͷͰɺ࠷దͳςετ෼ׂ͕ߦ͍͑ͯͳ͍
  34. timing dataͷϑΝΠϧΛ֬ೝ͢Δํ๏ cat ${CIRCLE_INTERNAL_TASK_DATA}/circle-test-results/results.json

  35. Կ͕μϝ͔ͩͬͨ...😱 • timing dataΛ֬ೝͯ͠ΈͯΘ͔ͬͨͷ͕ɺ֤ςετέʔε͕ ͲͷςετϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λࣔ͢ fi le keyͷ 
 ஋͕nullͩͬͨ

  36. Կ͕μϝ͔ͩͬͨ...😱 fi le͕null😱

  37. Կ͕μϝ͔ͩͬͨ...·ͱΊΔͱ...😱 • ֤ςετέʔεͷ࣮ߦ࣌ؒͳͲ͸هࡌ͞Ε͍ͯΔ͕ 
 ֤ςετέʔε͕ͲͷϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λ 
 ࣔ͢஋͕nullͳͷͰɺςετϑΝΠϧ͝ͱͷ࣮ߦ͕࣌ؒ 
 ܭࢉͰ͖ͳ͍ •

    ݁Ռɺ͏·͘ςετ෼ׂͰ͖ͳ͍Θʂͬͯײͩͬͨ͡
  38. fi le keyͷ஋͕nullͳݪҼΛ୳Δ • timing data͸ςετϝλσʔλ͔Βੜ੒͞ΕΔ • ςετϝλσʔλ = ςετ݁Ռ͕هࡌ͞ΕͨJunit

    XMLΛ 
 ֬ೝͯ͠Έͨ
  39. <testsuites> <testsuite errors="0" failures="0" tests="2" skipped="0" name=“test.sample_1" > <properties> <property

    name="platform" value="vm" /> </properties> <testcase classname="test.sample_1" name=“String…” time="0.027" /> <testcase classname="test.sample_1" name=“String.trim…” time="0.005" /> </testsuite> <testsuite errors="0" failures="0" tests="4" skipped="0" name="test.sample_2" > <properties> <property name="platform" value="vm" /> </properties> <testcase classname="test.sample_2" name="String .s…” time="0.027" /> </testsuite> </testsuites> ग़ྗ͞ΕͨJunit XMLͷ಺༰
  40. fi le keyͷ஋͕nullͳݪҼΛ୳Δ • testcase elementʹ֤ςετέʔεͷ݁Ռ͕هࡌ͞Ε͍ͯΔ • ͦ͜ʹ֤ςετέʔε͕هࡌ͞Ε͍ͯΔςετϑΝΠϧ໊͕ 
 ॻ͔Ε͍ͯͳ͍͕ͳ͍͜ͱ͕ݪҼͬΆ͍͜ͱ͕Θ͔ͬͨ

    • ݪҼ͕Θ͔ͬͨͷͰtestcase elementʹ fi le attributeͷ 
 ௥ՃΛࢼΈͨ
  41. ࢼͯ͠Έͨमਖ਼ • dart-junitreportͷ಺෦Ͱར༻͞Ε͍ͯΔdart-testreportϥΠϒϥϦͷ मਖ਼ • dart-junitreportͰJunit XMLग़ྗ͢Δࡍɺtestcase elementʹ 
 fi

    le attributeͷ௥Ճ • fi le attributeʹهࡌ͞Εͨ fi le pathΛCI্ͰsedίϚϯυΛ 
 ར༻ͯ͠ॻ͖׵͑
  42. मਖ਼ͯ͠Έͨ݁Ռ • Dart testͰ΋timing dataΛར༻ͯ͠࠷దͳςετ෼ׂ͕ 
 Ͱ͖ΔΑ͏ʹͳͬͨʂ • ΊͰ͍ͨ🎉🎉🎉🎉

  43. ׬੒ܗ͕ͪ͜Β💪

  44. CircleCIͰtiming dataʹج͍ͮͨςετ෼ׂΛ 
 DartͰར༻Ͱ͖ΔΑ͏ʹ͢ΔαϯϓϧϓϩδΣΫτ https://github.com/operando/dart-test-CircleCI-timing-data ׬੒ܗ✨

  45. αϯϓϧϓϩδΣΫτͷCircleCIͷ݁Ռ https://app.circleci.com/pipelines/github/operando/dart-test-CircleCI-timing- data/14/work fl ows/e4884496-b4b0-4d5b-af7a-0209097affac/jobs/16 ׬੒ܗ✨

  46. version: 2.1 jobs: test: docker: - image: dart:2.16.1 parallelism: 2

    steps: - checkout - run: name: Download dependencies command: pub get - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ workflows: version: 2 build: jobs: - test .circleci/con fi g.ymlͷ಺༰ 
 ௕͍….จࣈখ͍͞…😇
  47. - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart"

    | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ େࣄͳͷ͸͜ͷ෦෼ʂ
  48. - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart"

    | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ forkͨ͠ϦϙδτϦʹमਖ਼ΛೖΕͨͷͰ 
 ࣗ਎ͷϦϙδτϦ͔Βऔಘ
  49. - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart"

    | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ Junit XMLͰग़ྗ͠ɺsedͰ fi le attributeͷ಺༰Λॻ͖׵͑
  50. - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart"

    | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ ग़ྗͨ͠Junit XMLΛอଘʂ
  51. - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart"

    | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ ͜ΕͰ͜ͷ෦෼ͷςετ෼ׂ͕͏·͘ಈ͍ͨʂ
  52. ΍ͬͨͶ😊ʂʂ

  53. Thanks🙏