CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

文字起こし
https://github.com/operando/Notes/tree/master/%E9%96%8B%E7%99%BA%C3%97%E3%83%86%E3%82%B9%E3%83%88%20LT%E4%BC%9A%20-%20vol.2%20%23devtestlt

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにするサンプルプロジェクト
https://github.com/operando/dart-test-CircleCI-timing-data

開発×テスト LT会 - vol.2 #devtestlt
https://rakus.connpass.com/event/237602/

Transcript

1. CircleCIͰtiming dataʹ 
 ج͍ͮͨςετ෼ׂΛ 
 DartͰར༻Ͱ͖ΔΑ͏ʹͨ͠࿩ ։ൃ×ςετ LTձ - vol.2 #devtestlt
2. ࣗݾ঺հ • Shinobu Okano • @operandoOS / shinobu.dart • 10X, Inc. / Software Engineer • Flutter / Dart / Android / Automation
3. CircleCIͰςετ 
 ճͯ͠·͔͢ʁ
4. timing dataͷ࿩Λ ͢Δ্ͷͰલఏ஌ࣝ
5. CircleCIͷςετϝλσʔλ • ςετ݁Ռ͕هࡌ͞ΕͨϑΝΠϧͷ͜ͱʢࡶʣ • ςετϝλσʔλΛ׆༻͢ΔͱTest InsightsͳͲ͕ར༻Ͱ͖Δ • timing data͸ςετϝλσʔλ͔Βੜ੒͞ΕΔ
6. CircleCIͷςετϝλσʔλ Collecting test data https://circleci.com/docs/2.0/collect-test-data/
7. CircleCIͰςετϝλσʔλΛอଘ͢Δ • store_test_resultsεςοϓͰςετ݁Ռ 
 ʢςετϝλσʔλʣΛอଘ͢Δ • https://circleci.com/docs/ja/2.0/con fi guration-reference/ #storetestresults
8. CircleCIͰςετϝλσʔλΛอଘ͢Δ - store_test_results: path: test-results
9. CircleCIͷςετϝλσʔλͷϑΥʔϚοτ • CircleCIͰςετ݁Ռ͕ςετϝλσʔλͱೝࣝ͞ΕΔϑΥʔϚοτ͸ҎԼ • JUnit XML • Cucumber JSON • ଟ͘ͷݴޠϥΠϒϥϦͰαϙʔτ͍ͯ͠Δͷ͸JUnit XML • ຊࢿྉͰ΋JUnit XMLΛѻ͍·͢
10. JUnit XML Schema • https://github.com/windyroad/JUnit-Schema • https://github.com/sj26/rspec_junit_formatter/blob/ main/lib/rspec_junit_formatter.rb#L9-L11 • http://windyroad.org/dl/Open%20Source/JUnit.xsd
11. CircleCIͷςετ෼ׂͱฒྻ࣮ߦ • 1ͭͷ࣮ߦ؀ڥʢExecutorʣͰશςετΛ࣮ߦ͢ΔͱCIͷ࣮ߦ͕࣌ؒ௕͘ͳΔ • CircleCIʹ͸ςετΛෳ਺ͷExecutorʹ෼ׂͤ͞ΔͨΊͷίϚϯυ͕͋Δ • ίϚϯυͰ෼ׂͨ͠ςετΛෳ਺ͷExecutorͰฒྻ࣮ߦͤ͞Δ͜ͱʹΑΓ 
 શςετ࣮ߦ͕ૣ͘ͳΓɺCIͷ࣮ߦ࣌ؒ΋୹͘ͳΔ
12. CircleCIͷςετ෼ׂͱฒྻ࣮ߦ Running Tests in Parallel https://circleci.com/docs/2.0/parallelism-faster-jobs/
13. CircleCIͷςετ෼ׂͱ ฒྻ࣮ߦ https://circleci.com/docs/ja/2.0/parallelism-faster-jobs/ • ެࣜυΩϡϝϯτͷਤΛ͓आΓͯ͠ 
 આ໌͢Δͱ͜Μͳײ͡ • Ұ൪Լ͕timing dataʹج͍ͮͨ 
 ύλʔϯͷઆ໌
14. ςετ෼ׂ • ෼ׂͷ࢓ํʹΑͬͯશମͷςετ࣮ߦ࣌ؒʹӨڹ͢Δ • ద౰ʹ෼ׂͯ͠ฒྻ࣮ߦͤ͞Δͱ؀ڥ͝ͱʹ࣮ߦ࣌ؒʹ 
 ͹Β͖͕ͭग़Δ • Մೳͳ͔͗Γ؀ڥ͝ͱͷ࣮ߦ࣌ؒͷ͹Β͖͕ͭ 
 খ͘͞ͳΔΑ͏ʹ෼ׂ͢Δͷ͕Α͍✨
15. ࠷దͳςετ෼ׂ • ֤ςετέʔεͷ࣮ߦʹ͔͔Δ࣌ؒΛܭଌ͢Δ • ܭଌͨ࣌ؒ͠Λར༻͠શମͷςετ͕࣌ؒ୹͘ͳΔΑ͏ 
 ͍͍ײ͡ͷΞϧΰϦζϜͰ͍͍ײ͡ʹ෼ׂ͢Δ • ʮ͍͍ײ͡ʯ͕େࣄ😊
16. CircleCIͷςετϝλσʔλ circleci tests split --split-by=timingsʹ͍ͭͯ 
 ௐ΂ͨɾࣗ࡞ͯ͠Έͨ htthttps://hoshinotsuyoshi.com/post/circleci_tests_split/
17. CircleCIͷςετ෼ׂ • ҎԼͷ෼ׂํ๏Λαϙʔτ͍ͯ͠Δ • timing data • ϑΝΠϧ໊ɾΫϥε໊ • ϑΝΠϧαΠζ • CircleCI͸timing dataʹΑΔ෼ׂ͕࠷ྑͷํ๏ͱͯ͠Δ
18. CircleCIͷtiming data • ֤ςετέʔεͷ࣮ߦ͕࣌ؒهࡌ͞Ε͍ͯΔJSONϑΝΠϧ • ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ੒͢Δ • timing dataͷ಺༰͸֬ೝͰ͖Δ
19. CircleCIͷtiming data • ֤ςετέʔεͷ݁Ռʹ͸ɺΫϥε໊ͱϑΝΠϧ໊ͳͲ͕ 
 هࡌ͞Ε͍ͯΔ͜ͱΛCircleCI͸ظ଴͍ͯ͠Δ • ֤ςετϑΝΠϧͷςετ࣮ߦʹ͔͔Δ͕࣌ؒΘ͔Δσʔλߏ଄ • σϑΥϧτͷ෼ׂํ๏͕ɺϑΝΠϧ໊ʹج͍ͮͨ෼ׂͳͷͰ্ه͕ඞཁ • ςετϝλσʔλʹͦΕΒͷ৘ใ͕هࡌ͞Ε͍ͯͳ͍ͱtiming dataΛ 
 ར༻ͯ͠΋ࢥͬͨͱ͓Γʹςετ෼ׂ͕࣮ߦ͞Εͳ͍
20. timing dataΛར༻ͯ͠ 
 ςετ෼ׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ circleci tests glob "test/**/*.rb" | circleci tests split --split-by=timings > /tmp/tests-to-run bundle exec rspec $(cat /tmp/tests-to-run)
21. • circleci tests splitͰςετϑΝΠϧͷҰཡऔಘ • --split-by=timingsͰtiming dataΛར༻࣮͠ߦ؀ڥ͝ͱʹ 
 ςετ෼ׂΛߦ͏Α͏ʹࢦఆ • bundle exec rspec΁࣮ߦ؀ڥ͝ͱʹ෼ׂ͞ΕͨςετΛ 
 ࣮ߦ͢ΔΑ͏ʹࢦఆ timing dataΛར༻ͯ͠ 
 ςετ෼ׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ
22. Dartͷtestʹ͍ͭͯ
23. Why Dart test? • ࢲ͕ॴଐ͢Δձࣾ 10XͰ࡞͍ͬͯΔαʔϏε StailerͰ͸ 
 API࣮૷ʹαʔόʔαΠυDartΛ࢖༻ • αʔόʔαΠυDartͰ͸CIʹCircleCIΛར༻͍ͯ͠Δ • DartͰςετΛॻ͘৔߹ʮtestʯύοέʔδ͕ϝδϟʔ • https://pub.dev/packages/test
24. Dart sharding tests • testύοέʔδʹ͸ςετ෼ׂ͢Δػೳ͕͋Δʂʂ • ݱࡏ 10XͰ͸testͷػೳΛ࢖ͬͯCircleCI্ͰςετΛ෼ׂɾฒྻ࣮ߦͯ͠ ͍Δ • ͔͠͠ɺ͜ͷ෼ׂํ๏͸֤ςετέʔεͷ࣮ߦ࣌ؒͳͲΛݩʹ 
 ςετ෼ׂ͞ΕΔΘ͚Ͱ͸ͳ͍ • ͦͷͨΊExecutor͝ͱʹ࣮ߦ࣌ؒͷ͹Β͖͕ͭେ͖͍ͷ͕՝୊
25. Dart sharding tests dart test --total-shards 3 --shard-index 0 path/to/test.dart dart test --total-shards 3 --shard-index 1 path/to/test.dart dart test --total-shards 3 --shard-index 2 path/to/test.dart
26. Dart sharding tests testύοέʔδͷςετ෼ׂͷ࣮૷ https://github.com/dart-lang/test/blob/2afbff8cc059b37b4b1a69fc2ddffc53290e4780/ pkgs/test_core/lib/src/runner.dart#L428-L443
27. Dart testsͰCircleCIͷtiming dataΛར༻ͯ͠ 
 ΋ͬͱ͍͍ײ͡ʹςετ෼ׂɾฒྻ࣮ߦ͍ͨ͠ʂ ͱࢥͬͨΘ͚Ͱ…
28. Dart testͷ݁ՌΛJunit XMLͰग़ྗ͢Δ • Dart test͸ςετ݁ՌΛJSONग़ྗ͢Δػೳ͕͋Δ • ͦͷJSONग़ྗ͞Εͨ݁ՌΛJunit XMLʹม׵͢Δπʔϧ͕͋Δ • https://github.com/TOPdesk/dart-junitreport • ͜ΕΛ࢖ͬͯDart testͷςετ݁ՌΛJunit XMLͰग़ྗ͢Δ
29. ग़ྗͨ͠Junit XMLΛςετϝλσʔλͱͯ͠ CircleCIʹೝࣝͤ͞Δ • store_test_resultsεςοϓʹJunit XMLΛग़ྗͨ͠σΟϨΫτϦΛ ࢦఆ͢Δ • ͜ΕͰςετ݁ՌͷJunit XML͕CircleCIʹςετϝλσʔλ 
 ͱͯ͠ೝࣝ͞ΕΔ • ೝࣝͨ͠ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ੒͢Δ
30. ςετ࣮ߦͷίϚϯυΛॻ͖׵͑Δ • timing dataΛར༻ͯ͠ςετ෼ׂΛߦ͍ɺ෼ׂͨ͠ϑΝΠϧ ͷΈͰςετ࣮ߦ͢ΔΑ͏ʹίϚϯυΛॻ͖׵͑Δ TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded
31. Α͠ɺ͜ΕͰ׬ᘳ✨✨✨✨
32. Ͱ͸...ͳ͔ͬͨ...💥💥💥💥
33. Կ͕ى͖͔ͨ...😨 • "No timing found for <ϑΝΠϧ໊>"ͱ͍͏ϝοηʔδ͕ग़ͨ • ςετϑΝΠϧʹର͢Δtiming data͕ଘࡏ͠ͳ͍ͥʂͬͯݴͬͯΔ • timing dataͷϑΝΠϧ͸ੜ੒͞Ε͍͕ͯͨɺԿ͔͠ΒͷݪҼͰ 
 ͏·͘ར༻͞Ε͍ͯͳ͍ͷͰɺ࠷దͳςετ෼ׂ͕ߦ͍͑ͯͳ͍
34. timing dataͷϑΝΠϧΛ֬ೝ͢Δํ๏ cat ${CIRCLE_INTERNAL_TASK_DATA}/circle-test-results/results.json
35. Կ͕μϝ͔ͩͬͨ...😱 • timing dataΛ֬ೝͯ͠ΈͯΘ͔ͬͨͷ͕ɺ֤ςετέʔε͕ ͲͷςετϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λࣔ͢ fi le keyͷ 
 ஋͕nullͩͬͨ
36. Կ͕μϝ͔ͩͬͨ...😱 fi le͕null😱
37. Կ͕μϝ͔ͩͬͨ...·ͱΊΔͱ...😱 • ֤ςετέʔεͷ࣮ߦ࣌ؒͳͲ͸هࡌ͞Ε͍ͯΔ͕ 
 ֤ςετέʔε͕ͲͷϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λ 
 ࣔ͢஋͕nullͳͷͰɺςετϑΝΠϧ͝ͱͷ࣮ߦ͕࣌ؒ 
 ܭࢉͰ͖ͳ͍ • ݁Ռɺ͏·͘ςετ෼ׂͰ͖ͳ͍Θʂͬͯײͩͬͨ͡
38. fi le keyͷ஋͕nullͳݪҼΛ୳Δ • timing data͸ςετϝλσʔλ͔Βੜ੒͞ΕΔ • ςετϝλσʔλ = ςετ݁Ռ͕هࡌ͞ΕͨJunit XMLΛ 
 ֬ೝͯ͠Έͨ
39. <testsuites> <testsuite errors="0" failures="0" tests="2" skipped="0" name=“test.sample_1" > <properties> <property name="platform" value="vm" /> </properties> <testcase classname="test.sample_1" name=“String…” time="0.027" /> <testcase classname="test.sample_1" name=“String.trim…” time="0.005" /> </testsuite> <testsuite errors="0" failures="0" tests="4" skipped="0" name="test.sample_2" > <properties> <property name="platform" value="vm" /> </properties> <testcase classname="test.sample_2" name="String .s…” time="0.027" /> </testsuite> </testsuites> ग़ྗ͞ΕͨJunit XMLͷ಺༰
40. fi le keyͷ஋͕nullͳݪҼΛ୳Δ • testcase elementʹ֤ςετέʔεͷ݁Ռ͕هࡌ͞Ε͍ͯΔ • ͦ͜ʹ֤ςετέʔε͕هࡌ͞Ε͍ͯΔςετϑΝΠϧ໊͕ 
 ॻ͔Ε͍ͯͳ͍͕ͳ͍͜ͱ͕ݪҼͬΆ͍͜ͱ͕Θ͔ͬͨ • ݪҼ͕Θ͔ͬͨͷͰtestcase elementʹ fi le attributeͷ 
 ௥ՃΛࢼΈͨ
41. ࢼͯ͠Έͨमਖ਼ • dart-junitreportͷ಺෦Ͱར༻͞Ε͍ͯΔdart-testreportϥΠϒϥϦͷ मਖ਼ • dart-junitreportͰJunit XMLग़ྗ͢Δࡍɺtestcase elementʹ 
 fi le attributeͷ௥Ճ • fi le attributeʹهࡌ͞Εͨ fi le pathΛCI্ͰsedίϚϯυΛ 
 ར༻ͯ͠ॻ͖׵͑
42. मਖ਼ͯ͠Έͨ݁Ռ • Dart testͰ΋timing dataΛར༻ͯ͠࠷దͳςετ෼ׂ͕ 
 Ͱ͖ΔΑ͏ʹͳͬͨʂ • ΊͰ͍ͨ🎉🎉🎉🎉
43. ׬੒ܗ͕ͪ͜Β💪
44. CircleCIͰtiming dataʹج͍ͮͨςετ෼ׂΛ 
 DartͰར༻Ͱ͖ΔΑ͏ʹ͢ΔαϯϓϧϓϩδΣΫτ https://github.com/operando/dart-test-CircleCI-timing-data ׬੒ܗ✨
45. αϯϓϧϓϩδΣΫτͷCircleCIͷ݁Ռ https://app.circleci.com/pipelines/github/operando/dart-test-CircleCI-timing- data/14/work fl ows/e4884496-b4b0-4d5b-af7a-0209097affac/jobs/16 ׬੒ܗ✨
46. version: 2.1 jobs: test: docker: - image: dart:2.16.1 parallelism: 2 steps: - checkout - run: name: Download dependencies command: pub get - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ workflows: version: 2 build: jobs: - test .circleci/con fi g.ymlͷ಺༰ 
 ௕͍….จࣈখ͍͞…😇
47. - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ େࣄͳͷ͸͜ͷ෦෼ʂ
48. - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ forkͨ͠ϦϙδτϦʹमਖ਼ΛೖΕͨͷͰ 
 ࣗ਎ͷϦϙδτϦ͔Βऔಘ
49. - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ Junit XMLͰग़ྗ͠ɺsedͰ fi le attributeͷ಺༰Λॻ͖׵͑
50. - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ ग़ྗͨ͠Junit XMLΛอଘʂ
51. - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ ͜ΕͰ͜ͷ෦෼ͷςετ෼ׂ͕͏·͘ಈ͍ͨʂ
52. ΍ͬͨͶ😊ʂʂ
53. Thanks🙏