CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.
文字起こし https://github.com/operando/Notes/tree/master/%E9%96%8B%E7%99%BA%C3%97%E3%83%86%E3%82%B9%E3%83%88%20LT%E4%BC%9A%20-%20vol.2%20%23devtestlt
CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにするサンプルプロジェクト https://github.com/operando/dart-test-CircleCI-timing-data
開発×テスト LT会 - vol.2 #devtestlt https://rakus.connpass.com/event/237602/
CircleCIͰtiming dataʹ ج͍ͮͨςετׂΛ DartͰར༻Ͱ͖ΔΑ͏ʹͨ͠։ൃ×ςετ LTձ - vol.2 #devtestlt
View Slide
ࣗݾհ• Shinobu Okano• @operandoOS / shinobu.dart• 10X, Inc. / Software Engineer• Flutter / Dart / Android / Automation
CircleCIͰςετ ճͯ͠·͔͢ʁ
timing dataͷΛ͢Δ্ͷͰલఏࣝ
CircleCIͷςετϝλσʔλ• ςετ݁Ռ͕هࡌ͞ΕͨϑΝΠϧͷ͜ͱʢࡶʣ• ςετϝλσʔλΛ׆༻͢ΔͱTest InsightsͳͲ͕ར༻Ͱ͖Δ• timing dataςετϝλσʔλ͔Βੜ͞ΕΔ
CircleCIͷςετϝλσʔλCollecting test datahttps://circleci.com/docs/2.0/collect-test-data/
CircleCIͰςετϝλσʔλΛอଘ͢Δ• store_test_resultsεςοϓͰςετ݁Ռ ʢςετϝλσʔλʣΛอଘ͢Δ• https://circleci.com/docs/ja/2.0/configuration-reference/#storetestresults
CircleCIͰςετϝλσʔλΛอଘ͢Δ- store_test_results:path: test-results
CircleCIͷςετϝλσʔλͷϑΥʔϚοτ• CircleCIͰςετ݁Ռ͕ςετϝλσʔλͱೝࣝ͞ΕΔϑΥʔϚοτҎԼ• JUnit XML• Cucumber JSON• ଟ͘ͷݴޠϥΠϒϥϦͰαϙʔτ͍ͯ͠ΔͷJUnit XML• ຊࢿྉͰJUnit XMLΛѻ͍·͢
JUnit XML Schema• https://github.com/windyroad/JUnit-Schema• https://github.com/sj26/rspec_junit_formatter/blob/main/lib/rspec_junit_formatter.rb#L9-L11• http://windyroad.org/dl/Open%20Source/JUnit.xsd
CircleCIͷςετׂͱฒྻ࣮ߦ• 1ͭͷ࣮ߦڥʢExecutorʣͰશςετΛ࣮ߦ͢ΔͱCIͷ࣮ߦ͕࣌ؒ͘ͳΔ• CircleCIʹςετΛෳͷExecutorʹׂͤ͞ΔͨΊͷίϚϯυ͕͋Δ• ίϚϯυͰׂͨ͠ςετΛෳͷExecutorͰฒྻ࣮ߦͤ͞Δ͜ͱʹΑΓ શςετ࣮ߦ͕ૣ͘ͳΓɺCIͷ࣮ߦ࣌ؒ͘ͳΔ
CircleCIͷςετׂͱฒྻ࣮ߦRunning Tests in Parallelhttps://circleci.com/docs/2.0/parallelism-faster-jobs/
CircleCIͷςετׂͱฒྻ࣮ߦhttps://circleci.com/docs/ja/2.0/parallelism-faster-jobs/• ެࣜυΩϡϝϯτͷਤΛ͓आΓͯ͠ આ໌͢Δͱ͜Μͳײ͡• Ұ൪Լ͕timing dataʹج͍ͮͨ ύλʔϯͷઆ໌
ςετׂ• ׂͷํʹΑͬͯશମͷςετ࣮ߦ࣌ؒʹӨڹ͢Δ• దʹׂͯ͠ฒྻ࣮ߦͤ͞Δͱڥ͝ͱʹ࣮ߦ࣌ؒʹ Β͖͕ͭग़Δ• Մೳͳ͔͗Γڥ͝ͱͷ࣮ߦ࣌ؒͷΒ͖͕ͭ খ͘͞ͳΔΑ͏ʹׂ͢Δͷ͕Α͍✨
࠷దͳςετׂ• ֤ςετέʔεͷ࣮ߦʹ͔͔Δ࣌ؒΛܭଌ͢Δ• ܭଌͨ࣌ؒ͠Λར༻͠શମͷςετ͕࣌ؒ͘ͳΔΑ͏ ͍͍ײ͡ͷΞϧΰϦζϜͰ͍͍ײ͡ʹׂ͢Δ• ʮ͍͍ײ͡ʯ͕େࣄ😊
CircleCIͷςετϝλσʔλcircleci tests split --split-by=timingsʹ͍ͭͯ ௐͨɾࣗ࡞ͯ͠Έͨhtthttps://hoshinotsuyoshi.com/post/circleci_tests_split/
CircleCIͷςετׂ• ҎԼͷׂํ๏Λαϙʔτ͍ͯ͠Δ• timing data• ϑΝΠϧ໊ɾΫϥε໊• ϑΝΠϧαΠζ• CircleCItiming dataʹΑΔׂ͕࠷ྑͷํ๏ͱͯ͠Δ
CircleCIͷtiming data• ֤ςετέʔεͷ࣮ߦ͕࣌ؒهࡌ͞Ε͍ͯΔJSONϑΝΠϧ• ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ͢Δ• timing dataͷ༰֬ೝͰ͖Δ
CircleCIͷtiming data• ֤ςετέʔεͷ݁ՌʹɺΫϥε໊ͱϑΝΠϧ໊ͳͲ͕ هࡌ͞Ε͍ͯΔ͜ͱΛCircleCIظ͍ͯ͠Δ• ֤ςετϑΝΠϧͷςετ࣮ߦʹ͔͔Δ͕࣌ؒΘ͔Δσʔλߏ• σϑΥϧτͷׂํ๏͕ɺϑΝΠϧ໊ʹجׂ͍ͮͨͳͷͰ্ه͕ඞཁ• ςετϝλσʔλʹͦΕΒͷใ͕هࡌ͞Ε͍ͯͳ͍ͱtiming dataΛ ར༻ͯ͠ࢥͬͨͱ͓Γʹςετׂ͕࣮ߦ͞Εͳ͍
timing dataΛར༻ͯ͠ ςετׂ࣮ͯ͠ߦ͢Δྫʢrspecʣcircleci tests glob "test/**/*.rb" | circleci tests split --split-by=timings > /tmp/tests-to-runbundle exec rspec $(cat /tmp/tests-to-run)
• circleci tests splitͰςετϑΝΠϧͷҰཡऔಘ• --split-by=timingsͰtiming dataΛར༻࣮͠ߦڥ͝ͱʹ ςετׂΛߦ͏Α͏ʹࢦఆ• bundle exec rspec࣮ߦڥ͝ͱʹׂ͞ΕͨςετΛ ࣮ߦ͢ΔΑ͏ʹࢦఆtiming dataΛར༻ͯ͠ ςετׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ
Dartͷtestʹ͍ͭͯ
Why Dart test?• ࢲ͕ॴଐ͢Δձࣾ 10XͰ࡞͍ͬͯΔαʔϏε StailerͰ API࣮ʹαʔόʔαΠυDartΛ༻• αʔόʔαΠυDartͰCIʹCircleCIΛར༻͍ͯ͠Δ• DartͰςετΛॻ͘߹ʮtestʯύοέʔδ͕ϝδϟʔ• https://pub.dev/packages/test
Dart sharding tests• testύοέʔδʹςετׂ͢Δػೳ͕͋Δʂʂ• ݱࡏ 10XͰtestͷػೳΛͬͯCircleCI্ͰςετΛׂɾฒྻ࣮ߦ͍ͯ͠Δ• ͔͠͠ɺ͜ͷׂํ๏֤ςετέʔεͷ࣮ߦ࣌ؒͳͲΛݩʹ ςετׂ͞ΕΔΘ͚Ͱͳ͍• ͦͷͨΊExecutor͝ͱʹ࣮ߦ࣌ؒͷΒ͖͕ͭେ͖͍ͷ͕՝
Dart sharding testsdart test --total-shards 3 --shard-index 0 path/to/test.dartdart test --total-shards 3 --shard-index 1 path/to/test.dartdart test --total-shards 3 --shard-index 2 path/to/test.dart
Dart sharding teststestύοέʔδͷςετׂͷ࣮https://github.com/dart-lang/test/blob/2afbff8cc059b37b4b1a69fc2ddffc53290e4780/pkgs/test_core/lib/src/runner.dart#L428-L443
Dart testsͰCircleCIͷtiming dataΛར༻ͯ͠ ͬͱ͍͍ײ͡ʹςετׂɾฒྻ࣮ߦ͍ͨ͠ʂͱࢥͬͨΘ͚Ͱ…
Dart testͷ݁ՌΛJunit XMLͰग़ྗ͢Δ• Dart testςετ݁ՌΛJSONग़ྗ͢Δػೳ͕͋Δ• ͦͷJSONग़ྗ͞Εͨ݁ՌΛJunit XMLʹม͢Δπʔϧ͕͋Δ• https://github.com/TOPdesk/dart-junitreport• ͜ΕΛͬͯDart testͷςετ݁ՌΛJunit XMLͰग़ྗ͢Δ
ग़ྗͨ͠Junit XMLΛςετϝλσʔλͱͯ͠CircleCIʹೝࣝͤ͞Δ• store_test_resultsεςοϓʹJunit XMLΛग़ྗͨ͠σΟϨΫτϦΛࢦఆ͢Δ• ͜ΕͰςετ݁ՌͷJunit XML͕CircleCIʹςετϝλσʔλ ͱͯ͠ೝࣝ͞ΕΔ• ೝࣝͨ͠ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ͢Δ
ςετ࣮ߦͷίϚϯυΛॻ͖͑Δ• timing dataΛར༻ͯ͠ςετׂΛߦ͍ɺׂͨ͠ϑΝΠϧͷΈͰςετ࣮ߦ͢ΔΑ͏ʹίϚϯυΛॻ͖͑ΔTESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded
Α͠ɺ͜ΕͰᘳ✨✨✨✨
Ͱ...ͳ͔ͬͨ...💥💥💥💥
Կ͕ى͖͔ͨ...😨• "No timing found for <ϑΝΠϧ໊>"ͱ͍͏ϝοηʔδ͕ग़ͨ• ςετϑΝΠϧʹର͢Δtiming data͕ଘࡏ͠ͳ͍ͥʂͬͯݴͬͯΔ• timing dataͷϑΝΠϧੜ͞Ε͍͕ͯͨɺԿ͔͠ΒͷݪҼͰ ͏·͘ར༻͞Ε͍ͯͳ͍ͷͰɺ࠷దͳςετׂ͕ߦ͍͑ͯͳ͍
timing dataͷϑΝΠϧΛ֬ೝ͢Δํ๏cat ${CIRCLE_INTERNAL_TASK_DATA}/circle-test-results/results.json
Կ͕μϝ͔ͩͬͨ...😱• timing dataΛ֬ೝͯ͠ΈͯΘ͔ͬͨͷ͕ɺ֤ςετέʔε͕ͲͷςετϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λࣔ͢file keyͷ ͕nullͩͬͨ
Կ͕μϝ͔ͩͬͨ...😱file͕null😱
Կ͕μϝ͔ͩͬͨ...·ͱΊΔͱ...😱• ֤ςετέʔεͷ࣮ߦ࣌ؒͳͲهࡌ͞Ε͍ͯΔ͕ ֤ςετέʔε͕ͲͷϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λ ͕ࣔ͢nullͳͷͰɺςετϑΝΠϧ͝ͱͷ࣮ߦ͕࣌ؒ ܭࢉͰ͖ͳ͍• ݁Ռɺ͏·͘ςετׂͰ͖ͳ͍Θʂͬͯײͩͬͨ͡
file keyͷ͕nullͳݪҼΛ୳Δ• timing dataςετϝλσʔλ͔Βੜ͞ΕΔ• ςετϝλσʔλ = ςετ݁Ռ͕هࡌ͞ΕͨJunit XMLΛ ֬ೝͯ͠Έͨ
ग़ྗ͞ΕͨJunit XMLͷ༰
file keyͷ͕nullͳݪҼΛ୳Δ• testcase elementʹ֤ςετέʔεͷ݁Ռ͕هࡌ͞Ε͍ͯΔ• ͦ͜ʹ֤ςετέʔε͕هࡌ͞Ε͍ͯΔςετϑΝΠϧ໊͕ ॻ͔Ε͍ͯͳ͍͕ͳ͍͜ͱ͕ݪҼͬΆ͍͜ͱ͕Θ͔ͬͨ• ݪҼ͕Θ͔ͬͨͷͰtestcase elementʹfile attributeͷ ՃΛࢼΈͨ
ࢼͯ͠Έͨमਖ਼• dart-junitreportͷ෦Ͱར༻͞Ε͍ͯΔdart-testreportϥΠϒϥϦͷमਖ਼• dart-junitreportͰJunit XMLग़ྗ͢Δࡍɺtestcase elementʹ file attributeͷՃ•file attributeʹهࡌ͞Εͨfile pathΛCI্ͰsedίϚϯυΛ ར༻ͯ͠ॻ͖͑
मਖ਼ͯ͠Έͨ݁Ռ• Dart testͰtiming dataΛར༻ͯ͠࠷దͳςετׂ͕ Ͱ͖ΔΑ͏ʹͳͬͨʂ• ΊͰ͍ͨ🎉🎉🎉🎉
ܗ͕ͪ͜Β💪
CircleCIͰtiming dataʹج͍ͮͨςετׂΛ DartͰར༻Ͱ͖ΔΑ͏ʹ͢ΔαϯϓϧϓϩδΣΫτhttps://github.com/operando/dart-test-CircleCI-timing-dataܗ✨
αϯϓϧϓϩδΣΫτͷCircleCIͷ݁Ռhttps://app.circleci.com/pipelines/github/operando/dart-test-CircleCI-timing-data/14/workflows/e4884496-b4b0-4d5b-af7a-0209097affac/jobs/16ܗ✨
version: 2.1jobs:test:docker:- image: dart:2.16.1parallelism: 2steps:- checkout- run:name: Download dependenciescommand: pub get- run:name: Testcommand: |TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded- run:name: Install JUnitReportwhen: alwayscommand: dart pub global activate --source git https://github.com/operando/dart-junitreport.git- run:name: Convert tests to JUnitwhen: alwayscommand: |$HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xmlsed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml- store_test_results:path: test_results/- store_artifacts:path: test_results/workflows:version: 2build:jobs:- test.circleci/config.ymlͷ༰ ͍….จࣈখ͍͞…😇
- run:name: Testcommand: |TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded- run:name: Install JUnitReportwhen: alwayscommand: dart pub global activate --source git https://github.com/operando/dart-junitreport.git- run:name: Convert tests to JUnitwhen: alwayscommand: |$HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json--output test_results/tests_report_junit.xmlsed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml- store_test_results:path: test_results/- store_artifacts:path: test_results/େࣄͳͷ͜ͷ෦ʂ
- run:name: Testcommand: |TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded- run:name: Install JUnitReportwhen: alwayscommand: dart pub global activate --source git https://github.com/operando/dart-junitreport.git- run:name: Convert tests to JUnitwhen: alwayscommand: |$HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json--output test_results/tests_report_junit.xmlsed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml- store_test_results:path: test_results/- store_artifacts:path: test_results/forkͨ͠ϦϙδτϦʹमਖ਼ΛೖΕͨͷͰ ࣗͷϦϙδτϦ͔Βऔಘ
- run:name: Testcommand: |TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded- run:name: Install JUnitReportwhen: alwayscommand: dart pub global activate --source git https://github.com/operando/dart-junitreport.git- run:name: Convert tests to JUnitwhen: alwayscommand: |$HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json--output test_results/tests_report_junit.xmlsed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml- store_test_results:path: test_results/- store_artifacts:path: test_results/Junit XMLͰग़ྗ͠ɺsedͰfile attributeͷ༰Λॻ͖͑
- run:name: Testcommand: |TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded- run:name: Install JUnitReportwhen: alwayscommand: dart pub global activate --source git https://github.com/operando/dart-junitreport.git- run:name: Convert tests to JUnitwhen: alwayscommand: |$HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json--output test_results/tests_report_junit.xmlsed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml- store_test_results:path: test_results/- store_artifacts:path: test_results/ग़ྗͨ͠Junit XMLΛอଘʂ
- run:name: Testcommand: |TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded- run:name: Install JUnitReportwhen: alwayscommand: dart pub global activate --source git https://github.com/operando/dart-junitreport.git- run:name: Convert tests to JUnitwhen: alwayscommand: |$HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json--output test_results/tests_report_junit.xmlsed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml- store_test_results:path: test_results/- store_artifacts:path: test_results/͜ΕͰ͜ͷ෦ͷςετׂ͕͏·͘ಈ͍ͨʂ
ͬͨͶ😊ʂʂ
Thanks🙏