CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

by operandoOS

Slide 1

Slide 1 text

CircleCIͰtiming dataʹ   ج͍ͮͨςετ෼ׂΛ   DartͰར༻Ͱ͖ΔΑ͏ʹͨ͠࿩ ։ൃ×ςετ LTձ - vol.2 #devtestlt

Slide 2

Slide 2 text

ࣗݾ঺հ • Shinobu Okano • @operandoOS / shinobu.dart • 10X, Inc. / Software Engineer • Flutter / Dart / Android / Automation

Slide 3

Slide 3 text

CircleCIͰςετ   ճͯ͠·͔͢ʁ

Slide 4

Slide 4 text

timing dataͷ࿩Λ ͢Δ্ͷͰલఏ஌ࣝ

Slide 5

Slide 5 text

CircleCIͷςετϝλσʔλ • ςετ݁Ռ͕هࡌ͞ΕͨϑΝΠϧͷ͜ͱʢࡶʣ • ςετϝλσʔλΛ׆༻͢ΔͱTest InsightsͳͲ͕ར༻Ͱ͖Δ • timing data͸ςετϝλσʔλ͔Βੜ੒͞ΕΔ

Slide 6

Slide 6 text

CircleCIͷςετϝλσʔλ Collecting test data https://circleci.com/docs/2.0/collect-test-data/

Slide 7

Slide 7 text

CircleCIͰςετϝλσʔλΛอଘ͢Δ • store_test_resultsεςοϓͰςετ݁Ռ   ʢςετϝλσʔλʣΛอଘ͢Δ • https://circleci.com/docs/ja/2.0/con fi guration-reference/ #storetestresults

Slide 8

Slide 8 text

CircleCIͰςετϝλσʔλΛอଘ͢Δ - store_test_results: path: test-results

Slide 9

Slide 9 text

CircleCIͷςετϝλσʔλͷϑΥʔϚοτ • CircleCIͰςετ݁Ռ͕ςετϝλσʔλͱೝࣝ͞ΕΔϑΥʔϚοτ͸ҎԼ • JUnit XML • Cucumber JSON • ଟ͘ͷݴޠϥΠϒϥϦͰαϙʔτ͍ͯ͠Δͷ͸JUnit XML • ຊࢿྉͰ΋JUnit XMLΛѻ͍·͢

Slide 10

Slide 10 text

JUnit XML Schema • https://github.com/windyroad/JUnit-Schema • https://github.com/sj26/rspec_junit_formatter/blob/ main/lib/rspec_junit_formatter.rb#L9-L11 • http://windyroad.org/dl/Open%20Source/JUnit.xsd

Slide 11

Slide 11 text

CircleCIͷςετ෼ׂͱฒྻ࣮ߦ • 1ͭͷ࣮ߦ؀ڥʢExecutorʣͰશςετΛ࣮ߦ͢ΔͱCIͷ࣮ߦ͕࣌ؒ௕͘ͳΔ • CircleCIʹ͸ςετΛෳ਺ͷExecutorʹ෼ׂͤ͞ΔͨΊͷίϚϯυ͕͋Δ • ίϚϯυͰ෼ׂͨ͠ςετΛෳ਺ͷExecutorͰฒྻ࣮ߦͤ͞Δ͜ͱʹΑΓ   શςετ࣮ߦ͕ૣ͘ͳΓɺCIͷ࣮ߦ࣌ؒ΋୹͘ͳΔ

Slide 12

Slide 12 text

CircleCIͷςετ෼ׂͱฒྻ࣮ߦ Running Tests in Parallel https://circleci.com/docs/2.0/parallelism-faster-jobs/

Slide 13

Slide 13 text

CircleCIͷςετ෼ׂͱ ฒྻ࣮ߦ https://circleci.com/docs/ja/2.0/parallelism-faster-jobs/ • ެࣜυΩϡϝϯτͷਤΛ͓आΓͯ͠   આ໌͢Δͱ͜Μͳײ͡ • Ұ൪Լ͕timing dataʹج͍ͮͨ   ύλʔϯͷઆ໌

Slide 14

Slide 14 text

ςετ෼ׂ • ෼ׂͷ࢓ํʹΑͬͯશମͷςετ࣮ߦ࣌ؒʹӨڹ͢Δ • ద౰ʹ෼ׂͯ͠ฒྻ࣮ߦͤ͞Δͱ؀ڥ͝ͱʹ࣮ߦ࣌ؒʹ   ͹Β͖͕ͭग़Δ • Մೳͳ͔͗Γ؀ڥ͝ͱͷ࣮ߦ࣌ؒͷ͹Β͖͕ͭ   খ͘͞ͳΔΑ͏ʹ෼ׂ͢Δͷ͕Α͍✨

Slide 15

Slide 15 text

࠷దͳςετ෼ׂ • ֤ςετέʔεͷ࣮ߦʹ͔͔Δ࣌ؒΛܭଌ͢Δ • ܭଌͨ࣌ؒ͠Λར༻͠શମͷςετ͕࣌ؒ୹͘ͳΔΑ͏   ͍͍ײ͡ͷΞϧΰϦζϜͰ͍͍ײ͡ʹ෼ׂ͢Δ • ʮ͍͍ײ͡ʯ͕େࣄ😊

Slide 16

Slide 16 text

CircleCIͷςετϝλσʔλ circleci tests split --split-by=timingsʹ͍ͭͯ   ௐ΂ͨɾࣗ࡞ͯ͠Έͨ htthttps://hoshinotsuyoshi.com/post/circleci_tests_split/

Slide 17

Slide 17 text

CircleCIͷςετ෼ׂ • ҎԼͷ෼ׂํ๏Λαϙʔτ͍ͯ͠Δ • timing data • ϑΝΠϧ໊ɾΫϥε໊ • ϑΝΠϧαΠζ • CircleCI͸timing dataʹΑΔ෼ׂ͕࠷ྑͷํ๏ͱͯ͠Δ

Slide 18

Slide 18 text

CircleCIͷtiming data • ֤ςετέʔεͷ࣮ߦ͕࣌ؒهࡌ͞Ε͍ͯΔJSONϑΝΠϧ • ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ੒͢Δ • timing dataͷ಺༰͸֬ೝͰ͖Δ

Slide 19

Slide 19 text

CircleCIͷtiming data • ֤ςετέʔεͷ݁Ռʹ͸ɺΫϥε໊ͱϑΝΠϧ໊ͳͲ͕   هࡌ͞Ε͍ͯΔ͜ͱΛCircleCI͸ظ଴͍ͯ͠Δ • ֤ςετϑΝΠϧͷςετ࣮ߦʹ͔͔Δ͕࣌ؒΘ͔Δσʔλߏ଄ • σϑΥϧτͷ෼ׂํ๏͕ɺϑΝΠϧ໊ʹج͍ͮͨ෼ׂͳͷͰ্ه͕ඞཁ • ςετϝλσʔλʹͦΕΒͷ৘ใ͕هࡌ͞Ε͍ͯͳ͍ͱtiming dataΛ   ར༻ͯ͠΋ࢥͬͨͱ͓Γʹςετ෼ׂ͕࣮ߦ͞Εͳ͍

Slide 20

Slide 20 text

timing dataΛར༻ͯ͠   ςετ෼ׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ circleci tests glob "test/**/*.rb" | circleci tests split --split-by=timings > /tmp/tests-to-run bundle exec rspec $(cat /tmp/tests-to-run)

Slide 21

Slide 21 text

• circleci tests splitͰςετϑΝΠϧͷҰཡऔಘ • --split-by=timingsͰtiming dataΛར༻࣮͠ߦ؀ڥ͝ͱʹ   ςετ෼ׂΛߦ͏Α͏ʹࢦఆ • bundle exec rspec΁࣮ߦ؀ڥ͝ͱʹ෼ׂ͞ΕͨςετΛ   ࣮ߦ͢ΔΑ͏ʹࢦఆ timing dataΛར༻ͯ͠   ςετ෼ׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ

Slide 22

Slide 22 text

Dartͷtestʹ͍ͭͯ

Slide 23

Slide 23 text

Why Dart test? • ࢲ͕ॴଐ͢Δձࣾ 10XͰ࡞͍ͬͯΔαʔϏε StailerͰ͸   API࣮૷ʹαʔόʔαΠυDartΛ࢖༻ • αʔόʔαΠυDartͰ͸CIʹCircleCIΛར༻͍ͯ͠Δ • DartͰςετΛॻ͘৔߹ʮtestʯύοέʔδ͕ϝδϟʔ • https://pub.dev/packages/test

Slide 24

Slide 24 text

Dart sharding tests • testύοέʔδʹ͸ςετ෼ׂ͢Δػೳ͕͋Δʂʂ • ݱࡏ 10XͰ͸testͷػೳΛ࢖ͬͯCircleCI্ͰςετΛ෼ׂɾฒྻ࣮ߦͯ͠ ͍Δ • ͔͠͠ɺ͜ͷ෼ׂํ๏͸֤ςετέʔεͷ࣮ߦ࣌ؒͳͲΛݩʹ   ςετ෼ׂ͞ΕΔΘ͚Ͱ͸ͳ͍ • ͦͷͨΊExecutor͝ͱʹ࣮ߦ࣌ؒͷ͹Β͖͕ͭେ͖͍ͷ͕՝୊

Slide 25

Slide 25 text

Dart sharding tests dart test --total-shards 3 --shard-index 0 path/to/test.dart dart test --total-shards 3 --shard-index 1 path/to/test.dart dart test --total-shards 3 --shard-index 2 path/to/test.dart

Slide 26

Slide 26 text

Dart sharding tests testύοέʔδͷςετ෼ׂͷ࣮૷ https://github.com/dart-lang/test/blob/2afbff8cc059b37b4b1a69fc2ddffc53290e4780/ pkgs/test_core/lib/src/runner.dart#L428-L443

Slide 27

Slide 27 text

Dart testsͰCircleCIͷtiming dataΛར༻ͯ͠   ΋ͬͱ͍͍ײ͡ʹςετ෼ׂɾฒྻ࣮ߦ͍ͨ͠ʂ ͱࢥͬͨΘ͚Ͱ…

Slide 28

Slide 28 text

Dart testͷ݁ՌΛJunit XMLͰग़ྗ͢Δ • Dart test͸ςετ݁ՌΛJSONग़ྗ͢Δػೳ͕͋Δ • ͦͷJSONग़ྗ͞Εͨ݁ՌΛJunit XMLʹม׵͢Δπʔϧ͕͋Δ • https://github.com/TOPdesk/dart-junitreport • ͜ΕΛ࢖ͬͯDart testͷςετ݁ՌΛJunit XMLͰग़ྗ͢Δ

Slide 29

Slide 29 text

ग़ྗͨ͠Junit XMLΛςετϝλσʔλͱͯ͠ CircleCIʹೝࣝͤ͞Δ • store_test_resultsεςοϓʹJunit XMLΛग़ྗͨ͠σΟϨΫτϦΛ ࢦఆ͢Δ • ͜ΕͰςετ݁ՌͷJunit XML͕CircleCIʹςετϝλσʔλ   ͱͯ͠ೝࣝ͞ΕΔ • ೝࣝͨ͠ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ੒͢Δ

Slide 30

Slide 30 text

ςετ࣮ߦͷίϚϯυΛॻ͖׵͑Δ • timing dataΛར༻ͯ͠ςετ෼ׂΛߦ͍ɺ෼ׂͨ͠ϑΝΠϧ ͷΈͰςετ࣮ߦ͢ΔΑ͏ʹίϚϯυΛॻ͖׵͑Δ TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded

Slide 31

Slide 31 text

Α͠ɺ͜ΕͰ׬ᘳ✨✨✨✨

Slide 32

Slide 32 text

Ͱ͸...ͳ͔ͬͨ...💥💥💥💥

Slide 33

Slide 33 text

Կ͕ى͖͔ͨ...😨 • "No timing found for <ϑΝΠϧ໊>"ͱ͍͏ϝοηʔδ͕ग़ͨ • ςετϑΝΠϧʹର͢Δtiming data͕ଘࡏ͠ͳ͍ͥʂͬͯݴͬͯΔ • timing dataͷϑΝΠϧ͸ੜ੒͞Ε͍͕ͯͨɺԿ͔͠ΒͷݪҼͰ   ͏·͘ར༻͞Ε͍ͯͳ͍ͷͰɺ࠷దͳςετ෼ׂ͕ߦ͍͑ͯͳ͍

Slide 34

Slide 34 text

timing dataͷϑΝΠϧΛ֬ೝ͢Δํ๏ cat ${CIRCLE_INTERNAL_TASK_DATA}/circle-test-results/results.json

Slide 35

Slide 35 text

Կ͕μϝ͔ͩͬͨ...😱 • timing dataΛ֬ೝͯ͠ΈͯΘ͔ͬͨͷ͕ɺ֤ςετέʔε͕ ͲͷςετϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λࣔ͢ fi le keyͷ   ஋͕nullͩͬͨ

Slide 36

Slide 36 text

Կ͕μϝ͔ͩͬͨ...😱 fi le͕null😱

Slide 37

Slide 37 text

Կ͕μϝ͔ͩͬͨ...·ͱΊΔͱ...😱 • ֤ςετέʔεͷ࣮ߦ࣌ؒͳͲ͸هࡌ͞Ε͍ͯΔ͕   ֤ςετέʔε͕ͲͷϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λ   ࣔ͢஋͕nullͳͷͰɺςετϑΝΠϧ͝ͱͷ࣮ߦ͕࣌ؒ   ܭࢉͰ͖ͳ͍ • ݁Ռɺ͏·͘ςετ෼ׂͰ͖ͳ͍Θʂͬͯײͩͬͨ͡

Slide 38

Slide 38 text

fi le keyͷ஋͕nullͳݪҼΛ୳Δ • timing data͸ςετϝλσʔλ͔Βੜ੒͞ΕΔ • ςετϝλσʔλ = ςετ݁Ռ͕هࡌ͞ΕͨJunit XMLΛ   ֬ೝͯ͠Έͨ

Slide 39

Slide 39 text

ग़ྗ͞ΕͨJunit XMLͷ಺༰

Slide 40

Slide 40 text

fi le keyͷ஋͕nullͳݪҼΛ୳Δ • testcase elementʹ֤ςετέʔεͷ݁Ռ͕هࡌ͞Ε͍ͯΔ • ͦ͜ʹ֤ςετέʔε͕هࡌ͞Ε͍ͯΔςετϑΝΠϧ໊͕   ॻ͔Ε͍ͯͳ͍͕ͳ͍͜ͱ͕ݪҼͬΆ͍͜ͱ͕Θ͔ͬͨ • ݪҼ͕Θ͔ͬͨͷͰtestcase elementʹ fi le attributeͷ   ௥ՃΛࢼΈͨ

Slide 41

Slide 41 text

ࢼͯ͠Έͨमਖ਼ • dart-junitreportͷ಺෦Ͱར༻͞Ε͍ͯΔdart-testreportϥΠϒϥϦͷ मਖ਼ • dart-junitreportͰJunit XMLग़ྗ͢Δࡍɺtestcase elementʹ   fi le attributeͷ௥Ճ • fi le attributeʹهࡌ͞Εͨ fi le pathΛCI্ͰsedίϚϯυΛ   ར༻ͯ͠ॻ͖׵͑

Slide 42

Slide 42 text

मਖ਼ͯ͠Έͨ݁Ռ • Dart testͰ΋timing dataΛར༻ͯ͠࠷దͳςετ෼ׂ͕   Ͱ͖ΔΑ͏ʹͳͬͨʂ • ΊͰ͍ͨ🎉🎉🎉🎉

Slide 43

Slide 43 text

׬੒ܗ͕ͪ͜Β💪

Slide 44

Slide 44 text

CircleCIͰtiming dataʹج͍ͮͨςετ෼ׂΛ   DartͰར༻Ͱ͖ΔΑ͏ʹ͢ΔαϯϓϧϓϩδΣΫτ https://github.com/operando/dart-test-CircleCI-timing-data ׬੒ܗ✨

Slide 45

Slide 45 text

αϯϓϧϓϩδΣΫτͷCircleCIͷ݁Ռ https://app.circleci.com/pipelines/github/operando/dart-test-CircleCI-timing- data/14/work fl ows/e4884496-b4b0-4d5b-af7a-0209097affac/jobs/16 ׬੒ܗ✨

Slide 46

Slide 46 text

version: 2.1 jobs: test: docker: - image: dart:2.16.1 parallelism: 2 steps: - checkout - run: name: Download dependencies command: pub get - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ workflows: version: 2 build: jobs: - test .circleci/con fi g.ymlͷ಺༰   ௕͍….จࣈখ͍͞…😇

Slide 47

Slide 47 text

- run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ େࣄͳͷ͸͜ͷ෦෼ʂ