CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

CircleCIͰtiming dataʹ
 
ج͍ͮͨςετ෼ׂΛ
 
DartͰར༻Ͱ͖ΔΑ͏ʹͨ͠࿩
։ൃ×ςετ LTձ - vol.2 #devtestlt

View Slide

ࣗݾ঺հ
• Shinobu Okano

• @operandoOS / shinobu.dart

• 10X, Inc. / Software Engineer

• Flutter / Dart / Android / Automation

View Slide

CircleCIͰςετ
 
ճͯ͠·͔͢ʁ

View Slide

timing dataͷ࿩Λ
͢Δ্ͷͰલఏ஌ࣝ

View Slide

CircleCIͷςετϝλσʔλ
• ςετ݁Ռ͕هࡌ͞ΕͨϑΝΠϧͷ͜ͱʢࡶʣ

• ςετϝλσʔλΛ׆༻͢ΔͱTest InsightsͳͲ͕ར༻Ͱ͖Δ

• timing data͸ςετϝλσʔλ͔Βੜ੒͞ΕΔ

View Slide

Collecting test data
https://circleci.com/docs/2.0/collect-test-data/

View Slide

CircleCIͰςετϝλσʔλΛอଘ͢Δ
• store_test_resultsεςοϓͰςετ݁Ռ
 
ʢςετϝλσʔλʣΛอଘ͢Δ

• https://circleci.com/docs/ja/2.0/con
fi
guration-reference/
#storetestresults

View Slide

CircleCIͰςετϝλσʔλΛอଘ͢Δ
- store_test_results:
path: test-results

View Slide

CircleCIͷςετϝλσʔλͷϑΥʔϚοτ
• CircleCIͰςετ݁Ռ͕ςετϝλσʔλͱೝࣝ͞ΕΔϑΥʔϚοτ͸ҎԼ

• JUnit XML

• Cucumber JSON

• ଟ͘ͷݴޠϥΠϒϥϦͰαϙʔτ͍ͯ͠Δͷ͸JUnit XML

• ຊࢿྉͰ΋JUnit XMLΛѻ͍·͢

View Slide

JUnit XML Schema
• https://github.com/windyroad/JUnit-Schema

• https://github.com/sj26/rspec_junit_formatter/blob/
main/lib/rspec_junit_formatter.rb#L9-L11

• http://windyroad.org/dl/Open%20Source/JUnit.xsd

View Slide

CircleCIͷςετ෼ׂͱฒྻ࣮ߦ
• 1ͭͷ࣮ߦ؀ڥʢExecutorʣͰશςετΛ࣮ߦ͢ΔͱCIͷ࣮ߦ͕࣌ؒ௕͘ͳΔ

• CircleCIʹ͸ςετΛෳ਺ͷExecutorʹ෼ׂͤ͞ΔͨΊͷίϚϯυ͕͋Δ

• ίϚϯυͰ෼ׂͨ͠ςετΛෳ਺ͷExecutorͰฒྻ࣮ߦͤ͞Δ͜ͱʹΑΓ
 
શςετ࣮ߦ͕ૣ͘ͳΓɺCIͷ࣮ߦ࣌ؒ΋୹͘ͳΔ

View Slide

CircleCIͷςετ෼ׂͱฒྻ࣮ߦ
Running Tests in Parallel
https://circleci.com/docs/2.0/parallelism-faster-jobs/

View Slide

CircleCIͷςετ෼ׂͱ
ฒྻ࣮ߦ
https://circleci.com/docs/ja/2.0/parallelism-faster-jobs/
• ެࣜυΩϡϝϯτͷਤΛ͓आΓͯ͠
 
આ໌͢Δͱ͜Μͳײ͡

• Ұ൪Լ͕timing dataʹج͍ͮͨ
 
ύλʔϯͷઆ໌

View Slide

ςετ෼ׂ
• ෼ׂͷ࢓ํʹΑͬͯશମͷςετ࣮ߦ࣌ؒʹӨڹ͢Δ

• ద౰ʹ෼ׂͯ͠ฒྻ࣮ߦͤ͞Δͱ؀ڥ͝ͱʹ࣮ߦ࣌ؒʹ
 
͹Β͖͕ͭग़Δ

• Մೳͳ͔͗Γ؀ڥ͝ͱͷ࣮ߦ࣌ؒͷ͹Β͖͕ͭ
 
খ͘͞ͳΔΑ͏ʹ෼ׂ͢Δͷ͕Α͍✨

View Slide

࠷దͳςετ෼ׂ
• ֤ςετέʔεͷ࣮ߦʹ͔͔Δ࣌ؒΛܭଌ͢Δ

• ܭଌͨ࣌ؒ͠Λར༻͠શମͷςετ͕࣌ؒ୹͘ͳΔΑ͏
 
͍͍ײ͡ͷΞϧΰϦζϜͰ͍͍ײ͡ʹ෼ׂ͢Δ

• ʮ͍͍ײ͡ʯ͕େࣄ😊

View Slide

circleci tests split --split-by=timingsʹ͍ͭͯ
 
ௐ΂ͨɾࣗ࡞ͯ͠Έͨ
htthttps://hoshinotsuyoshi.com/post/circleci_tests_split/

View Slide

CircleCIͷςετ෼ׂ
• ҎԼͷ෼ׂํ๏Λαϙʔτ͍ͯ͠Δ

• timing data

• ϑΝΠϧ໊ɾΫϥε໊

• ϑΝΠϧαΠζ

• CircleCI͸timing dataʹΑΔ෼ׂ͕࠷ྑͷํ๏ͱͯ͠Δ

View Slide

CircleCIͷtiming data
• ֤ςετέʔεͷ࣮ߦ͕࣌ؒهࡌ͞Ε͍ͯΔJSONϑΝΠϧ

• ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ੒͢Δ

• timing dataͷ಺༰͸֬ೝͰ͖Δ

View Slide

CircleCIͷtiming data
• ֤ςετέʔεͷ݁Ռʹ͸ɺΫϥε໊ͱϑΝΠϧ໊ͳͲ͕
 
هࡌ͞Ε͍ͯΔ͜ͱΛCircleCI͸ظ଴͍ͯ͠Δ

• ֤ςετϑΝΠϧͷςετ࣮ߦʹ͔͔Δ͕࣌ؒΘ͔Δσʔλߏ଄

• σϑΥϧτͷ෼ׂํ๏͕ɺϑΝΠϧ໊ʹج͍ͮͨ෼ׂͳͷͰ্ه͕ඞཁ

• ςετϝλσʔλʹͦΕΒͷ৘ใ͕هࡌ͞Ε͍ͯͳ͍ͱtiming dataΛ
 
ར༻ͯ͠΋ࢥͬͨͱ͓Γʹςετ෼ׂ͕࣮ߦ͞Εͳ͍

View Slide

timing dataΛར༻ͯ͠
 
ςετ෼ׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ
circleci tests glob "test/**/*.rb" | circleci tests split --split-by=timings > /tmp/tests-to-run
bundle exec rspec $(cat /tmp/tests-to-run)

View Slide

• circleci tests splitͰςετϑΝΠϧͷҰཡऔಘ

• --split-by=timingsͰtiming dataΛར༻࣮͠ߦ؀ڥ͝ͱʹ
 
ςετ෼ׂΛߦ͏Α͏ʹࢦఆ

• bundle exec rspec΁࣮ߦ؀ڥ͝ͱʹ෼ׂ͞ΕͨςετΛ
 
࣮ߦ͢ΔΑ͏ʹࢦఆ
timing dataΛར༻ͯ͠
 
ςετ෼ׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ

View Slide

Dartͷtestʹ͍ͭͯ

View Slide

Why Dart test?
• ࢲ͕ॴଐ͢Δձࣾ 10XͰ࡞͍ͬͯΔαʔϏε StailerͰ͸
 
API࣮૷ʹαʔόʔαΠυDartΛ࢖༻

• αʔόʔαΠυDartͰ͸CIʹCircleCIΛར༻͍ͯ͠Δ

• DartͰςετΛॻ͘৔߹ʮtestʯύοέʔδ͕ϝδϟʔ

• https://pub.dev/packages/test

View Slide

Dart sharding tests
• testύοέʔδʹ͸ςετ෼ׂ͢Δػೳ͕͋Δʂʂ

• ݱࡏ 10XͰ͸testͷػೳΛ࢖ͬͯCircleCI্ͰςετΛ෼ׂɾฒྻ࣮ߦͯ͠
͍Δ

• ͔͠͠ɺ͜ͷ෼ׂํ๏͸֤ςετέʔεͷ࣮ߦ࣌ؒͳͲΛݩʹ
 
ςετ෼ׂ͞ΕΔΘ͚Ͱ͸ͳ͍

• ͦͷͨΊExecutor͝ͱʹ࣮ߦ࣌ؒͷ͹Β͖͕ͭେ͖͍ͷ͕՝୊

View Slide

Dart sharding tests
dart test --total-shards 3 --shard-index 0 path/to/test.dart

View Slide

Dart sharding tests
testύοέʔδͷςετ෼ׂͷ࣮૷
https://github.com/dart-lang/test/blob/2afbff8cc059b37b4b1a69fc2ddffc53290e4780/
pkgs/test_core/lib/src/runner.dart#L428-L443

View Slide

Dart testsͰCircleCIͷtiming dataΛར༻ͯ͠
 
΋ͬͱ͍͍ײ͡ʹςετ෼ׂɾฒྻ࣮ߦ͍ͨ͠ʂ
ͱࢥͬͨΘ͚Ͱ…

View Slide

Dart testͷ݁ՌΛJunit XMLͰग़ྗ͢Δ
• Dart test͸ςετ݁ՌΛJSONग़ྗ͢Δػೳ͕͋Δ

• ͦͷJSONग़ྗ͞Εͨ݁ՌΛJunit XMLʹม׵͢Δπʔϧ͕͋Δ

• https://github.com/TOPdesk/dart-junitreport

• ͜ΕΛ࢖ͬͯDart testͷςετ݁ՌΛJunit XMLͰग़ྗ͢Δ

View Slide

ग़ྗͨ͠Junit XMLΛςετϝλσʔλͱͯ͠
CircleCIʹೝࣝͤ͞Δ
• store_test_resultsεςοϓʹJunit XMLΛग़ྗͨ͠σΟϨΫτϦΛ
ࢦఆ͢Δ

• ͜ΕͰςετ݁ՌͷJunit XML͕CircleCIʹςετϝλσʔλ
 
ͱͯ͠ೝࣝ͞ΕΔ

• ೝࣝͨ͠ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ੒͢Δ

View Slide

ςετ࣮ߦͷίϚϯυΛॻ͖׵͑Δ
• timing dataΛར༻ͯ͠ςετ෼ׂΛߦ͍ɺ෼ׂͨ͠ϑΝΠϧ
ͷΈͰςετ࣮ߦ͢ΔΑ͏ʹίϚϯυΛॻ͖׵͑Δ
TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)
dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded

View Slide

Α͠ɺ͜ΕͰ׬ᘳ✨✨✨✨

View Slide

Ͱ͸...ͳ͔ͬͨ...💥💥💥💥

View Slide

Կ͕ى͖͔ͨ...😨
• "No timing found for <ϑΝΠϧ໊>"ͱ͍͏ϝοηʔδ͕ग़ͨ

• ςετϑΝΠϧʹର͢Δtiming data͕ଘࡏ͠ͳ͍ͥʂͬͯݴͬͯΔ

• timing dataͷϑΝΠϧ͸ੜ੒͞Ε͍͕ͯͨɺԿ͔͠ΒͷݪҼͰ
 
͏·͘ར༻͞Ε͍ͯͳ͍ͷͰɺ࠷దͳςετ෼ׂ͕ߦ͍͑ͯͳ͍

View Slide

timing dataͷϑΝΠϧΛ֬ೝ͢Δํ๏
cat ${CIRCLE_INTERNAL_TASK_DATA}/circle-test-results/results.json

View Slide

Կ͕μϝ͔ͩͬͨ...😱
• timing dataΛ֬ೝͯ͠ΈͯΘ͔ͬͨͷ͕ɺ֤ςετέʔε͕
ͲͷςετϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λࣔ͢
fi
le keyͷ
 
஋͕nullͩͬͨ

View Slide

Կ͕μϝ͔ͩͬͨ...😱
fi
le͕null😱

View Slide

Կ͕μϝ͔ͩͬͨ...·ͱΊΔͱ...😱
• ֤ςετέʔεͷ࣮ߦ࣌ؒͳͲ͸هࡌ͞Ε͍ͯΔ͕
 
֤ςετέʔε͕ͲͷϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λ
 
ࣔ͢஋͕nullͳͷͰɺςετϑΝΠϧ͝ͱͷ࣮ߦ͕࣌ؒ
 
ܭࢉͰ͖ͳ͍

• ݁Ռɺ͏·͘ςετ෼ׂͰ͖ͳ͍Θʂͬͯײͩͬͨ͡

View Slide

fi
le keyͷ஋͕nullͳݪҼΛ୳Δ
• timing data͸ςετϝλσʔλ͔Βੜ੒͞ΕΔ

• ςετϝλσʔλ = ςετ݁Ռ͕هࡌ͞ΕͨJunit XMLΛ
 
֬ೝͯ͠Έͨ

View Slide

ग़ྗ͞ΕͨJunit XMLͷ಺༰

View Slide

fi
le keyͷ஋͕nullͳݪҼΛ୳Δ
• testcase elementʹ֤ςετέʔεͷ݁Ռ͕هࡌ͞Ε͍ͯΔ

• ͦ͜ʹ֤ςετέʔε͕هࡌ͞Ε͍ͯΔςετϑΝΠϧ໊͕
 
ॻ͔Ε͍ͯͳ͍͕ͳ͍͜ͱ͕ݪҼͬΆ͍͜ͱ͕Θ͔ͬͨ

• ݪҼ͕Θ͔ͬͨͷͰtestcase elementʹ
fi
le attributeͷ
 
௥ՃΛࢼΈͨ

View Slide

ࢼͯ͠Έͨमਖ਼
• dart-junitreportͷ಺෦Ͱར༻͞Ε͍ͯΔdart-testreportϥΠϒϥϦͷ
मਖ਼

• dart-junitreportͰJunit XMLग़ྗ͢Δࡍɺtestcase elementʹ
 
fi
le attributeͷ௥Ճ

•
fi
le attributeʹهࡌ͞Εͨ
fi
le pathΛCI্ͰsedίϚϯυΛ
 
ར༻ͯ͠ॻ͖׵͑

View Slide

मਖ਼ͯ͠Έͨ݁Ռ
• Dart testͰ΋timing dataΛར༻ͯ͠࠷దͳςετ෼ׂ͕
 
Ͱ͖ΔΑ͏ʹͳͬͨʂ

• ΊͰ͍ͨ🎉🎉🎉🎉

View Slide

׬੒ܗ͕ͪ͜Β💪

View Slide

CircleCIͰtiming dataʹج͍ͮͨςετ෼ׂΛ
 
DartͰར༻Ͱ͖ΔΑ͏ʹ͢ΔαϯϓϧϓϩδΣΫτ
https://github.com/operando/dart-test-CircleCI-timing-data
׬੒ܗ✨

View Slide

αϯϓϧϓϩδΣΫτͷCircleCIͷ݁Ռ
https://app.circleci.com/pipelines/github/operando/dart-test-CircleCI-timing-
data/14/work
fl
ows/e4884496-b4b0-4d5b-af7a-0209097affac/jobs/16
׬੒ܗ✨

View Slide

version: 2.1
jobs:
test:
docker:
- image: dart:2.16.1
parallelism: 2
steps:
- checkout
- run:
name: Download dependencies
command: pub get
- run:
name: Test
command: |
- run:
name: Install JUnitReport
when: always
command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git
- run:
name: Convert tests to JUnit
when: always
command: |
$HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml
sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml
path: test_results/
- store_artifacts:
path: test_results/
workflows:
version: 2
build:
jobs:
- test
.circleci/con
fi
g.ymlͷ಺༰
 
௕͍….จࣈখ͍͞…😇

View Slide

- run:
name: Test
command: |
- run:
when: always
- run:
when: always
command: |
$HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json
--output test_results/tests_report_junit.xml
path: test_results/
- store_artifacts:
path: test_results/
େࣄͳͷ͸͜ͷ෦෼ʂ

View Slide

- run:
name: Test
command: |
- run:
when: always
- run:
when: always
command: |
path: test_results/
- store_artifacts:
path: test_results/
forkͨ͠ϦϙδτϦʹमਖ਼ΛೖΕͨͷͰ
 
ࣗ਎ͷϦϙδτϦ͔Βऔಘ

View Slide

- run:
name: Test
command: |
- run:
when: always
- run:
when: always
command: |
path: test_results/
- store_artifacts:
path: test_results/
Junit XMLͰग़ྗ͠ɺsedͰ
fi
le attributeͷ಺༰Λॻ͖׵͑

View Slide

- run:
name: Test
command: |
- run:
when: always
- run:
when: always
command: |
path: test_results/
- store_artifacts:
path: test_results/
ग़ྗͨ͠Junit XMLΛอଘʂ

View Slide

- run:
name: Test
command: |
- run:
when: always
- run:
when: always
command: |
path: test_results/
- store_artifacts:
path: test_results/
͜ΕͰ͜ͷ෦෼ͷςετ෼ׂ͕͏·͘ಈ͍ͨʂ

View Slide

΍ͬͨͶ😊ʂʂ

View Slide

Thanks🙏

View Slide

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

operandoOS

More Decks by operandoOS

Other Decks in Technology

Featured

Transcript