$30 off During Our Annual Pro Sale. View Details »

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

operandoOS
March 15, 2022

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.

文字起こし
https://github.com/operando/Notes/tree/master/%E9%96%8B%E7%99%BA%C3%97%E3%83%86%E3%82%B9%E3%83%88%20LT%E4%BC%9A%20-%20vol.2%20%23devtestlt

CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにするサンプルプロジェクト
https://github.com/operando/dart-test-CircleCI-timing-data

開発×テスト LT会 - vol.2 #devtestlt
https://rakus.connpass.com/event/237602/

operandoOS

March 15, 2022
Tweet

More Decks by operandoOS

Other Decks in Technology

Transcript

  1. CircleCIͰtiming dataʹ

    ج͍ͮͨςετ෼ׂΛ

    DartͰར༻Ͱ͖ΔΑ͏ʹͨ͠࿩
    ։ൃ×ςετ LTձ - vol.2 #devtestlt

    View Slide

  2. ࣗݾ঺հ
    • Shinobu Okano


    • @operandoOS / shinobu.dart


    • 10X, Inc. / Software Engineer


    • Flutter / Dart / Android / Automation

    View Slide

  3. CircleCIͰςετ

    ճͯ͠·͔͢ʁ

    View Slide

  4. timing dataͷ࿩Λ
    ͢Δ্ͷͰલఏ஌ࣝ

    View Slide

  5. CircleCIͷςετϝλσʔλ
    • ςετ݁Ռ͕هࡌ͞ΕͨϑΝΠϧͷ͜ͱʢࡶʣ


    • ςετϝλσʔλΛ׆༻͢ΔͱTest InsightsͳͲ͕ར༻Ͱ͖Δ


    • timing data͸ςετϝλσʔλ͔Βੜ੒͞ΕΔ

    View Slide

  6. CircleCIͷςετϝλσʔλ
    Collecting test data
    https://circleci.com/docs/2.0/collect-test-data/

    View Slide

  7. CircleCIͰςετϝλσʔλΛอଘ͢Δ
    • store_test_resultsεςοϓͰςετ݁Ռ

    ʢςετϝλσʔλʣΛอଘ͢Δ


    • https://circleci.com/docs/ja/2.0/con
    fi
    guration-reference/
    #storetestresults

    View Slide

  8. CircleCIͰςετϝλσʔλΛอଘ͢Δ
    - store_test_results:
    path: test-results

    View Slide

  9. CircleCIͷςετϝλσʔλͷϑΥʔϚοτ
    • CircleCIͰςετ݁Ռ͕ςετϝλσʔλͱೝࣝ͞ΕΔϑΥʔϚοτ͸ҎԼ


    • JUnit XML


    • Cucumber JSON


    • ଟ͘ͷݴޠϥΠϒϥϦͰαϙʔτ͍ͯ͠Δͷ͸JUnit XML


    • ຊࢿྉͰ΋JUnit XMLΛѻ͍·͢

    View Slide

  10. JUnit XML Schema
    • https://github.com/windyroad/JUnit-Schema


    • https://github.com/sj26/rspec_junit_formatter/blob/
    main/lib/rspec_junit_formatter.rb#L9-L11


    • http://windyroad.org/dl/Open%20Source/JUnit.xsd

    View Slide

  11. CircleCIͷςετ෼ׂͱฒྻ࣮ߦ
    • 1ͭͷ࣮ߦ؀ڥʢExecutorʣͰશςετΛ࣮ߦ͢ΔͱCIͷ࣮ߦ͕࣌ؒ௕͘ͳΔ


    • CircleCIʹ͸ςετΛෳ਺ͷExecutorʹ෼ׂͤ͞ΔͨΊͷίϚϯυ͕͋Δ


    • ίϚϯυͰ෼ׂͨ͠ςετΛෳ਺ͷExecutorͰฒྻ࣮ߦͤ͞Δ͜ͱʹΑΓ

    શςετ࣮ߦ͕ૣ͘ͳΓɺCIͷ࣮ߦ࣌ؒ΋୹͘ͳΔ

    View Slide

  12. CircleCIͷςετ෼ׂͱฒྻ࣮ߦ
    Running Tests in Parallel
    https://circleci.com/docs/2.0/parallelism-faster-jobs/

    View Slide

  13. CircleCIͷςετ෼ׂͱ
    ฒྻ࣮ߦ
    https://circleci.com/docs/ja/2.0/parallelism-faster-jobs/
    • ެࣜυΩϡϝϯτͷਤΛ͓आΓͯ͠

    આ໌͢Δͱ͜Μͳײ͡


    • Ұ൪Լ͕timing dataʹج͍ͮͨ

    ύλʔϯͷઆ໌

    View Slide

  14. ςετ෼ׂ
    • ෼ׂͷ࢓ํʹΑͬͯશମͷςετ࣮ߦ࣌ؒʹӨڹ͢Δ


    • ద౰ʹ෼ׂͯ͠ฒྻ࣮ߦͤ͞Δͱ؀ڥ͝ͱʹ࣮ߦ࣌ؒʹ

    ͹Β͖͕ͭग़Δ


    • Մೳͳ͔͗Γ؀ڥ͝ͱͷ࣮ߦ࣌ؒͷ͹Β͖͕ͭ

    খ͘͞ͳΔΑ͏ʹ෼ׂ͢Δͷ͕Α͍✨

    View Slide

  15. ࠷దͳςετ෼ׂ
    • ֤ςετέʔεͷ࣮ߦʹ͔͔Δ࣌ؒΛܭଌ͢Δ


    • ܭଌͨ࣌ؒ͠Λར༻͠શମͷςετ͕࣌ؒ୹͘ͳΔΑ͏

    ͍͍ײ͡ͷΞϧΰϦζϜͰ͍͍ײ͡ʹ෼ׂ͢Δ


    • ʮ͍͍ײ͡ʯ͕େࣄ😊

    View Slide

  16. CircleCIͷςετϝλσʔλ
    circleci tests split --split-by=timingsʹ͍ͭͯ

    ௐ΂ͨɾࣗ࡞ͯ͠Έͨ
    htthttps://hoshinotsuyoshi.com/post/circleci_tests_split/

    View Slide

  17. CircleCIͷςετ෼ׂ
    • ҎԼͷ෼ׂํ๏Λαϙʔτ͍ͯ͠Δ


    • timing data


    • ϑΝΠϧ໊ɾΫϥε໊


    • ϑΝΠϧαΠζ


    • CircleCI͸timing dataʹΑΔ෼ׂ͕࠷ྑͷํ๏ͱͯ͠Δ

    View Slide

  18. CircleCIͷtiming data
    • ֤ςετέʔεͷ࣮ߦ͕࣌ؒهࡌ͞Ε͍ͯΔJSONϑΝΠϧ


    • ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ੒͢Δ


    • timing dataͷ಺༰͸֬ೝͰ͖Δ

    View Slide

  19. CircleCIͷtiming data
    • ֤ςετέʔεͷ݁Ռʹ͸ɺΫϥε໊ͱϑΝΠϧ໊ͳͲ͕

    هࡌ͞Ε͍ͯΔ͜ͱΛCircleCI͸ظ଴͍ͯ͠Δ


    • ֤ςετϑΝΠϧͷςετ࣮ߦʹ͔͔Δ͕࣌ؒΘ͔Δσʔλߏ଄


    • σϑΥϧτͷ෼ׂํ๏͕ɺϑΝΠϧ໊ʹج͍ͮͨ෼ׂͳͷͰ্ه͕ඞཁ


    • ςετϝλσʔλʹͦΕΒͷ৘ใ͕هࡌ͞Ε͍ͯͳ͍ͱtiming dataΛ

    ར༻ͯ͠΋ࢥͬͨͱ͓Γʹςετ෼ׂ͕࣮ߦ͞Εͳ͍

    View Slide

  20. timing dataΛར༻ͯ͠

    ςετ෼ׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ
    circleci tests glob "test/**/*.rb" | circleci tests split --split-by=timings > /tmp/tests-to-run
    bundle exec rspec $(cat /tmp/tests-to-run)

    View Slide

  21. • circleci tests splitͰςετϑΝΠϧͷҰཡऔಘ


    • --split-by=timingsͰtiming dataΛར༻࣮͠ߦ؀ڥ͝ͱʹ

    ςετ෼ׂΛߦ͏Α͏ʹࢦఆ


    • bundle exec rspec΁࣮ߦ؀ڥ͝ͱʹ෼ׂ͞ΕͨςετΛ

    ࣮ߦ͢ΔΑ͏ʹࢦఆ
    timing dataΛར༻ͯ͠

    ςετ෼ׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ

    View Slide

  22. Dartͷtestʹ͍ͭͯ

    View Slide

  23. Why Dart test?
    • ࢲ͕ॴଐ͢Δձࣾ 10XͰ࡞͍ͬͯΔαʔϏε StailerͰ͸

    API࣮૷ʹαʔόʔαΠυDartΛ࢖༻


    • αʔόʔαΠυDartͰ͸CIʹCircleCIΛར༻͍ͯ͠Δ


    • DartͰςετΛॻ͘৔߹ʮtestʯύοέʔδ͕ϝδϟʔ


    • https://pub.dev/packages/test

    View Slide

  24. Dart sharding tests
    • testύοέʔδʹ͸ςετ෼ׂ͢Δػೳ͕͋Δʂʂ


    • ݱࡏ 10XͰ͸testͷػೳΛ࢖ͬͯCircleCI্ͰςετΛ෼ׂɾฒྻ࣮ߦͯ͠
    ͍Δ


    • ͔͠͠ɺ͜ͷ෼ׂํ๏͸֤ςετέʔεͷ࣮ߦ࣌ؒͳͲΛݩʹ

    ςετ෼ׂ͞ΕΔΘ͚Ͱ͸ͳ͍


    • ͦͷͨΊExecutor͝ͱʹ࣮ߦ࣌ؒͷ͹Β͖͕ͭେ͖͍ͷ͕՝୊

    View Slide

  25. Dart sharding tests
    dart test --total-shards 3 --shard-index 0 path/to/test.dart
    dart test --total-shards 3 --shard-index 1 path/to/test.dart
    dart test --total-shards 3 --shard-index 2 path/to/test.dart

    View Slide

  26. Dart sharding tests
    testύοέʔδͷςετ෼ׂͷ࣮૷
    https://github.com/dart-lang/test/blob/2afbff8cc059b37b4b1a69fc2ddffc53290e4780/
    pkgs/test_core/lib/src/runner.dart#L428-L443

    View Slide

  27. Dart testsͰCircleCIͷtiming dataΛར༻ͯ͠

    ΋ͬͱ͍͍ײ͡ʹςετ෼ׂɾฒྻ࣮ߦ͍ͨ͠ʂ
    ͱࢥͬͨΘ͚Ͱ…

    View Slide

  28. Dart testͷ݁ՌΛJunit XMLͰग़ྗ͢Δ
    • Dart test͸ςετ݁ՌΛJSONग़ྗ͢Δػೳ͕͋Δ


    • ͦͷJSONग़ྗ͞Εͨ݁ՌΛJunit XMLʹม׵͢Δπʔϧ͕͋Δ


    • https://github.com/TOPdesk/dart-junitreport


    • ͜ΕΛ࢖ͬͯDart testͷςετ݁ՌΛJunit XMLͰग़ྗ͢Δ

    View Slide

  29. ग़ྗͨ͠Junit XMLΛςετϝλσʔλͱͯ͠
    CircleCIʹೝࣝͤ͞Δ
    • store_test_resultsεςοϓʹJunit XMLΛग़ྗͨ͠σΟϨΫτϦΛ
    ࢦఆ͢Δ


    • ͜ΕͰςετ݁ՌͷJunit XML͕CircleCIʹςετϝλσʔλ

    ͱͯ͠ೝࣝ͞ΕΔ


    • ೝࣝͨ͠ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ੒͢Δ

    View Slide

  30. ςετ࣮ߦͷίϚϯυΛॻ͖׵͑Δ
    • timing dataΛར༻ͯ͠ςετ෼ׂΛߦ͍ɺ෼ׂͨ͠ϑΝΠϧ
    ͷΈͰςετ࣮ߦ͢ΔΑ͏ʹίϚϯυΛॻ͖׵͑Δ
    TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)
    dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded

    View Slide

  31. Α͠ɺ͜ΕͰ׬ᘳ✨✨✨✨

    View Slide

  32. Ͱ͸...ͳ͔ͬͨ...💥💥💥💥

    View Slide

  33. Կ͕ى͖͔ͨ...😨
    • "No timing found for <ϑΝΠϧ໊>"ͱ͍͏ϝοηʔδ͕ग़ͨ


    • ςετϑΝΠϧʹର͢Δtiming data͕ଘࡏ͠ͳ͍ͥʂͬͯݴͬͯΔ


    • timing dataͷϑΝΠϧ͸ੜ੒͞Ε͍͕ͯͨɺԿ͔͠ΒͷݪҼͰ

    ͏·͘ར༻͞Ε͍ͯͳ͍ͷͰɺ࠷దͳςετ෼ׂ͕ߦ͍͑ͯͳ͍

    View Slide

  34. timing dataͷϑΝΠϧΛ֬ೝ͢Δํ๏
    cat ${CIRCLE_INTERNAL_TASK_DATA}/circle-test-results/results.json

    View Slide

  35. Կ͕μϝ͔ͩͬͨ...😱
    • timing dataΛ֬ೝͯ͠ΈͯΘ͔ͬͨͷ͕ɺ֤ςετέʔε͕
    ͲͷςετϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λࣔ͢
    fi
    le keyͷ

    ஋͕nullͩͬͨ

    View Slide

  36. Կ͕μϝ͔ͩͬͨ...😱
    fi
    le͕null😱

    View Slide

  37. Կ͕μϝ͔ͩͬͨ...·ͱΊΔͱ...😱
    • ֤ςετέʔεͷ࣮ߦ࣌ؒͳͲ͸هࡌ͞Ε͍ͯΔ͕

    ֤ςετέʔε͕ͲͷϑΝΠϧʹهࡌ͞Ε͍ͯΔͷ͔Λ

    ࣔ͢஋͕nullͳͷͰɺςετϑΝΠϧ͝ͱͷ࣮ߦ͕࣌ؒ

    ܭࢉͰ͖ͳ͍


    • ݁Ռɺ͏·͘ςετ෼ׂͰ͖ͳ͍Θʂͬͯײͩͬͨ͡

    View Slide

  38. fi
    le keyͷ஋͕nullͳݪҼΛ୳Δ
    • timing data͸ςετϝλσʔλ͔Βੜ੒͞ΕΔ


    • ςετϝλσʔλ = ςετ݁Ռ͕هࡌ͞ΕͨJunit XMLΛ

    ֬ೝͯ͠Έͨ

    View Slide
















  39. ग़ྗ͞ΕͨJunit XMLͷ಺༰

    View Slide

  40. fi
    le keyͷ஋͕nullͳݪҼΛ୳Δ
    • testcase elementʹ֤ςετέʔεͷ݁Ռ͕هࡌ͞Ε͍ͯΔ


    • ͦ͜ʹ֤ςετέʔε͕هࡌ͞Ε͍ͯΔςετϑΝΠϧ໊͕

    ॻ͔Ε͍ͯͳ͍͕ͳ͍͜ͱ͕ݪҼͬΆ͍͜ͱ͕Θ͔ͬͨ


    • ݪҼ͕Θ͔ͬͨͷͰtestcase elementʹ
    fi
    le attributeͷ

    ௥ՃΛࢼΈͨ

    View Slide

  41. ࢼͯ͠Έͨमਖ਼
    • dart-junitreportͷ಺෦Ͱར༻͞Ε͍ͯΔdart-testreportϥΠϒϥϦͷ
    मਖ਼


    • dart-junitreportͰJunit XMLग़ྗ͢Δࡍɺtestcase elementʹ

    fi
    le attributeͷ௥Ճ



    fi
    le attributeʹهࡌ͞Εͨ
    fi
    le pathΛCI্ͰsedίϚϯυΛ

    ར༻ͯ͠ॻ͖׵͑

    View Slide

  42. मਖ਼ͯ͠Έͨ݁Ռ
    • Dart testͰ΋timing dataΛར༻ͯ͠࠷దͳςετ෼ׂ͕

    Ͱ͖ΔΑ͏ʹͳͬͨʂ


    • ΊͰ͍ͨ🎉🎉🎉🎉

    View Slide

  43. ׬੒ܗ͕ͪ͜Β💪

    View Slide

  44. CircleCIͰtiming dataʹج͍ͮͨςετ෼ׂΛ

    DartͰར༻Ͱ͖ΔΑ͏ʹ͢ΔαϯϓϧϓϩδΣΫτ
    https://github.com/operando/dart-test-CircleCI-timing-data
    ׬੒ܗ✨

    View Slide

  45. αϯϓϧϓϩδΣΫτͷCircleCIͷ݁Ռ
    https://app.circleci.com/pipelines/github/operando/dart-test-CircleCI-timing-
    data/14/work
    fl
    ows/e4884496-b4b0-4d5b-af7a-0209097affac/jobs/16
    ׬੒ܗ✨

    View Slide

  46. version: 2.1
    jobs:
    test:
    docker:
    - image: dart:2.16.1
    parallelism: 2
    steps:
    - checkout
    - run:
    name: Download dependencies
    command: pub get
    - run:
    name: Test
    command: |
    TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)
    dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded
    - run:
    name: Install JUnitReport
    when: always
    command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git
    - run:
    name: Convert tests to JUnit
    when: always
    command: |
    $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml
    sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml
    - store_test_results:
    path: test_results/
    - store_artifacts:
    path: test_results/
    workflows:
    version: 2
    build:
    jobs:
    - test
    .circleci/con
    fi
    g.ymlͷ಺༰

    ௕͍….จࣈখ͍͞…😇

    View Slide

  47. - run:
    name: Test
    command: |
    TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)
    dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded
    - run:
    name: Install JUnitReport
    when: always
    command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git
    - run:
    name: Convert tests to JUnit
    when: always
    command: |
    $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json
    --output test_results/tests_report_junit.xml
    sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml
    - store_test_results:
    path: test_results/
    - store_artifacts:
    path: test_results/
    େࣄͳͷ͸͜ͷ෦෼ʂ

    View Slide

  48. - run:
    name: Test
    command: |
    TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)
    dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded
    - run:
    name: Install JUnitReport
    when: always
    command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git
    - run:
    name: Convert tests to JUnit
    when: always
    command: |
    $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json
    --output test_results/tests_report_junit.xml
    sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml
    - store_test_results:
    path: test_results/
    - store_artifacts:
    path: test_results/
    forkͨ͠ϦϙδτϦʹमਖ਼ΛೖΕͨͷͰ

    ࣗ਎ͷϦϙδτϦ͔Βऔಘ

    View Slide

  49. - run:
    name: Test
    command: |
    TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)
    dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded
    - run:
    name: Install JUnitReport
    when: always
    command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git
    - run:
    name: Convert tests to JUnit
    when: always
    command: |
    $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json
    --output test_results/tests_report_junit.xml
    sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml
    - store_test_results:
    path: test_results/
    - store_artifacts:
    path: test_results/
    Junit XMLͰग़ྗ͠ɺsedͰ
    fi
    le attributeͷ಺༰Λॻ͖׵͑

    View Slide

  50. - run:
    name: Test
    command: |
    TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)
    dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded
    - run:
    name: Install JUnitReport
    when: always
    command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git
    - run:
    name: Convert tests to JUnit
    when: always
    command: |
    $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json
    --output test_results/tests_report_junit.xml
    sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml
    - store_test_results:
    path: test_results/
    - store_artifacts:
    path: test_results/
    ग़ྗͨ͠Junit XMLΛอଘʂ

    View Slide

  51. - run:
    name: Test
    command: |
    TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings)
    dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded
    - run:
    name: Install JUnitReport
    when: always
    command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git
    - run:
    name: Convert tests to JUnit
    when: always
    command: |
    $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json
    --output test_results/tests_report_junit.xml
    sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml
    - store_test_results:
    path: test_results/
    - store_artifacts:
    path: test_results/
    ͜ΕͰ͜ͷ෦෼ͷςετ෼ׂ͕͏·͘ಈ͍ͨʂ

    View Slide

  52. ΍ͬͨͶ😊ʂʂ

    View Slide

  53. Thanks🙏

    View Slide