Upgrade to Pro — share decks privately, control downloads, hide ads and more …

クラウド上のモバイル端末・シミュレータを活用した大規模並列テスト / Large Parallel Testing with Real Devices or Simulators on Cloud

Kuniwak
November 30, 2019

クラウド上のモバイル端末・シミュレータを活用した大規模並列テスト / Large Parallel Testing with Real Devices or Simulators on Cloud

AWS Device Farm やコンテナ化した Android Emulator をクラウド上で複数起動し、並列に自動テストを実行するノウハウと経験則をお話します。また、この自動テストではテストシナリオの保守コストを下げるためにシナリオを必要としないテストエージェントを活用しています。この種のテストエージェントによって、クラッシュやメモリリーク、パフォーマンスイシューを自動で見つける方法を解説します。

システムテスト自動化カンファレンス2019
https://testautomationresearch.connpass.com/event/144768/

Kuniwak

November 30, 2019
Tweet

More Decks by Kuniwak

Other Decks in Programming

Transcript

  1. Ϋϥ΢υ্ͷϞόΠϧ୺຤Λ

    ׆༻ͨ͠େن໛ฒྻςετͷ

    ࣮ߦ؀ڥʹ͍ͭͯ
    γεςϜςετࣗಈԽΧϯϑΝϨϯε
    ,VOJXBL !PSHB@DIFN

    View Slide

  2. "CPVUNF

    View Slide

  3. ,VOJXBL
    w ॴଐˠ
    w HJUIVCDPN,VOJXBL
    w RJJUBDPN,VOJXBL
    w ೰ΊΔςετϚϯ

    View Slide

  4. ൃදͰ఻͍͑ͨ͜ͱ

    View Slide

  5. -JOU΍୯ମςετͰൃݟͰ͖ͳ͍όάΛɺ

    อक༰қͳςετͰݟ͚ͭΔج൫ͱͯ͠

    ʮEFWGBSNʯ͕ศརͰ͢

    View Slide

  6. $*ʹࡌ͍ͤͨ΋ͷ
    EFWGBSNͷ঺հ
    ӡ༻։࢝࣌఺Ͱͷײ૝

    View Slide

  7. $*ʹࡌ͍ͤͨ΋ͷ
    EFWGBSNͷ঺հ
    ӡ༻։࢝࣌఺Ͱͷײ૝
    ͱͯ΋ॏཁ

    View Slide

  8. ϞόΠϧήʔϜΛ

    ແ࡞ҝʹૢ࡞ͯ͠

    Ϋϥογϡ΍

    ੑೳྼԽΛ

    ݕग़͢Δςετ
    ࠨҠಈ ӈҠಈ
    δϟϯϓ
    ແ࡞ҝʹ࣮ߦ

    View Slide

  9. ͜ͷςετͷҐஔ෇͚͔Β

    ཧ૝తͳ༻๏༻ྔΛઆ໌͠·͢
    ʮಉ͡ํ๏Ͱ΍ͬͨͷʹશવμϝʯΛආ͚ΔͨΊʹ

    View Slide

  10. υϝΠϯݻ༗ͷ݁߹ϨΠϠʔ

    ϞϯΩʔςετ
    શ෦ΛҰ౓ʹઆ໌͢Δͱࠞཚ͢Δͱࢥ͏ͷͰɺ

    ॱ൪ʹઆ໌͍͖ͯ͠·͢

    View Slide

  11. υϝΠϯݻ༗ͷ݁߹ϨΠϠʔ

    ϞϯΩʔςετ
    ·ͣ͸ʮ݁߹ϨΠϠʔʯʹ͍ͭͯͰɺ

    ͜͜ͰԿΛݟ͚ͭΔ΂͖͔ΛΈͯΈ·͠ΐ͏

    View Slide

  12. όάൃݟ͸ଟ૚͕جຊ
    ݁߹ϨΠϠʔ
    w ߏจϋΠϥΠτ
    w -JOU࣮ߦ
    w ੩తܕݕࠪ
    w ୯ମςετ
    w ݁߹ςετ
    w &&ςετ
    ͍Ζ͍Ζͳ૚ʹόάൃݟͷखஈ͕͋ΓɺͲͷ૚Ͱ

    ͲͷόάΛݟ͚ͭΔ͔͕ͱͯ΋ॏཁɿ
    w ຊ൪؀ڥͰͷ

    ΤϥʔϨϙʔλʔ
    w ͓໰͍߹Θͤ

    View Slide

  13. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    ଟ૚ͷόάൃݟखஈΛʮ͋ΔنଇʯͰฒ΂ͨ΋ͷ

    View Slide

  14. ߏจϋΠϥΠτ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    όά9Λݟ͚ͭͨ৔ॴ
    όά9
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    -JOU࣮ߦ

    View Slide

  15. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    όά"
    खલͷํͰࢭ·Δͱ
    ॴཁ͕࣌ؒ୹͍
    όά#

    View Slide

  16. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    όά"
    खલͷํͰࢭ·Δͱ
    ॴཁ͕࣌ؒ୹͍
    όά#
    όά͸ͳΔ΂͘खલͰ

    ൃݟ͢Δͷ͕ޮՌత

    View Slide

  17. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    όά" όά#
    ಛघͳόά͸ޙͷํͰ

    ͔͠ݟ͚ͭΒΕͳ͍

    View Slide

  18. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    όά"
    ಛघͳόά͸ޙͷํͰ

    ͔͠ݟ͚ͭΒΕͳ͍
    όά#
    खલͰऔΓ͜΅ͨ͠

    ΋ͷ͸ԼͰड͚ࢭΊΔ

    View Slide

  19. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    ͜ͷൃදͰൃݟ͍ͨ͠

    όά͸ɺ୯ମςετͰ

    औΓ͜΅͞ΕΔ͕

    ΤϥʔϨϙʔτΑΓ

    ૣ͘ݟ͚ͭΒΕΔόά

    View Slide

  20. -JOU΍୯ମςετͰ

    όάΛݟ্͚ͭͨͰ

    ซ༻͢Δͷ͕େલఏ
    ཁ͢Δʹ

    View Slide

  21. ݁߹ςετ͸อक͕ͱͯ΋େม
    ͨͩ͠

    View Slide

  22. ݁߹͢Δ΄Ͳɺࢼ͞ͳ͍ͱ

    ͍͚ͳ͍૊Έ߹Θ͕ͤ૿͑Δ
    ݁߹͢Δ΄Ͳɺݸʑͷཁૉͷ

    มߋʹରͯ͠ςετ͕੬͘ͳΔ
    ܽ఺
    ܽ఺

    View Slide

  23. υϝΠϯݻ༗ͷ݁߹ϨΠϠʔ

    ϞϯΩʔςετ
    ͦ͜Ͱ

    View Slide

  24. ಛఆͷૢ࡞खॱΛఆΊͣɺ

    ແ࡞ҝʹૢ࡞͢Δςετ
    ϞϯΩʔςετͱ͸

    View Slide

  25. ແ࡞ҝʹૢ࡞͢ΔͷͰ૊Έ߹Θ͕ͤ
    ૿͑ͯ΋ίʔυ͸͋·Γ૿͑ͳ͍
    Ұൠతͳੑ࣭͔֬͠ೝ͠ͳ͍ͷͰɺ

    ݸʑͷཁૉ͕มߋ͞Εͯ΋յΕͳ͍
    ར఺
    ར఺

    View Slide

  26. ήʔϜͰ͸ແ࡞ҝʹૢ࡞͢Δ
    ͚ͩͰ͸ෆे෼ͳ͜ͱ͕ଟ͍
    ͨͩ͠

    View Slide

  27. ͋Δॱ൪Ͱૢ࡞͢Δͱ

    ඞࡴٕ͕ൃಈ͢ΔΑ͏ͳ

    ΞΫγϣϯ͕͋Δͱ͖ɺ

    ແ࡞ҝͩͱΞΫγϣϯ͸

    ΄ͱΜͲൃੜ͠ͳ͍
    ΞΫγϣϯҙຯΛ΋ͬͨ࿈ଓૢ࡞ʹ

    Α࣮ͬͯߦ͞ΕΔΠϕϯτ
    ΞΫγϣϯ

    View Slide

  28. ՄೳͳΞΫγϣϯΛ

    ެฏʹൃੜͤ͞ͳ͍ͱ

    ςετ͕ෆ׬શʹͳΔ
    ͭ·Γ

    View Slide

  29. υϝΠϯݻ༗ͷ݁߹ϨΠϠʔ

    ϞϯΩʔςετ
    ͦ͜Ͱ

    View Slide

  30.  ͭʹߜΓࠐΉ
    ⁞ՄೳͳΞΫγϣϯΛ

    ໰͍߹Θͤ
    ⁠ΞΫγϣϯΛ࣮ߦ
    ඞࡴٕ
    ࠨӈҠಈ
    δϟϯϓ
    ඞࡴٕ
    ࠨӈҠಈ
    ඞࡴٕ
    ࣮ߦ
    ϦλΠΞ

    View Slide

  31. ެฏͳΞΫγϣϯ͕Մೳʹͳͬͨ
    ͭ·Γ

    View Slide

  32. ͜͜·Ͱͷ·ͱΊ
    -JOU΍୯ମςετͰ͸

    ݟ͚ͭΒΕͳ͍όάΛ

    อक༰қͳςετͰݟ͚͍ͭͨ

    View Slide

  33. $*ʹࡌͤΔࡍͷ՝୊

    View Slide

  34. Ͳ͜ͰςετΛಈ͔͔͢
    ΞϓϦΛͲ͔͜Ͱىಈ͢Δඞཁ͕͋Δ͕ɺ

    ࣮ػݻ༗ͷ໰୊Λݟ͚͍ͭͨΘ͚Ͱ͸ͳ͍
    ϞϯΩʔςετ͸େྔʹฒྻ࣮ߦ͢Δ΄Ͳ
    ୹࣌ؒͰଟ͘ͷ໰୊Λݟ͚ͭΒΕΔͷͰ

    ߴ଎Ͱ҆ՁͳγϛϡϨʔλΛ࢖͑ΔͳΒ

    ͦ͏͍ͨ͠

    View Slide

  35. γϛϡϨʔλΫϥ΢υʜʁ
    γϛϡϨʔλΛΫϥ΢υͰ࣮ߦ͢Δ

    αʔϏε͸Θ͔ͣʹ͋Δ͕࢖͍ͮΒ͍ʜ
    ࢓ํͳ͍ͷͰ࣮ػΛΫϥ΢υͰ࣮ߦͰ͖Δ

    σόΠεϑΝʔϜ͕ୈҰީิʹͳΔ

    View Slide

  36. طଘͷσόΠεϑΝʔϜ
    w "84%FWJDF'BSN
    w )FBETQJO
    w 3FNPUF5FTU,JU
    w 74"QQ$FOUFS
    w 'JSFCBTF5FTU-BC
    w 4BVDF-BCT

    3FBM%FWJDF$MPVE
    w

    View Slide

  37. ͨ͘͞Μ͋Γ͗ͯ͢

    ͲΕΛબ΂͹ʜʂʁ

    View Slide

  38. ࠓճ͸"84%FWJDF'BSN
    "84%FWJDF'BSN͸NJOͰ

    ͍ͭͰ΋ฒྻ਺Λ্͛Լ͛Ͱ͖Δ
    ଞ͸୆਺͕ܾ·ͬͨظؒܖ໿͕ଟ͘ɺ

    $*த͚ͩฒྻ਺Λ֬อ͢Δͱ͍ͬͨ

    ࢖͍ํͱ૬ੑ͕ѱ͍
    IUUQTBXTBNB[PODPNKQEFWJDFGBSNQSJDJOH

    View Slide

  39. ͨͩ͠ෆ҆ཁૉ΋ʜ
    ւ֎੡ͷ୺຤͕ଟ͘ɺ೔ຊ੡ͷ୺຤͸

    ͋·Γἧ͍ͬͯͳ͍
    কདྷతʹࠃ಺ͷ୺຤͕ଟ͘

    ैྔ՝ۚͷσόΠεϑΝʔϜ͕ग़ͨΒ

    ৐Γ׵͑ΔՄೳੑ͕͋Δ

    View Slide

  40. ݱঢ়ͷํ਑
    ͍ͭผͷγϛϡϨʔλΫϥ΢υ΍

    σόΠεΫϥ΢υ΁Ҡߦ͢Δ͔Θ͔Βͳ͍
    ͜ͷ͕͍͖࣌ͭͯ΋ɺ

    Ҡߦίετͷগͳ͍ঢ়ଶΛอ͍ͪͨ

    View Slide

  41. ͦ͜ͰEFWGBSN
    γϛϡϨʔλσόΠεΫϥ΢υͷ

    ੾Γସָ͕͑ͰɺσόΠεΫϥ΢υ্Ͱ

    ΞϓϦͷىಈঢ়گΛ؅ཧ͢Δπʔϧ

    View Slide

  42. IUUQTHJUIVCDPN%F/"EFWGBSN

    View Slide

  43. $*ʹࡌ͍ͤͨ΋ͷ
    EFWGBSNͷ঺հ
    ӡ༻։࢝࣌఺Ͱͷײ૝

    View Slide

  44. EFWGBSNͰͰ͖Δ͜ͱ
    w ࢦఆͨ͠σόΠεΫϥ΢υ04ػछ

    ୆਺ΞϓϦҾ਺ͰΞϓϦΛىಈ
    w ΞϓϦ͕Ϋϥογϡͨ͠ΒίϚϯυΛ

    ҟৗऴྃɺࢦఆ࣌ؒੜଘͨ͠Βਖ਼ৗऴྃ
    w ݱࡏ͸"84%FWJDF'BSNͷΈͷରԠ

    View Slide

  45. ͑ɺͦΕ͚ͩʜ

    View Slide

  46. ཧ༝͕ʜ͋Γ·͢ʜ

    View Slide

  47. EFWGBSNͷࢥ૝
    ࣮ػɾγϛϡϨʔλɾσόΠεΫϥ΢υΛ

    ໰Θͣɺಉ͡ΠϯλʔϑΣʔεͰ

    ىಈɾऴྃ͢Δ͜ͱͷΈʹઐ೦͢Δ
    ಛఆͷ৚݅Ͱ͔͠αϙʔτͰ͖ͳ͍ػೳ͸

    αϙʔτ͠ͳ͍ʢػೳͷڞ௨ू߹ओٛʣ

    View Slide

  48. $ devfarm run-ios --os-version 12.0 \

    --device 'apple iphone xs' \

    --instance-group 'example' \

    --platform 'aws-device-farm' \

    --ipa 'path/to/app.ipa' \

    --args-json '["-ARG1", "HELLO_DEVFARM"]' \

    --lifetime-sec 900 &
    platform status
    aws-device-farm launching
    J04ΞϓϦΛ෼࣮ؒߦʢΫϥογϡ͠ͳ͔ͬͨΒਖ਼ৗऴྃʣ

    View Slide

  49. $ devfarm status --instance-group 'example'

    platform device os state note
    aws-device-farm apple iphone xs ios ACTIVE
    ىಈ͍ͯ͠ΔΞϓϦͷ֬ೝ

    View Slide

  50. $ devfarm halt --instance-group 'example'

    platform status
    aws-device-farm halting
    ΞϓϦͷڧ੍ऴྃ

    View Slide

  51. instance_groups:
    :
    # for Android
    - platform: # required
    android: # required
    device: # required
    apk: # required
    app_id: # required
    intent_extras: [] # optional
    lifetime_sec: 900 # required
    $ devfarm run-all planfile.yml
    ෳ਺ΞϓϦͷҰׅىಈ

    View Slide

  52. "84%FWJDF'BSNͰ୆ฒྻ࣮ߦ͍ͯ͠Δ༷ࢠ

    View Slide

  53. EFWGBSNͱ͸
    ࣮ػͱγϛϡϨʔλͷ੾Γସָ͕͑Ͱɺ

    σόΠεΫϥ΢υ্ͰΞϓϦͷىಈঢ়گΛ
    ؅ཧ͢Δπʔϧ
    $-*ͳͷͰ$*αʔϏε͔Β࣮ߦ͠΍͍͢

    View Slide

  54. ฐࣾͷEFWGBSNར༻
    EFWGBSNʹ͸ςετ࣮ߦΛ։࢝͢Δ

    ࢓૊ΈΛ༻ҙ͠ͳ͍ํ਑
    ઐ༻ʹϏϧυͨ͠ήʔϜΞϓϦͷىಈͱ

    ಉ࣌ʹϞϯΩʔςετΛ։͍࢝ͤͯ͞Δ

    View Slide

  55. $*ʹࡌ͍ͤͨ΋ͷ
    EFWGBSNͷ঺հ
    ӡ༻։࢝௚લ࣌఺Ͱͷײ૝

    View Slide

  56. "OESPJEͷηοτΞοϓ͕ෆ҆ఆʜ
    "84%FWJDF'BSNଆͱͷ੾Γ෼͚͕

    Ͱ͖͍ͯͳ͍͕ɺ"OESPJE͕ͷ

    ֬཰Ͱ͏·͘ىಈͯ͘͠Εͳ͍ʜ
    ݱࡏ"84΁໰͍߹Θͤதʜ
    ࠷ѱπʔϧଆͰSFUSZରԠʹͳΔ͔΋

    View Slide

  57. ୆ηοτΞοϓʹࣦഊ༷ͨ͠ࢠ

    View Slide

  58. ଟ͘ͷσόΠεϑΝʔϜ͕

    ಛఆͷϑϨʔϜϫʔΫʹಛԽ͗͢͠
    "QQJVN΍9$6*5FTUͳͲͷಛఆͷ

    ϑϨʔϜϫʔΫͷ࣮ߦํ๏ʹґଘ͍ͯ͠Δ
    ࠓճ͸ىಈ؂ࢹऴ͚ྃͩΛ͔͕ͤͨͬͨ͞

    ͍Ζ͍Ζͱେมͩͬͨʜ

    View Slide

  59. ྫ"84%FWJDF'BSNͰඞཁͳ΋ͷͱҾ਺ͱͷؔ܎

    View Slide

  60. Run all by ./test/health-check/planfile.yml:
    searching to skip creating AWS Device Farm projects
    finding the AWS Device Farm project: "devfarm-health_check"
    the AWS Device Farm project was found
    skipped to create an AWS Device Farm project (because the project already exists)
    searching AWS Device Farm device
    the AWS Device Farm device found
    searching AWS Device Farm device pool to skip creation
    listing AWS Device Farm device pools
    found the AWS Device Farm device pool
    skipping to create AWS Device Farm device pool (because already exists)
    generating AWS Device Farm custom test spec
    AWS Device Farm custom test spec was successfully generated
    uploading AWS Device Farm custom test spec
    searching AWS Device Farm upload to skip upload
    listing AWS Device Farm uploads
    found AWS Device Farm upload
    found completed AWS Device Farm upload
    generating AWS Device Farm test package
    searching AWS Device Farm upload to skip upload
    listing AWS Device Farm uploads
    ྫ࣮ߦϩά͔Β࡯ͤΒΕΔେม͞
    ·࣮ͩߦ࢝·ͬͯͳ͍ʜ

    View Slide

  61. ͦΕͰ΋ɺςετͷ஌໊౓Λ্͛Δ

    ઌۦ͚͔ͭීٴͷج൫ͱͳΓ͍ͨ
    ىಈ؂ࢹऴྃΛඪ४Ͱ҆ఆͯ͠αϙʔτͯ͘͠ΕΔ

    σόΠεϑΝʔϜ͕૿͑ΔͨΊʹ͸஌໊౓͕ඞཁ
    υϝΠϯݻ༗ɾ݁߹ϨΠϠʔɾϞϯΩʔςετ͸

    อक༰қͰޮՌతͳςετͳͷͰɺ

    ͜ΕΛ͖͔͚ͬʹ޿·ͬͯ͘ΕͨΒخ͍͠

    View Slide

  62. ·ͱΊ
    EFWGBSN͸-JOU΍୯ମςετͰ

    ݟ͚ͭΒΕͳ͍όάΛอक༰қͳ

    ςετͰݟ͚ͭΔͨΊͷ044Ͱ͢

    View Slide

  63. '"2ͳͥ"QQJVNΛ࢖Θͳ͍ͷʁ
    কདྷతʹΑΓݡ͍ΦʔτύΠϩοτʹ͢Δ

    Մೳੑ͕͋ͬͯ"QQJVNͩͱ΍ΓͮΒ͍
    ىಈ؂ࢹऴྃͷͨΊʹ"QQJVNΛ

    ಺෦తʹ࢖͏ࢼΈ΋͕͋ͬͨ

    "84%FWJDF'BSN্ͩͱσόοάࠔ೉

    View Slide