クラウド上のモバイル端末・シミュレータを活用した大規模並列テスト / Large Parallel Testing with Real Devices or Simulators on Cloud

AWS Device Farm やコンテナ化した Android Emulator をクラウド上で複数起動し、並列に自動テストを実行するノウハウと経験則をお話します。また、この自動テストではテストシナリオの保守コストを下げるためにシナリオを必要としないテストエージェントを活用しています。この種のテストエージェントによって、クラッシュやメモリリーク、パフォーマンスイシューを自動で見つける方法を解説します。

システムテスト自動化カンファレンス2019
https://testautomationresearch.connpass.com/event/144768/

Transcript

1. Ϋϥ΢υ্ͷϞόΠϧ୺຤Λ

   ׆༻ͨ͠େن໛ฒྻςετͷ

   ࣮ߦ؀ڥʹ͍ͭͯ
   γεςϜςετࣗಈԽΧϯϑΝϨϯε
   ,VOJXBL !PSHB@DIFN
   
   

   View Slide

2. "CPVUNF
   

   View Slide

3. ,VOJXBL
   w ॴଐˠ
   w HJUIVCDPN,VOJXBL
   w RJJUBDPN,VOJXBL
   w ೰ΊΔςετϚϯ
   

   View Slide

4. ൃදͰ఻͍͑ͨ͜ͱ
   

   View Slide

5. -JOU΍୯ମςετͰൃݟͰ͖ͳ͍όάΛɺ

   อक༰қͳςετͰݟ͚ͭΔج൫ͱͯ͠

   ʮEFWGBSNʯ͕ศརͰ͢
   

   View Slide

6. $*ʹࡌ͍ͤͨ΋ͷ
   EFWGBSNͷ঺հ
   ӡ༻։࢝࣌఺Ͱͷײ૝
   

   View Slide

7. $*ʹࡌ͍ͤͨ΋ͷ
   EFWGBSNͷ঺հ
   ӡ༻։࢝࣌఺Ͱͷײ૝
   ͱͯ΋ॏཁ
   

   View Slide

8. ϞόΠϧήʔϜΛ

   ແ࡞ҝʹૢ࡞ͯ͠

   Ϋϥογϡ΍

   ੑೳྼԽΛ

   ݕग़͢Δςετ
   ࠨҠಈ ӈҠಈ
   δϟϯϓ
   ແ࡞ҝʹ࣮ߦ
   

   View Slide

9. ͜ͷςετͷҐஔ෇͚͔Β

   ཧ૝తͳ༻๏༻ྔΛઆ໌͠·͢
   ʮಉ͡ํ๏Ͱ΍ͬͨͷʹશવμϝʯΛආ͚ΔͨΊʹ
   

   View Slide

10. υϝΠϯݻ༗ͷ݁߹ϨΠϠʔ

    ϞϯΩʔςετ
    શ෦ΛҰ౓ʹઆ໌͢Δͱࠞཚ͢Δͱࢥ͏ͷͰɺ

    ॱ൪ʹઆ໌͍͖ͯ͠·͢
    

    View Slide

11. υϝΠϯݻ༗ͷ݁߹ϨΠϠʔ

    ϞϯΩʔςετ
    ·ͣ͸ʮ݁߹ϨΠϠʔʯʹ͍ͭͯͰɺ

    ͜͜ͰԿΛݟ͚ͭΔ΂͖͔ΛΈͯΈ·͠ΐ͏
    

    View Slide

12. όάൃݟ͸ଟ૚͕جຊ
    ݁߹ϨΠϠʔ
    w ߏจϋΠϥΠτ
    w -JOU࣮ߦ
    w ੩తܕݕࠪ
    w ୯ମςετ
    w ݁߹ςετ
    w &&ςετ
    ͍Ζ͍Ζͳ૚ʹόάൃݟͷखஈ͕͋ΓɺͲͷ૚Ͱ

    ͲͷόάΛݟ͚ͭΔ͔͕ͱͯ΋ॏཁɿ
    w ຊ൪؀ڥͰͷ

    ΤϥʔϨϙʔλʔ
    w ͓໰͍߹Θͤ
    

    View Slide

13. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    ଟ૚ͷόάൃݟखஈΛʮ͋ΔنଇʯͰฒ΂ͨ΋ͷ
    

    View Slide

14. ߏจϋΠϥΠτ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    όά9Λݟ͚ͭͨ৔ॴ
    όά9
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    -JOU࣮ߦ
    

    View Slide

15. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    όά"
    खલͷํͰࢭ·Δͱ
    ॴཁ͕࣌ؒ୹͍
    όά#
    

    View Slide

16. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    όά"
    खલͷํͰࢭ·Δͱ
    ॴཁ͕࣌ؒ୹͍
    όά#
    όά͸ͳΔ΂͘खલͰ

    ൃݟ͢Δͷ͕ޮՌత
    

    View Slide

17. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    όά" όά#
    ಛघͳόά͸ޙͷํͰ

    ͔͠ݟ͚ͭΒΕͳ͍
    

    View Slide

18. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    όά"
    ಛघͳόά͸ޙͷํͰ

    ͔͠ݟ͚ͭΒΕͳ͍
    όά#
    खલͰऔΓ͜΅ͨ͠

    ΋ͷ͸ԼͰड͚ࢭΊΔ
    

    View Slide

19. ߏจϋΠϥΠτ
    -JOU࣮ߦ
    ੩తܕݕࠪ
    ୯ମςετ
    ݁߹ςετ
    &&ςετ
    ຊ൪؀ڥͰͷΤϥʔϨϙʔλʔ
    ͜ͷൃදͰൃݟ͍ͨ͠

    όά͸ɺ୯ମςετͰ

    औΓ͜΅͞ΕΔ͕

    ΤϥʔϨϙʔτΑΓ

    ૣ͘ݟ͚ͭΒΕΔόά
    

    View Slide

20. -JOU΍୯ମςετͰ

    όάΛݟ্͚ͭͨͰ

    ซ༻͢Δͷ͕େલఏ
    ཁ͢Δʹ
    

    View Slide

21. ݁߹ςετ͸อक͕ͱͯ΋େม
    ͨͩ͠
    

    View Slide

22. ݁߹͢Δ΄Ͳɺࢼ͞ͳ͍ͱ

    ͍͚ͳ͍૊Έ߹Θ͕ͤ૿͑Δ
    ݁߹͢Δ΄Ͳɺݸʑͷཁૉͷ

    มߋʹରͯ͠ςετ͕੬͘ͳΔ
    ܽ఺
    ܽ఺
    

    View Slide

23. υϝΠϯݻ༗ͷ݁߹ϨΠϠʔ

    ϞϯΩʔςετ
    ͦ͜Ͱ
    

    View Slide

24. ಛఆͷૢ࡞खॱΛఆΊͣɺ

    ແ࡞ҝʹૢ࡞͢Δςετ
    ϞϯΩʔςετͱ͸
    

    View Slide

25. ແ࡞ҝʹૢ࡞͢ΔͷͰ૊Έ߹Θ͕ͤ
    ૿͑ͯ΋ίʔυ͸͋·Γ૿͑ͳ͍
    Ұൠతͳੑ࣭͔֬͠ೝ͠ͳ͍ͷͰɺ

    ݸʑͷཁૉ͕มߋ͞Εͯ΋յΕͳ͍
    ར఺
    ར఺
    

    View Slide

26. ήʔϜͰ͸ແ࡞ҝʹૢ࡞͢Δ
    ͚ͩͰ͸ෆे෼ͳ͜ͱ͕ଟ͍
    ͨͩ͠
    

    View Slide

27. ͋Δॱ൪Ͱૢ࡞͢Δͱ

    ඞࡴٕ͕ൃಈ͢ΔΑ͏ͳ

    ΞΫγϣϯ͕͋Δͱ͖ɺ

    ແ࡞ҝͩͱΞΫγϣϯ͸

    ΄ͱΜͲൃੜ͠ͳ͍
    ΞΫγϣϯҙຯΛ΋ͬͨ࿈ଓૢ࡞ʹ

    Α࣮ͬͯߦ͞ΕΔΠϕϯτ
    ΞΫγϣϯ
    

    View Slide

28. ՄೳͳΞΫγϣϯΛ

    ެฏʹൃੜͤ͞ͳ͍ͱ

    ςετ͕ෆ׬શʹͳΔ
    ͭ·Γ
    

    View Slide

29. υϝΠϯݻ༗ͷ݁߹ϨΠϠʔ

    ϞϯΩʔςετ
    ͦ͜Ͱ
    

    View Slide

30.  ͭʹߜΓࠐΉ
    ⁞ՄೳͳΞΫγϣϯΛ

    ໰͍߹Θͤ
    ⁠ΞΫγϣϯΛ࣮ߦ
    ඞࡴٕ
    ࠨӈҠಈ
    δϟϯϓ
    ඞࡴٕ
    ࠨӈҠಈ
    ඞࡴٕ
    ࣮ߦ
    ϦλΠΞ
    

    View Slide

31. ެฏͳΞΫγϣϯ͕Մೳʹͳͬͨ
    ͭ·Γ
    

    View Slide

32. ͜͜·Ͱͷ·ͱΊ
    -JOU΍୯ମςετͰ͸

    ݟ͚ͭΒΕͳ͍όάΛ

    อक༰қͳςετͰݟ͚͍ͭͨ
    

    View Slide

33. $*ʹࡌͤΔࡍͷ՝୊
    

    View Slide

34. Ͳ͜ͰςετΛಈ͔͔͢
    ΞϓϦΛͲ͔͜Ͱىಈ͢Δඞཁ͕͋Δ͕ɺ

    ࣮ػݻ༗ͷ໰୊Λݟ͚͍ͭͨΘ͚Ͱ͸ͳ͍
    ϞϯΩʔςετ͸େྔʹฒྻ࣮ߦ͢Δ΄Ͳ
    ୹࣌ؒͰଟ͘ͷ໰୊Λݟ͚ͭΒΕΔͷͰ

    ߴ଎Ͱ҆ՁͳγϛϡϨʔλΛ࢖͑ΔͳΒ

    ͦ͏͍ͨ͠
    

    View Slide

35. γϛϡϨʔλΫϥ΢υʜʁ
    γϛϡϨʔλΛΫϥ΢υͰ࣮ߦ͢Δ

    αʔϏε͸Θ͔ͣʹ͋Δ͕࢖͍ͮΒ͍ʜ
    ࢓ํͳ͍ͷͰ࣮ػΛΫϥ΢υͰ࣮ߦͰ͖Δ

    σόΠεϑΝʔϜ͕ୈҰީิʹͳΔ
    

    View Slide

36. طଘͷσόΠεϑΝʔϜ
    w "84%FWJDF'BSN
    w )FBETQJO
    w 3FNPUF5FTU,JU
    w 74"QQ$FOUFS
    w 'JSFCBTF5FTU-BC
    w 4BVDF-BCT

    3FBM%FWJDF$MPVE
    w
    

    View Slide

37. ͨ͘͞Μ͋Γ͗ͯ͢

    ͲΕΛબ΂͹ʜʂʁ
    

    View Slide

38. ࠓճ͸"84%FWJDF'BSN
    "84%FWJDF'BSN͸NJOͰ

    ͍ͭͰ΋ฒྻ਺Λ্͛Լ͛Ͱ͖Δ
    ଞ͸୆਺͕ܾ·ͬͨظؒܖ໿͕ଟ͘ɺ

    $*த͚ͩฒྻ਺Λ֬อ͢Δͱ͍ͬͨ

    ࢖͍ํͱ૬ੑ͕ѱ͍
    IUUQTBXTBNB[PODPNKQEFWJDFGBSNQSJDJOH
    

    View Slide

39. ͨͩ͠ෆ҆ཁૉ΋ʜ
    ւ֎੡ͷ୺຤͕ଟ͘ɺ೔ຊ੡ͷ୺຤͸

    ͋·Γἧ͍ͬͯͳ͍
    কདྷతʹࠃ಺ͷ୺຤͕ଟ͘

    ैྔ՝ۚͷσόΠεϑΝʔϜ͕ग़ͨΒ

    ৐Γ׵͑ΔՄೳੑ͕͋Δ
    

    View Slide

40. ݱঢ়ͷํ਑
    ͍ͭผͷγϛϡϨʔλΫϥ΢υ΍

    σόΠεΫϥ΢υ΁Ҡߦ͢Δ͔Θ͔Βͳ͍
    ͜ͷ͕͍͖࣌ͭͯ΋ɺ

    Ҡߦίετͷগͳ͍ঢ়ଶΛอ͍ͪͨ
    

    View Slide

41. ͦ͜ͰEFWGBSN
    γϛϡϨʔλσόΠεΫϥ΢υͷ

    ੾Γସָ͕͑ͰɺσόΠεΫϥ΢υ্Ͱ

    ΞϓϦͷىಈঢ়گΛ؅ཧ͢Δπʔϧ
    

    View Slide

42. IUUQTHJUIVCDPN%F/"EFWGBSN
    

    View Slide

43. $*ʹࡌ͍ͤͨ΋ͷ
    EFWGBSNͷ঺հ
    ӡ༻։࢝࣌఺Ͱͷײ૝
    

    View Slide

44. EFWGBSNͰͰ͖Δ͜ͱ
    w ࢦఆͨ͠σόΠεΫϥ΢υ04ػछ

    ୆਺ΞϓϦҾ਺ͰΞϓϦΛىಈ
    w ΞϓϦ͕Ϋϥογϡͨ͠ΒίϚϯυΛ

    ҟৗऴྃɺࢦఆ࣌ؒੜଘͨ͠Βਖ਼ৗऴྃ
    w ݱࡏ͸"84%FWJDF'BSNͷΈͷରԠ
    

    View Slide

45. ͑ɺͦΕ͚ͩʜ
    

    View Slide

46. ཧ༝͕ʜ͋Γ·͢ʜ
    

    View Slide

47. EFWGBSNͷࢥ૝
    ࣮ػɾγϛϡϨʔλɾσόΠεΫϥ΢υΛ

    ໰Θͣɺಉ͡ΠϯλʔϑΣʔεͰ

    ىಈɾऴྃ͢Δ͜ͱͷΈʹઐ೦͢Δ
    ಛఆͷ৚݅Ͱ͔͠αϙʔτͰ͖ͳ͍ػೳ͸

    αϙʔτ͠ͳ͍ʢػೳͷڞ௨ू߹ओٛʣ
    

    View Slide

48. $ devfarm run-ios --os-version 12.0 \

    --device 'apple iphone xs' \

    --instance-group 'example' \

    --platform 'aws-device-farm' \

    --ipa 'path/to/app.ipa' \

    --args-json '["-ARG1", "HELLO_DEVFARM"]' \

    --lifetime-sec 900 &
    platform status
    aws-device-farm launching
    J04ΞϓϦΛ෼࣮ؒߦʢΫϥογϡ͠ͳ͔ͬͨΒਖ਼ৗऴྃʣ
    

    View Slide

49. $ devfarm status --instance-group 'example'

    platform device os state note
    aws-device-farm apple iphone xs ios ACTIVE
    ىಈ͍ͯ͠ΔΞϓϦͷ֬ೝ
    

    View Slide

50. $ devfarm halt --instance-group 'example'

    platform status
    aws-device-farm halting
    ΞϓϦͷڧ੍ऴྃ
    

    View Slide

51. instance_groups:
    :
    # for Android
    - platform: # required
    android: # required
    device: # required
    apk: # required
    app_id: # required
    intent_extras: [] # optional
    lifetime_sec: 900 # required
    $ devfarm run-all planfile.yml
    ෳ਺ΞϓϦͷҰׅىಈ
    

    View Slide

52. "84%FWJDF'BSNͰ୆ฒྻ࣮ߦ͍ͯ͠Δ༷ࢠ
    

    View Slide

53. EFWGBSNͱ͸
    ࣮ػͱγϛϡϨʔλͷ੾Γସָ͕͑Ͱɺ

    σόΠεΫϥ΢υ্ͰΞϓϦͷىಈঢ়گΛ
    ؅ཧ͢Δπʔϧ
    $-*ͳͷͰ$*αʔϏε͔Β࣮ߦ͠΍͍͢
    

    View Slide

54. ฐࣾͷEFWGBSNར༻
    EFWGBSNʹ͸ςετ࣮ߦΛ։࢝͢Δ

    ࢓૊ΈΛ༻ҙ͠ͳ͍ํ਑
    ઐ༻ʹϏϧυͨ͠ήʔϜΞϓϦͷىಈͱ

    ಉ࣌ʹϞϯΩʔςετΛ։͍࢝ͤͯ͞Δ
    

    View Slide

55. $*ʹࡌ͍ͤͨ΋ͷ
    EFWGBSNͷ঺հ
    ӡ༻։࢝௚લ࣌఺Ͱͷײ૝
    

    View Slide

56. "OESPJEͷηοτΞοϓ͕ෆ҆ఆʜ
    "84%FWJDF'BSNଆͱͷ੾Γ෼͚͕

    Ͱ͖͍ͯͳ͍͕ɺ"OESPJE͕ͷ

    ֬཰Ͱ͏·͘ىಈͯ͘͠Εͳ͍ʜ
    ݱࡏ"84΁໰͍߹Θͤதʜ
    ࠷ѱπʔϧଆͰSFUSZରԠʹͳΔ͔΋
    

    View Slide

57. ୆ηοτΞοϓʹࣦഊ༷ͨ͠ࢠ
    

    View Slide

58. ଟ͘ͷσόΠεϑΝʔϜ͕

    ಛఆͷϑϨʔϜϫʔΫʹಛԽ͗͢͠
    "QQJVN΍9$6*5FTUͳͲͷಛఆͷ

    ϑϨʔϜϫʔΫͷ࣮ߦํ๏ʹґଘ͍ͯ͠Δ
    ࠓճ͸ىಈ؂ࢹऴ͚ྃͩΛ͔͕ͤͨͬͨ͞

    ͍Ζ͍Ζͱେมͩͬͨʜ
    

    View Slide

59. ྫ"84%FWJDF'BSNͰඞཁͳ΋ͷͱҾ਺ͱͷؔ܎
    

    View Slide

60. Run all by ./test/health-check/planfile.yml:
    searching to skip creating AWS Device Farm projects
    finding the AWS Device Farm project: "devfarm-health_check"
    the AWS Device Farm project was found
    skipped to create an AWS Device Farm project (because the project already exists)
    searching AWS Device Farm device
    the AWS Device Farm device found
    searching AWS Device Farm device pool to skip creation
    listing AWS Device Farm device pools
    found the AWS Device Farm device pool
    skipping to create AWS Device Farm device pool (because already exists)
    generating AWS Device Farm custom test spec
    AWS Device Farm custom test spec was successfully generated
    uploading AWS Device Farm custom test spec
    searching AWS Device Farm upload to skip upload
    listing AWS Device Farm uploads
    found AWS Device Farm upload
    found completed AWS Device Farm upload
    generating AWS Device Farm test package
    searching AWS Device Farm upload to skip upload
    listing AWS Device Farm uploads
    ྫ࣮ߦϩά͔Β࡯ͤΒΕΔେม͞
    ·࣮ͩߦ࢝·ͬͯͳ͍ʜ
    

    View Slide

61. ͦΕͰ΋ɺςετͷ஌໊౓Λ্͛Δ

    ઌۦ͚͔ͭීٴͷج൫ͱͳΓ͍ͨ
    ىಈ؂ࢹऴྃΛඪ४Ͱ҆ఆͯ͠αϙʔτͯ͘͠ΕΔ

    σόΠεϑΝʔϜ͕૿͑ΔͨΊʹ͸஌໊౓͕ඞཁ
    υϝΠϯݻ༗ɾ݁߹ϨΠϠʔɾϞϯΩʔςετ͸

    อक༰қͰޮՌతͳςετͳͷͰɺ

    ͜ΕΛ͖͔͚ͬʹ޿·ͬͯ͘ΕͨΒخ͍͠
    

    View Slide

62. ·ͱΊ
    EFWGBSN͸-JOU΍୯ମςετͰ

    ݟ͚ͭΒΕͳ͍όάΛอक༰қͳ

    ςετͰݟ͚ͭΔͨΊͷ044Ͱ͢
    

    View Slide

63. '"2ͳͥ"QQJVNΛ࢖Θͳ͍ͷʁ
    কདྷతʹΑΓݡ͍ΦʔτύΠϩοτʹ͢Δ

    Մೳੑ͕͋ͬͯ"QQJVNͩͱ΍ΓͮΒ͍
    ىಈ؂ࢹऴྃͷͨΊʹ"QQJVNΛ

    ಺෦తʹ࢖͏ࢼΈ΋͕͋ͬͨ

    "84%FWJDF'BSN্ͩͱσόοάࠔ೉
    

    View Slide