Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Everyone hates slow native application tests an...

Avatar for Rebecca Stafford Rebecca Stafford
May 13, 2015
23

Everyone hates slow native application tests and so do we (Alan Nichols and Adrian Negoita)

A talk given by Alan Nichols at the South West Test Meetup #1

Avatar for Rebecca Stafford

Rebecca Stafford

May 13, 2015

Transcript

  1. Context • Mobile application for Android • Implemented with Xamarin

    framework • Automation in calabash-android with Cucumber and Ruby
  2. Why? • Automation suite was growing fast (~40 minutes/tablet) •

    Number of supported tablets increased • Trialling 3rd party components increased the permutations
  3. Complexity • One tablet - running tests on my local

    machine; • Two tablets - started to think about moving away from my local machine; • Three tablets and four 3rd party application - moving away from my local machine.
  4. Cloud services Xamarin Test Cloud • Pros • Real devices

    to run your tests; • Easy integration with calabash automation; • Our app was developed using Xamarin framework. • Cons • Our tablets were not available into their supported devices. • Couldn’t test with third party components
  5. Emulators • Pros • Price; • Simple to use; •

    Fast and stable. • Cons • We wanted to test mobile network behaviour (3g); • Increased risk - not testing what users will use; • Wouldn’t be testing the whole bundle (app + hardware).
  6. In house device lab • Pros: • Hardware was already

    there; • We had the space to create it; • 3rd Party Components • Cons: • Running on one tablet at a time; • Getting result reports slow; • No benefit for releases.
  7. In house parallel device lab • Pros: • Our product

    is not just the app is the whole bundle (app + hardware); • Easy to interact with devices if needed; • Fast results from all platforms; • Being able to monitor and to analyse the behaviour of the tablets; • Faster releases. • Cons: • Maintaining all the hardware.
  8. Implementation • Rake multitask; • Task list containing one task

    for each device; • Hardware identification using the Android hardware IDs • Mapping file between Android IDs and users; • Environment variable for users (sent from TeamCity).
  9. Challenges Challenge 1: Environment Variables - Running on multiple threads

    results in conflicts; Solution: Thread-safe variables that will remove all the thread conflicts; eg. Thread.current['NAME_OF_THE_VARIABLE']
  10. Challenges Challenge 2: Getting a device ID into the cucumber

    task: • couldn’t use environment vars anymore; • cucumber task was running on a different thread than the one used to create it; Solution: • parse the cucumber parameters and use the report path one to grab the device ID
  11. Future work • Rerun mechanism for the failing scenarios; •

    We will be moving some of the tests to use a mock back end… • Emulators for some tests? • Splitting tests for one model between multiple devices? • BYOD?
  12. Questions? The blog post this presentation was based on is

    available here - http://tech.just-eat.com/2015/03/18/everyone-hates-slow-native- application-tests-and-so-do-we/