Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Justin Ison (Isonic), Mobile App Crawling — A year automating myself out of a job, CodeFest 2017

CodeFest
January 25, 2018

Justin Ison (Isonic), Mobile App Crawling — A year automating myself out of a job, CodeFest 2017

https://2017.codefest.ru/lecture/1143

In todays Agile development world we do not have a lot of time to review our applications on every new CD build. Especially, when you consider different OS’s, locales, and resolutions your mobile app supports.

Adding an app crawler to your CI pipeline is ideal for environments when your app is in constant change or development. A app crawler will look at the new build when others in your company may not. A app crawler might find a issue where automated test were not programmed to.

CodeFest

January 25, 2018
Tweet

More Decks by CodeFest

Other Decks in Programming

Transcript

  1. The Agile World Moves Fast It’s up to us to

    keep up… I have noticed a trend: • Increasingly shorter time to market • Desire for continued feature updates to keep ahead of competition • Adoption of QA-less teams, requiring developers to test also This means less time is usually allocated to exploratory and manually testing your application. IMO there are not enough software options available in the market to help fill the quality gaps because of this shortened release and testing time. Because of this: • Many companies (big and small) are relying more and more on the endusers/customers to report issues. • Some companies rely almost solely on telemetry (analytics). • Analytics only provide you with so much. You are not going to get UI, locale, or design flaws from analytics.
  2. My Goal Automate Exploratory Testing! Know more about my application

    for every build • Capture elements with and without accessibility labels • Capture images for every supported resolution and orientation • Know when errors or exceptions occur • Detect language or locale issues • Capture performance data • Generate unique telemetry data Ability to replay a test after code fix Be cross-platform!
  3. So what led me here? I once worked for a

    company that was very big on dog-fooding our applications. However, there was a slight problem with this. • People tended to only test the platform they preferred. We were not getting much feedback for our less popular platforms. • We had a very small QA team that could not manual and exploratory test all of our platforms. So this led me to the idea of creating something that didn’t care about a apps current state and would test whatever was given to it, reporting back it’s findings. I also didn’t want to replace UI Test Automation, but rather include another tool to fill the missing quality gaps as I mentioned previously. The pain of automating and re-automating tests • Constantly fixing and revising tests due to code or design changes
  4. What is an app crawler? • An app crawler is

    much like a web crawler (spider, robot). It’s a program that is designed to emulate user interactions. A self-automating program that interacts with components that are available in the UI much like a user would.
  5. Device Orientation • Majority of people test only in portrait

    orientation. • How will your apps usability and design work in landscape?
  6. Device Resolution • Most people only test the device and

    resolution that is available to them. • How will your designs look on every resolution you support?
  7. OS Versions •Most people only test the device OS that

    is available to them. •How will your app look and function for every OS version you support?
  8. Language • Majority of people test only in their native

    tongue. • How will your designs work for languages such as Arabic or Hebrew? • How will your designs look in languages with longer strings like Deutsch?
  9. Metadata Overload! Now you can see, there are many many

    different variables for which to account. • How do you know your app will work correctly for all combinations in shorter sprints? You could do this all with UI Test Automation. • Then again UI tests only test what is programmed to test. • But you would need an army of engineers to pull it off.
  10. • Or perhaps an army of zombie testers. • Automating

    every possible combination would be next to impossible to maintain.
  11. Some Challenges • Hybrid web views that will never end

    • Detecting unique locators • Rescuing the app when it gets stuck • Handling authentication login or accidental logout • Gestures and when to appropriately use them
  12. Finding Unique Locators Difficulty finding uniqueness in the DOM •

    Parent and children classes can execute same functions, but not necessarily always. Text values can change and so then do the locators attributes. • You can potentially get into an endless loop of adding new objects/elements and then interacting with these objects Views are layered on top of views • Getting just the elements that are visible in the top layer
  13. Performance The Forgotten Test It's not enough to only have

    automated tests to ensure quality and catch regressions. • It's imperative to know more about what’s happening under-the- hood of your mobile application. Specifically, monitoring the performance. • It's not only important to capture this information, but storing it for historical lookup, benchmarking and detecting trends.
  14. Language Detection As you’ve seen, there are a lot of

    screenshots generated. I was manually scanning every image for abnormalities. So, I thought there had to be a better way! I dug into learning about all the open source translation libraries. I tried several, but none of them worked very well. Some worked great when submitting a sentence of 4 or more words, but that is not going to work well for applications when one word titles are used. I relented and used a paid service like google translations. They provided the best results compared with any open source tools I tried.
  15. Log Monitoring When performing exploratory testing, it’s very important to

    monitor the logs at the same time. Many errors go unnoticed in the UI, such as network or memory errors. Same applies to automated exploratory testing! I wanted to: • Detect any errors and record them • Capture exceptions and shut down • Take screenshot when an error or exception occurs
  16. Exception Capture --- beginning of system
 6 11:56:08.177 15803 15803

    E AndroidRuntime: FATAL EXCEPTION: main
 6 11:56:08.177 15803 15803 E AndroidRuntime: Process: com.amazonaws.devicefarm.android.referenceapp, PID: 15803
 6 11:56:08.177 15803 15803 E AndroidRuntime: java.lang.ArithmeticException: divide by zero
 6 11:56:08.177 15803 15803 E AndroidRuntime: at .amazonaws.devicefarm.android.referenceapp.Fragments.crashFragment.onCrashButton(crashFragment.java:49)
 6 11:56:08.177 15803 15803 E AndroidRuntime: at com.amazonaws.devicefarm.android.referenceapp.Fragments.crashFragment$ wInjector$1.doClick(crashFragment$$ViewInjector.java:17)
 6 11:56:08.177 15803 15803 E AndroidRuntime: at erknife.internal.DebouncingOnClickListener.onClick(DebouncingOnClickListener.java:22)
 6 11:56:08.177 15803 15803 E AndroidRuntime: at android.view.View.performClick(View.java:5198)
 6 11:56:08.177 15803 15803 E AndroidRuntime: at android.view.View$PerformClick.run(View.java:21147)
 6 11:56:08.177 15803 15803 E AndroidRuntime: at android.os.Handler.handleCallback(Handler.java:739)
 6 11:56:08.177 15803 15803 E AndroidRuntime: at android.os.Handler.dispatchMessage(Handler.java:95)
 6 11:56:08.177 15803 15803 E AndroidRuntime: at android.os.Looper.loop(Looper.java:148)
 6 11:56:08.177 15803 15803 E AndroidRuntime: at android.app.ActivityThread.main(ActivityThread.java:5417)
 6 11:56:08.177 15803 15803 E AndroidRuntime: at java.lang.reflect.Method.invoke(Native Method)
 6 11:56:08.177 15803 15803 E AndroidRuntime: at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726) 6 11:56:08.177 15803 15803 E AndroidRuntime: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)
 6 11:56:08.191 1190 1190 E EGL_emulation: tid 1190: eglCreateSyncKHR(1370): error 0x3004 (EGL_BAD_ATTRIBUTE)
  17. Replay Test • Validate if a crash is repeatable. •

    Rerun the test after a code fix is made. • Compare performance data from before and after each run.
  18. Automatic Tests! I wanted to take it one step further.

    Reviewing screenshots for every single language, resolution and orientation on every build became very cumbersome, and also prone to human mistake. How can I automate that process? • Image validation with Applitools!
  19. 02-25 14:50:23.111 7813 7813 E AndroidRuntime: FATAL EXCEPTION: main 02-25

    14:50:23.111 7813 7813 E AndroidRuntime: Process: com.snapchat.android, PID: 7813 02-25 14:50:23.111 7813 7813 E AndroidRuntime: java.lang.NullPointerException: Attempt to invoke virtual method 'android.content.Context android.view.View.getContext()' on a null object reference 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at fra.a(SourceFile:409) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at com.snapchat.android.app.feature.search.ui.SearchFragment.onDelegatedBackPressed(SourceFile:299) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at com.snapchat.android.LandingPageActivity.onBackPressed(SourceFile:50480) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.app.Activity.onKeyUp(Activity.java:2511) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at com.snapchat.android.LandingPageActivity.onKeyUp(SourceFile:4014) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.KeyEvent.dispatch(KeyEvent.java:2698) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.app.Activity.dispatchKeyEvent(Activity.java:2764) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at com.android.internal.policy.PhoneWindow$DecorView.dispatchKeyEvent(PhoneWindow.java:2353) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$ViewPostImeInputStage.processKeyEvent(ViewRootImpl.java:4639) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$ViewPostImeInputStage.onProcess(ViewRootImpl.java:4595) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$InputStage.deliver(ViewRootImpl.java:4108) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$InputStage.onDeliverToNext(ViewRootImpl.java:4161) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$InputStage.forward(ViewRootImpl.java:4127) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$AsyncInputStage.forward(ViewRootImpl.java:4273) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$InputStage.apply(ViewRootImpl.java:4135) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$AsyncInputStage.apply(ViewRootImpl.java:4330) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$InputStage.deliver(ViewRootImpl.java:4108) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$InputStage.onDeliverToNext(ViewRootImpl.java:4161) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$InputStage.forward(ViewRootImpl.java:4127) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$InputStage.apply(ViewRootImpl.java:4135) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$InputStage.deliver(ViewRootImpl.java:4108) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$InputStage.onDeliverToNext(ViewRootImpl.java:4161) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$InputStage.forward(ViewRootImpl.java:4127) 02-25 14:50:23.111 7813 7813 E AndroidRuntime: at android.view.ViewRootImpl$AsyncInputStage.forward(ViewRootImpl.java:4306) Snapchat Crash
  20. Conclusion Hopefully I’ve demonstrated the benefits of adding a crawler

    to your build process. I do believe with the rate of technology changes, app crawling could be the future of UI Test Automation or some form of it. I plan to open source the crawling engine I have built. • If anyone is interested in helping out, please feel free to contact me.