Автоматизация регрессии аналитики

Автоматизация регрессии аналитики

Talk by Andrii Rogulin

Андрей расскажет нам об автоматизации регрессии аналитики, а именно:
- Как упростить жизнь Data Analytics и QA.
- Как (почти) избавиться от Charles при проверке аналитики
- Как подружить UI тесты, сбор и проверку аналитики
- Решения и инструменты

This talk was made for CocoaFriday #4 ( https://cocoaheads.org.ua/cocoafriday/4) which took place June 14, 2019

Video: https://youtu.be/jwsohbSNZ2M

Db84cf61fdada06b63f43f310b68b462?s=128

CocoaHeads Ukraine

June 14, 2019
Tweet

Transcript

  1. Better Analytics Andrii Rogulin

  2. – Arthur C. Nielsen, Market Researcher & Founder of ACNielsen

    The price of light is less than the cost of darkness.
  3. Why is it so important? A/B tests Conversion to subscription

    Retention User behaviors …
  4. Pareto Rule

  5. Charles

  6. Analytic events manual debug in Charles

  7. Basically… it works

  8. None
  9. None
  10. None
  11. - But what can we do? - Automate it! -

    But what will we get? Reduced time for analytics testing CI-driven analytics checks Reduced risk of human-driven mistakes Avoid breaks in analytics during refactoring
  12. Appium Why not native? It’s cross-platform It can work with

    Java
  13. Setup Homebrew Node+npm Maven (or Gradle) Appium Appium Desktop (optional)

    Appium Doctor (optional but recommended) System variables: - JAVA_HOME - M2_HOME - M2 Java libs selenium-java TestNG Appium Allure ...
  14. Selenium
 (Open-Source lib) Web Driver Agent iOS Simulator Your Application

    Appium
 Server Your
 UI tests code TestNG + Maven + Java How it works
  15. Manual Run

  16. { "platformName": "iOS", "platformVersion": "12.1", "deviceName": "iPhone XR", "app": "/Users/Me/Dev/MyAwesomeProject/Project.ipa"

    } Basic Setup Appium server
  17. WebDriverAgent

  18. Inspecting UI elements

  19. Run from scripts

  20. Build fo Simulator (fastlane) fastlane `gym` command does not work

    You’ll get an .app file
  21. Wrap .ipa

  22. Setting up device capabilities in your UI tests project

  23. Setting up profiles (for Maven) pom.xml <profile> <id>My-UItests=Profile-12.1</id> <properties> <hub.url>http://127.0.0.1:4723/wd/hub</hub.url>

    <directory>project.xml</directory> <app.path>${project.basedir}/../Project.ipa</app.path> <language>en</language> </properties> </profile> TestNG xml path to your .ipa file
  24. and... run mvn clean test -P 'My-UItests-Profile-12.1'

  25. None
  26. Unit tests 
 vs 
 High order tests (that’s all

    about analytics)
  27. Unit tests - Tests will vary depending on application’s architecture

    - Tests will be different on each platform (iOS, Android, etc)
  28. High order tests - Architecturally independent - Platform independent (in

    some way)
  29. Get the most out of your 
 High order tests

  30. The touchpoint is in files

  31. CocoaLumberjack

  32. File rolling by: - file size - time - session

    Multiple files management - number of files File reuse policy
  33. How things are going… generally You’re likely to have such

    code somewhere regardless to your app architecture
  34. Or how things can go

  35. Couple words about CI How Jenkins see it:

  36. Couple words about CI What actually happens: 1) Cleaning build

    folder 2) Updating dependencies 3) Build for simulator 4) Wrapping .app to .ipa
  37. Couple words about CI 5) Pulling Automation tests repo to

    subfolder 6) Checking out needed automation tests branch 7) Starting Appium 8) Running UI tests 9) Killing Appium after all
  38. Allure report of failed test

  39. Expected result: Actual result: Duplicated events

  40. While you sleep, CI runs your tests

  41. None
  42. Bonus - Handy analytics log to be passed as a

    new expected result to automation tests - On-device autonomous analytics testing DON’T FORGET TO REMOVE IT IN THE APP STORE BUILDS
  43. None
  44. Results - Analytics regression tests that can be run on

    CI - Analytics log file to be passed around - Time economy for manual tests - Handy real-time analytics logging for manual tests - Autonomous analytics on-device testing and sharing - Reduced chance of human-driven mistakes while doing routine analytics checks - Platform and architectural independent tests and Analytics in NOT the only thing you can log and share with your 
 UI tests ;)
  45. Thanks! This presentation would not be possible
 without our QA

    team Andrii Rogulin