Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Автоматизация регрессии аналитики

Автоматизация регрессии аналитики

Talk by Andrii Rogulin

Андрей расскажет нам об автоматизации регрессии аналитики, а именно:
- Как упростить жизнь Data Analytics и QA.
- Как (почти) избавиться от Charles при проверке аналитики
- Как подружить UI тесты, сбор и проверку аналитики
- Решения и инструменты

This talk was made for CocoaFriday #4 ( https://cocoaheads.org.ua/cocoafriday/4) which took place June 14, 2019

Video: https://youtu.be/jwsohbSNZ2M

CocoaHeads Ukraine

June 14, 2019
Tweet

More Decks by CocoaHeads Ukraine

Other Decks in Programming

Transcript

  1. Better Analytics
    Andrii Rogulin

    View Slide

  2. – Arthur C. Nielsen, Market Researcher & Founder of ACNielsen
    The price of light is less than the cost of darkness.

    View Slide

  3. Why is it so important?
    A/B tests
    Conversion to subscription
    Retention
    User behaviors

    View Slide

  4. Pareto Rule

    View Slide

  5. Charles

    View Slide

  6. Analytic events manual debug in Charles

    View Slide

  7. Basically… it works

    View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. - But what can we do?
    - Automate it!
    - But what will we get?
    Reduced time for analytics testing
    CI-driven analytics checks
    Reduced risk of human-driven mistakes
    Avoid breaks in analytics during refactoring

    View Slide

  12. Appium
    Why not native?
    It’s cross-platform
    It can work with Java

    View Slide

  13. Setup
    Homebrew
    Node+npm
    Maven (or Gradle)
    Appium
    Appium Desktop (optional)
    Appium Doctor (optional but recommended)
    System variables:
    - JAVA_HOME
    - M2_HOME
    - M2
    Java libs
    selenium-java
    TestNG
    Appium
    Allure
    ...

    View Slide

  14. Selenium

    (Open-Source lib)
    Web Driver Agent
    iOS Simulator
    Your Application
    Appium

    Server
    Your

    UI tests
    code
    TestNG + Maven + Java
    How it works

    View Slide

  15. Manual Run

    View Slide

  16. {
    "platformName": "iOS",
    "platformVersion": "12.1",
    "deviceName": "iPhone XR",
    "app": "/Users/Me/Dev/MyAwesomeProject/Project.ipa"
    }
    Basic Setup
    Appium server

    View Slide

  17. WebDriverAgent

    View Slide

  18. Inspecting UI elements

    View Slide

  19. Run from scripts

    View Slide

  20. Build fo Simulator (fastlane)
    fastlane `gym`
    command does not
    work
    You’ll get an .app file

    View Slide

  21. Wrap .ipa

    View Slide

  22. Setting up device capabilities in your UI tests project

    View Slide

  23. Setting up profiles (for Maven)
    pom.xml

    My-UItests=Profile-12.1

    http://127.0.0.1:4723/wd/hub
    project.xml
    ${project.basedir}/../Project.ipa
    en


    TestNG xml
    path to your .ipa file

    View Slide

  24. and... run
    mvn clean test -P 'My-UItests-Profile-12.1'

    View Slide

  25. View Slide

  26. Unit tests 

    vs 

    High order tests
    (that’s all about analytics)

    View Slide

  27. Unit tests
    - Tests will vary depending on
    application’s architecture
    - Tests will be different on each
    platform (iOS, Android, etc)

    View Slide

  28. High order tests
    - Architecturally independent
    - Platform independent (in some way)

    View Slide

  29. Get the most out of your 

    High order tests

    View Slide

  30. The touchpoint
    is in files

    View Slide

  31. CocoaLumberjack

    View Slide

  32. File rolling by:
    - file size
    - time
    - session
    Multiple files management
    - number of files
    File reuse policy

    View Slide

  33. How things are going… generally
    You’re likely to have such code somewhere regardless to your app architecture

    View Slide

  34. Or how things can go

    View Slide

  35. Couple words about CI
    How Jenkins see it:

    View Slide

  36. Couple words about CI
    What actually happens:
    1) Cleaning build folder
    2) Updating dependencies
    3) Build for simulator
    4) Wrapping .app to .ipa

    View Slide

  37. Couple words about CI
    5) Pulling Automation tests repo to subfolder
    6) Checking out needed automation tests branch
    7) Starting Appium
    8) Running UI tests
    9) Killing Appium after all

    View Slide

  38. Allure report of failed test

    View Slide

  39. Expected result:
    Actual result: Duplicated events

    View Slide

  40. While you sleep, CI runs your tests

    View Slide

  41. View Slide

  42. Bonus
    - Handy analytics log to be passed as
    a new expected result to automation
    tests
    - On-device autonomous analytics
    testing
    DON’T FORGET TO REMOVE IT IN THE
    APP STORE BUILDS

    View Slide

  43. View Slide

  44. Results
    - Analytics regression tests that can be
    run on CI
    - Analytics log file to be passed around
    - Time economy for manual tests
    - Handy real-time analytics logging for
    manual tests
    - Autonomous analytics on-device
    testing and sharing
    - Reduced chance of human-driven
    mistakes while doing routine analytics
    checks
    - Platform and architectural
    independent tests
    and
    Analytics in NOT the only thing
    you can log and share with your 

    UI tests ;)

    View Slide

  45. Thanks!
    This presentation would not be possible

    without our QA team
    Andrii Rogulin

    View Slide