Автоматизация регрессии аналитики

Talk by Andrii Rogulin

Андрей расскажет нам об автоматизации регрессии аналитики, а именно:
- Как упростить жизнь Data Analytics и QA.
- Как (почти) избавиться от Charles при проверке аналитики
- Как подружить UI тесты, сбор и проверку аналитики
- Решения и инструменты

This talk was made for CocoaFriday #4 ( https://cocoaheads.org.ua/cocoafriday/4) which took place June 14, 2019

Video: https://youtu.be/jwsohbSNZ2M

Transcript

1. Better Analytics
   Andrii Rogulin
   

   View Slide

2. – Arthur C. Nielsen, Market Researcher & Founder of ACNielsen
   The price of light is less than the cost of darkness.
   

   View Slide

3. Why is it so important?
   A/B tests
   Conversion to subscription
   Retention
   User behaviors
   …
   

   View Slide

4. Pareto Rule
   

   View Slide

5. Charles
   

   View Slide

6. Analytic events manual debug in Charles
   

   View Slide

7. Basically… it works
   

   View Slide

8. View Slide

9. View Slide

10. View Slide

11. - But what can we do?
    - Automate it!
    - But what will we get?
    Reduced time for analytics testing
    CI-driven analytics checks
    Reduced risk of human-driven mistakes
    Avoid breaks in analytics during refactoring
    

    View Slide

12. Appium
    Why not native?
    It’s cross-platform
    It can work with Java
    

    View Slide

13. Setup
    Homebrew
    Node+npm
    Maven (or Gradle)
    Appium
    Appium Desktop (optional)
    Appium Doctor (optional but recommended)
    System variables:
    - JAVA_HOME
    - M2_HOME
    - M2
    Java libs
    selenium-java
    TestNG
    Appium
    Allure
    ...
    

    View Slide

14. Selenium

    (Open-Source lib)
    Web Driver Agent
    iOS Simulator
    Your Application
    Appium

    Server
    Your

    UI tests
    code
    TestNG + Maven + Java
    How it works
    

    View Slide

15. Manual Run
    

    View Slide

16. {
    "platformName": "iOS",
    "platformVersion": "12.1",
    "deviceName": "iPhone XR",
    "app": "/Users/Me/Dev/MyAwesomeProject/Project.ipa"
    }
    Basic Setup
    Appium server
    

    View Slide

17. WebDriverAgent
    

    View Slide

18. Inspecting UI elements
    

    View Slide

19. Run from scripts
    

    View Slide

20. Build fo Simulator (fastlane)
    fastlane `gym`
    command does not
    work
    You’ll get an .app file
    

    View Slide

21. Wrap .ipa
    

    View Slide

22. Setting up device capabilities in your UI tests project
    

    View Slide

23. Setting up profiles (for Maven)
    pom.xml
    
    My-UItests=Profile-12.1
    
    http://127.0.0.1:4723/wd/hub
    project.xml
    ${project.basedir}/../Project.ipa
    en
    
    
    TestNG xml
    path to your .ipa file
    

    View Slide

24. and... run
    mvn clean test -P 'My-UItests-Profile-12.1'
    

    View Slide

25. View Slide

26. Unit tests 

    vs 

    High order tests
    (that’s all about analytics)
    

    View Slide

27. Unit tests
    - Tests will vary depending on
    application’s architecture
    - Tests will be different on each
    platform (iOS, Android, etc)
    

    View Slide

28. High order tests
    - Architecturally independent
    - Platform independent (in some way)
    

    View Slide

29. Get the most out of your 

    High order tests
    

    View Slide

30. The touchpoint
    is in files
    

    View Slide

31. CocoaLumberjack
    

    View Slide

32. File rolling by:
    - file size
    - time
    - session
    Multiple files management
    - number of files
    File reuse policy
    

    View Slide

33. How things are going… generally
    You’re likely to have such code somewhere regardless to your app architecture
    

    View Slide

34. Or how things can go
    

    View Slide

35. Couple words about CI
    How Jenkins see it:
    

    View Slide

36. Couple words about CI
    What actually happens:
    1) Cleaning build folder
    2) Updating dependencies
    3) Build for simulator
    4) Wrapping .app to .ipa
    

    View Slide

37. Couple words about CI
    5) Pulling Automation tests repo to subfolder
    6) Checking out needed automation tests branch
    7) Starting Appium
    8) Running UI tests
    9) Killing Appium after all
    

    View Slide

38. Allure report of failed test
    

    View Slide

39. Expected result:
    Actual result: Duplicated events
    

    View Slide

40. While you sleep, CI runs your tests
    

    View Slide

41. View Slide

42. Bonus
    - Handy analytics log to be passed as
    a new expected result to automation
    tests
    - On-device autonomous analytics
    testing
    DON’T FORGET TO REMOVE IT IN THE
    APP STORE BUILDS
    

    View Slide

43. View Slide

44. Results
    - Analytics regression tests that can be
    run on CI
    - Analytics log file to be passed around
    - Time economy for manual tests
    - Handy real-time analytics logging for
    manual tests
    - Autonomous analytics on-device
    testing and sharing
    - Reduced chance of human-driven
    mistakes while doing routine analytics
    checks
    - Platform and architectural
    independent tests
    and
    Analytics in NOT the only thing
    you can log and share with your 

    UI tests ;)
    

    View Slide

45. Thanks!
    This presentation would not be possible

    without our QA team
    Andrii Rogulin
    

    View Slide