Автоматизация регрессии аналитики

Talk by Andrii Rogulin

Андрей расскажет нам об автоматизации регрессии аналитики, а именно:
- Как упростить жизнь Data Analytics и QA.
- Как (почти) избавиться от Charles при проверке аналитики
- Как подружить UI тесты, сбор и проверку аналитики
- Решения и инструменты

This talk was made for CocoaFriday #4 ( https://cocoaheads.org.ua/cocoafriday/4) which took place June 14, 2019

Video: https://youtu.be/jwsohbSNZ2M

Transcript

1. Better Analytics Andrii Rogulin
2. – Arthur C. Nielsen, Market Researcher & Founder of ACNielsen The price of light is less than the cost of darkness.
3. Why is it so important? A/B tests Conversion to subscription Retention User behaviors …
4. Pareto Rule
5. Charles
6. Analytic events manual debug in Charles
7. Basically… it works
11. - But what can we do? - Automate it! - But what will we get? Reduced time for analytics testing CI-driven analytics checks Reduced risk of human-driven mistakes Avoid breaks in analytics during refactoring
12. Appium Why not native? It’s cross-platform It can work with Java
13. Setup Homebrew Node+npm Maven (or Gradle) Appium Appium Desktop (optional) Appium Doctor (optional but recommended) System variables: - JAVA_HOME - M2_HOME - M2 Java libs selenium-java TestNG Appium Allure ...
14. Selenium
 (Open-Source lib) Web Driver Agent iOS Simulator Your Application Appium
 Server Your
 UI tests code TestNG + Maven + Java How it works
15. Manual Run
16. { "platformName": "iOS", "platformVersion": "12.1", "deviceName": "iPhone XR", "app": "/Users/Me/Dev/MyAwesomeProject/Project.ipa" } Basic Setup Appium server
17. WebDriverAgent
18. Inspecting UI elements
19. Run from scripts
20. Build fo Simulator (fastlane) fastlane `gym` command does not work You’ll get an .app file
21. Wrap .ipa
22. Setting up device capabilities in your UI tests project
23. Setting up profiles (for Maven) pom.xml <profile> <id>My-UItests=Profile-12.1</id> <properties> <hub.url>http://127.0.0.1:4723/wd/hub</hub.url> <directory>project.xml</directory> <app.path>${project.basedir}/../Project.ipa</app.path> <language>en</language> </properties> </profile> TestNG xml path to your .ipa file
24. and... run mvn clean test -P 'My-UItests-Profile-12.1'
26. Unit tests 
 vs 
 High order tests (that’s all about analytics)
27. Unit tests - Tests will vary depending on application’s architecture - Tests will be different on each platform (iOS, Android, etc)
28. High order tests - Architecturally independent - Platform independent (in some way)
29. Get the most out of your 
 High order tests
30. The touchpoint is in files
31. CocoaLumberjack
32. File rolling by: - file size - time - session Multiple files management - number of files File reuse policy
33. How things are going… generally You’re likely to have such code somewhere regardless to your app architecture
34. Or how things can go
35. Couple words about CI How Jenkins see it:
36. Couple words about CI What actually happens: 1) Cleaning build folder 2) Updating dependencies 3) Build for simulator 4) Wrapping .app to .ipa
37. Couple words about CI 5) Pulling Automation tests repo to subfolder 6) Checking out needed automation tests branch 7) Starting Appium 8) Running UI tests 9) Killing Appium after all
38. Allure report of failed test
39. Expected result: Actual result: Duplicated events
40. While you sleep, CI runs your tests
42. Bonus - Handy analytics log to be passed as a new expected result to automation tests - On-device autonomous analytics testing DON’T FORGET TO REMOVE IT IN THE APP STORE BUILDS
44. Results - Analytics regression tests that can be run on CI - Analytics log file to be passed around - Time economy for manual tests - Handy real-time analytics logging for manual tests - Autonomous analytics on-device testing and sharing - Reduced chance of human-driven mistakes while doing routine analytics checks - Platform and architectural independent tests and Analytics in NOT the only thing you can log and share with your 
 UI tests ;)
45. Thanks! This presentation would not be possible
 without our QA team Andrii Rogulin