Aim at 100% Push Notification - Knowledge Obtained by Data Analysis

Hisato Shoji
LINE / LINE B2B App Dev Team / Engineering Manager

https://linedevday.linecorp.com/2021/ja/sessions/133
https://linedevday.linecorp.com/2021/en/sessions/133
https://linedevday.linecorp.com/2021/ko/sessions/133

Transcript

2. Agenda - Introduction - Logging / Data Analysis - Dogfooding - Wrap up
3. - Hisato Shoji - LINE B2B app dev team manager - iOS/Android app of LINE Official Account - LINE Official Account related features in LINE app Self-introduction
4. - For people who working on iOS/Android development - Why & how app dev team takes data and analyzes it, while performing dogfooding activities About this session
5. - B2B2C service to support communication between LINE users and business owners - Over 10M accounts worldwide LINE Official Account
6. LINE Official Account - iOS / Android / PC browser / Web API support - Service available globally, including Japan, Thailand, Taiwan, Indonesia etc
7. - Chat with LINE users - VoIP with LINE users - Issuing coupons - Posting to Timeline - etc iOS / Android app
8. - Notification delivery delay - Notification delivery failure - Pointed out via App Store / Google Play / CS / user interviews Major user feedback
9. But…
10. Logging / Data Analysis
11. Notification system overview LINE User LINE Server Notification Service Extension LINE Official Account App Chat with an LINE Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server
12. Logging - Using inhouse logging system, for easy comparison with server side log etc - iOS: Implemented UNNotificationServiceExtension to download and show attached images of chats, in addition to collecting log for analysis - Android: Implemented FirebaseMessagingService and collect log normally - Note: Text in user message is not collected.
13. Why app developers analyze data? - Company has a data scientist team - Need knowledge about how APNS, Notification Service Extension, FCM, Android Doze, or Background Restriction works - For faster try-and-error cycle - App developers were interested in data analysis J
14. How many delayed Notifications? 92.48% 99.08% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% ANDROID IOS Rate of Notifications reached to app within 1 min
15. How many delayed Notifications? 96.00% 94.05% 92.32% 88.60% 99.36% 99.41% 99.12% 98.46% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% JP TW TH ID Rate of Notifications reached to app within 1 min Android iOS
16. Average time consumed by each part of Android LINE Server -> FCM Server* 43.65 ms FCM Server* -> FirebaseMessageingService 70065.36 ms JSON parsing + Notification API call 35.62 ms Which part takes the most time? * RemoteMessage.getSentTime()
17. Android Doze mode https://developer.android.com/topic/performance/power/power-details
18. Android App Standby Buckets https://developer.android.com/topic/performance/power/power-details
19. How many delayed Notifications? 94.92% 94.34% 64.72% 62.72% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% STANDBY_BUCKET_ACTIVE STANDBY_BUCKET_W ORKING_SET STANDBY_BUCKET_FREQUENT STANDBY_BUCKET_RARE Rate of Notifications reached to app within 1 min
20. How many delayed Notifications? 96.00% 94.05% 92.32% 88.60% 97.68% 96.21% 94.48% 89.03% 99.36% 99.41% 99.12% 98.46% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% JP TW TH ID Rate of Notifications reached to app within 1 min Android Android (Active & Working Set only) iOS
21. How to check App Standby Buckets - UsageStatsManager.getAppStandbyBucket() - RemoteMessage.getPriority() - RemoteMessage.getOriginalPriority() - Stopped sending unnecessary FCM to Android device, reduced 17% of delayed notifications
22. Other suspected points of Android - Background Restriction - Do Not Disturb mode
23. - ActivityManager.isBackgroundRestricted() Background Restriction https://developer.android.com/topic/performance/power/power-details
24. - ActivityManager.isBackgroundRestricted() Background Restriction https://developer.android.com/topic/performance/power/power-details
25. Background Restriction - 0.27% of session, 1.88% of Notification is BackgroundRestriction enabled. 92.27% 84.23% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% Background Restriction=false Background Restriction=true Rate of Notification reached to app within 1 min
26. Background Restriction - Not much impact due to background restriction - Some manufacturers automatically turn on the restriction for unused apps (?) 94.50% 94.71% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% Background Restriction=false Background Restriction=true Rate of Notification reached to app within 1 min (App Standby Bucket Active or Frequent)
27. Do not Disturb mode - 0.19% of Notifications affected by DnD - NotificationManager .getNotificationPolicy() .suppressedVisualEffects
28. Wrap up of Logging / Data Analysis Android - Pay attention to spec of Android App Standby Buckets, minimize number of FCM calls with high priority setting on server side. - Background Restriction, Do not Disturb mode do not have much of an impact (for our case at least) iOS - Utilized Notification Service Extension to collect logs - Proven by data; Relatively fewer Notification delays compared to Android
29. Dogfooding
30. - Fitting in to routine tasks of development team - Use as attendance tool in app development team for WFH Dogfooding
31. Notification system overview LINE User LINE Server Notification Service Extension LINE Official Account App Chat with an LINE Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server
32. Notification system overview LINE User LINE Server Notification Service Extension LINE Official Account App Chat with an LINE Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server
33. Notification system overview LINE User LINE Server Notification Service Extension LINE Official Account App Chat with an Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server
34. Android Notification quota 2021-09-22 15:00:46.533 1390-4456/? E/NotificationService: Package has already posted or enqueued 25 notifications. Not showing more. package=com.linecorp.lineoa services/core/java/com/android/server/notification/NotificationManagerService.java
35. Android Notification quota - Android OS has a Notification quota since Android 2.3.1. - NotificationManager.notify() is ignored and LogCat appears in system side process - No document in Android Developers (?) - Limit changes by Android OS version
36. Android Notification quota % adb shell dumpsys notification --- AggregatedStats{ key='com.linecorp.lineoa’, numRateViolations=0 --- numAlertViolations=12 numQuotaViolations=29 - Status can be checked by adb shell - No Android API to get the quota status - NotificationManager .getActiveNotifications() can be used to check status
37. How many pending Notifications? 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Number of pending Notifications on Android
38. Findings & Next Actions Findings - Proven by data; iOS has relatively fewer Notification delays compared to Android. - Android App Standby Bucket causes Notification delay, need to pay attention. - Android has Notification quota, which can cause missing Notification. Next Actions - Collect logs and analyze how many rate of Notifications reach to app from server - Develop troubleshooting feature on iOS/Android app
39. Logging / Data Analysis - To prove the problems really happen, to find explanatory variables - To confirm resolution after bug fix release Wrap up Logging / Data Analysis - To proof the problems really happens, try to find out explanatory variables - To confirm resolution after bug fix release - Analyze data by app developers Dogfooding - To frequently monitor features in real use case - Fitting in to routine tasks of development team
40. Thank you