No content

Agenda - Introduction - Logging / Data Analysis - Dogfooding - Wrap up

- Hisato Shoji - LINE B2B app dev team manager - iOS/Android app of LINE Official Account - LINE Official Account related features in LINE app Self-introduction

- For people who working on iOS/Android development - Why & how app dev team takes data and analyzes it, while performing dogfooding activities About this session

- B2B2C service to support communication between LINE users and business owners - Over 10M accounts worldwide LINE Official Account

LINE Official Account - iOS / Android / PC browser / Web API support - Service available globally, including Japan, Thailand, Taiwan, Indonesia etc

- Chat with LINE users - VoIP with LINE users - Issuing coupons - Posting to Timeline - etc iOS / Android app

- Notification delivery delay - Notification delivery failure - Pointed out via App Store / Google Play / CS / user interviews Major user feedback

But…

Logging / Data Analysis

Notification system overview LINE User LINE Server Notification Service Extension LINE Official Account App Chat with an LINE Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server

Logging - Using inhouse logging system, for easy comparison with server side log etc - iOS: Implemented UNNotificationServiceExtension to download and show attached images of chats, in addition to collecting log for analysis - Android: Implemented FirebaseMessagingService and collect log normally - Note: Text in user message is not collected.

Why app developers analyze data? - Company has a data scientist team - Need knowledge about how APNS, Notification Service Extension, FCM, Android Doze, or Background Restriction works - For faster try-and-error cycle - App developers were interested in data analysis J

How many delayed Notifications? 92.48% 99.08% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% ANDROID IOS Rate of Notifications reached to app within 1 min

How many delayed Notifications? 96.00% 94.05% 92.32% 88.60% 99.36% 99.41% 99.12% 98.46% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% JP TW TH ID Rate of Notifications reached to app within 1 min Android iOS

Average time consumed by each part of Android LINE Server -> FCM Server* 43.65 ms FCM Server* -> FirebaseMessageingService 70065.36 ms JSON parsing + Notification API call 35.62 ms Which part takes the most time? * RemoteMessage.getSentTime()

Android Doze mode https://developer.android.com/topic/performance/power/power-details

Android App Standby Buckets https://developer.android.com/topic/performance/power/power-details

How many delayed Notifications? 94.92% 94.34% 64.72% 62.72% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% STANDBY_BUCKET_ACTIVE STANDBY_BUCKET_W ORKING_SET STANDBY_BUCKET_FREQUENT STANDBY_BUCKET_RARE Rate of Notifications reached to app within 1 min

How many delayed Notifications? 96.00% 94.05% 92.32% 88.60% 97.68% 96.21% 94.48% 89.03% 99.36% 99.41% 99.12% 98.46% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% JP TW TH ID Rate of Notifications reached to app within 1 min Android Android (Active & Working Set only) iOS

How to check App Standby Buckets - UsageStatsManager.getAppStandbyBucket() - RemoteMessage.getPriority() - RemoteMessage.getOriginalPriority() - Stopped sending unnecessary FCM to Android device, reduced 17% of delayed notifications

Other suspected points of Android - Background Restriction - Do Not Disturb mode

- ActivityManager.isBackgroundRestricted() Background Restriction https://developer.android.com/topic/performance/power/power-details

- ActivityManager.isBackgroundRestricted() Background Restriction https://developer.android.com/topic/performance/power/power-details

Background Restriction - 0.27% of session, 1.88% of Notification is BackgroundRestriction enabled. 92.27% 84.23% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% Background Restriction=false Background Restriction=true Rate of Notification reached to app within 1 min

Background Restriction - Not much impact due to background restriction - Some manufacturers automatically turn on the restriction for unused apps (?) 94.50% 94.71% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% Background Restriction=false Background Restriction=true Rate of Notification reached to app within 1 min (App Standby Bucket Active or Frequent)

Do not Disturb mode - 0.19% of Notifications affected by DnD - NotificationManager .getNotificationPolicy() .suppressedVisualEffects

Wrap up of Logging / Data Analysis Android - Pay attention to spec of Android App Standby Buckets, minimize number of FCM calls with high priority setting on server side. - Background Restriction, Do not Disturb mode do not have much of an impact (for our case at least) iOS - Utilized Notification Service Extension to collect logs - Proven by data; Relatively fewer Notification delays compared to Android

Dogfooding

- Fitting in to routine tasks of development team - Use as attendance tool in app development team for WFH Dogfooding

Notification system overview LINE User LINE Server Notification Service Extension LINE Official Account App Chat with an LINE Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server

Notification system overview LINE User LINE Server Notification Service Extension LINE Official Account App Chat with an LINE Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server

Notification system overview LINE User LINE Server Notification Service Extension LINE Official Account App Chat with an Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server

Android Notification quota 2021-09-22 15:00:46.533 1390-4456/? E/NotificationService: Package has already posted or enqueued 25 notifications. Not showing more. package=com.linecorp.lineoa services/core/java/com/android/server/notification/NotificationManagerService.java

Android Notification quota - Android OS has a Notification quota since Android 2.3.1. - NotificationManager.notify() is ignored and LogCat appears in system side process - No document in Android Developers (?) - Limit changes by Android OS version

Android Notification quota % adb shell dumpsys notification --- AggregatedStats{ key='com.linecorp.lineoa’, numRateViolations=0 --- numAlertViolations=12 numQuotaViolations=29 - Status can be checked by adb shell - No Android API to get the quota status - NotificationManager .getActiveNotifications() can be used to check status

How many pending Notifications? 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Number of pending Notifications on Android

Findings & Next Actions Findings - Proven by data; iOS has relatively fewer Notification delays compared to Android. - Android App Standby Bucket causes Notification delay, need to pay attention. - Android has Notification quota, which can cause missing Notification. Next Actions - Collect logs and analyze how many rate of Notifications reach to app from server - Develop troubleshooting feature on iOS/Android app

Logging / Data Analysis - To prove the problems really happen, to find explanatory variables - To confirm resolution after bug fix release Wrap up Logging / Data Analysis - To proof the problems really happens, try to find out explanatory variables - To confirm resolution after bug fix release - Analyze data by app developers Dogfooding - To frequently monitor features in real use case - Fitting in to routine tasks of development team

Thank you