$30 off During Our Annual Pro Sale. View Details »

Aim at 100% Push Notification - Knowledge Obtained by Data Analysis

Aim at 100% Push Notification - Knowledge Obtained by Data Analysis

LINE DEVDAY 2021
PRO

November 11, 2021
Tweet

More Decks by LINE DEVDAY 2021

Other Decks in Technology

Transcript

  1. None
  2. Agenda - Introduction - Logging / Data Analysis - Dogfooding

    - Wrap up
  3. - Hisato Shoji - LINE B2B app dev team manager

    - iOS/Android app of LINE Official Account - LINE Official Account related features in LINE app Self-introduction
  4. - For people who working on iOS/Android development - Why

    & how app dev team takes data and analyzes it, while performing dogfooding activities About this session
  5. - B2B2C service to support communication between LINE users and

    business owners - Over 10M accounts worldwide LINE Official Account
  6. LINE Official Account - iOS / Android / PC browser

    / Web API support - Service available globally, including Japan, Thailand, Taiwan, Indonesia etc
  7. - Chat with LINE users - VoIP with LINE users

    - Issuing coupons - Posting to Timeline - etc iOS / Android app
  8. - Notification delivery delay - Notification delivery failure - Pointed

    out via App Store / Google Play / CS / user interviews Major user feedback
  9. But…

  10. Logging / Data Analysis

  11. Notification system overview LINE User LINE Server Notification Service Extension

    LINE Official Account App Chat with an LINE Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server
  12. Logging - Using inhouse logging system, for easy comparison with

    server side log etc - iOS: Implemented UNNotificationServiceExtension to download and show attached images of chats, in addition to collecting log for analysis - Android: Implemented FirebaseMessagingService and collect log normally - Note: Text in user message is not collected.
  13. Why app developers analyze data? - Company has a data

    scientist team - Need knowledge about how APNS, Notification Service Extension, FCM, Android Doze, or Background Restriction works - For faster try-and-error cycle - App developers were interested in data analysis J
  14. How many delayed Notifications? 92.48% 99.08% 80.00% 82.00% 84.00% 86.00%

    88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% ANDROID IOS Rate of Notifications reached to app within 1 min
  15. How many delayed Notifications? 96.00% 94.05% 92.32% 88.60% 99.36% 99.41%

    99.12% 98.46% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% JP TW TH ID Rate of Notifications reached to app within 1 min Android iOS
  16. Average time consumed by each part of Android LINE Server

    -> FCM Server* 43.65 ms FCM Server* -> FirebaseMessageingService 70065.36 ms JSON parsing + Notification API call 35.62 ms Which part takes the most time? * RemoteMessage.getSentTime()
  17. Android Doze mode https://developer.android.com/topic/performance/power/power-details

  18. Android App Standby Buckets https://developer.android.com/topic/performance/power/power-details

  19. How many delayed Notifications? 94.92% 94.34% 64.72% 62.72% 40.00% 50.00%

    60.00% 70.00% 80.00% 90.00% 100.00% STANDBY_BUCKET_ACTIVE STANDBY_BUCKET_W ORKING_SET STANDBY_BUCKET_FREQUENT STANDBY_BUCKET_RARE Rate of Notifications reached to app within 1 min
  20. How many delayed Notifications? 96.00% 94.05% 92.32% 88.60% 97.68% 96.21%

    94.48% 89.03% 99.36% 99.41% 99.12% 98.46% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% JP TW TH ID Rate of Notifications reached to app within 1 min Android Android (Active & Working Set only) iOS
  21. How to check App Standby Buckets - UsageStatsManager.getAppStandbyBucket() - RemoteMessage.getPriority()

    - RemoteMessage.getOriginalPriority() - Stopped sending unnecessary FCM to Android device, reduced 17% of delayed notifications
  22. Other suspected points of Android - Background Restriction - Do

    Not Disturb mode
  23. - ActivityManager.isBackgroundRestricted() Background Restriction https://developer.android.com/topic/performance/power/power-details

  24. - ActivityManager.isBackgroundRestricted() Background Restriction https://developer.android.com/topic/performance/power/power-details

  25. Background Restriction - 0.27% of session, 1.88% of Notification is

    BackgroundRestriction enabled. 92.27% 84.23% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% Background Restriction=false Background Restriction=true Rate of Notification reached to app within 1 min
  26. Background Restriction - Not much impact due to background restriction

    - Some manufacturers automatically turn on the restriction for unused apps (?) 94.50% 94.71% 80.00% 82.00% 84.00% 86.00% 88.00% 90.00% 92.00% 94.00% 96.00% 98.00% 100.00% Background Restriction=false Background Restriction=true Rate of Notification reached to app within 1 min (App Standby Bucket Active or Frequent)
  27. Do not Disturb mode - 0.19% of Notifications affected by

    DnD - NotificationManager .getNotificationPolicy() .suppressedVisualEffects
  28. Wrap up of Logging / Data Analysis Android - Pay

    attention to spec of Android App Standby Buckets, minimize number of FCM calls with high priority setting on server side. - Background Restriction, Do not Disturb mode do not have much of an impact (for our case at least) iOS - Utilized Notification Service Extension to collect logs - Proven by data; Relatively fewer Notification delays compared to Android
  29. Dogfooding

  30. - Fitting in to routine tasks of development team -

    Use as attendance tool in app development team for WFH Dogfooding
  31. Notification system overview LINE User LINE Server Notification Service Extension

    LINE Official Account App Chat with an LINE Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server
  32. Notification system overview LINE User LINE Server Notification Service Extension

    LINE Official Account App Chat with an LINE Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server
  33. Notification system overview LINE User LINE Server Notification Service Extension

    LINE Official Account App Chat with an Official Account LINE Official Account App PC Browser LINE Official Account Server Google FCM Server OS GUI iPhone Firebase Messaging Service Android OS GUI Apple APNS Server
  34. Android Notification quota 2021-09-22 15:00:46.533 1390-4456/? E/NotificationService: Package has already

    posted or enqueued 25 notifications. Not showing more. package=com.linecorp.lineoa services/core/java/com/android/server/notification/NotificationManagerService.java
  35. Android Notification quota - Android OS has a Notification quota

    since Android 2.3.1. - NotificationManager.notify() is ignored and LogCat appears in system side process - No document in Android Developers (?) - Limit changes by Android OS version
  36. Android Notification quota % adb shell dumpsys notification --- AggregatedStats{

    key='com.linecorp.lineoa’, numRateViolations=0 --- numAlertViolations=12 numQuotaViolations=29 - Status can be checked by adb shell - No Android API to get the quota status - NotificationManager .getActiveNotifications() can be used to check status
  37. How many pending Notifications? 0 10000 20000 30000 40000 50000

    60000 70000 80000 90000 100000 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Number of pending Notifications on Android
  38. Findings & Next Actions Findings - Proven by data; iOS

    has relatively fewer Notification delays compared to Android. - Android App Standby Bucket causes Notification delay, need to pay attention. - Android has Notification quota, which can cause missing Notification. Next Actions - Collect logs and analyze how many rate of Notifications reach to app from server - Develop troubleshooting feature on iOS/Android app
  39. Logging / Data Analysis - To prove the problems really

    happen, to find explanatory variables - To confirm resolution after bug fix release Wrap up Logging / Data Analysis - To proof the problems really happens, try to find out explanatory variables - To confirm resolution after bug fix release - Analyze data by app developers Dogfooding - To frequently monitor features in real use case - Fitting in to routine tasks of development team
  40. Thank you