Crash reporting for large Android teams

As part of this presentation we:
- Review how crash-reporting tools work, by intercepting uncaught exceptions
- Review how the Babylon team managed crash reporting as it scaled from 5 engineers to about 30 engineers sitting in numerous cross-functional teams.
- Review the New Relic mobile SDK and the New Relic Query Language (NRQL)

Transcript

1. Crash reporting for large
   Android teams
   @skaliakoudas
   

   View Slide

2. My name is Sakis Kaliakoudas
   

   View Slide

3. My name is Sakis Kaliakoudas
   I am an Android engineering manager for a team of 25 Android engineers
   

   View Slide

4. My name is Sakis Kaliakoudas
   I am an Android engineering manager for a team of 25 Android engineers
   Currently working at Babylon Health
   

   View Slide

5. My name is Sakis Kaliakoudas
   I am an Android engineering manager for a team of 25 Android engineers
   Currently working at Babylon Health
   On a mission to provide affordable and accessible healthcare to everyone on the
   planet!
   

   View Slide

6. What is a crash?
   

   View Slide

7. What is a crash?
   private fun likelyToCrash() {
   throw NullPointerException()
   }
   

   View Slide

8. What is a crash?
   private fun likelyToCrash() {
   throw NullPointerException()
   }
   

   View Slide

9. How does crash reporting work?
   

   View Slide

10. How does crash reporting work?
    

    View Slide

11. How does crash reporting work?
    

    View Slide

12. How does crash reporting work?
    

    View Slide

13. How does crash reporting work?
    

    View Slide

14. How does crash reporting work?
    

    View Slide

15. How does crash reporting work?
    

    View Slide

16. Android has a default uncaught exception handler
    How does Android terminate an app
    

    View Slide

17. Android has a default uncaught exception handler
    It is called com.android.internal.os.RuntimeInit.KillApplicationHandler
    How does Android terminate an app
    

    View Slide

18. @Override
    public void uncaughtException(Thread t, Throwable e) {
    try {
    // Log the crash
    // if profiling stop profiling
    // Show the crash dialog
    } catch (Throwable t) {
    // more logging
    } finally {
    // Try everything to make sure this process goes away.
    Process.killProcess(Process.myPid());
    System.exit(10);
    }
    How does Android terminate an app
    

    View Slide

19. @Override
    public void uncaughtException(Thread t, Throwable e) {
    try {
    // Log the crash
    // if profiling stop profiling
    // Show the crash dialog
    } catch (Throwable t) {
    // more logging
    } finally {
    // Try everything to make sure this process goes away.
    Process.killProcess(Process.myPid());
    System.exit(10);
    }
    How does Android terminate an app
    

    View Slide

20. @Override
    public void uncaughtException(Thread t, Throwable e) {
    try {
    // Log the crash
    // Stop profiling
    // Show the crash dialog
    } catch (Throwable t) {
    // more logging
    } finally {
    // Try everything to make sure this process goes away.
    Process.killProcess(Process.myPid());
    System.exit(10);
    }
    How does Android terminate an app
    

    View Slide

21. /**
    * Set the default handler invoked when a thread abruptly terminates
    * due to an uncaught exception, and no other handler has been defined
    * for that thread.
    */
    Thread.setDefaultUncaughtExceptionHandler()
    Overriding the default exception handler
    

    View Slide

22. private fun changeUncaughtExceptionHandler() {
    val handler = Thread.UncaughtExceptionHandler { thread, throwable ->
    reportException(throwable)
    }
    Thread.setDefaultUncaughtExceptionHandler(handler)
    }
    Overriding the default exception handler
    

    View Slide

23. private fun changeUncaughtExceptionHandler() {
    val handler = Thread.UncaughtExceptionHandler { thread, throwable ->
    reportException(throwable)
    }
    Thread.setDefaultUncaughtExceptionHandler(handler)
    }
    Overriding the default exception handler
    

    View Slide

24. private fun changeUncaughtExceptionHandler() {
    val handler = Thread.UncaughtExceptionHandler { thread, throwable ->
    reportException(throwable)
    }
    Thread.setDefaultUncaughtExceptionHandler(handler)
    }
    Overriding the default exception handler
    

    View Slide

25. private fun changeUncaughtExceptionHandler() {
    val handler = Thread.UncaughtExceptionHandler { thread, throwable ->
    reportException(throwable)
    }
    Thread.setDefaultUncaughtExceptionHandler(handler)
    }
    Overriding the default exception handler
    

    View Slide

26. private fun changeUncaughtExceptionHandler() {
    val handler = Thread.UncaughtExceptionHandler { thread, throwable ->
    reportException(throwable)
    }
    Thread.setDefaultUncaughtExceptionHandler(handler)
    }
    Not calling the default uncaught exception handler for Android
    Overriding the default exception handler
    

    View Slide

27. private fun changeUncaughtExceptionHandler() {
    val handler = Thread.UncaughtExceptionHandler { thread, throwable ->
    reportException(throwable)
    }
    Thread.setDefaultUncaughtExceptionHandler(handler)
    }
    Not calling the default uncaught exception handler for Android
    Overriding the default exception handler
    Nothing there to terminate our application process – it will get stuck
    

    View Slide

28. private fun changeUncaughtExceptionHandler() {
    val originalHandler = Thread.getDefaultUncaughtExceptionHandler()
    val handler = Thread.UncaughtExceptionHandler { thread, throwable ->
    reportException(throwable)
    }
    Thread.setDefaultUncaughtExceptionHandler(handler)
    }
    Overriding the default exception handler
    

    View Slide

29. private fun changeUncaughtExceptionHandler() {
    val originalHandler = Thread.getDefaultUncaughtExceptionHandler()
    val handler = Thread.UncaughtExceptionHandler { thread, throwable ->
    reportException(throwable)
    originalHandler.uncaughtException(thread, throwable)
    }
    Thread.setDefaultUncaughtExceptionHandler(handler)
    }
    Overriding the default exception handler
    

    View Slide

30. Initially using Fabric Crashlytics
    Babylon – the start
    

    View Slide

31. Initially using Fabric Crashlytics
    Babylon – the start
    

    View Slide

32. Initially using Fabric Crashlytics
    Babylon – the start
    

    View Slide

33. Initially using Fabric Crashlytics
    That kind of made sense
    Team was pretty small
    Product was relatively small
    The whole company was structured around departments
    Babylon – the start
    

    View Slide

34. Initially using Fabric Crashlytics
    That kind of made sense
    Team was pretty small
    Product was relatively small
    The whole company was structured around departments
    Every engineer would monitor the crash-rate in Fabric and try to fix the issues
    Babylon – the start
    

    View Slide

35. Initially using Fabric Crashlytics
    That kind of made sense
    Team was pretty small
    Product was relatively small
    The whole company was structured around departments
    Every engineer would monitor the crash-rate in Fabric and try to fix the issues
    We were aiming to have about 99.5% crash-free sessions.
    Babylon – the start
    

    View Slide

36. Babylon – growing
    

    View Slide

37. Team gradually growing
    Babylon – growing
    

    View Slide

38. Team gradually growing
    We moved to the Spotify organizational model – structured around product teams
    Babylon – growing
    

    View Slide

39. Team gradually growing
    We moved to the Spotify organizational model – structured around product teams
    The app crash-rate became less meaningful
    Babylon – growing
    

    View Slide

40. Team gradually growing
    We moved to the Spotify organizational model – structured around product teams
    The app crash-rate became less meaningful
    Engineers didn’t have easy access to crashes from their areas
    Babylon – growing
    

    View Slide

41. Babylon – the dream
    

    View Slide

42. Every Android engineer should be able to see just the crashes for their area
    Babylon – the dream
    

    View Slide

43. Every Android engineer should be able to see just the crashes for their area
    Every product team can have their own crash rate KPI
    Babylon – the dream
    

    View Slide

44. Every Android engineer should be able to see just the crashes for their area
    Every product team can have their own crash rate KPI
    Every product team can get alerted for the crashes in their area
    Babylon – the dream
    

    View Slide

45. Every Android engineer should be able to see just the crashes for their area
    Every product team can have their own crash rate KPI
    Every product team can get alerted for the crashes in their area
    As many breadcrumbs as possible
    Babylon – the dream
    

    View Slide

46. What is a breadcrumb
    

    View Slide

47. A breadcrumb is an event that helps create a digital trail that
    allows engineers to diagnose a problem more easily
    What is a breadcrumb
    

    View Slide

48. What is a breadcrumb
    

    View Slide

49. What is a breadcrumb
    

    View Slide

50. What is a breadcrumb
    

    View Slide

51. What is a breadcrumb
    

    View Slide

52. What is a breadcrumb
    

    View Slide

53. What is a breadcrumb
    

    View Slide

54. Investigating other crash reporting tools
    

    View Slide

55. Main requirement was around having flexibility with the crash data
    Investigating other crash reporting tools
    

    View Slide

56. Main requirement was around having flexibility with the crash data
    We decided to do a Proof Of Concept (POC) for:
    Firebase with Big Query support
    New Relic
    Investigating other crash reporting tools
    

    View Slide

57. POCs – Firebase with big query
    

    View Slide

58. POCs – Firebase with big query
    Ability to export your crash data into big query
    

    View Slide

59. POCs – Firebase with big query
    Ability to export your crash data into big query
    We tried it out during a hackathon
    

    View Slide

60. POCs – Firebase with big query
    Ability to export your crash data into big query
    We tried it out during a hackathon
    Looked pretty promising!
    

    View Slide

61. POCs – Firebase with big query
    Ability to export your crash data into big query
    We tried it out during a hackathon
    Looked pretty promising!
    While promising, it would involve a lot of effort
    

    View Slide

62. POCs – Firebase with big query
    Ability to export your crash data into big query
    We tried it out during a hackathon
    Looked pretty promising!
    While promising, it would involve a lot of effort
    We decided not to proceed further with this
    

    View Slide

63. POCs – New Relic
    

    View Slide

64. Looking at what the tool offered, it looked like a good candidate
    POCs – New Relic
    

    View Slide

65. Looking at what the tool offered, it looked like a good candidate
    We decided to spend a bit of time to set it up in the codebase
    POCs – New Relic
    

    View Slide

66. Looking at what the tool offered, it looked like a good candidate
    We decided to spend a bit of time to set it up in the codebase
    We reached out to New Relic to start a trial
    POCs – New Relic
    

    View Slide

67. POCs – New Relic
    

    View Slide

68. Send more breadcrumbs
    POCs – New Relic
    

    View Slide

69. Send more breadcrumbs
    Analytic events
    POCs – New Relic
    

    View Slide

70. Send more breadcrumbs
    Analytic events
    Page views
    POCs – New Relic
    

    View Slide

71. Send more breadcrumbs
    Analytic events
    Page views
    Product team names
    POCs – New Relic
    

    View Slide

72. Send more breadcrumbs
    Analytic events
    Page views
    Product team names
    Create some dashboards, with a focus on metrics around product team crash rates
    POCs – New Relic
    

    View Slide

73. Send more breadcrumbs
    Analytic events
    Page views
    Product team names
    Create some dashboards, with a focus on metrics around product team crash rates
    Assess New Relic alerts
    POCs – New Relic
    

    View Slide

74. Sending analytic events
    

    View Slide

75. This was pretty straightforward because of the layers of abstraction in place
    Sending analytic events
    

    View Slide

76. This was pretty straightforward because of the layers of abstraction in place
    All analytic frameworks are wrapped
    Sending analytic events
    

    View Slide

77. This was pretty straightforward because of the layers of abstraction in place
    All analytic frameworks are wrapped
    All wrappers implement the same interface, TrackingGateway
    Sending analytic events
    

    View Slide

78. This was pretty straightforward because of the layers of abstraction in place
    All analytic frameworks are wrapped
    All wrappers implement the same interface, TrackingGateway
    Each wrapper class is added to a java.util.Set through Dagger
    Sending analytic events
    

    View Slide

79. This was pretty straightforward because of the layers of abstraction in place
    All analytic frameworks are wrapped
    All wrappers implement the same interface, TrackingGateway
    Each wrapper class is added to a java.util.Set through Dagger
    That Set is injected into a Usecase that deals with forwarding events to all trackers
    Sending analytic events
    

    View Slide

80. Sending analytic events
    interface TrackingGateway {
    fun track(action: Action)
    }
    

    View Slide

81. Sending analytic events
    interface TrackingGateway {
    fun track(action: Action)
    }
    class NewRelicTrackingGateway : TrackingGateway {
    override fun track(action: Action) {
    NewRelic.recordBreadcrumb(action.name, action.data)
    }
    }
    

    View Slide

82. @Module
    internal abstract class AnalyticsModule {
    @Binds
    @IntoSet
    abstract fun bind(tracker: NewRelicTrackingGateway): TrackingGateway
    }
    Sending analytic events
    

    View Slide

83. @Module
    internal abstract class AnalyticsModule {
    @Binds
    @IntoSet
    abstract fun bind(tracker: NewRelicTrackingGateway): TrackingGateway
    }
    Sending analytic events
    

    View Slide

84. Sending analytic events
    

    View Slide

85. Sending analytic events
    

    View Slide

86. Sending analytic events
    

    View Slide

87. Sending analytic events
    

    View Slide

88. Sending page views
    

    View Slide

89. Page views are mostly tracked automatically
    Sending page views
    

    View Slide

90. Page views are mostly tracked automatically
    Main implementation mechanism uses the Application.ActivityLifecycleCallbacks
    Sending page views
    

    View Slide

91. Page views are mostly tracked automatically
    Main implementation mechanism uses the Application.ActivityLifecycleCallbacks
    Mechanism for sending the page views to New Relic is similar to the analytic events
    Sending page views
    

    View Slide

92. interface TrackingGateway {
    fun track(action: Action)
    }
    Sending page views
    

    View Slide

93. interface TrackingGateway {
    fun track(action: Action)
    fun trackScreenView(event: ScreenViewTrackingEvent)
    }
    Sending page views
    

    View Slide

94. interface TrackingGateway {
    fun track(action: Action)
    fun trackScreenView(event: ScreenViewTrackingEvent)
    }
    Sending page views
    data class ScreenViewTrackingEvent(
    val name: String,
    val screen: Any
    )
    

    View Slide

95. class NewRelicTrackingGateway : TrackingGateway {
    override fun track(action: Action) {
    NewRelic.recordBreadcrumb(action.name, action.data)
    }
    override fun trackScreenView(event: ScreenViewTrackingEvent) {
    val breadcrumbName = "ScreenView: '${event.name}'"
    NewRelic.recordBreadcrumb(breadcrumbName)
    }
    }
    Sending page views
    

    View Slide

96. Sending product team names
    

    View Slide

97. Sending product team names
    The idea:
    

    View Slide

98. Sending product team names
    The idea:
    Associate in the codebase each screen with the product team that owns it
    

    View Slide

99. Sending product team names
    The idea:
    Associate in the codebase each screen with the product team that owns it
    As users navigate in the app, send that team name for each screen to New Relic
    

    View Slide

100. Sending product team names
     The idea:
     Associate in the codebase each screen with the product team that owns it
     As users navigate in the app, send that team name for each screen to New Relic
     When a crash occurs, use New Relic to associate the crash with the last team name
     reported.
     

     View Slide

101. Sending product team names
     

     View Slide

102. Sending product team names
     Book appointment screen Monitor screen Home screen
     

     View Slide

103. Sending product team names
     Book appointment screen
     Product team: Appointments
     Monitor screen
     Product team: Monitor
     Home screen
     Product team: Core experience
     

     View Slide

104. Attributes
     Team: no value
     New Relic backend
     Sending product team names
     

     View Slide

105. New Relic backend
     Sending product team names
     NewRelic.setAttribute(“Team", “Appointments”)
     Attributes
     Team: no value
     

     View Slide

106. New Relic backend
     Sending product team names
     NewRelic.setAttribute(“Team", “Appointments”)
     Attributes
     Team: Appointments
     

     View Slide

107. New Relic backend
     Sending product team names
     NewRelic.setAttribute(“Team", “Monitor”)
     Attributes
     Team: Appointments
     

     View Slide

108. New Relic backend
     Sending product team names
     NewRelic.setAttribute(“Team", “Monitor”)
     Attributes
     Team: Monitor
     

     View Slide

109. New Relic backend
     NewRelic.setAttribute(“Team", “Core Experience”)
     Sending product team names
     Attributes
     Team: Monitor
     

     View Slide

110. New Relic backend
     NewRelic.setAttribute(“Team", “Core Experience”)
     Sending product team names
     Attributes
     Team: Core Experience
     

     View Slide

111. New Relic backend
     Sending product team names
     NewRelic.setAttribute(“Team", “Core Experience”)
     Attributes
     Team: Core Experience
     

     View Slide

112. Connecting screens with teams
     

     View Slide

113. Our app is one activity per screen at the moment
     Connecting screens with teams
     

     View Slide

114. Our app is one activity per screen at the moment
     Based on this we decided to add an annotation on every activity
     Connecting screens with teams
     

     View Slide

115. @Retention(AnnotationRetention.RUNTIME)
     @Target(AnnotationTarget.CLASS)
     annotation class OwnedByTeams(val teams: Array)
     Connecting screens with teams
     

     View Slide

116. @Retention(AnnotationRetention.RUNTIME)
     @Target(AnnotationTarget.CLASS)
     annotation class OwnedByTeams(val teams: Array)
     enum class Team(val teamName: String) {
     TRIAGE("Triage"),
     CHAT_PLATFORM("Chat platform"),
     APPOINTMENTS("Appointments"),
     MONITOR("Monitor"),
     HEALTHCHECK("Healthcheck"),
     MAPLE("Maple"),
     PAYMENTS_AND_ELIGIBILITY("Payments and Eligibility")
     }
     Connecting screens with teams
     

     View Slide

117. @OwnedByTeams (teams = [Team.APPOINTMENTS])
     class AppointmentDetailsActivity : AppCompatActivity() {
     ...
     }
     Connecting screens with teams
     

     View Slide

118. @OwnedByTeams (teams = [Team.APPOINTMENTS, Team.ANOTHER_TEAM])
     class AppointmentDetailsActivity : AppCompatActivity() {
     ...
     }
     Connecting screens with teams
     

     View Slide

119. class NewRelicTrackingGateway : TrackingGateway {
     override fun track(action: Action) {
     NewRelic.recordBreadcrumb(action.name, action.data)
     }
     override fun trackScreenView(event: ScreenViewTrackingEvent) {
     val breadcrumbName = "ScreenView: '${event.name}'"
     NewRelic.recordBreadcrumb(breadcrumbName)
     }
     }
     Sending product team names
     

     View Slide

120. override fun trackScreenView(event: ScreenViewTrackingEvent) {
     val breadcrumbName = "ScreenView: '${event.name}'"
     NewRelic.recordBreadcrumb(breadcrumbName)
     }
     }
     Sending product team names
     

     View Slide

121. override fun trackScreenView(event: ScreenViewTrackingEvent) {
     val breadcrumbName = "ScreenView: '${event.name}’”
     NewRelic.recordBreadcrumb(breadcrumbName)
     if (event.screen is Activity) {
     event.screen::class.java.getAnnotation(OwnedByTeams::class.java)?.let {
     NewRelic.setAttribute(“Team",
     it.teams.joinToString { team -> team.teamName })
     }
     }
     }
     Sending product team names
     

     View Slide

122. Enforcing the @OwnedByTeams
     

     View Slide

123. Using a tool called Arch Unit (https://www.archunit.org/)
     Enforcing the @OwnedByTeams
     

     View Slide

124. Using a tool called Arch Unit (https://www.archunit.org/)
     Allows to write unit tests for your architecture with a nice API
     Enforcing the @OwnedByTeams
     

     View Slide

125. Using a tool called Arch Unit (https://www.archunit.org/)
     Allows to write unit tests for your architecture with a nice API
     Enforcing the @OwnedByTeams
     

     View Slide

126. Enforcing the @OwnedByTeams
     @Test
     fun `all activities should be annotated with OwnedByTeams annotation`() {
     val classes = ClassFileImporter().importPackages("com.babylon")
     val classesToCheck = classes().that().areAssignableTo(AppCompatActivity::class.java)
     classesToCheck.should().beAnnotatedWith(OwnedByTeams::class.java)
     .because(“You should always assign an owner to a screen")
     .check(classes)
     }
     

     View Slide

127. Enforcing the @OwnedByTeams
     @Test
     fun `all activities should be annotated with OwnedByTeams annotation`() {
     val classes = ClassFileImporter().importPackages("com.babylon")
     val classesToCheck = classes().that().areAssignableTo(AppCompatActivity::class.java)
     classesToCheck.should().beAnnotatedWith(OwnedByTeams::class.java)
     .because(“You should always assign an owner to a screen")
     .check(classes)
     }
     

     View Slide

128. Enforcing the @OwnedByTeams
     @Test
     fun `all activities should be annotated with OwnedByTeams annotation`() {
     val classes = ClassFileImporter().importPackages("com.babylon")
     val classesToCheck = classes().that().areAssignableTo(AppCompatActivity::class.java)
     classesToCheck.should().beAnnotatedWith(OwnedByTeams::class.java)
     .because(“You should always assign an owner to a screen")
     .check(classes)
     }
     

     View Slide

129. Enforcing the @OwnedByTeams
     @Test
     fun `all activities should be annotated with OwnedByTeams annotation`() {
     val classes = ClassFileImporter().importPackages("com.babylon")
     val classesToCheck = classes().that().areAssignableTo(AppCompatActivity::class.java)
     classesToCheck.should().beAnnotatedWith(OwnedByTeams::class.java)
     .because(“You should always assign an owner to a screen")
     .check(classes)
     }
     

     View Slide

130. Getting crash-rates per team
     

     View Slide

131. At this point we had all the relevant data in New Relic
     Getting crash-rates per team
     

     View Slide

132. At this point we had all the relevant data in New Relic
     We just had to create queries
     Getting crash-rates per team
     

     View Slide

133. At this point we had all the relevant data in New Relic
     We just had to create queries
     New Relic has its own query language called NRQL
     Getting crash-rates per team
     

     View Slide

134. At this point we had all the relevant data in New Relic
     We just had to create queries
     New Relic has its own query language called NRQL
     Similar to SQL, but not as powerful
     Getting crash-rates per team
     

     View Slide

135. MobileSession
     App version
     Country
     Device model
     OS version
     Session duration
     MobileCrash
     Exception
     Available size on disk
     Orientation
     Architecture
     MobileHandledException
     (non-fatal exceptions)
     Data captured by New Relic
     MobileRequest
     Bytes sent & received
     Connection type (2G, 3G etc.)
     Request URL
     Response time
     REST status code
     MobileRequestError
     Similar to “MobileRequest”
     Error type (cellular issue or REST error)
     Response body
     MobileBreadcrumb
     

     View Slide

136. NRQL examples
     SELECT *
     FROM MobileCrash
     SINCE last week
     

     View Slide

137. SELECT count(*)
     FROM MobileRequest
     WHERE requestPath LIKE '%patient%’
     SINCE last week
     TIMESERIES 5 hours
     NRQL examples
     

     View Slide

138. SELECT average(bytesReceived)
     FROM MobileRequest
     WHERE requestMethod = 'PATCH’
     AND requestPath LIKE '%appointment%’
     SINCE last month
     NRQL examples
     

     View Slide

139. SELECT (filter(count(sessionId),
     WHERE category ='Crash' AND Team = 'Appointments' ) / count(sessionId)) * 100
     FROM MobileCrash, MobileSession
     SINCE 1 week ago
     Crash-rates per product team
     

     View Slide

140. SELECT (filter(count(sessionId),
     WHERE category ='Crash' AND Team = 'Appointments' ) / count(sessionId)) * 100
     FROM MobileCrash, MobileSession
     SINCE 1 week ago
     Crash-rates per product team
     

     View Slide

141. SELECT (filter(count(sessionId),
     WHERE category ='Crash' AND Team = 'Appointments' ) / count(sessionId)) * 100
     FROM MobileCrash, MobileSession
     SINCE 1 week ago
     Crash-rates per product team
     

     View Slide

142. SELECT (filter(count(sessionId),
     WHERE category ='Crash' AND Team = 'Appointments' ) / count(sessionId)) * 100
     FROM MobileCrash, MobileSession
     SINCE 1 week ago
     Crash-rates per product team
     

     View Slide

143. SELECT (filter(count(sessionId),
     WHERE category ='Crash' AND Team = 'Appointments' ) / count(sessionId)) * 100
     FROM MobileCrash, MobileSession
     SINCE 1 week ago
     Crash-rates per product team
     

     View Slide

144. SELECT (filter(count(sessionId),
     WHERE category ='Crash' AND Team = 'Appointments' ) / count(sessionId)) * 100
     FROM MobileCrash, MobileSession
     SINCE 1 week ago
     Crash-rates per product team
     

     View Slide

145. Crash-rates per product team
     

     View Slide

146. Breadcrumbs in New Relic
     HTTP Response: 200 to request in event 92
     612 ms
     https://prod.babylonpartners.com/api/v2/video_sessions/1233
     eventType: MobileCrash
     ScreenView ‘VideoConsultationsActivity’
     Incoming videocall/notification_accepted
     

     View Slide

147. Breadcrumbs in New Relic
     HTTP Response: 200 to request in event 92
     612 ms
     https://prod.babylonpartners.com/api/v2/video_sessions/1233
     eventType: MobileCrash
     ScreenView ‘VideoConsultationsActivity’
     Incoming videocall/notification_accepted
     

     View Slide

148. Breadcrumbs in New Relic
     HTTP Response: 200 to request in event 92
     612 ms
     https://prod.babylonpartners.com/api/v2/video_sessions/1233
     eventType: MobileCrash
     ScreenView ‘VideoConsultationsActivity’
     Incoming videocall/notification_accepted
     

     View Slide

149. Breadcrumbs in New Relic
     HTTP Response: 200 to request in event 92
     612 ms
     https://prod.babylonpartners.com/api/v2/video_sessions/1233
     eventType: MobileCrash
     ScreenView ‘VideoConsultationsActivity’
     Incoming videocall/notification_accepted
     

     View Slide

150. Breadcrumbs in New Relic
     HTTP Response: 200 to request in event 92
     612 ms
     https://prod.babylonpartners.com/api/v2/video_sessions/1233
     eventType: MobileCrash
     ScreenView ‘VideoConsultationsActivity’
     Incoming videocall/notification_accepted
     

     View Slide

151. Alerts
     

     View Slide

152. Alerts
     New Relic provides a very flexible alerting framework that can integrate with many
     tools
     

     View Slide

153. Going for New Relic
     

     View Slide

154. Going for New Relic
     We went for it
     

     View Slide

155. Bonus: Ownership in debug builds
     

     View Slide

156. Bonus: Ownership in debug builds
     

     View Slide

157. Bonus: Ownership in other places
     

     View Slide

158. Bonus: Ownership in other places
     Feature flags
     

     View Slide

159. Bonus: Ownership in other places
     Feature flags
     Notifications on slack grouped by teams
     

     View Slide

160. Bonus: Ownership in other places
     Feature flags
     Notifications on slack grouped by teams
     Some UI Tests
     

     View Slide

161. The future
     

     View Slide

162. KPIs per team
     The future
     

     View Slide

163. KPIs per team
     Using New Relic alerts
     The future
     

     View Slide

164. KPIs per team
     Using New Relic alerts
     Using this for iOS as well
     The future
     

     View Slide

165. KPIs per team
     Using New Relic alerts
     Using this for iOS as well
     Defining higher level organizational structures of ownership
     The future
     

     View Slide

166. Thanks!
     

     View Slide

167. With 1 activity and multiple fragments the only change would be moving the
     ownership annotation from each activity to each fragment
     Q: How would the ownership mechanism
     work with a single activity app?
     

     View Slide

168. In the process of modularizing the app. This mechanism can stay as is even with
     feature modules
     Q: Is your app modularized, and how
     would that affect this mechanism?
     

     View Slide

169. Q: What’s a good crash-free rate?
     

     View Slide

170. There’s no one universal answer
     Q: What’s a good crash-free rate?
     

     View Slide

171. There’s no one universal answer
     NASA would probably not tolerate a single crash!
     Q: What’s a good crash-free rate?
     

     View Slide

172. There’s no one universal answer.
     NASA would probably not tolerate a single crash!
     For the Babylon Android team, the number started from about 99.5%
     Q: What’s a good crash-free rate?
     

     View Slide

173. There’s no one universal answer.
     NASA would probably not tolerate a single crash!
     For the Babylon Android team, the number started from about 99.5%
     Currently sitting at about 99.9%
     Q: What’s a good crash-free rate?
     

     View Slide

174. There’s no one universal answer.
     NASA would probably not tolerate a single crash!
     For the Babylon Android team, the number started from about 99.5%
     Currently sitting at about 99.9%
     Seems like it is easier to increase it with our MVI architecture
     Q: What’s a good crash-free rate?
     

     View Slide

175. https://github.com/babylonhealth/orbit-mvi
     Q: What’s a good crash-free rate?
     

     View Slide

176. Other questions?
     

     View Slide