Rewind and Resolve: A deep dive into building Session Replay for Android

Understanding and debugging production issues can feel like solving a puzzle with missing pieces. We will dive deep into the journey of building a session replay feature for the Sentry Android SDK that helps developers capture those elusive bugs, respects user privacy, and maintains low overhead.

In this talk, we'll explore the technical aspects, including:

- Taking screenshots of a Window
- Redacting the screenshots to ensure no sensitive user data is leaked
- Encoding the screenshots into a video
- Collecting supporting data like breadcrumbs, network requests, logs, touches to improve debuggability
- Maintaining low performance overhead while implementing all of the above.

We'll also demonstrate how our tool empowers Android engineers to gain actionable insights when solving errors, crashes, and ANRs.

Whether you're an Android developer looking to improve your debugging toolkit or someone interested in the technical guts of the Android OS, the session will cover it all. Join us for a glimpse into the future of mobile debugging – it's a replay you won't want to miss!

Transcript

1. @romtsn Rewind and Resolve: A deep dive into building Session Replay
2. What is Sentry? getsentry/sentry @getsentry sentry.io • Application Monitoring in Production • Developer Tool by devs for devs • Open-Source • Free* • Self-host • Generous free tier
3. What is Sentry?
5. What are we building?
7. What are we building? • A video-like reproduction of user activity • Connected to errors/traces • High fi delity/accuracy • Low performance overhead • Possibility to strip/mask PII and privacy-concerned elements
8. A note on privacy Masking
9. Session Replay approaches
10. Session Replay approaches • Raw video recording/replay • View Hierarchy capture + transform into html elements • Canvas Draw commands capture + redraw on a web HTML5 canvas • Periodic screenshots capture + stitch them together into a video Raw video recording View Hierarchy Canvas Draw commands Periodic screenshots capture
11. Session Replay approaches Fidelity/ Accuracy Network throughput CPU overhead Memory overhead 99 % 10-20 MB 15-20 % 35-45 MB 60 % 10-20 kB 7-10 % 10-15 MB 90 % 10-15 kB ~5 % 5-7 MB 99 % 35-55 kB 3-5 % 20-25 MB Raw video recording View Hierarchy Canvas Draw commands Periodic screenshots
12. Session Replay approaches Fidelity/ Accuracy Network throughput CPU overhead Memory overhead 99 % 10-20 MB 15-20 % 35-45 MB 60 % 10-20 kB 7-10 % 10-15 MB 90 % 10-15 kB ~5 % 5-7 MB 99 % 35-55 kB 3-5 % 20-25 MB Raw video recording View Hierarchy Canvas Draw commands Periodic screenshots
13. Let’s build some Replay!
14. Key Pieces ReplayVideo • Actual video bytes that we created from periodically taking screenshots ReplayVideo ReplayRecording • Video metadata • Breadcrumbs • Gestures • Network requests • Logs ReplayEvent • Tags • Error/Trace connection • Internal metadata Backend
15. Key Pieces ReplayVideo • Actual video bytes that we created from periodically taking screenshots ReplayVideo ReplayRecording • Video metadata • Breadcrumbs • Gestures • Network requests • Logs ReplayEvent • Tags • Error/Trace connection • Internal metadata Backend
16. ReplayVideo - algorithm • Choose frame rate (screenshots per second) - we chose 1 fps • Take a screenshot of the entire screen at that frame rate • Capture View Hierarchy of the same frame to mask sensitive data • Apply masking • Store screenshot to disk • Create a video segment from the stored screenshots - we chose 5 seconds
17. Taking a screenshot
18. data class RecorderConfig( val frameRate: Int ) val recorderConfig = RecorderConfig( frameRate = 1 )
19. data class RecorderConfig( val frameRate: Int ) val recorderConfig = RecorderConfig( frameRate = 1 ) val executor = Executors.newSingleThreadScheduledExecutor("SentryRecorder")
20. data class RecorderConfig( val frameRate: Int ) val recorderConfig = RecorderConfig( frameRate = 1 ) val executor = Executors.newSingleThreadScheduledExecutor("SentryRecorder") executor.scheduleAtFixedRate( task = { capture() }, initialDelay = 0L, period = 1000L / recorderConfig.frameRate, unit = TimeUnit.MILLISECONDS ) fun capture() { / / here goes the capturing logic }
21. ... fun capture() { / / here goes the capturing logic }
22. ... @RequiresApi(26) fun capture() { / / here goes the capturing logic }
23. ... private val mainHandler = Handler(Looper.getMainLooper()) @RequiresApi(26) fun capture() { / / here goes the capturing logic Handler(mainHandler).post { } }
24. ... private val mainHandler = Handler(Looper.getMainLooper()) @RequiresApi(26) fun capture() { / / here goes the capturing logic Handler(mainHandler).post { PixelCopy.request( window = TODO(), bitmap = TODO(), listener = { copyResult: Int -> }, listenerThread = mainHandler ) } }
25. Getting the current window github.com/square/curtains
26. ... private val mainHandler = Handler(Looper.getMainLooper()) private val rootViews = ArrayList<WeakReference<View >> () Curtains.onRootViewsChangedListeners += OnRootViewsChangedListener { root, added -> if (added) { rootViews.add(WeakReference<View>(root)) } else { rootViews.removeAll { it.get() == root } } } @RequiresApi(26) fun capture() { / / here goes the capturing logic . .. }
27. rootViews.removeAll { it.get() == root } } } @RequiresApi(26) fun capture() { / / here goes the capturing logic val root = rootViews.lastOrNull() ?. get() if (root = = null || !root.isShown) { // sanity check return } Handler(mainHandler).post { PixelCopy.request( window = root.phoneWindow, bitmap = TODO(), listener = { copyResult: Int -> }, listenerThread = mainHandler ) } }
28. ?. if.(root = = null || !root.isShown).{ // sanity check return } val bitmap = Bitmap.createBitmap( width = TODO("width"), height = TODO("height"), Bitmap.Config.ARGB_8888 ) Handler(mainHandler).post { PixelCopy.request( window = root.phoneWindow, bitmap = bitmap, listener = { copyResult: Int -> }, listenerThread = mainHandler ) } }
29. ?. if.(root = = null || !root.isShown).{ // sanity check return } / / PixelCopy takes screenshots including system bars, so we have to get the real size here val wm = context.getSystemService(Context.WINDOW_SERVICE) as WindowManager val screenBounds = if (VERSION.SDK_INT > = VERSION_CODES.R) { wm.currentWindowMetrics.bounds } else { val screenBounds = Point() @Suppress("DEPRECATION") wm.defaultDisplay.getRealSize(screenBounds) Rect(0, 0, screenBounds.x, screenBounds.y) } val bitmap = Bitmap.createBitmap( width = screenBounds.width(), height = screenBounds.height(), Bitmap.Config.ARGB_8888 ) Handler(mainHandler).post { PixelCopy.request( window = root.phoneWindow,
30. Bitmap.Config.ARGB_8888 ) Handler(mainHandler).post { PixelCopy.request( window = root.phoneWindow, bitmap = bitmap, listener = { copyResult: Int -> if (copyResult ! = PixelCopy.SUCCESS) { bitmap.recycle() return@request } / / hurray, we have a screenshot! bitmap }, listenerThread = mainHandler ) } }
31. Masking screenshots
32. ImageView TextView Button (TextView) NavigationIcon (ImageView) NavigationTextView (TextView)
33. Masking algorithm • Walk the view tree from its root • If it’s a TextView or ImageView • Check if it’s visible and retrieve its bounds • Retrieve other necessary info for masking • Skip otherwise
34. sealed class ViewHierarchyNode( val shouldMask: Boolean, val bounds: Rect, ) { class GenericViewHierarchyNode( shouldMask: Boolean, bounds: Rect ): ViewHierarchyNode(shouldMask, bounds) class ImageViewHierarchyNode( shouldMask: Boolean, bounds: Rect ): ViewHierarchyNode(shouldMask, bounds) class TextViewHierarchyNode( shouldMask: Boolean, bounds: Rect ): ViewHierarchyNode(shouldMask, bounds) }
35. ): ViewHierarchyNode(shouldMask, bounds) class TextViewHierarchyNode( shouldMask: Boolean, bounds: Rect ): ViewHierarchyNode(shouldMask, bounds) } { companion object { fun fromView(view: View) { val bounds = TODO("bounds") return when (view) { is TextView -> TextViewHierarchyNode(true, bounds) is ImageView -> ImageViewHierarchyNode(true, bounds) else -> GenericViewHierarchyNode(false, null) } } } }
36. class TextViewHierarchyNode( shouldMask: Boolean, bounds: Rect ): ViewHierarchyNode(shouldMask, bounds) } { companion object { fun fromView(view: View) { val bounds = Rect() val isVisible = view.getGlobalVisibleRect(bounds) return when (view) { is TextView -> TextViewHierarchyNode(isVisible, bounds) is ImageView -> ImageViewHierarchyNode(isVisible, bounds) else -> GenericViewHierarchyNode(false, null) } } } }
37. is ImageView -> ImageViewHierarchyNode(isVisible, bounds) else -> GenericViewHierarchyNode(false, null) } } } } fun View.traverse(nodesToMask: MutableList<ViewHierarchyNode>) { val node = ViewHierarchyNode.fromView(this) if (node.shouldMask) { nodesToMask += node } if (this !is ViewGroup || this.childCount = = 0) { return } val childNodes = ArrayList<ViewHierarchyNode>(this.childCount) for (i in 0 until childCount) { val child = getChildAt(i) if (child != null) { child.traverse(nodesToMask) } } }
38. Handler(mainHandler).post { PixelCopy.request( window = root.phoneWindow, bitmap = bitmap, listener = { copyResult: Int -> if (copyResult ! = PixelCopy.SUCCESS) { bitmap.recycle() return@request } / / hurray, we have a screenshot! val nodesToMask = mutableListOf<ViewHierarchyNode>() root.traverse(nodesToMask) }, listenerThread = mainHandler ) } }
39. ! = bitmap.recycle() return@request } / / hurray, we have a screenshot! val nodesToMask = mutableListOf<ViewHierarchyNode>() root.traverse(nodesToMask) / / can switch back to bg thread, main thread stuff is done at this point executor.submit { } }, listenerThread = mainHandler ) } }
40. / / hurray, we have a screenshot! val nodesToMask = mutableListOf<ViewHierarchyNode>() root.traverse(nodesToMask) / / can switch back to bg thread, main thread stuff is done at this point executor.submit { val canvas = Canvas(bitmap) val paint = Paint().apply { setColor(Color.BLACK) } for (node in nodesToMask) { canvas.drawRect(node.bounds, paint) } } }, listenerThread = mainHandler ) } }
41. / / hurray, we have a screenshot! val nodesToMask = mutableListOf<ViewHierarchyNode>() root.traverse(nodesToMask) / / can switch back to bg thread, main thread stuff is done at this point executor.submit { val canvas = Canvas(bitmap) val paint = Paint().apply { setColor(Color.BLACK) } for (node in nodesToMask) { canvas.drawRect(node.bounds, paint) } } }, listenerThread = mainHandler ) } }
42. Wrong Color Text Layout is not considered
43. Getting the text layout
44. Getting the text layout
45. bounds: Rect ): ViewHierarchyNode(shouldMask, bounds) class TextViewHierarchyNode( shouldMask: Boolean, bounds: Rect, layout: Layout ): ViewHierarchyNode(shouldMask, bounds) } { companion object { fun fromView(view: View) { val bounds = Rect() val isVisible = view.getGlobalVisibleRect(bounds) return when (view) { is TextView -> TextViewHierarchyNode(isVisible, bounds, view.layout) is ImageView -> ImageViewHierarchyNode(isVisible, bounds) else -> GenericViewHierarchyNode(false, null) } } } }
46. fun Layout.getVisibleRects(globalBounds: Rect): List<Rect> { val rects = mutableListOf<Rect>() for (i in 0 until lineCount) { val lineStart = getLineStart(i) val lineVisibleEnd = getLineVisibleEnd(i) val lineTop = getLineTop(i) val lineBottom = getLineBottom(i) val rect = Rect() rect.left = globalBounds.left + lineStart rect.right = rect.left + (lineEnd - lineStart) rect.top = globalBounds.top + lineTop rect.bottom = rect.top + (lineBottom - lineTop) rects += rect } return rects } LineStart LineVisibleEnd
47. LineTop LineBottom fun Layout.getVisibleRects(globalBounds: Rect): List<Rect> { val rects = mutableListOf<Rect>() for (i in 0 until lineCount) { val lineStart = getLineStart(i) val lineVisibleEnd = getLineVisibleEnd(i) val lineTop = getLineTop(i) val lineBottom = getLineBottom(i) val rect = Rect() rect.left = globalBounds.left + lineStart rect.right = rect.left + (lineEnd - lineStart) rect.top = globalBounds.top + lineTop rect.bottom = rect.top + (lineBottom - lineTop) rects += rect } return rects }
48. fun Layout.getVisibleRects(globalBounds: Rect): List<Rect> { val rects = mutableListOf<Rect>() for (i in 0 until lineCount) { val lineStart = getLineStart(i) val lineVisibleEnd = getLineVisibleEnd(i) val lineTop = getLineTop(i) val lineBottom = getLineBottom(i) val rect = Rect() rect.left = globalBounds.left + lineStart rect.right = rect.left + (lineEnd - lineStart) rect.top = globalBounds.top + lineTop rect.bottom = rect.top + (lineBottom - lineTop) rects += rect } return rects }
49. fun Layout.getVisibleRects(globalBounds: Rect): List<Rect> { val rects = mutableListOf<Rect>() for (i in 0 until lineCount) { val lineStart = getLineStart(i) val lineVisibleEnd = getLineVisibleEnd(i) val lineTop = getLineTop(i) val lineBottom = getLineBottom(i) val rect = Rect() rect.left = globalBounds.left + lineStart rect.right = rect.left + (lineEnd - lineStart) rect.top = globalBounds.top + lineTop rect.bottom = rect.top + (lineBottom - lineTop) rects += rect } return rects }
50. fun Layout.getVisibleRects(globalBounds: Rect): List<Rect> { val rects = mutableListOf<Rect>() for (i in 0 until lineCount) { val lineStart = getLineStart(i) val lineVisibleEnd = getLineVisibleEnd(i) val lineTop = getLineTop(i) val lineBottom = getLineBottom(i) val rect = Rect() rect.left = globalBounds.left + lineStart rect.right = rect.left + (lineEnd - lineStart) rect.top = globalBounds.top + lineTop rect.bottom = rect.top + (lineBottom - lineTop) rects += rect } return rects }
51. executor.submit { val canvas = Canvas(bitmap) val paint = Paint().apply { setColor(Color.BLACK) } for (node in nodesToMask) { when (node) { is TextViewHierarchyNode -> { val rects = node.layout.getVisibleRects(node.bounds) for (rect in rects) { canvas.drawRect(rect, paint) } } else -> canvas.drawRect(node.bounds, paint) } } } }, listenerThread = mainHandler ) } }
52. Getting the dominant color
53. Getting the dominant color of text
54. setColor(Color.BLACK) } for (node in nodesToMask) { when (node) { is TextViewHierarchyNode -> { val rects = node.layout.getVisibleRects(node.bounds) for (rect in rects) { canvas.drawRect(rect, node.layout.paint) } } else -> canvas.drawRect(node.bounds, paint) } } } }, listenerThread = mainHandler ) } }
55. Getting the dominant color of images 1px 1px
56. Getting the dominant color of images 1px 1px
57. for (node in nodesToMask) { when (node) { is TextViewHierarchyNode -> { val rects = node.layout.getVisibleRects(node.bounds) for (rect in rects) { canvas.drawRect(rect, node.layout.paint) } } is ImageViewHierarchyNode -> { val singlePixelBitmap = Bitmap.createBitmap(1, 1, ARGB_8888) val singlePixelCanvas = Canvas(singlePixelBitmap) canvas.drawRect(node.bounds, paint) } else -> canvas.drawRect(node.bounds, paint) } } } }, listenerThread = mainHandler ) } }
58. Getting the dominant color of images
59. } is ImageViewHierarchyNode -> { val singlePixelBitmap = Bitmap.createBitmap(1, 1, ARGB_8888) val singlePixelCanvas = Canvas(singlePixelBitmap) // draw part of the screenshot (bounds) to a single pixel bitmap singlePixelCanvas.drawBitmap( bitmap = bitmap, src = node.bounds, dst = Rect(0, 0, 1, 1), paint = null ) canvas.drawRect(node.bounds, paint) } else -> canvas.drawRect(node.bounds, paint) } } } }, listenerThread = mainHandler ) } }
60. Getting the dominant color of images
61. // draw part of the screenshot (bounds) to a single pixel bitmap singlePixelCanvas.drawBitmap( bitmap = bitmap, src = node.bounds, dst = Rect(0, 0, 1, 1), paint = null ) // get the dominant color from the single-pixel bitmap paint.setColor(singlePixelBitmap.getPixel(0, 0)) canvas.drawRect(node.bounds, paint) } else -> canvas.drawRect(node.bounds, paint) } } } }, listenerThread = mainHandler ) } }
63. Jetpack Compose Replay • PixelCopy API still works and takes screenshots of Compose screens • Masking is tricky • No View classes, just @Composable functions • A lot of internal or private stu ff that we need • Changes rapidly in a breaking way
64. ViewGroup ImageView TextView Button LayoutNode LayoutNode LayoutNode LayoutNode
65. ViewGroup ImageView TextView Button LayoutNode LayoutNode LayoutNode LayoutNode {isContainer:true} {role: Button, text: “Click”} Semantics Tree {text: “Hello”} {role: Image, contentDescription: “Image”}
66. fun View.traverse(nodesToMask: MutableList<ViewHierarchyNode>) { val node = if (class.java.name.contains("AndroidComposeView")) { ComposeViewHierarchyNode.fromView(this) return } else { ViewHierarchyNode.fromView(this) } if (node.shouldMask) { nodesToMask += node } if (this !is ViewGroup || this.childCount = = 0) { return } val childNodes = ArrayList<ViewHierarchyNode>(this.childCount) for (i in 0 until childCount) { val child = getChildAt(i) if (child != null) {
67. object ComposeViewHierarchyNode { fun fromView(view: View) { val rootNode = (view as? Owner) ?. root ? : return } }
68. /** * Owner implements the connection to the underlying view system. On Android, this connects * to Android [views][android.view.View] and all layout, draw, input, and accessibility is hooked * through them. */ internal interface Owner { /** * The root layout node in the component tree. */ val root: LayoutNode ... ...
69. @Suppress("INVISIBLE_MEMBER", "INVISIBLE_REFERENCE") // sorry internal, but we need you object ComposeViewHierarchyNode { fun fromView(view: View) { val rootNode = (view as? Owner) ?. root ?: return } }
70. @Suppress("INVISIBLE_MEMBER", "INVISIBLE_REFERENCE") // sorry internal, but we need you object ComposeViewHierarchyNode { fun fromView(view: View) { val rootNode = (view as? Owner) ?. root ?: return for (layoutNode in rootNode.children) { } } }
71. @Suppress("INVISIBLE_MEMBER", "INVISIBLE_REFERENCE") // sorry internal, but we need you object ComposeViewHierarchyNode { fun fromView(view: View) { val rootNode = (view as? Owner) ?. root ?: return for (layoutNode in rootNode.children) { val bounds = layoutNode.coordinates.boundsInWindow() } } }
72. @Suppress("INVISIBLE_MEMBER", "INVISIBLE_REFERENCE") // sorry internal, but we need you object ComposeViewHierarchyNode { fun fromView(view: View) { val rootNode = (view as? Owner) ?. root ?: return for (layoutNode in rootNode.children) { val bounds = layoutNode.coordinates.boundsInWindow() val semantics = layoutNode.collapsedSemantics val isVisible = layoutNode.isPlaced } } }
73. @Suppress("INVISIBLE_MEMBER", "INVISIBLE_REFERENCE") // sorry internal, but we need you object ComposeViewHierarchyNode { fun fromView(view: View) { val rootNode = (view as? Owner) ?. root ?: return for (layoutNode in rootNode.children) { val bounds = layoutNode.coordinates.boundsInWindow() val semantics = layoutNode.collapsedSemantics val isVisible = layoutNode.isPlaced return when { semantics.contains(SemanticsProperties.Text) || semantics.contains(SemanticsActions.SetText) -> { } } } } }
74. @Suppress("INVISIBLE_MEMBER", "INVISIBLE_REFERENCE") // sorry internal, but we need you object ComposeViewHierarchyNode { fun fromView(view: View) { val rootNode = (view as? Owner) ?. root ?: return for (layoutNode in rootNode.children) { val bounds = layoutNode.coordinates.boundsInWindow() val semantics = layoutNode.collapsedSemantics val isVisible = layoutNode.isPlaced return when { semantics.contains(SemanticsProperties.Text) || semantics.contains(SemanticsActions.SetText) -> { val textLayoutResults = mutableListOf<TextLayoutResult>() semantics.getOrNull(SemanticsActions.GetTextLayoutResult) ?. action ?. invoke(textLayoutResults) } } } } }
75. TextLayout AndroidTextLayout ComposeTextLayout /** * An abstraction over [android.text.Layout] with * different implementations for Views and Compose. */ interface TextLayout { val lineCount: Int val textPaint: TextPaint fun getLineVisibleEnd(line: Int): Int fun getLineTop(line: Int): Int fun getLineBottom(line: Int): Int fun getLineStart(line: Int): Int }
76. object ComposeViewHierarchyNode { fun fromView(view: View) { val rootNode = (view as? Owner) ?. root ?: return for (layoutNode in rootNode.children) { val bounds = layoutNode.coordinates.boundsInWindow() val semantics = layoutNode.collapsedSemantics val isVisible = layoutNode.isPlaced return when { semantics.contains(SemanticsProperties.Text) || semantics.contains(SemanticsActions.SetText) -> { val textLayoutResults = mutableListOf<TextLayoutResult>() semantics.getOrNull(SemanticsActions.GetTextLayoutResult) ?. action ?. invoke(textLayoutResults) TextViewHierarchyNode(isVisible, bounds, ComposeTextLayout(textLayoutResults.first()) } } } } }
77. val rootNode = (view as? Owner) ?. root ?: return for (layoutNode in rootNode.children) { val bounds = layoutNode.coordinates.boundsInWindow() val semantics = layoutNode.collapsedSemantics val isVisible = layoutNode.isPlaced return when { semantics.contains(SemanticsProperties.Text) || semantics.contains(SemanticsActions.SetText) -> { val textLayoutResults = mutableListOf<TextLayoutResult>() semantics.getOrNull(SemanticsActions.GetTextLayoutResult) ?. action ?. invoke(textLayoutResults) TextViewHierarchyNode(isVisible, bounds, ComposeTextLayout(textLayoutResults.first()) } semantics.contains(SemanticsProperties.ContentDescription) - > { ImageViewHierarchyNode(isVisible, bounds) } } } } }
78. internal data class ContentPainterElement( private val painter: Painter, private val alignment: Alignment, private val contentScale: ContentScale, private val alpha: Float, private val colorFilter: ColorFilter?, ) : ModifierNodeElement<ContentPainterNode>() private data class PainterElement( val painter: Painter, val sizeToIntrinsics: Boolean, val alignment: Alignment, val contentScale: ContentScale, val alpha: Float, val colorFilter: ColorFilter? ) : ModifierNodeElement<PainterNode>() private data class PainterModifierNodeElement( val painter: Painter, val sizeToIntrinsics: Boolean, val alignment: Alignment, val contentScale: ContentScale, val alpha: Float, val colorFilter: ColorFilter? ) : ModifierNodeElement<PainterNode>()
79. internal data class ContentPainterElement( private val painter: Painter, private val alignment: Alignment, private val contentScale: ContentScale, private val alpha: Float, private val colorFilter: ColorFilter?, ) : ModifierNodeElement<ContentPainterNode>() private data class PainterElement( val painter: Painter, val sizeToIntrinsics: Boolean, val alignment: Alignment, val contentScale: ContentScale, val alpha: Float, val colorFilter: ColorFilter? ) : ModifierNodeElement<PainterNode>() private data class PainterModifierNodeElement( val painter: Painter, val sizeToIntrinsics: Boolean, val alignment: Alignment, val contentScale: ContentScale, val alpha: Float, val colorFilter: ColorFilter? ) : ModifierNodeElement<PainterNode>()
80. internal fun LayoutNode.findPainter(): Painter? { val modifierInfos = getModifierInfo() for (index in modifierInfos.indices) { val modifier = modifierInfos[index].modifier if (modifier :: class.java.name.contains("Painter")) { return try { modifier :: class.java.getDeclaredField("painter") .apply { isAccessible = true } .get(modifier) as? Painter } catch (e: Throwable) { null } } } return null }
81. val rootNode = (view as? Owner) ?. root ?: return for (layoutNode in rootNode.children) { val bounds = layoutNode.coordinates.boundsInWindow() val semantics = layoutNode.collapsedSemantics val isVisible = layoutNode.isPlaced return when { semantics.contains(SemanticsProperties.Text) || semantics.contains(SemanticsActions.SetText) -> { val textLayoutResults = mutableListOf<TextLayoutResult>() semantics.getOrNull(SemanticsActions.GetTextLayoutResult) ?. action ?. invoke(textLayoutResults) TextViewHierarchyNode(isVisible, bounds, ComposeTextLayout(textLayoutResults.first()) } layoutNode.findPainter() ! = null - > { ImageViewHierarchyNode(isVisible, bounds) } } } } }
82. Storing screenshots to disk
83. // get the dominant color from the single-pixel bitmap paint.setColor(singlePixelBitmap.getPixel(0, 0)) canvas.drawRect(node.visibleRect, paint) } else -> canvas.drawRect(node.visibleRect, paint) } } // store masked screenshot to disk with current timestamp val frameTimestamp = System.currentTimeMillis() File(replayCacheDir, "$frameTimestamp.jpg").outputStream().use { bitmap.compress(JPEG, 80, it) it.flush() } bitmap.recycle() } }, listenerThread = mainHandler ) } }
84. // get the dominant color from the single-pixel bitmap paint.setColor(singlePixelBitmap.getPixel(0, 0)) canvas.drawRect(node.visibleRect, paint) } else -> canvas.drawRect(node.visibleRect, paint) } } // store masked screenshot to disk with current timestamp val frameTimestamp = System.currentTimeMillis() File(replayCacheDir, "$frameTimestamp.jpg").outputStream().use { bitmap.compress(JPEG, 80, it) it.flush() } bitmap.recycle() } }, listenerThread = mainHandler ) } }
85. Creating a video segment
86. Creating a video segment • Use MediaCodec to encode screenshots into video frames • Use MediaMuxer to create an .mp4 video out of encoded frames • Brute-force MediaCodec settings to fi nd the most optimal ones for your case
87. val mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC) val mediaFormat = MediaFormat.createVideoFormat( MediaFormat.MIMETYPE_VIDEO_AVC, width, height ).apply { format.setInteger(MediaFormat.KEY_BIT_RATE, bitRate) format.setFloat(MediaFormat.KEY_FRAME_RATE, frameRate) format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5) }
88. val mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC) val mediaFormat = MediaFormat.createVideoFormat( MediaFormat.MIMETYPE_VIDEO_AVC, width, height ).apply { format.setInteger(MediaFormat.KEY_BIT_RATE, bitRate) format.setFloat(MediaFormat.KEY_FRAME_RATE, frameRate) format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5) }
89. val mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC) val mediaFormat = MediaFormat.createVideoFormat( MediaFormat.MIMETYPE_VIDEO_AVC, width, height ).apply { format.setInteger(MediaFormat.KEY_BIT_RATE, bitRate) format.setFloat(MediaFormat.KEY_FRAME_RATE, frameRate) format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5) }
90. val mediaCodec = MediaCodec.createEncoderByType(MediaFormat.MIMETYPE_VIDEO_AVC) val mediaFormat = MediaFormat.createVideoFormat( MediaFormat.MIMETYPE_VIDEO_AVC, width, height ).apply { format.setInteger(MediaFormat.KEY_BIT_RATE, bitRate) format.setFloat(MediaFormat.KEY_FRAME_RATE, frameRate) format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5) } private var surface: Surface? = null fun start() { mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE) surface = mediaCodec.createInputSurface() mediaCodec.start() }
91. format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5) } private var surface: Surface? = null fun start() { mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE) surface = mediaCodec.createInputSurface() mediaCodec.start() } val muxer = MediaMuxer(videoFilePath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4) fun encode(image: Bitmap) { val canvas = surface ?. lockHardwareCanvas() canvas ?. drawBitmap(image, 0f, 0f, null) surface ? . unlockCanvasAndPost(canvas) mediaMuxer.writeSampleData(mediaCodec.outputBuffers) }
92. format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5) } private var surface: Surface? = null fun start() { mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE) surface = mediaCodec.createInputSurface() mediaCodec.start() } val muxer = MediaMuxer(videoFilePath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4) fun encode(image: Bitmap) { val canvas = surface ?. lockHardwareCanvas() canvas ?. drawBitmap(image, 0f, 0f, null) surface ? . unlockCanvasAndPost(canvas) mediaMuxer.writeSampleData(mediaCodec.outputBuffers) }
93. format.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 5) } private var surface: Surface? = null fun start() { mediaCodec.configure(mediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE) surface = mediaCodec.createInputSurface() mediaCodec.start() } val muxer = MediaMuxer(videoFilePath, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4) fun encode(image: Bitmap) { val canvas = surface ?. lockHardwareCanvas() canvas ?. drawBitmap(image, 0f, 0f, null) surface ? . unlockCanvasAndPost(canvas) mediaMuxer.writeSampleData(mediaCodec.outputBuffers) }
94. // paint.setColor(singlePixelBitmap.getPixel(0, 0)) canvas.drawRect(node.bounds, paint) } else -> canvas.drawRect(node.bounds, paint) } } val encoder = VideoEncoder(width, height, bitRate, frameRate) val time = System.currentTimeMillis() if (time - lastSegmentSent >= 5000L) { encoder.start() for (screenshot in screenshotsFromDisk()) { encoder.encode(frame) } encoder.release() } } }, listenerThread = mainHandler ) } }
95. Video encoding gotchas
96. Adjust to block size /** * Maximum number of macroblocks in the frame. * * Video frames are conceptually divided into 16-by-16 pixel blocks called macroblocks. * Most coding standards operate on these 16-by-16 pixel blocks; thus, codec performance * is characterized using such blocks. * * @hide * / @TestApi public int getMaxMacroBlocks() { return saturateLongToInt(mWidth * (long)mHeight); }
97. /** * Since codec block size is 16, so we have to adjust the width and height to it, otherwise * the codec might fail to configure on some devices */ private fun Int.adjustToBlockSize(): Int { val remainder = this % 16 return if (remainder <= 8) { this - remainder } else { this + (16 - remainder) } } val bitmap = Bitmap.createBitmap( width = screenBounds.width().adjustToBlockSize(), height = screenBounds.height().adjustToBlockSize(), Bitmap.Config.ARGB_8888 )
98. Hack for Xiaomi devices fun encode(image: Bitmap) { // it seems that Xiaomi devices have problems with hardware canvas, so we have to use // lockCanvas instead https: // stackoverflow.com/a/73520742 val canvas = if (Build.MANUFACTURER.contains("xiaomi", ignoreCase = true)) { surface ?. lockCanvas(null) } else { surface ?. lockHardwareCanvas() } canvas ?. drawBitmap(image, 0f, 0f, null) surface ? . unlockCanvasAndPost(canvas) mediaMuxer.writeSampleData(mediaCodec.outputBuffers) }
99. Key Pieces ReplayVideo • Actual video bytes that we created from periodically taking screenshots ReplayVideo ReplayRecording • Video metadata • Breadcrumbs • Gestures • Network requests • Logs ReplayEvent • Tags • Error/Trace connection • Internal metadata Backend
101. Keeping Performance overhead low
102. Keeping Performance overhead low • Downscale screenshots
103. = = || // sanity check return } / / PixelCopy takes screenshots including system bars, so we have to get the real size here val wm = context.getSystemService(Context.WINDOW_SERVICE) as WindowManager val screenBounds = if (VERSION.SDK_INT > = VERSION_CODES.R) { wm.currentWindowMetrics.bounds } else { val screenBounds = Point() @Suppress("DEPRECATION") wm.defaultDisplay.getRealSize(screenBounds) Rect(0, 0, screenBounds.x, screenBounds.y) } val.bitmap = Bitmap.createBitmap( width = screenBounds.width(), height = screenBounds.height(), Bitmap.Config.ARGB_8888 ) } 1440px 2880px
104. wm.currentWindowMetrics.bounds } else { val screenBounds = Point() @Suppress("DEPRECATION") wm.defaultDisplay.getRealSize(screenBounds) Rect(0, 0, screenBounds.x, screenBounds.y) } / / use the baseline density of 1x (mdpi) val (height, width) = (screenBounds.height() / context.resources.displayMetrics.density) to (screenBounds.width() / context.resources.displayMetrics.density) val.bitmap = Bitmap.createBitmap( width = width, height = height, Bitmap.Config.ARGB_8888 ) } 412 px 823 px
105. @Suppress("DEPRECATION") wm.defaultDisplay.getRealSize(screenBounds) Rect(0, 0, screenBounds.x, screenBounds.y) } / / use the baseline density of 1x (mdpi) val (height, width) = (screenBounds.height() / context.resources.displayMetrics.density) to (screenBounds.width() / context.resources.displayMetrics.density) val scaleFactorX = width / screenBounds.width() val scaleFactorY = height / screenBounds.height() val.bitmap = Bitmap.createBitmap( width = width, height = height, Bitmap.Config.ARGB_8888 ) }
106. bitmap.recycle() return@request } / / hurray, we have a screenshot! val nodesToMask = mutableListOf<ViewHierarchyNode>() root.traverse(nodesToMask) / / can switch back to bg thread, main thread stuff is done at this point executor.submit { val canvas = Canvas(bitmap) val preScaledMatrix = Matrix().apply { preScale(scaleFactorX, scaleFactorY) } canvas.setMatrix(preScaledMatrix) val paint = Paint().apply { setColor(Color.BLACK) } for (node in nodesToMask) { when (node) { is TextViewHierarchyNode -> { val rects = node.layout.getVisibleRects(node.visibleRect) for (rect in rects) { canvas.drawRect(rect, node.layout.paint) } } ->
107. Keeping Performance overhead low • Downscale screenshots • Only record changes on screen
108. ... private val mainHandler = Handler(Looper.getMainLooper()) private val rootViews = ArrayList<WeakReference<View >> () Curtains.onRootViewsChangedListeners += OnRootViewsChangedListener { root, added -> if (added) { rootViews.add(WeakReference<View>(root)) root.viewTreeObserver ?. addOnDrawListener(this) } else { rootViews.removeAll { it.get() == root } root.viewTreeObserver ?. removeOnDrawListener(this) } } override fun onDraw() { } @RequiresApi(26) fun capture() { / / here goes the capturing logic . .. }
109. } else { rootViews.removeAll { it.get() == root } root.viewTreeObserver ?. removeOnDrawListener(this) } } val contentChanged = AtomicBoolean(false) override fun onDraw() { contentChanged.set(true) } @RequiresApi(26) fun capture() { / / here goes the capturing logic if (!contentChanged.get()) { // nothing changed since last frame, so we don't need to capture it return } . .. }
110. Keeping Performance overhead low • Downscale screenshots • Only record changes on screen • Speed up Compose bounds retrieval
113. /** * The boundaries of this layout relative to the window's origin. */ fun LayoutCoordinates.boundsInWindow(): Rect { val root = findRootCoordinates() ... } Root Row Text Button Image Button Text
114. /** * The boundaries of this layout inside the root composable. */ fun LayoutCoordinates.boundsInRoot(): Rect = findRootCoordinates().localBoundingBoxOf(this) /** * The boundaries of this layout relative to the window's origin. */ fun LayoutCoordinates.boundsInWindow(): Rect { val root = findRootCoordinates() val bounds = boundsInRoot() ... }
115. /** * The boundaries of this layout inside the root composable. */ fun LayoutCoordinates.boundsInRoot(): Rect = findRootCoordinates().localBoundingBoxOf(this) /** * The boundaries of this layout relative to the window's origin. */ fun LayoutCoordinates.boundsInWindow(): Rect { val root = findRootCoordinates() val bounds = boundsInRoot() ... }
117. /** * The boundaries of this layout inside the root composable. */ fun LayoutCoordinates.boundsInRoot(): Rect = findRootCoordinates().localBoundingBoxOf(this) /** * The boundaries of this layout relative to the window's origin. */ fun LayoutCoordinates.boundsInWindow(): Rect { val root = findRootCoordinates() val bounds = boundsInRoot() ... } Root Row Text Button Image Button Text
118. /** * The boundaries of this layout relative to the window's origin. * A faster copy of https: // github.com/androidx/ . .. /LayoutCoordinates.kt#L187 */ fun LayoutCoordinates.boundsInWindow(root: LayoutCoordinates): Rect { val bounds = root.localBoundingBoxOf(this) ... }
119. @Suppress("INVISIBLE_MEMBER", "INVISIBLE_REFERENCE") // sorry internal, but we need you object ComposeViewHierarchyNode { fun fromView(view: View) { val rootNode = (view as? Owner) ?. root ?: return for (layoutNode in rootNode.children) { val bounds = layoutNode.coordinates.boundsInWindow(rootNode.coordinates) } } }
120. ~35-40% main thread time saved
121. Keeping Performance overhead low • Downscale screenshots • Only record changes on screen • Speed up Compose bounds retrieval
122. Hybrid SDKs
123. Hybrid SDKs RN/Flutter SDK Android SDK iOS SDK Just Works* ™
124. What’s Next • Session Replay Beta available for Android, iOS, React Native and Flutter • Stable Release early January • Rage and Dead Clicks detection • CPU/Memory/Battery stats • Layout Inspector
125. References • Sentry Android SDK - github.com/getsentry/sentry-java • Sentry Session Replay - docs.sentry.io/product/explore/session-replay/ mobile/ • PixelCopy - developer.android.com/reference/android/view/PixelCopy • PocketCasts Android - github.com/Automattic/pocket-casts-android • Flutter screen recorder - github.com/fzyzcjy/ fl utter_screen_recorder • Curtains - github.com/square/curtains • MediaCodec - developer.android.com/reference/android/media/MediaCodec
126. Thank you! @romtsn