The Flutter Journey of Building a Live Streaming App — With a Side of Performance Tuning

The Flutter Journey of Building a Live Streaming App — With a Side of Performance Tuning

https://www.fluttercon.dev/agenda

Transcript

1. © DeNA Co., Ltd. 1 Fluttercon Europe 2025
2. © DeNA Co., Ltd. 2 I'm developing a live streaming app
3. © DeNA Co., Ltd. 3 There are various live streaming apps around the world
4. © DeNA Co., Ltd. 4 If you were to develop a new live streaming app, how would you develop it?
5. © DeNA Co., Ltd. 5 Building a live streaming app involves many considerations 1. How do you implement streaming and viewing functionality? 2. Minimizing latency from streaming to viewing 3. The screen contains many functionalities: a. Chat / Virtual Gifts / Polls / Filters for video / Collaborative streaming 4. Having many functionalities means placing numerous components on the screen 5. Performance tuning is essential considering all these functionalities
6. © DeNA Co., Ltd. 6 1. How we built a live streaming app with Flutter 2. What challenges we faced 3. How we improved them Our improvements to Flutter live streaming development
7. © DeNA Co., Ltd. 7 The Flutter Journey of Building a Live Streaming App - With a Side of Performance Tuning - Fluttercon Europe 2025 Sugiy / Yugo Sugiyama
8. © DeNA Co., Ltd. 8 Sorry, my English is not very good (Please excuse my accent 😅)
9. © DeNA Co., Ltd. 9 README 1. Sugiy / Yugo Sugiyama 2. X: @u5_03, LinkedIn: @sugiy 3. Flutter/iOS Engineer a. at DeNA Co., Ltd.(2023/07~) 4. iOS (2018~), Flutter (2023~) engineer 5. Came from Japan 󰏦 6. This is my first time in Europe 󰎾 7. This is also my first time presenting overseas
10. © DeNA Co., Ltd. 10 10 01. About play-by-sports The live streaming service we’re developing
11. © DeNA Co., Ltd. 11 play-by-sports service introduction 1. A sports-themed live streaming service 2. Viewers and streamers can enjoy watching the same sports footage⚾ 3. Available for iOS and Android app 4. Base live streaming functionality is implemented using AWS services a. Considering schedule and maintenance, building everything from scratch wasn't realistic
12. © DeNA Co., Ltd. 12 Requirements: 1. Display the camera feed and cast it to the server 2. Cast two videos to viewers with low latency 3. Allow viewers to check comments 4. Play movie for virtual gifts Solution for 1 & 2: AWS Chime SDK 1. WebRTC low-latency synchronization 2. Cost-effective pricing play-by-sports architecture(Streaming functionality)
13. © DeNA Co., Ltd. 13 Requirements: 1. Sports game footage and streamer video playback 2. Minimal latency with streamer’s screen 3. Viewer can post and check comments 4. Play movie for virtual gifts Solution for 1 & 2: AWS IVS Player 1. Low latency capabilities 2. Cost-effective pricing 3. Internal expertise available play-by-sports architecture (Viewing functionality)
14. © DeNA Co., Ltd. 14 play-by-sports live streaming system Architecture Overview Viewers Play game footage source Stream play game footage Cast streamer’s camera feed Deliver via HLS Combine play game footage and streamer video into a single video Video transfer via WebRTC Video transfer via RTMP Server Streamers ChimeSDK IVS Player
15. © DeNA Co., Ltd. 15 15 02. play-by-sports core function implementation How to implement viewing and streaming functions
16. © DeNA Co., Ltd. 16 Challenge: 1. Native SDKs are used for streaming and viewing functionalities a. Amazon Chime SDK for streaming b. IVS Player SDK for viewing 2. Required to show Native UI display in Flutter screen a. PlatformView b. Texture Widget Native Video Display
17. © DeNA Co., Ltd. 17 PlatformView / Texture Widget PlatformView: 1. Embeds native iOS/Android Views into Flutter WidgetTree 2. UIView (iOS) / View (Android) display 3. Used in official packages (webview_flutter, google_maps_flutter) Texture Widget: 1. Bridges Flutter and native rendering 2. Embeds camera/video footage into Flutter 3. Used in camera, video_player packages
18. © DeNA Co., Ltd. 18 Our Implementation Choices Policy: Prefer Texture Widget if possible due to PlatformView UI bugs Streaming Screen: 1. iOS → Texture Widget(ChimeSDK Player + AVCaptureSession CVPixelBuffer support) 2. Android → PlatformView(Screen stutters issues, limited Android expertise) Viewing Screen: 1. iOS → PlatformView(SDK limitations) 2. Android → PlatformView(SDK limitations, limited Android expertise)
19. © DeNA Co., Ltd. 19 Problems we are faced Issues Encountered: 1. Battery consumption (Xcode Energy Impact > "Very High") 2. Severe device heating 3. The application sometimes crashed, more often on the viewing screen, and especially on the streaming screen 4. Video stutters under high load (especially noticeable on the streaming screen) Root Cause: 1. High performance load indicated by metrics
20. © DeNA Co., Ltd. 20 20 03. Performance tuning we challenged Mainly about the collaboration between Flutter and native implementation
21. © DeNA Co., Ltd. 21 1. Frame rate a. Easy to measure by using Flutter debug function 2. Device surface temperature a. Decided based on the company's QA policy. b. To avoid low-temperature burns during long periods of use. 3. CPU load a. Relatively easy to measure and compare before and after of tuning Key Metrics
22. © DeNA Co., Ltd. 22 How to measure 1. Platform-specific measurements tools (Flutter, iOS, Android) a. Flutter i. Performance Overlay / Flutter DevTools: UI b. iOS i. Debug gauges: Quick resource monitor ii. Instruments: Deep performance profiler c. Android i. Android Studio Profiler: Real-time CPU/memory view 2. Manual measurement: Verification using a thermometer
23. © DeNA Co., Ltd. 23 Measuring device temperature
24. © DeNA Co., Ltd. 24 How we used iOS tools 1. Debug gauges a. Quick check such as CPU, Memory, Network, Energy Impact 2. Instruments: a. CPU Profiler: Compare CPU cycles of the execution across builds b. Time Profiler: Find the bottleneck of the code
25. © DeNA Co., Ltd. 25 How we used Android tools 1. Android Studio Profiler a. Find CPU Hotspots– Analyze method traces and thread activity
26. © DeNA Co., Ltd. 26 Points of interest from the measurements • PlatformView and Texture Widget load
27. © DeNA Co., Ltd. 27 Viewing Screen Optimizations 1. ABR quality restrictions (HD over-specification) a. Adaptive Bitrate (ABR) automatically adjusts video quality based on network conditions. b. The maximum quality, which had been set to HD (1080p) by default, has now been restricted to 720p. 2. ShaderMask improvements a. Optimize ShaderMask by creating shaders only once.
28. © DeNA Co., Ltd. 28 Streaming Screen Optimizations 1. Some viewing screen improvements also applied to the streaming screen 2. Lowered resolution and reduced frame rate (down to 20fps) 3. Fundamental improvements to PlatformView and the Texture widget were needed.
29. © DeNA Co., Ltd. 29 How to improve this?
30. © DeNA Co., Ltd. 30 💡
31. © DeNA Co., Ltd. 31 Just combine the two videos into one?
32. © DeNA Co., Ltd. 32 In other words... 1. Main issue: two videos (game footage + streamer video) must be shown a. iOS: 2×Texture, Android: 2× PlatformView→heavy 2. If combined natively into one frame/view, only one Texture/PlatformView needed 3. How to implement? a. iOS→Metal merges two pixelBuffers into one b. Android→merge both into a single native view
33. © DeNA Co., Ltd. 33 What is Metal? Why Metal over CIImage? • Metal: Apple's low-level GPU framework for parallel computing • MSL (Metal Shading Language): C++ based language for GPU kernels • Key functionalities: Direct GPU access with minimal overhead • Flutter's UI rendering on iOS also uses Metal Approach Performance Memory Complexity Use Case Core Graphics Slow(CPU only) Moderate Simple Static images CIImage Medium (CPU/GPU) High overhead Simple Basic filters Metal Fast (GPU parallel) Low overhead Complex Real-time video We selected Metal
34. © DeNA Co., Ltd. 34 Implementation Architecture Video Compositing Pipeline 1. Input: 2 CVPixelBuffers (Game + Streamer video) 2. Output: Single frame (Game on top 16:9, Streamer below) Key Optimizations 1. Texture Cache: Reuse Metal textures via CVMetalTextureCache 2. PixelBuffer Pool: Pre-allocated buffers for output 3. Layout Caching: Avoid recalculation for same dimensions 4. 16x16 Thread Groups: Efficient GPU parallelization for Apple GPU architecture
35. © DeNA Co., Ltd. 35 Development Strategy & Best Practices Our Approach: 1. Write clear specifications a. Clear code logic input leads to accurate code generation 2. Let AI generate MSL shader code from specs 3. Carefully execute pre-defined test cases & Performance check Key Success Factors: 1. Precise requirements > GPU expertise 2. Extensive logging for validation 3. Focus on correctness, not optimization
36. © DeNA Co., Ltd. 36 iOS performance tuning result Successes: 1. Significant performance improvements 2. CPU Cycles(Giga cycles per second): 4.74Gc/s →0.911Gc/s Problems: 1. High implementation/maintenance costs 2. Potential technical debt concerns
37. © DeNA Co., Ltd. 37 Detail implementation of Android Approach: 1. Vertical video player stacking to merge into single view a. We used ConstraintLayout b. ConstraintLayout is similar to UIKit's AutoLayout implementation on iOS Benefits: 1. Using a single PlatformView improves performance and reduces the risk of issues
38. © DeNA Co., Ltd. 38 About Android PlatformView 1. Android has three types of PlatformViews. 2. TLHC is the latest, but it isn’t always the best choice. 3. Pick the approach based on your implementation and use case. Mode Render mechanism Pros Cons/Limits Virtual Display (VD) Render to virtual display → Texture Stable layering; Flutter transforms/clipping work Extra copy; input/a11y quirks; higher cost Hybrid Composition (HC) Android View in host hierarchy; OS composites Best platform fidelity; works with WebView/Map Limited transforms; possible jank; pre-Android10 heavier Texture Layer Hybrid Composition (TLHC) Like HC, but draw into a Texture Good Flutter-side effects & perf Breaks with SurfaceView; jitter; needs fallback
39. © DeNA Co., Ltd. 39 1. z-index issue (TLHC) 1. A video view rendered with a PlatformView, which should have been in the backmost layer, was mistakenly displayed in the frontmost layer. a. All UI implemented in the Flutter layer on the front will be hidden behind it. 2. Known issue that occurs when the PlatformView uses a SurfaceView. a. Ref:https://github.com/flutter/flutter/wiki/Android-Platform-Views/50847a99dbafa8c3dd8 8eb98a7f834fffb0bb94b#texture-layer-hybrid-composition 3. It was resolved by switching to HC.
40. © DeNA Co., Ltd. 40 Widget stale UI issue (HC) 1. After closing a overlay screen, part of it remains as a ghost(~Android 13). 2. Improved with Hybrid Composition, but only effective on Android 14 and above. a. ref:https://github.com/flutter/flutter/pull/142399 3. The cause was SurfaceView, so on 13↓we switched to the ChimeSDK TextureView player. 4. TextureView has performance drawbacks. Close modal
41. © DeNA Co., Ltd. 41 Result of performance tuning After tuning Elapsed time Elapsed time iOS iOS 1. Device heating: reduced 2. Crashes: Nearly eliminated 3. Video stuttering: No reported issues 4. Battery consumption: Reduced user complaints
42. © DeNA Co., Ltd. 42 Future Plans 1. Performance metrics still show no margin; further optimization is required a. Energy Impact remains Very High b. Deeper understanding needed of Flutter rendering, native UI mechanisms (PlatformView, Texture Widget) and performance tools c. Automating and labor-saving performance measurement 2. Provide Debug Menu switches to enable/disable individual functionalities a. Facilitate performance measurement of each functionality independently
43. © DeNA Co., Ltd. 43 43 How to implement complicated UI 04. play-by-sports UI implementation
44. © DeNA Co., Ltd. 44 UI for Viewing and Streaming to organize many functionalities 1. Common components with separate base widgets 1.1. Background gradient 1.2. Live / Streamer Information 1.3. Comment list / information from service 1.4. Buttons to navigate to various functions 2. Shared UI elements between streaming/viewing 3. State injection into viewing / streaming widgets 4. CustomMultiChildLayout for component placement
45. © DeNA Co., Ltd. 45 What is CustomMultiChildLayout widget? 1. A widget that controls the layout of multiple child widgets, placing each one at arbitrary positions. 2. Enables us to build custom layouts similar to Column, Row, or Stack. 3. Each child widget is assigned an ID, so we can define conditions and rules per ID. 4. The parent widget’s size can be considered when determining the size and position of its children.
46. © DeNA Co., Ltd. 46 Previous Work Reference 1. Presented at FlutterKaigi2024 (Japanese Flutter conference) a. ‘Use CustomMultiChildLayout to create any layout you can imagine!’ b. https://ulog.sugiy.com/en/flutter-kaigi-2024/ 2. Examples: circular, piano keyboard, Google Calendar-like timeline UI
47. © DeNA Co., Ltd. 47 Why CustomMultiChildLayout widget? Why CustomMultiChildLayout: 1. Complex layouts difficult with Row/Column/Stack 2. Centralized layout logic management 3. Reduced fragmented display logic 4. Easier design modification tracking Trade-offs: 1. High learning costs a. Team pros/cons evaluation required
48. © DeNA Co., Ltd. 48 48 05. Summary Review of the presentation
49. © DeNA Co., Ltd. 49 1. Developing a live streaming app requires considering many factors. 2. Requiring strong commitment to develop and maintain over time. 3. It’s important to consult with other engineers by solving issues. Summary
50. © DeNA Co., Ltd. 50 Reference 1. Flutter performance profiling a. https://docs.flutter.dev/perf/ui-performance 2. Texture class a. https://api.flutter.dev/flutter/widgets/Texture-class.html 3. Flutter performance profiling a. https://docs.flutter.dev/perf/ui-performance 4. Metal a. https://developer.apple.com/metal/
51. © DeNA Co., Ltd. 51 Thanks for listening! • Check this slide from this QR code→ • If you have any questions… ◦ ask me after the talk ◦ mention @u5_03 on X