involves many considerations 1. How do you implement streaming and viewing functionality? 2. Minimizing latency from streaming to viewing 3. The screen contains many functionalities: a. Chat / Virtual Gifts / Polls / Filters for video / Collaborative streaming 4. Having many functionalities means placing numerous components on the screen 5. Performance tuning is essential considering all these functionalities
Sugiyama 2. X: @u5_03, LinkedIn: @sugiy 3. Flutter/iOS Engineer a. at DeNA Co., Ltd.(2023/07~) 4. iOS (2018~), Flutter (2023~) engineer 5. Came from Japan 6. This is my first time in Europe 7. This is also my first time presenting overseas
sports-themed live streaming service 2. Viewers and streamers can enjoy watching the same sports footage⚾ 3. Available for iOS and Android app 4. Base live streaming functionality is implemented using AWS services a. Considering schedule and maintenance, building everything from scratch wasn't realistic
feed and cast it to the server 2. Cast two videos to viewers with low latency 3. Allow viewers to check comments 4. Play movie for virtual gifts Solution for 1 & 2: AWS Chime SDK 1. WebRTC low-latency synchronization 2. Cost-effective pricing play-by-sports architecture(Streaming functionality)
and streamer video playback 2. Minimal latency with streamer’s screen 3. Viewer can post and check comments 4. Play movie for virtual gifts Solution for 1 & 2: AWS IVS Player 1. Low latency capabilities 2. Cost-effective pricing 3. Internal expertise available play-by-sports architecture (Viewing functionality)
Overview Viewers Play game footage source Stream play game footage Cast streamer’s camera feed Deliver via HLS Combine play game footage and streamer video into a single video Video transfer via WebRTC Video transfer via RTMP Server Streamers ChimeSDK IVS Player
used for streaming and viewing functionalities a. Amazon Chime SDK for streaming b. IVS Player SDK for viewing 2. Required to show Native UI display in Flutter screen a. PlatformView b. Texture Widget Native Video Display
Encountered: 1. Battery consumption (Xcode Energy Impact > "Very High") 2. Severe device heating 3. The application sometimes crashed, more often on the viewing screen, and especially on the streaming screen 4. Video stutters under high load (especially noticeable on the streaming screen) Root Cause: 1. High performance load indicated by metrics
to measure by using Flutter debug function 2. Device surface temperature a. Decided based on the company's QA policy. b. To avoid low-temperature burns during long periods of use. 3. CPU load a. Relatively easy to measure and compare before and after of tuning Key Metrics
measurements tools (Flutter, iOS, Android) a. Flutter i. Performance Overlay / Flutter DevTools: UI b. iOS i. Debug gauges: Quick resource monitor ii. Instruments: Deep performance profiler c. Android i. Android Studio Profiler: Real-time CPU/memory view 2. Manual measurement: Verification using a thermometer
1. Debug gauges a. Quick check such as CPU, Memory, Network, Energy Impact 2. Instruments: a. CPU Profiler: Compare CPU cycles of the execution across builds b. Time Profiler: Find the bottleneck of the code
quality restrictions (HD over-specification) a. Adaptive Bitrate (ABR) automatically adjusts video quality based on network conditions. b. The maximum quality, which had been set to HD (1080p) by default, has now been restricted to 720p. 2. ShaderMask improvements a. Optimize ShaderMask by creating shaders only once.
viewing screen improvements also applied to the streaming screen 2. Lowered resolution and reduced frame rate (down to 20fps) 3. Fundamental improvements to PlatformView and the Texture widget were needed.
issue: two videos (game footage + streamer video) must be shown a. iOS: 2×Texture, Android: 2× PlatformView→heavy 2. If combined natively into one frame/view, only one Texture/PlatformView needed 3. How to implement? a. iOS→Metal merges two pixelBuffers into one b. Android→merge both into a single native view
over CIImage? • Metal: Apple's low-level GPU framework for parallel computing • MSL (Metal Shading Language): C++ based language for GPU kernels • Key functionalities: Direct GPU access with minimal overhead • Flutter's UI rendering on iOS also uses Metal Approach Performance Memory Complexity Use Case Core Graphics Slow(CPU only) Moderate Simple Static images CIImage Medium (CPU/GPU) High overhead Simple Basic filters Metal Fast (GPU parallel) Low overhead Complex Real-time video We selected Metal
Our Approach: 1. Write clear specifications a. Clear code logic input leads to accurate code generation 2. Let AI generate MSL shader code from specs 3. Carefully execute pre-defined test cases & Performance check Key Success Factors: 1. Precise requirements > GPU expertise 2. Extensive logging for validation 3. Focus on correctness, not optimization
1. Vertical video player stacking to merge into single view a. We used ConstraintLayout b. ConstraintLayout is similar to UIKit's AutoLayout implementation on iOS Benefits: 1. Using a single PlatformView improves performance and reduces the risk of issues
has three types of PlatformViews. 2. TLHC is the latest, but it isn’t always the best choice. 3. Pick the approach based on your implementation and use case. Mode Render mechanism Pros Cons/Limits Virtual Display (VD) Render to virtual display → Texture Stable layering; Flutter transforms/clipping work Extra copy; input/a11y quirks; higher cost Hybrid Composition (HC) Android View in host hierarchy; OS composites Best platform fidelity; works with WebView/Map Limited transforms; possible jank; pre-Android10 heavier Texture Layer Hybrid Composition (TLHC) Like HC, but draw into a Texture Good Flutter-side effects & perf Breaks with SurfaceView; jitter; needs fallback
A video view rendered with a PlatformView, which should have been in the backmost layer, was mistakenly displayed in the frontmost layer. a. All UI implemented in the Flutter layer on the front will be hidden behind it. 2. Known issue that occurs when the PlatformView uses a SurfaceView. a. Ref:https://github.com/flutter/flutter/wiki/Android-Platform-Views/50847a99dbafa8c3dd8 8eb98a7f834fffb0bb94b#texture-layer-hybrid-composition 3. It was resolved by switching to HC.
1. After closing a overlay screen, part of it remains as a ghost(~Android 13). 2. Improved with Hybrid Composition, but only effective on Android 14 and above. a. ref:https://github.com/flutter/flutter/pull/142399 3. The cause was SurfaceView, so on 13↓we switched to the ChimeSDK TextureView player. 4. TextureView has performance drawbacks. Close modal
still show no margin; further optimization is required a. Energy Impact remains Very High b. Deeper understanding needed of Flutter rendering, native UI mechanisms (PlatformView, Texture Widget) and performance tools c. Automating and labor-saving performance measurement 2. Provide Debug Menu switches to enable/disable individual functionalities a. Facilitate performance measurement of each functionality independently
to organize many functionalities 1. Common components with separate base widgets 1.1. Background gradient 1.2. Live / Streamer Information 1.3. Comment list / information from service 1.4. Buttons to navigate to various functions 2. Shared UI elements between streaming/viewing 3. State injection into viewing / streaming widgets 4. CustomMultiChildLayout for component placement
A widget that controls the layout of multiple child widgets, placing each one at arbitrary positions. 2. Enables us to build custom layouts similar to Column, Row, or Stack. 3. Each child widget is assigned an ID, so we can define conditions and rules per ID. 4. The parent widget’s size can be considered when determining the size and position of its children.
at FlutterKaigi2024 (Japanese Flutter conference) a. ‘Use CustomMultiChildLayout to create any layout you can imagine!’ b. https://ulog.sugiy.com/en/flutter-kaigi-2024/ 2. Examples: circular, piano keyboard, Google Calendar-like timeline UI
app requires considering many factors. 2. Requiring strong commitment to develop and maintain over time. 3. It’s important to consult with other engineers by solving issues. Summary
a. https://docs.flutter.dev/perf/ui-performance 2. Texture class a. https://api.flutter.dev/flutter/widgets/Texture-class.html 3. Flutter performance profiling a. https://docs.flutter.dev/perf/ui-performance 4. Metal a. https://developer.apple.com/metal/
u503
javiergs
pvcresin
.jpg){kind=avatar}
97vaibhav
marcoroth
sobolevn
kinocoboy2
teamlab
jrsaruo
moro
po3rin
addyosmani
wjessup
destraynor
rmw
kevinliebholz
reverentgeek
jonrohan
lauravandoore
chrislema
stephaniewalter
davidbonilla
csswizardry
