Tweet
Tweet

Slide 1

Slide 1 text

Friday, June 7, 2024 droidcon San Francisco Seamless mobile real-time communication with WebRTC Renaud Mathieu

Slide 2

Slide 2 text

Renaud Mathieu β€’ Freelance β€’ Based in Paris, France πŸ₯–πŸ· β€’ Member of the Paris Android User Group β€’ Proud of being here today with you About me renaudmathieu.com

Slide 3

Slide 3 text

Web Real-Time Communication

Slide 4

Slide 4 text

πŸ“Ή Video πŸ”ˆ Audio πŸ“¦ Data

Slide 5

Slide 5 text

πŸ“œ History

Slide 6

Slide 6 text

2009 History Timeline

Slide 7

Slide 7 text

Early Beginnings 2009 History Timeline

Slide 8

Slide 8 text

Early Beginnings 2009 2011 Development & Standardization History Timeline

Slide 9

Slide 9 text

Early Beginnings 2009 2011 Development & Standardization Browser Adoption 2012-2013 History Timeline

Slide 10

Slide 10 text

Early Beginnings 2009 2011 Development & Standardization Browser Adoption 2012-2013 Industry Adoption 2014-2015 History Timeline

Slide 11

Slide 11 text

Early Beginnings 2009 2011 Development & Standardization Browser Adoption 2012-2013 Industry Adoption 2014-2015 2018 WebRTC 1.0 History Timeline

Slide 12

Slide 12 text

Early Beginnings 2009 2011 Development & Standardization Browser Adoption 2012-2013 Industry Adoption 2014-2015 2018 WebRTC 1.0 2020-Present Ubiquity History Timeline

Slide 13

Slide 13 text

πŸ“ Architecture

Slide 14

Slide 14 text

Client A Client C Client B πŸ•Έ Mesh Number of edges= NΓ—(Nβˆ’1) / 2

Slide 15

Slide 15 text

Server Client A Client C Client B πŸ”— Multipoint Conference Unit

Slide 16

Slide 16 text

Server Client A Client C Client B πŸ“¦ πŸ“¦ πŸ“¦ πŸ”— Multipoint Conference Unit

Slide 17

Slide 17 text

Server Client A Client B πŸ‘₯ WebRTC πŸ‘‹πŸͺͺ πŸ€™πŸͺͺ πŸ’ πŸ’ 🀝

Slide 18

Slide 18 text

πŸ‘₯ WebRTC 1.Signaling Phase 2. Peer-to-Peer Connection Phase

Slide 19

Slide 19 text

ICE (Interactive Connectivity Establishment) πŸ‘₯ WebRTC

Slide 20

Slide 20 text

STUN (Session Traversal Utilities for NAT) ICE (Interactive Connectivity Establishment) πŸ‘₯ WebRTC

Slide 21

Slide 21 text

TURN (Traversal Using Relays around NAT) πŸ‘₯ WebRTC STUN (Session Traversal Utilities for NAT) ICE (Interactive Connectivity Establishment)

Slide 22

Slide 22 text

Turn Server STUN Server NAT NAT Client A Client B STUN Server Signaling Server Summary

Slide 23

Slide 23 text

Additional informations β€’ WebRTC transmits data using UDP β€’ No Standard regarding to the Signaling Server β€’ πŸ”Ž chrome://webrtc-internals

Slide 24

Slide 24 text

πŸ§‘πŸ’» Signaling Server

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

https://start.ktor.io

Slide 28

Slide 28 text

import kotlinx.serialization.SerialName import kotlinx.serialization.Serializable @Serializable data class WsPacket( @SerialName("id") val id: WsMessage, @SerialName("data") val data: String, )

Slide 29

Slide 29 text

import kotlinx.serialization.Serializable @Serializable enum class WsMessage { WsError, WsIceCandidates, WsIceServers, WsInit, WsSDPAnswer, WsSDPOffer, }

Slide 30

Slide 30 text

class Connection( val session: DefaultWebSocketSession ) { companion object { val lastId = AtomicInteger(0) } val name = "client${lastId.getAndIncrement()}" }

Slide 31

Slide 31 text

@OptIn(InternalAPI :: class) fun Application.configureSockets() { install(WebSockets) { pingPeriod = Duration.ofSeconds(15) timeout = Duration.ofSeconds(15) maxFrameSize = Long.MAX_VALUE masking = false contentConverter = KotlinxWebsocketSerializationConverter(Json) } val connections = Collections.synchronizedSet(LinkedHashSet()) routing { webSocket("/connect") // Let’s GO } } }

Slide 32

Slide 32 text

@OptIn(InternalAPI :: class) fun Application.configureSockets() { install(WebSockets) { pingPeriod = Duration.ofSeconds(15) timeout = Duration.ofSeconds(15) maxFrameSize = Long.MAX_VALUE masking = false contentConverter = KotlinxWebsocketSerializationConverter(Json) } val connections = Collections.synchronizedSet(LinkedHashSet()) routing { webSocket("/connect") // Let’s GO } } }

Slide 33

Slide 33 text

@OptIn(InternalAPI :: class) fun Application.configureSockets() { install(WebSockets) { pingPeriod = Duration.ofSeconds(15) timeout = Duration.ofSeconds(15) maxFrameSize = Long.MAX_VALUE masking = false contentConverter = KotlinxWebsocketSerializationConverter(Json) } val connections = Collections.synchronizedSet(LinkedHashSet()) routing { webSocket("/connect") // Let’s GO } } }

Slide 34

Slide 34 text

webSocket("/connect") { // Register this new client val thisConnection = Connection(this) connections += thisConnection try { while (true) { val wsPacket = receiveDeserialized() when (wsPacket.id) { WsMessage.WsInit, WsMessage.WsError, WsMessage.WsIceCandidates, WsMessage.WsIceServers, WsMessage.WsSDPAnswer, WsMessage.WsSDPOffer -> { // DO SOMETHING } } } } catch (e: Exception) { println(e.localizedMessage) } finally { println("Removing $thisConnection!") connections -= thisConnection } }

Slide 35

Slide 35 text

webSocket("/connect") { // Register this new client val thisConnection = Connection(this) connections += thisConnection try { while (true) { val wsPacket = receiveDeserialized() when (wsPacket.id) { WsMessage.WsInit, WsMessage.WsError, WsMessage.WsIceCandidates, WsMessage.WsIceServers, WsMessage.WsSDPAnswer, WsMessage.WsSDPOffer -> { // DO SOMETHING } } } } catch (e: Exception) { println(e.localizedMessage) } finally { println("Removing $thisConnection!") connections -= thisConnection } }

Slide 36

Slide 36 text

webSocket("/connect") { // Register this new client val thisConnection = Connection(this) connections += thisConnection try { while (true) { val wsPacket = receiveDeserialized() when (wsPacket.id) { WsMessage.WsInit, WsMessage.WsError, WsMessage.WsIceCandidates, WsMessage.WsIceServers, WsMessage.WsSDPAnswer, WsMessage.WsSDPOffer -> { // DO SOMETHING } } } } catch (e: Exception) { println(e.localizedMessage) } finally { println("Removing $thisConnection!") connections -= thisConnection } }

Slide 37

Slide 37 text

WsMessage.WsIceCandidates, WsMessage.WsIceServers, WsMessage.WsSDPAnswer, WsMessage.WsSDPOffer -> { connections .filter { connection -> connection.name != thisConnection.name } .forEach { it.session.send(wsPacket.data) } }

Slide 38

Slide 38 text

WebRTC Simple Example title> head>

WebRTC Connection h1> video> video> Start Call button> const signalingServerUrl = "ws: // localhost:8080/connect"; const signalingSocket = new WebSocket(signalingServerUrl); let localPeerConnection; let remotePeerConnection; let localStream; signalingSocket.onmessage = async (message) => { const data = JSON.parse(message.data); if (data.offer) { await handleOffer(data.offer); } else if (data.answer) {

Slide 39

Slide 39 text

β–ΆοΈŽ

Slide 40

Slide 40 text

πŸ§‘πŸ’» Android

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

πŸ’‘States

Slide 47

Slide 47 text

WebSocketDataSource

Slide 48

Slide 48 text

WebSocketDataSource WebRTCDataSource

Slide 49

Slide 49 text

ConnectionRepository WebSocketDataSource WebRTCDataSource

Slide 50

Slide 50 text

ConnectionRepository WebSocketDataSource WebRTCDataSource ViewModel

Slide 51

Slide 51 text

WebSocketDataSource

Slide 52

Slide 52 text

class WebSocketDataSource { private var session: DefaultClientWebSocketSession? = null private var pingJob: Job = SupervisorJob() private val client = HttpClient { install(Logging) { level = LogLevel.ALL } install(WebSockets) { contentConverter = KotlinxWebsocketSerializationConverter(Json) } } val webSocketDataSourceStream: Flow = receiveMessages() .onStart { openWebSocketSession() sendMessage(WSPacket(id = WsMessage.WsInit, data = "")) } private suspend fun openWebSocketSession() { session = client.webSocketSession( method = HttpMethod.Get, host = Environment.HOST, path = Environment.PATH, port = 8080 ) {

Slide 53

Slide 53 text

private suspend fun openWebSocketSession() { session = client.webSocketSession( method = HttpMethod.Get, host = Environment.HOST, path = Environment.PATH, port = 8080 ) { url.protocol = URLProtocol.WS } } private suspend fun closeWebSocketSession() { session ?. close() } fun close() { client.close() session = null } suspend fun sendMessage(wsPacket: WSPacket) { logger.d { "Sending: $wsPacket" } try { session ?. ensureActive() if (session == null) { logger.e { "Unable to send message since session is null. $wsPacket" } } else { ?.

Slide 54

Slide 54 text

suspend fun sendMessage(wsPacket: WSPacket) { logger.d { "Sending: $wsPacket" } try { session ?. ensureActive() if (session == null) { logger.e { "Unable to send message since session is null. $wsPacket" } } else { session ?. sendSerialized(wsPacket) } } catch (e: Exception) { logger.e(throwable = e.cause, message = { e.message.orEmpty() }) } } private fun receiveMessages(): Flow { return flow { emit(session !! ) } // This is needed because session is not initialized yet. .flatMapLatest { session -> session.incoming.consumeAsFlow() .map { session.converter !! .deserialize(it) } .onEach { logger.d { "Receiving: $it" } } } .timeout(TIMEOUT_DURATION) .catch { logger.i { "Timeout exception was triggered" } } .onCompletion { throw CancellationException("WebSocket connection terminated") } } }

Slide 55

Slide 55 text

ConnectionRepository WebSocketDataSource WebRTCDataSource ViewModel

Slide 56

Slide 56 text

WebRTCDataSource

Slide 57

Slide 57 text

class WebRTCDataSource( private val peerConnection: PeerConnection = PeerConnection(defaultRtcConfig), private var dataChannel: DataChannel? = null ) { private var peerConnectionJob: Job = SupervisorJob() private val coroutineScope: CoroutineScope = CoroutineScope(Dispatchers.Default) var onIceCandidateStream: Flow? = null var peerConnectionState: StateFlow? = null init { dataChannel = peerConnection.createDataChannel( label = "input", id = 0, ordered = true, maxRetransmitTimeMs = -1, maxRetransmits = -1, protocol = "", negotiated = false ) onIceCandidateStream = peerConnection.onIceCandidate peerConnectionState = peerConnection.onConnectionStateChange

Slide 58

Slide 58 text

onIceCandidateStream = peerConnection.onIceCandidate peerConnectionState = peerConnection.onConnectionStateChange .stateIn( scope = coroutineScope + peerConnectionJob, started = SharingStarted.WhileSubscribed(), initialValue = PeerConnectionState.New ) } val onMessageDataChannelStream = dataChannel ?. onMessage ?. shareIn( scope = coroutineScope + peerConnectionJob, started = SharingStarted.WhileSubscribed() ) fun sendCommand(key: NavKey, state: NavState) { logger.d { "sendCommand: ${key.name} ${state.name}" } val code = key.ordinal + state.ordinal.times(128) dataChannel ?. send(byteArrayOf(code.toByte())) } suspend fun createOffer(): SessionDescription = peerConnection.createOffer(defaultOfferAnswerOptions) .also { peerConnection.setLocalDescription(it) }

Slide 59

Slide 59 text

suspend fun createOffer(): SessionDescription = peerConnection.createOffer(defaultOfferAnswerOptions) .also { peerConnection.setLocalDescription(it) } suspend fun createAnswer(wsSDPAnswerData: WsSDPAnswerData) = SessionDescription( type = SessionDescriptionType.Answer, sdp = wsSDPAnswerData.sdp ).also { peerConnection.setRemoteDescription(it) } fun createRtcConfiguration(iceServers: List) = RtcConfiguration( iceServers = iceServers.map { iceServer -> IceServer( urls = iceServer.urls, password = iceServer.credential.orEmpty(), username = iceServer.username.orEmpty() ) } ) fun setConfiguration(configuration: RtcConfiguration) { peerConnection.setConfiguration(configuration) } fun addIceCandidates(iceCandidates: List) { iceCandidates.forEach { wsIceCandidate -> peerConnection.addIceCandidate( IceCandidate(

Slide 60

Slide 60 text

sdpMid = wsIceCandidate.sdpMid, sdpMLineIndex = wsIceCandidate.sdpMLineIndex, candidate = wsIceCandidate.candidate, ) ) } } fun closeWebRTCSession() { logger.i { "closeWebRTCConnection" } peerConnectionJob.cancel() dataChannel = null peerConnection.close() } companion object { val defaultRtcConfig = RtcConfiguration( iceServers = listOf( IceServer( urls = listOf( "stun:stun1.l.google.com:19302", "stun:stun2.l.google.com:19302" ) ), ) )

Slide 61

Slide 61 text

ConnectionRepository WebSocketDataSource WebRTCDataSource ViewModel

Slide 62

Slide 62 text

ConnectionRepository

Slide 63

Slide 63 text

class DefaultConnectionRepository( private val coroutineScope: CoroutineScope, private val webSocketDataSource: WebSocketDataSource, private val webRTCDataSource: WebRTCDataSource, var connectionInfo: ConnectionInfo, ) : ConnectionRepository { override val controllerState: StateFlow = combine( webRTCDataSource.peerConnectionState, webSocketDataSource.webSocketDataSourceStream, ) { peerConnectionState, packet -> when (packet.id) { WsMessage.WsError -> { val wsErrorCodeData: WsErrorCodeData = Json.decodeFromString(packet.data) PairingState.Failed(message = wsErrorCodeData.message) } else -> { when (peerConnectionState) { PeerConnectionState.New -> {

Slide 64

Slide 64 text

when (packet.id) { WsMessage.WsIceServers -> { val wsIceServersData = Json.decodeFromString(packet.data) val iceServers = wsIceServersData.iceServers val rtcConfigurations = webRTCDataSource.createRtcConfiguration(iceServers) webRTCDataSource.setConfiguration(rtcConfigurations) sendWsIceCandidate() val sessionDescription = webRTCDataSource.createOffer() webSocketDataSource.sendMessage( WSPacket( id = WsMessage.WsSDPOffer, data = Json.encodeToString( WsSDPOfferData( type = sessionDescription.type.name.lowercase(), sdp = sessionDescription.sdp, controllerId = connectionInfo.controllerId, )

Slide 65

Slide 65 text

PairingState.Connecting } WsMessage.WsIceCandidates -> { val wsIceCandidateDataList = Json.decodeFromString(packet.data) webRTCDataSource.addIceCandidates(wsIceCandidateDataList.iceCandidates) PairingState.Connecting } WsMessage.WsSDPAnswer -> { val wsSDPAnswerData = Json.decodeFromString(packet.data) webRTCDataSource.createAnswer(wsSDPAnswerData) PairingState.Connecting } else -> PairingState.Connecting } } PeerConnectionState.Connecting -> { when (packet.id) { WsMessage.WsSDPAnswer -> {

Slide 66

Slide 66 text

PeerConnectionState.Connecting -> { when (packet.id) { WsMessage.WsSDPAnswer -> { val wsSDPAnswerData = Json.decodeFromString(packet.data) webRTCDataSource.createAnswer(wsSDPAnswerData) PairingState.Connecting } WsMessage.WsIceCandidates -> { val wsIceCandidateDataList = Json.decodeFromString(packet.data) webRTCDataSource.addIceCandidates(wsIceCandidateDataList.iceCandidates) PairingState.Connecting } else -> PairingState.Connecting } } PeerConnectionState.Connected -> PairingState.Connected PeerConnectionState.Disconnected -> PairingState.Disconnected PeerConnectionState.Failed -> PairingState.Failed( message = "PeerConnectionState.Failed"

Slide 67

Slide 67 text

PeerConnectionState.Connected -> PairingState.Connected PeerConnectionState.Disconnected -> PairingState.Disconnected PeerConnectionState.Failed -> PairingState.Failed( message = "PeerConnectionState.Failed" ) PeerConnectionState.Closed -> PairingState.Closed } } } } .catch { emit(PairingState.Disconnected) } .onCompletion { webRTCDataSource.closeWebRTCSession() webSocketDataSource.close() } .stateIn( scope = coroutineScope, started = SharingStarted.WhileSubscribed(), initialValue = PairingState.New ) override suspend fun startSessions(connectionInfo: ConnectionInfo) {

Slide 68

Slide 68 text

private fun sendWsIceCandidate() { coroutineScope.launch { webRTCDataSource.onIceCandidateStream ?. collect { iceCandidate -> webSocketDataSource.sendMessage( WSPacket( id = WsMessage.WsIceCandidates, data = Json.encodeToString( WsIceCandidatesData( controllerId = connectionInfo.controllerId, iceCandidates = listOf( WsIceCandidateData( candidate = iceCandidate.candidate, sdpMLineIndex = iceCandidate.sdpMLineIndex, sdpMid = iceCandidate.sdpMid, ) ) ) ) ) ) } } } }

Slide 69

Slide 69 text

πŸ§‘πŸ’» Multiplatform

Slide 70

Slide 70 text

No content

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

@Composable expect fun Video( videoTrack: VideoStreamTrack, modifier: Modifier = Modifier, audioTrack: AudioStreamTrack? = null ) https: // github.com/shepeliev/webrtc-kmp

Slide 73

Slide 73 text

AndroidView( modifier = modifier, factory = { context -> SurfaceViewRenderer(context).apply { setScalingType( RendererCommon.ScalingType.SCALE_ASPECT_BALANCED, RendererCommon.ScalingType.SCALE_ASPECT_FIT ) renderer = this } }, ) https: // github.com/shepeliev/webrtc-kmp

Slide 74

Slide 74 text

@OptIn(ExperimentalForeignApi :: class) @Composable actual fun Video( videoTrack: VideoStreamTrack, modifier: Modifier, audioTrack: AudioStreamTrack?) { UIKitView( factory = { RTCMTLVideoView().apply { contentMode = UIViewContentMode.UIViewContentModeScaleAspectFit videoTrack.addRenderer(this) } }, modifier = modifier, ) } https: // github.com/shepeliev/webrtc-kmp

Slide 75

Slide 75 text

β–ΆοΈŽ

Slide 76

Slide 76 text

πŸ€” What is next?

Slide 77

Slide 77 text

πŸ‘¨βœˆ Selective Forwarding Unit Client A Client B

Slide 78

Slide 78 text

πŸ‘¨βœˆ Selective Forwarding Unit Client A Client B Client C

Slide 79

Slide 79 text

πŸ‘¨βœˆ Selective Forwarding Unit Client A Client B Client C Server PeerA β†’ SFU β†’ PeerB |β†’ PeerC PeerB β†’ SFU β†’ PeerA |β†’ PeerC PeerC β†’ SFU β†’ PeerA |β†’ PeerB

Slide 80

Slide 80 text

πŸ‘¨βœˆ Selective Forwarding Unit Server Client A Client B Client C Client D

Slide 81

Slide 81 text

πŸ‘¨βœˆ Selective Forwarding Unit Common technologies used with an SFU β€’ Simulcast β€’ Temporal scalability β€’ Insertable Streams β€’ Protocol Converter

Slide 82

Slide 82 text

What is next? Why WebRTC Companies Exist? β€’ Scalability and Performance β€’ Security β€’ Customization β€’ Easy to setup

Slide 83

Slide 83 text

What is next? Why WebRTC Companies Exist? β€’ GetStream β€’ LiveKit β€’ Dyte β€’ Zoom SDK β€’ Agora β€’ Vonage β€’ Twilio Video β€’ Mux β€’ Daily β€’ AWS Chime β€’ Whereby β€’ MirrorFly

Slide 84

Slide 84 text

What is next?

Slide 85

Slide 85 text

Thank you! X: @renaud_mathieu GitHub: @renaudmathieu