Audio: Characters in this framework 12 ৄղ Android Auto DroidKaigi 2018 Apps providing media contents Apps connecting to media app (incl. Android Auto and others) Android system service
ৄղ Android Auto Create MediaController from Token class MusicActivity : Activity() { var service: MusicService? = null var bound: Boolean = false val conn = object : ServiceConnection() { override fun onServiceConnected( cn: ComponentName, s: IBinder) { val binder = s as MusicBinder service = binder.getService() bound = true } } } 36 DroidKaigi 2018
ৄղ Android Auto Create MediaController from Token class MusicActivity : Activity() { var service: MusicService? = null var bound: Boolean = false val conn = object : ServiceConnection() { override fun onServiceConnected( cn: ComponentName, s: IBinder) { val binder = s as MusicBinder service = binder.getService() bound = true } } } 37 DroidKaigi 2018
ৄղ Android Auto Create MediaController from Token class MusicActivity : Activity() { var service: MusicService? = null var bound: Boolean = false val conn = object : ServiceConnection() { override fun onServiceConnected( cn: ComponentName, s: IBinder) { val binder = s as MusicBinder service = binder.getService() bound = true } } } 38 DroidKaigi 2018
ৄղ Android Auto Create MediaController from Token class MusicActivity : Activity() { var service: MusicService? = null var bound: Boolean = false val conn = object : ServiceConnection() { override fun onServiceConnected( cn: ComponentName, s: IBinder) { val binder = s as MusicBinder service = binder.getService() bound = true } } } 39 DroidKaigi 2018
ৄղ Android Auto Create MediaController from Token class MusicActivity : Activity() { var service: MusicService? = null var bound: Boolean = false val conn = object : ServiceConnection() { override fun onServiceConnected( cn: ComponentName, s: IBinder) { val binder = s as MusicBinder service = binder.getService() bound = true } } } 40 DroidKaigi 2018
ৄղ Android Auto Playback control via MC.TransportControls class MusicActivity : Activity() { var mediaController: MediaController? // … fun onPlayClick() { mediaController.transportControls.play() } fun onPauseClick() { mediaController.transportControls.pause() } } 45 DroidKaigi 2018
ৄղ Android Auto Playback control via MC.TransportControls class MusicActivity : Activity() { var mediaController: MediaController? // … fun onPlayClick() { mediaController.transportControls.play() } fun onPauseClick() { mediaController.transportControls.pause() } } 46 DroidKaigi 2018
What if the media app is dead…? 59 ৄղ Android Auto DroidKaigi 2018 MediaSession MediaSession.Token ISessionController MediaController ISessionController TransportControls play(), pause(), … play(), pause(), Process is dead!
When remote process is dead 61 ৄղ Android Auto DroidKaigi 2018 MediaSession MediaSession.Token ISessionController MediaController ISessionController Process is dead!
When remote process is dead 62 ৄղ Android Auto DroidKaigi 2018 MediaSession MediaSession.Token ISessionController MediaController ISessionController Process is dead! binderDied() ⇣ onDestroy()
When remote process is dead 63 ৄղ Android Auto DroidKaigi 2018 MediaSession MediaSession.Token ISessionController MediaController ISessionController Process is dead! Callback onSessionDestroy()
IBinder.DeathRecipient vs ServiceConnection ▸ IBinder.DeathRecipient#binderDied ▸ Monitor remote process death ▸ ServiceConnection#onServiceDisconnected ▸ Called from IBinder.DeathRecipient#binderDied ▸ Only called when service is bound to remote 67 ৄղ Android Auto DroidKaigi 2018
ৄղ Android Auto Get notified from media app class MusicActivity : Activity() { var service: MusicService? = null var mediaController: MediaController? = null val callback: MediaController.Callback = //… fun buildMediaController() { ɹ service?.let { mediaController = // … mediaController.registerCallback(callback) } } } 68 DroidKaigi 2018
ৄղ Android Auto Get notified from media app class MusicActivity : Activity() { var service: MusicService? = null var mediaController: MediaController? = null val callback: MediaController.Callback = object : MediaController.Callback() { fun onSessionDestroy() { // no more interaction from here! } } } 69 DroidKaigi 2018
ৄղ Android Auto Get notified from media app class MusicActivity : Activity() { var service: MusicService? = null var mediaController: MediaController? = null val callback: MediaController.Callback = object : MediaController.Callback() { fun onPlaybackStateChanged( state: PlaybackStateCompat) { // get new playback state! } } } 70 DroidKaigi 2018
ৄղ Android Auto Get notified from media app class MusicActivity : Activity() { var service: MusicService? = null var mediaController: MediaController? = null val callback: MediaController.Callback = object : MediaController.Callback() { fun onMetadataChanged( data: MediaMetadataCompat) { // get new media information! } } } 71 DroidKaigi 2018
Playback control IPC on Lock Screen in 4.x 73 ৄղ Android Auto DroidKaigi 2018 RemoteControlClient AudioManager AudioService RemoteControlDisplay LockScreen
Playback control IPC on Lock Screen in 4.x 74 ৄղ Android Auto DroidKaigi 2018 RemoteControlClient AudioManager AudioService RemoteControlDisplay LockScreen
Playback control IPC on Lock Screen in 4.4 75 ৄղ Android Auto DroidKaigi 2018 RemoteControlClient AudioManager AudioService RemoteController LockScreen NotificationListener Service
Playback control IPC on Lock Screen in 5+ 76 ৄղ Android Auto DroidKaigi 2018 MediaSession NotificationManager Service MediaController LockScreen MediaStyle Notification NotificationListener Service
Playback control IPC on Lock Screen in 5+ 77 ৄղ Android Auto DroidKaigi 2018 MediaSession NotificationManager Service MediaController LockScreen MediaStyle Notification NotificationListener Service
Get active sessions for the package 78 ৄղ Android Auto DroidKaigi 2018 MediaSessionManager MediaSessionService MediaSessionRecord MediaSessionRecord …… getActiveSessions()
Get active sessions for the package 79 ৄղ Android Auto DroidKaigi 2018 MediaSessionManager MediaSessionService MediaSessionRecord MediaSessionRecord …… getSessions()
Get active sessions for the package 80 ৄղ Android Auto DroidKaigi 2018 MediaSessionManager MediaSessionService MediaSessionRecord MediaSessionRecord …… ISessionController
Get active sessions for the package 81 ৄղ Android Auto DroidKaigi 2018 MediaSessionManager MediaSessionService MediaSessionRecord MediaSessionRecord …… MediaController
ৄղ Android Auto Get notified from media app class MusicActivity : Activity() { override fun onCreate(savedState: Bundle) { super.onCreate(savedState) val manager: MediaSessionManager = context.getSystemService( MEDIA_SESSION_SERVICE) val controllers: List = manager.getActiveSessions(null) // do whatever you want with media controller… } } 82 DroidKaigi 2018
ৄղ Android Auto Get notified from media app class MusicActivity : Activity() { override fun onCreate(savedState: Bundle) { super.onCreate(savedState) val manager: MediaSessionManager = context.getSystemService( MEDIA_SESSION_SERVICE) val controllers: List = manager.getActiveSessions(null) // do whatever you want with media controller… } } 83 DroidKaigi 2018
ৄղ Android Auto Get notified from media app class MusicActivity : Activity() { override fun onCreate(savedState: Bundle) { super.onCreate(savedState) val manager: MediaSessionManager = context.getSystemService( MEDIA_SESSION_SERVICE) val controllers: List = manager.getActiveSessions(null) // do whatever you want with media controller… } } 84 DroidKaigi 2018
ৄղ Android Auto Get notified from media app class MusicActivity : Activity() { override fun onCreate(savedState: Bundle) { super.onCreate(savedState) val manager: MediaSessionManager = context.getSystemService( MEDIA_SESSION_SERVICE) val controllers: List = manager.getActiveSessions(null) // do whatever you want with media controller… } } 85 DroidKaigi 2018
ৄղ Android Auto Connect to MediaBrowserService class MusicActivity : Activity() { lateinit var browser: MediaBrowserCompat val cb = object : MBC.ConnectionCallback { override fun onConnected() { // you can start subscribing/searching // media contents after this callback! } override fun onConnectionFailed() { // cannot connect to remote service! } } } 93 DroidKaigi 2018
ৄղ Android Auto Connect to MediaBrowserService class MusicActivity : Activity() { lateinit var browser: MediaBrowserCompat val cb = object : MBC.ConnectionCallback { override fun onConnected() { // you can start subscribing/searching // media contents after this callback! } override fun onConnectionFailed() { // cannot connect to remote service! } } } 94 DroidKaigi 2018
ৄղ Android Auto Connect to MediaBrowserService class MusicActivity : Activity() { lateinit var browser: MediaBrowserCompat val cb = object : MBC.ConnectionCallback { override fun onConnected() { // you can start subscribing/searching // media contents after this callback! } override fun onConnectionFailed() { // cannot connect to remote service! } } } 95 DroidKaigi 2018
Connection verification by 97 ৄղ Android Auto ✔Verify signature of the client ✔Verify the client package name DroidKaigi 2018 ✔Verify the client process user id
ৄղ Android Auto Load top level items class MusicActivity : Activity() { lateinit var browser: MediaBrowserCompat val scb = object : MBC.SubscriptionCallback() { override fun onChildrenLoaded( parentId: String, list: List) { // show media list under the parentId item } } } 117 DroidKaigi 2018
ৄղ Android Auto Load top level items class MusicActivity : Activity() { lateinit var browser: MediaBrowserCompat val scb = object : MBC.SubscriptionCallback() { override fun onChildrenLoaded( parentId: String, list: List) { // show media list under the parentId item } } } 118 DroidKaigi 2018
ৄղ Android Auto On select item: Load the next list 119 DroidKaigi 2018 __ROOT__ __FAVORITE__ __RECOMMEND__ __SUBSCRIPTION__ __FAVORITE__/1 __FAVORITE__/2 MediaItem BROWSABLE
ৄղ Android Auto Load media items synchronously class BrowserService : MediaBrowserServiceCompat() { override fun onLoadChildren( parentId: String, result: Result>) { val list: List = // load media items result.sendResult(list) } } 121 DroidKaigi 2018
ৄղ Android Auto Load media items synchronously class BrowserService : MediaBrowserServiceCompat() { override fun onLoadChildren( parentId: String, result: Result>) { val list: List = // load media items result.sendResult(list) } } 122 DroidKaigi 2018
ৄղ Android Auto Handle item selection class MusicActivity : Activity() { lateinit var browser: MediaBrowserCompat val scb: MBC.SubscriptionCallback = // … fun onItemSelected(item: MediaItem) { if (item.isBrowsable()) { // start loading more item under the item } else if (item.isPlayable() { // start playing item } } } 126 DroidKaigi 2018
ৄղ Android Auto Handle item selection class MusicActivity : Activity() { lateinit var browser: MediaBrowserCompat val scb: MBC.SubscriptionCallback = // … fun onItemSelected(item: MediaItem) { if (item.isBrowsable()) { // start loading more item under the item } else if (item.isPlayable() { // start playing item } } } 127 DroidKaigi 2018
ৄղ Android Auto Handle item selection: Load more! class MusicActivity : Activity() { lateinit var browser: MediaBrowserCompat val scb: MBC.SubscriptionCallback = // … fun onItemSelected(item: MediaItem) { if (item.isBrowsable()) { // start loading more item under the item } else if (item.isPlayable() { // start playing item } } } 128 DroidKaigi 2018
ৄղ Android Auto Handle item selection: Start playing! class MusicActivity : Activity() { lateinit var browser: MediaBrowserCompat val scb: MBC.SubscriptionCallback = // … fun onItemSelected(item: MediaItem) { if (item.isBrowsable()) { // start loading more item under the item } else if (item.isPlayable() { // start playing item } } } 129 DroidKaigi 2018
Wrap up: Audio Feature Framework 141 ৄղ Android Auto DroidKaigi 2018 MediaBrowserService MediaBrowser MediaController MediaSession MediaPlayer Provide media info and token
Messaging: Characters in this framework 147 ৄղ Android Auto DroidKaigi 2018 Messenger apps Apps listening to incoming messages (incl. Android Auto and others) Android system service
Message Notification for Android Auto 151 ৄղ Android Auto DroidKaigi 2018 NotificationCompat.CarExtender Visual settings(color and icon) Unread/Read status Voice input for reply
ৄղ Android Auto Building Notification for Android Auto val n = NotificationCompat.Builder(context) … .extend(CarExtender() .setColor(R.color.car_notification) .setLargeIcon(iconBitmap) .setUnreadConversation( CarExtender.UnreadConversation.Builder(“test”) .addMessage(“This is a message.”) .setReadPendingIntent(readPi) .setReplyAction(replyPi, remoteInput) .setLatestTimestamp(timestamp) .build()) ).build() 152 DroidKaigi 2018
ৄղ Android Auto Building Notification for Android Auto val n = NotificationCompat.Builder(context) … .extend(CarExtender() .setColor(R.color.car_notification) .setLargeIcon(iconBitmap) .setUnreadConversation( CarExtender.UnreadConversation.Builder(“test”) .addMessage(“This is a message.”) .setReadPendingIntent(readPi) .setReplyAction(replyPi, remoteInput) .setLatestTimestamp(timestamp) .build()) ).build() 153 DroidKaigi 2018
ৄղ Android Auto Building Notification for Android Auto val n = NotificationCompat.Builder(context) … .extend(CarExtender() .setColor(R.color.car_notification) .setLargeIcon(iconBitmap) .setUnreadConversation( CarExtender.UnreadConversation.Builder(“test”) .addMessage(“This is a message.”) .setReadPendingIntent(readPi) .setReplyAction(replyPi, remoteInput) .setLatestTimestamp(timestamp) .build()) ).build() 154 DroidKaigi 2018
ৄղ Android Auto Building Notification for Android Auto val n = NotificationCompat.Builder(context) … .extend(CarExtender() .setColor(R.color.car_notification) .setLargeIcon(iconBitmap) .setUnreadConversation( CarExtender.UnreadConversation.Builder(“test”) .addMessage(“This is a message.”) .setReadPendingIntent(readPi) .setReplyAction(replyPi, remoteInput) .setLatestTimestamp(timestamp) .build()) ).build() 155 DroidKaigi 2018
ৄղ Android Auto Building Notification for Android Auto val n = NotificationCompat.Builder(context) … .extend(CarExtender() .setColor(R.color.car_notification) .setLargeIcon(iconBitmap) .setUnreadConversation( CarExtender.UnreadConversation.Builder(“test”) .addMessage(“This is a message.”) .setReadPendingIntent(readPi) .setReplyAction(replyPi, remoteInput) .setLatestTimestamp(timestamp) .build()) ).build() 156 DroidKaigi 2018
ৄղ Android Auto Building Notification for Android Auto val n = NotificationCompat.Builder(context) … .extend(CarExtender() .setColor(R.color.car_notification) .setLargeIcon(iconBitmap) .setUnreadConversation( CarExtender.UnreadConversation.Builder(“test”) .addMessage(“This is a message.”) .setReadPendingIntent(readPi) .setReplyAction(replyPi, remoteInput) .setLatestTimestamp(timestamp) .build()) ).build() 157 DroidKaigi 2018
ৄղ Android Auto Building Notification for Android Auto val n = NotificationCompat.Builder(context) … .extend(CarExtender() .setColor(R.color.car_notification) .setLargeIcon(iconBitmap) .setUnreadConversation( CarExtender.UnreadConversation.Builder(“test”) .addMessage(“This is a message.”) .setReadPendingIntent(readPi) .setReplyAction(replyPi, remoteInput) .setLatestTimestamp(timestamp) .build()) ).build() 158 DroidKaigi 2018
ৄղ Android Auto Building Notification for Android Auto val n = NotificationCompat.Builder(context) … .extend(CarExtender() .setColor(R.color.car_notification) .setLargeIcon(iconBitmap) .setUnreadConversation( CarExtender.UnreadConversation.Builder(“test”) .addMessage(“This is a message.”) .setReadPendingIntent(readPi) .setReplyAction(replyPi, remoteInput) .setLatestTimestamp(timestamp) .build()) ).build() 159 DroidKaigi 2018
ৄղ Android Auto Building Notification for Android Auto val n = NotificationCompat.Builder(context) … .extend(CarExtender() .setColor(R.color.car_notification) .setLargeIcon(iconBitmap) .setUnreadConversation( CarExtender.UnreadConversation.Builder(“test”) .addMessage(“This is a message.”) .setReadPendingIntent(readPi) .setReplyAction(replyPi, remoteInput) .setLatestTimestamp(timestamp) .build()) ).build() 160 DroidKaigi 2018
ৄղ Android Auto Read/Unread status update val conversationId: Int = //ϝοηʔδͷΓऔΓΛද͢ID val intent = Intent(“your.app.action.READ_MESSAGE”) .addFlags(Intent.FLAG_INCLUDE_STOPPED_PACKAGE) .putExtra(“conversation_id”, conversationId); val readPi = PendingIntent.getBroadcast( applicationContext, conversationId, intent, PendingIntent.FLAG_UPDATE_CURRENT) 161 DroidKaigi 2018
ৄղ Android Auto Read/Unread status update val conversationId: Int = //ϝοηʔδͷΓऔΓΛද͢ID val intent = Intent(“your.app.action.READ_MESSAGE”) .addFlags(Intent.FLAG_INCLUDE_STOPPED_PACKAGE) .putExtra(“conversation_id”, conversationId); val readPi = PendingIntent.getBroadcast( applicationContext, conversationId, intent, PendingIntent.FLAG_UPDATE_CURRENT) 162 DroidKaigi 2018
ৄղ Android Auto Read/Unread status update val conversationId: Int = //ϝοηʔδͷΓऔΓΛද͢ID val intent = Intent(“your.app.action.READ_MESSAGE”) .addFlags(Intent.FLAG_INCLUDE_STOPPED_PACKAGE) .putExtra(“conversation_id”, conversationId); val readPi = PendingIntent.getBroadcast( applicationContext, conversationId, intent, PendingIntent.FLAG_UPDATE_CURRENT) 163 DroidKaigi 2018
ৄղ Android Auto Read/Unread status update val conversationId: Int = //ϝοηʔδͷΓऔΓΛද͢ID val intent = Intent(“your.app.action.READ_MESSAGE”) .addFlags(Intent.FLAG_INCLUDE_STOPPED_PACKAGE) .putExtra(“conversation_id”, conversationId); val readPi = PendingIntent.getBroadcast( applicationContext, conversationId, intent, PendingIntent.FLAG_UPDATE_CURRENT) 164 DroidKaigi 2018
ৄղ Android Auto Reply by voice input in Android Auto val conversationId: Int = //ϝοηʔδͷΓऔΓΛද͢ID val input = RemoteInput.Builder(“key_voice_reply”) … val intent = Intent(“your.app.action.REPLY_MESSAGE”) … val replyPi = PendingIntent.getBroadcast( applicationContext, conversationId, intent, PendingIntent.FLAG_UPDATE_CURRENT) 165 DroidKaigi 2018
ৄղ Android Auto Reply by voice input in Android Auto val conversationId: Int = //ϝοηʔδͷΓऔΓΛද͢ID val input = RemoteInput.Builder(“key_voice_reply”) … val intent = Intent(“your.app.action.REPLY_MESSAGE”) … val replyPi = PendingIntent.getBroadcast( applicationContext, conversationId, intent, PendingIntent.FLAG_UPDATE_CURRENT) 166 DroidKaigi 2018
ৄղ Android Auto Reply by voice input in Android Auto val conversationId: Int = //ϝοηʔδͷΓऔΓΛද͢ID val input = RemoteInput.Builder(“key_voice_reply”) .setLabel(“Reply by voice”) .build() val intent = Intent(“your.app.action.REPLY_MESSAGE”) … val replyPi = PendingIntent.getBroadcast( applicationContext, conversationId, intent, PendingIntent.FLAG_UPDATE_CURRENT) 167 DroidKaigi 2018
ৄղ Android Auto Reply by voice input in Android Auto val conversationId: Int = //ϝοηʔδͷΓऔΓΛද͢ID val input = RemoteInput.Builder(“key_voice_reply”) … val intent = Intent(“your.app.action.REPLY_MESSAGE”) .addFlags(Intent.FLAG_INCLUDE_STOPPED_PACKAGES) .putExtra(“conversation_id”, conversationId) val replyPi = PendingIntent.getBroadcast( applicationContext, conversationId, intent, PendingIntent.FLAG_UPDATE_CURRENT) 168 DroidKaigi 2018
ৄղ Android Auto Reply by voice input in Android Auto val conversationId: Int = //ϝοηʔδͷΓऔΓΛද͢ID val input = RemoteInput.Builder(“key_voice_reply”) … val intent = Intent(“your.app.action.REPLY_MESSAGE”) … val replyPi = PendingIntent.getBroadcast( applicationContext, conversationId, intent, PendingIntent.FLAG_UPDATE_CURRENT) 169 DroidKaigi 2018
ৄղ Android Auto Receive voice input result from Android Auto class ReplyReceiver : BroadcastReceiver() { override fun onReceive(c: Context, i: Intent) { val conversationId = i.getIntExtra(“conversation_id”) val result = RemoteInput.getResultsFromIntent(i) result?.let { val text = it.getCharSequence(“key_voice_reply”) … // sending reply } } } 170 DroidKaigi 2018
ৄղ Android Auto Receive voice input result from Android Auto class ReplyReceiver : BroadcastReceiver() { override fun onReceive(c: Context, i: Intent) { val conversationId = i.getIntExtra(“conversation_id”) val result = RemoteInput.getResultsFromIntent(i) result?.let { val text = it.getCharSequence(“key_voice_reply”) … // sending reply } } } 171 DroidKaigi 2018
ৄղ Android Auto Receive voice input result from Android Auto class ReplyReceiver : BroadcastReceiver() { override fun onReceive(c: Context, i: Intent) { val conversationId = i.getIntExtra(“conversation_id”) val result = RemoteInput.getResultsFromIntent(i) result?.let { val text = it.getCharSequence(“key_voice_reply”) … // sending reply } } } 172 DroidKaigi 2018
ৄղ Android Auto Get UnreadConversation data class MyListener : NotificationListenerService { override fun onNotificationPosted( sbn: StatusBarNotification) { val conv: Bundle = // …… // Array of Bundles containing messages val messages: Array = conv.getParcelableArray(“messages”) } } 190 DroidKaigi 2018
ৄղ Android Auto Reading incoming message texts class MyListener : NotificationListenerService { override fun onNotificationPosted( sbn: StatusBarNotification) { val messages: Array = // …… val list: List = messages.filter { it is Bundle }.map { (it as Bundle).getString(“text”) } } } 194 DroidKaigi 2018
ৄղ Android Auto Reading incoming message texts class MyListener : NotificationListenerService { override fun onNotificationPosted( sbn: StatusBarNotification) { val messages: Array = // …… val list: List = messages.filter { it is Bundle }.map { (it as Bundle).getString(“text”) } } } 195 DroidKaigi 2018
ৄղ Android Auto Reading incoming message texts class MyListener : NotificationListenerService { override fun onNotificationPosted( sbn: StatusBarNotification) { val messages: Array = // …… val list: List = messages.filter { it is Bundle }.map { (it as Bundle).getString(“text”) } } } 196 DroidKaigi 2018
Wrap up: Behind the scenes of Android Auto ▸ Interprocess Communication(IPC) ▸ The Binder ▸ Android Auto is based on the wrapped classes ▸ MediaSession and MediaController ▸ MediaBrowser and MediaBrowserService ▸ Google Assistant is also based on these classes! 225 ৄղ Android Auto DroidKaigi 2018
Wrap up: References ▸ More about IPC in Android ▸ Inside Binder ▸ https://www.slideshare.net/l_b__/binderandroid ▸ Deep Dive into Android IPC/Binder Framework ▸ Android Binder 226 ৄղ Android Auto DroidKaigi 2018
Wrap up: References ▸ API Docs: Media features ▸ Notification.MediaStyle ▸ MediaBrowser and MediaBrowserService ▸ MediaSession and MediaController ▸ MediaSessionManager 228 ৄղ Android Auto DroidKaigi 2018
keithyokoma
yokawasa
hhiroshell
masasuzu
yamamuteki
comucal
cimadai
yutoy
toshi0383
peisuke
yuya4
minorun365
yoshimi0227
lynnandtonic
tammyeverts
brianwarren
colly
addyosmani
eitanlees
myddelton
rasmusluckow
searls
deanohume
jonrohan
mojombo