Connect your Android Things with Firebase (DevFestDC)

Slide 1

Slide 1 text

Connect your Android Things with Firebase Doug Stevenson @CodingDoug

Slide 2

Slide 2 text

@CodingDoug

Slide 3

Slide 3 text

@CodingDoug 3.5mm audio in/out RJ45 ethernet USB Type-A (host) USB Type-C (power & adb) PICO-PI-IMX7 — System on Module (SoM) • ARM Cortex-A7 @ 1GHz • 512 MB • 4GB eMMC storage WiFi 40-pin expansion headers Micro USB (debug) Camera pins Touch screen pins

Slide 4

Slide 4 text

@CodingDoug Rainbow Hat peripheral 3 capacitative buttons 3 LEDs RGB 14-segment display 7 multicolor LEDs Breakout pins for more peripherals Temp & pressure sensor Piezo buzzer

Slide 5

Slide 5 text

@CodingDoug General Purpose Input/Output (GPIO) ● Digital inputs and outputs with an on/off state. ● Buttons, relays, and proximity sensors. Pulse Width Modulation (PWM) ● Variable control of a peripheral level. ● Servo motors, speakers, LEDs Types of peripheral I/O

Slide 6

Slide 6 text

@CodingDoug Inter-Integrated Circuit (I2C) ● Synchronous master serial bus allowing multiple slave devices addressed in software. ● Sensors, displays, advanced peripherals Inter-IC Sound (I2S) ● Synchronous serial bus connecting digital sound peripherals that support PCM audio data. ● Digital microphones and digital-analog converters (DAC) Types of peripheral I/O

Slide 7

Slide 7 text

@CodingDoug Serial Peripheral Interface (SPI) ● Synchronous master serial bus allowing multiple slave devices addressed in hardware. ● Sensors, displays, higher speed peripherals Universal Asynchronous Receiver Transmitter (UART) ● Asynchronous serial port used commonly in interrupt-driven applications. ● GPS, printers, RFID readers, barcode scanners Types of peripheral I/O

Slide 8

Slide 8 text

@CodingDoug Android Things Drivers dependencies { compileOnly ‘com.google.android.things:androidthings:1.0' implementation 'com.google.android.things.contrib:driver-rainbowhat:0.10' implementation ‘com.google.android.things.contrib:driver-button:0.6' implementation 'com.google.android.things.contrib:driver-bmx280:0.5' implementation 'com.google.android.things.contrib:driver-ht16k33:0.5' implementation 'com.google.android.things.contrib:driver-apa102:0.6' implementation ‘com.google.android.things.contrib:driver-pwmspeaker:0.4' }

Slide 9

Slide 9 text

@CodingDoug

Slide 10

Slide 10 text

@CodingDoug

Slide 11

Slide 11 text

@CodingDoug

Slide 12

Slide 12 text

@CodingDoug

Slide 13

Slide 13 text

@CodingDoug

Slide 14

Slide 14 text

@CodingDoug

Slide 15

Slide 15 text

@CodingDoug

Slide 16

Slide 16 text

@CodingDoug

Slide 17

Slide 17 text

@CodingDoug @CodingDoug

Slide 18

Slide 18 text

@CodingDoug JavaScript/TypeScript - node.js - express.js

Slide 19

Slide 19 text

@CodingDoug

Slide 20

Slide 20 text

@CodingDoug

Slide 21

Slide 21 text

@CodingDoug 1 2 3 4

Slide 22

Slide 22 text

@CodingDoug Cloud Storage Cloud Functions Cloud Firestore Cloud Messaging Cloud Firestore Cloud Functions Cloud Messaging capture picture upload picture download picture trigger write ring send message notify share auth token write answer trigger send msg notify read ring

Slide 23

Slide 23 text

@CodingDoug You can use the Android Camera2 API! Official sample: goo.gl/mSDPm3 Issues I discovered: ● Camera preview renders to TextureView, but it’s slow ○ Development device has no GPU ○ Ported to use SurfaceView instead ● Removed code that depends on autofocus ○ Portable code should check to see if camera has autofocus Capturing the picture

Slide 24

Slide 24 text

@CodingDoug Cloud Storage capture picture upload picture

Slide 25

Slide 25 text

@CodingDoug Uploading to Cloud Storage private fun uploadFile(file: File) { val sdf = SimpleDateFormat("yyyyMMddHHmmss", Locale.US) val storagePath = "/pictures/${sdf.format(Date())}.jpg" val ref = FirebaseStorage.getInstance().getReference(storagePath) ref.putFile(Uri.fromFile(file)) .addOnSuccessListener(this) { Log.i(TAG, "Picture uploaded”) } .addOnFailureListener(this) { e -> Log.i(TAG, "Upload failed", e) } .addOnCompleteListener(this) { file.delete() } }

Slide 26

Slide 26 text

@CodingDoug

Slide 27

Slide 27 text

@CodingDoug Cloud Storage Cloud Functions Cloud Firestore capture picture upload picture trigger write ring

Slide 28

Slide 28 text

@CodingDoug Storage upload trigger pt. 1 - add document to Firestore export const onRing = functions.storage.object().onFinalize(_onRing) async function _onRing(object: functions.storage.ObjectMetadata): Promise { const path = object.name // "pictures/20180608120000.jpg" const id = basename(path, '.jpg') // "20180608120000" try { // Add a document to Firestore with the details of this ring // const ring: Ring = { id: id, date: new Date(), imagePath: path, } await firestore.collection('rings').doc(id).set(ring) // continued...

Slide 29

Slide 29 text

@CodingDoug @CodingDoug

Slide 30

Slide 30 text

@CodingDoug Cloud Storage Cloud Functions Cloud Firestore Cloud Messaging capture picture upload picture trigger write ring send message

Slide 31

Slide 31 text

@CodingDoug Storage upload trigger pt. 2 - send notification to app // Send a notification to the app // const payload = { notification: { title: 'Ring Ring!', body: 'There is someone at the door!', click_action: 'com.hyperaware.doorbell.ANSWER_RING' }, data: { ring_id: id } } const response = await fcm.sendToTopic('rings', payload)

Slide 32

Slide 32 text

@CodingDoug Cloud Storage Cloud Functions Cloud Firestore Cloud Messaging capture picture upload picture trigger write ring send message notify

Slide 33

Slide 33 text

@CodingDoug Receiving the notification Background? Display notification Invoke Firebase MessagingService onMessageReceived (RemoteMessage) Display Ring UI message received yes no notification click ring_id ring_id

Slide 34

Slide 34 text

@CodingDoug Earlier: App subscribes to “rings” topic class MyInstanceIdService : FirebaseInstanceIdService() { companion object { private const val TAG = "MyInstanceIdService" } override fun onTokenRefresh() { super.onTokenRefresh() Log.d(TAG, "FCM token refresh: ${FirebaseInstanceId.getInstance().token!!}") FirebaseMessaging.getInstance().subscribeToTopic("rings") } }

Slide 35

Slide 35 text

@CodingDoug App handles incoming ring data message class OnRingMessagingService : FirebaseMessagingService() { override fun onMessageReceived(remoteMessage: RemoteMessage) { super.onMessageReceived(remoteMessage) if (remoteMessage.data.containsKey("ring_id")) { val ringId = remoteMessage.data["ring_id"] val intent = Intent(this, AnswerRingActivity::class.java) intent.putExtra("ring_id", ringId) startActivity(intent) } else { Log.w(TAG, "Data message received without ring_id") } } }

Slide 36

Slide 36 text

@CodingDoug App handles incoming ring (AnswerRingActivity) override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) val extras = intent.extras if (extras == null) { Log.e(TAG, "ring_id was not provided") finish() return } val ringId = extras.getString("ring_id") if (ringId.isEmpty()) { Log.e(TAG, "ring_id was empty") finish() return } // display it... }

Slide 37

Slide 37 text

@CodingDoug @CodingDoug

Slide 38

Slide 38 text

@CodingDoug

Slide 39

Slide 39 text

@CodingDoug Cloud Storage Cloud Functions Cloud Firestore Cloud Messaging capture picture upload picture download picture trigger write ring send message notify read ring

Slide 40

Slide 40 text

@CodingDoug Fetch ring data from Firestore private fun populateViews(ringId: String) { ringReference = FirebaseFirestore.getInstance().collection("rings").document(ringId) ringReference.get() .addOnSuccessListener(this) { snap -> if (snap.exists()) { val ring = snap.toObject(Ring::class.java) val ref = FirebaseStorage.getInstance().getReference(ring.imagePath!!) Glide.with(this@AnswerRingActivity).load(ref).into(ivGuest) } } .addOnFailureListener(this) { error -> Log.e(TAG, "Can't fetch ring $ringId", error) } }

Slide 41

Slide 41 text

@CodingDoug Glide Module — Cloud Storage for Firebase plugin @GlideModule class MyAppGlideModule : AppGlideModule() { override fun registerComponents(context: Context, glide: Glide, registry: Registry) { // Register FirebaseImageLoader from FirebaseUI to handle StorageReference registry.append(StorageReference::class.java, InputStream::class.java, FirebaseImageLoader.Factory()) } }

Slide 42

Slide 42 text

@CodingDoug Cloud Storage Cloud Functions Cloud Firestore Cloud Messaging Cloud Firestore capture picture upload picture download picture trigger write ring send message notify write answer read ring

Slide 43

Slide 43 text

@CodingDoug Update ring disposition in Firestore val disposition = button_click_true_or_false ringReference.update( "answer.uid", uid, "answer.disposition", disposition) .addOnCompleteListener(this) { Log.d(TAG, "Answer written to database") finish() } .addOnFailureListener(this, { e -> Log.d(TAG, "Answer not written to database", e) finish() })

Slide 44

Slide 44 text

@CodingDoug @CodingDoug

Slide 45

Slide 45 text

Slide 46

Slide 46 text

@CodingDoug Firestore trigger pt. 1 — send answer to Android Thing export const onAnswer = functions.firestore.document('/rings/{ringId}').onUpdate(_onAnswer) async function _onAnswer(change: functions.Change): Promise { const ringId = change.before.id // "20180608120000" const previous = change.before.data() as Ring const ring = change.after.data() as Ring // Only interested in rings that have a new answer if (previous.answer || !ring.answer) { console.log("This is not the update you're looking for.") return Promise.resolve() } // cont’d...

Slide 47

Slide 47 text

@CodingDoug Firestore trigger pt. 2 — send answer to Android Thing const payload = { data: { disposition: ring.answer.disposition.toString(), ring_id: ringId } } try { const response = await fcm.sendToTopic('answers', payload) console.log(`ring ${ringId} answer sent:`, response) } catch (err) { console.error(`ring ${ringId} answer error:`, err) } }

Slide 48

Slide 48 text

Slide 49

Slide 49 text

@CodingDoug Earlier: Thing subscribes to “answers” topic class MyInstanceIdService : FirebaseInstanceIdService() { companion object { private const val TAG = "MyInstanceIdService" } override fun onTokenRefresh() { super.onTokenRefresh() Log.d(TAG, "FCM token refresh: ${FirebaseInstanceId.getInstance().token!!}") FirebaseMessaging.getInstance().subscribeToTopic("answers") } }

Slide 50

Slide 50 text

@CodingDoug Handle incoming answer data message class OnAnswerMessagingService : FirebaseMessagingService() { override fun onMessageReceived(remoteMessage: RemoteMessage) { super.onMessageReceived(remoteMessage) if (remoteMessage.data.containsKey("disposition")) { val d = java.lang.Boolean.parseBoolean(remoteMessage.data["disposition"]) val intent = Intent(this, ResponseActivity::class.java) intent.putExtra("disposition", d) startActivity(intent) } else { Log.w(TAG, "Data message received without disposition") } } }

Slide 51

Slide 51 text

Slide 52

Slide 52 text

@CodingDoug This is a RecyclerView FirestoreRecyclerAdapter makes this super easy! FirebaseUI FTW https://goo.gl/oQsk4h

Slide 53

Slide 53 text

@CodingDoug What about security?

Slide 54

Slide 54 text

Slide 55

Slide 55 text

@CodingDoug Cloud Storage security rules — universal read and write service firebase.storage { match /b/{bucket}/o { match /{allPaths=**} { allow read, write: if true; } } }

Slide 56

Slide 56 text

@CodingDoug Firestore security rules — universal read and write service cloud.firestore { match /databases/{database}/documents { match /{document=**} { allow read, write: if true; } } }

Slide 57

Slide 57 text

@CodingDoug ● Yeah, it’s easy to get started without security ● All reads and writes should require user authentication (minimally) ● Also consider data validation rules ● For Storage: https://firebase.google.com/docs/storage/security/  For Firestore: https://firebase.google.com/docs/firestore/security/get-started What to do about security?

Slide 58

Slide 58 text

@CodingDoug Authentication login flows are hard (and boring)

Slide 59

Slide 59 text

@CodingDoug implementation "com.firebaseui:firebase-ui-auth:$firebase_ui_version" Authentication is easy (with FirebaseUI)

Slide 60

Slide 60 text

@CodingDoug App login with FirebaseUI — launch UI flow findViewById(R.id.btn_sign_in).setOnClickListener { startActivityForResult( AuthUI.getInstance() .createSignInIntentBuilder() .setAvailableProviders(listOf(AuthUI.IdpConfig.GoogleBuilder().build())) .build(), RC_SIGN_IN) }

Slide 61

Slide 61 text

@CodingDoug

Slide 62

Slide 62 text

@CodingDoug App login with FirebaseUI — handle sign in results override fun onActivityResult(requestCode: Int, resultCode: Int, data: Intent) { super.onActivityResult(requestCode, resultCode, data) if (requestCode == RC_SIGN_IN) { val response = IdpResponse.fromResultIntent(data) if (resultCode == Activity.RESULT_OK) { // handle login } } }

Slide 63

Slide 63 text

@CodingDoug ● UI for various provider logins aren’t supported on Android Things ● Android Thing may not even have a screen! ● Anonymous login works, not a great solution ● Best to get proper user credentials for a real login Phone apps are easy, but how does a Thing get signed in?

Slide 64

Slide 64 text

@CodingDoug Share an auth credential from app to Thing Authentication ID token But how? R/W protected data

Slide 65

Slide 65 text

@CodingDoug Nearby API https://developers.google.com/nearby/

Slide 66

Slide 66 text

@CodingDoug ● Peer-to-peer pub/sub messaging model ● Device pairing via combo of Wifi, BT, BLE, near-ultrasonic radio ● Data payload exchange via Google Cloud project - requires internet Nearby Messages

Slide 67

Slide 67 text

@CodingDoug ● Peer-to-peer networking, high bandwidth, low latency ● Direct data exchange via Wifi, BT, BLE - no internet required Nearby Connections

Slide 68

Slide 68 text

@CodingDoug Publish a Nearby Message pt. 1 — configuration val strategy = Strategy.Builder() .setDiscoveryMode(Strategy.DISCOVERY_MODE_BROADCAST) .setTtlSeconds(Strategy.TTL_SECONDS_MAX) .build() val publishOpts = PublishOptions.Builder() .setStrategy(strategy) .setCallback(object : PublishCallback() { override fun onExpired() { Log.d(TAG, "onExpired") } }) .build()

Slide 69

Slide 69 text

@CodingDoug Publish a Nearby Message pt. 2 — publish val client = Nearby.getMessagesClient(this) val message = Message("Hello, Firebase Thing!") client.publish(message, publishOpts) .addOnSuccessListener(this) { Log.e(TAG, "publish success") } .addOnFailureListener(this) { e -> Log.e(TAG, "publish failed", e) }

Slide 70

Slide 70 text

@CodingDoug Subscribe to a Nearby Message pt. 1 — configuration val strategy = Strategy.Builder() .setDiscoveryMode(Strategy.DISCOVERY_MODE_SCAN) .setTtlSeconds(Strategy.TTL_SECONDS_MAX) .build() val subscribeOpts = SubscribeOptions.Builder() .setStrategy(strategy) .setCallback(object : SubscribeCallback() { override fun onExpired() { Log.d(TAG, "onExpired") } }) .build()

Slide 71

Slide 71 text

@CodingDoug Subscribe to a Nearby Message pt. 2 — subscribe val client = Nearby.getMessagesClient(this) private val messageListener = object : MessageListener() { override fun onFound(message: Message) { // messsage.content contains payload } override fun onLost(message: Message) {} } client.subscribe(messageListener, subscribeOpts) .addOnSuccessListener(this) { Log.d(TAG, "subscribe success") } .addOnFailureListener(this) { e -> Log.e(TAG, "subscribe failure", e) }

Slide 72

Slide 72 text

@CodingDoug Log in Thing with Google and Firebase APIs private fun trySignIn() { Log.d(TAG, "Signing in with token " + token) val credential = GoogleAuthProvider.getCredential(token, null) FirebaseAuth.getInstance().signInWithCredential(credential) .addOnSuccessListener(this, { result -> val user = result.user Log.d(TAG, "signInWithCredential ${user.displayName} ${user.email}") finish() }) .addOnFailureListener(this, { e -> Log.e(TAG, "signInWithCredential onFailure", e) }) }

Slide 73

Slide 73 text

@CodingDoug Too much code to show here! 1. Both: Check for ACCESS_COARSE_LOCATION permission 2. Thing: Begin “advertising” with P2P_CLUSTER strategy 3. App: Begin “discovering” with P2P_CLUSTER strategy 4. Both: Connect to peer 5. Both: Send/receive Google Auth token 6. Both: Disconnect 7. Thing: Sign in with token using Google Auth and Firebase Auth APIs Use Nearby Connections to share an auth token

Slide 74

Slide 74 text

Slide 75

Slide 75 text

@CodingDoug What can be improved?

Slide 76

Slide 76 text

@CodingDoug ● Code cleanup ● Better UI ● Productization - currently only good for hobbyists ○ Can't expect a typical customer to manage a Firebase project ○ Currently no way to programmatically create a project ○ Restructure Firestore and Storage for multi-customer tenancy ○ Can’t use FCM topics securely - need to use device tokens This project needs…

Slide 77

Slide 77 text

@CodingDoug ● Better security ○ Allow users to confirm auth tokens received from Nearby (like BT pairing) ○ Tighter security rules for both Storage and Firestore This project needs…

Slide 78

Slide 78 text

@CodingDoug What features can be added?

Slide 79

Slide 79 text

@CodingDoug ● Facial sentiment detection via Cloud Functions ● Voice intercom (AudioTrack, AudioRecord) ○ Add speech to text with Google Cloud Speech API ● “Digital keys” ○ Guest app ○ QR codes Some ideas

Slide 80

Slide 80 text

Thank you! @CodingDoug https://github.com/CodingDoug/firebase-doorbell