Slide 1

Slide 1 text

DevFest NYC - Android Things Workshop Build a Motion Sensing Camera Rebecca Franks Android Engineering Lead at DVT @riggaroo

Slide 2

Slide 2 text

Joshua Leibstein GDG Johannesburg Organiser @JoshLiebe Oscar Salguero @OscarSalguero

Slide 3

Slide 3 text

Slides bit.ly/nyc-at-slides Workshop Feedback bit.ly/nyc-at-workshop

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

What we will build: Intro: 1. Create project and read physical button value 2. Read motion values from Motion Sensor 3. Turn LED on/off based on Motion Sensor 4. Take photo and display when Motion is detected Further reading - advanced section: 5. Upload information to Firebase (photo + log) 6. Send Push Notification 7. View list of photos

Slide 6

Slide 6 text

Android Things Theory

Slide 7

Slide 7 text

What is Android Things? Android Things is an extension of the Android platform for IoT and embedded devices.

Slide 8

Slide 8 text

Interactive Ads Vending Machines Point of Sale Stock Control Retail Cameras Gateways Access Control Smart Meters Business Asset Tracking Fleet Management Driver Assist Predictive Service Logistics Security Systems Smart Doorbells Routers Energy Monitors Home Android Things is ideal for powerful, intelligent devices that need to be secure.

Slide 9

Slide 9 text

Similarities Any other Android library... Android SDK Play Services Firebase Android Studio Cloud Platform

Slide 10

Slide 10 text

Differences No Play Store Deploy OTAs Subset of APIs Custom Hardware Single Purpose Device

Slide 11

Slide 11 text

Applications Launcher Phone Messaging Contacts Calendar Browser Settings Application Framework Activity Manager Window Manager Power Manager Resource Manager XMPP Service Content Providers Wallpapers System UI Package Manager Telephony Manager Location Manager Connectivity Manager View System Runtime Permissions Soft Keyboards Notifications Libraries Surface Manager Media Framework Chromium SSL HAL Audio Manager SQLite Open GL libc Core Libraries Android Runtime (ART) Android Runtime Linux Kernel Display Driver Camera Driver Bluetooth Driver Binder (IPC) Driver USB Driver Audio Driver WiFi Driver Power Management

Slide 12

Slide 12 text

Applications Launcher Phone Messaging Contacts Calendar Browser Settings Application Framework Activity Manager Window Manager Power Manager Resource Manager XMPP Service Content Providers Wallpapers System UI Package Manager Telephony Manager Location Manager Connectivity Manager View System Runtime Permissions Soft Keyboards Notifications Libraries Surface Manager Media Framework Chromium SSL HAL Audio Manager SQLite Open GL libc Core Libraries Android Runtime (ART) Android Runtime Linux Kernel Display Driver Camera Driver Bluetooth Driver Binder (IPC) Driver USB Driver Audio Driver WiFi Driver Power Management

Slide 13

Slide 13 text

Applications Launcher Phone Messaging Contacts Calendar Browser Settings Application Framework Activity Manager Window Manager Power Manager Resource Manager XMPP Service Wallpapers System UI Package Manager Telephony Manager Location Manager Connectivity Manager View System Soft Keyboards Notifications Libraries Surface Manager Media Framework Chromium SSL HAL Audio Manager SQLite Open GL libc Core Libraries Android Runtime (ART) Android Runtime Linux Kernel Display Driver Camera Driver Bluetooth Driver Binder (IPC) Driver USB Driver Audio Driver WiFi Driver Power Management Things Support Library Peripheral I/O Device Management User Drivers Connectivity

Slide 14

Slide 14 text

SoM Architecture Google Managed BSP

Slide 15

Slide 15 text

Distribution with Android Things Android Framework Hardware Libraries Linux Kernel Managed by Google Apps User Drivers Managed by Developers

Slide 16

Slide 16 text

Android Things Console ● Manage your Android Things IoT Product ● Download and install the latest Android Things system image ● Build factory images that contain OEM applications along with the system image ● Push over-the-air (OTA) updates partner.android.com/things/console

Slide 17

Slide 17 text

No content

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

Developing for Android Things

Slide 21

Slide 21 text

Developer Kits NXP I.MX7D Raspberry Pi 3

Slide 22

Slide 22 text

Hardware Integration Peripheral Driver Library - Github + many more... bit.ly/androidthings-github Button GPS PWM Servo RGB LED Strip Temperature Sensor Capacitive Touch Buttons Peripheral I/O - Low Level Access GPIO PWM I2C SPI UART I2S

Slide 23

Slide 23 text

Build a Motion Sensing Camera

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

things mobile cloud-functions Project Structure Runs on Android Things Device Runs on any Android Phone (companion app) Deployed to Firebase Cloud Functions (javascript)

Slide 26

Slide 26 text

Electronics Basics

Slide 27

Slide 27 text

Components Breadboard - Construction base for prototyping electronics Jumper Wire - Used to interconnect the components of a breadboard

Slide 28

Slide 28 text

LEDs - Emits visible light when an electric current passes through it Push Switches - Allows electricity to flow between its two contacts Components

Slide 29

Slide 29 text

Resistor -Used to reduce current flow Source: http://www.electronicshub.org/resistor-color-code/ Components and many more...

Slide 30

Slide 30 text

Ohm’s Law The current through a conductor between two points is directly proportional to the voltage across the two points. V = I * R V is Voltage (volts) I is Current (amps) R is Resistance (ohms) I = 0.023A V = 5V R = 5V / 0.023A R = ± 217 ohms

Slide 31

Slide 31 text

Hardware Setup

Slide 32

Slide 32 text

Pico i.MX7 Pinout * there was a change in pin naming in DP6

Slide 33

Slide 33 text

Red, Red, Brown = 220 ohm Brown, Black, Orange = 1k ohm

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

Working with a Motion Sensor

Slide 37

Slide 37 text

PIR Sensor

Slide 38

Slide 38 text

Adjust Sensitivity (Range)

Slide 39

Slide 39 text

Adjust Timeout

Slide 40

Slide 40 text

GPIO - General Purpose Input / Output - Programmable way to read true / false values (1 or 0) (push button, PIR Sensor) - Write true/false values (LED) - Configurable

Slide 41

Slide 41 text

Pico Setup: bit.ly/pico-pro-setup

Slide 42

Slide 42 text

Red, Red, Brown = 220 ohm Brown, Black, Orange = 1k ohm

Slide 43

Slide 43 text

Firmware Setup

Slide 44

Slide 44 text

//Boot into fastboot mode - old method $ adb reboot bootloader // Get list of fastboot devices $ fastboot devices 1b2f21d4e1fe0129 fastboot //Locate directory with factory image $ ./flash_all.sh //windows ./flash_all.bat // Get list of devices once flashed $ adb devices List of devices attached 1b2f21d4e1fe0129 device

Slide 45

Slide 45 text

//Start internet - Old Method $ adb shell am startservice -n com.google.wifisetup/.WifiSetupService -a WifiSetupService.Connect -e ssid "Jozihub Events" -e passphrase Jozihub#

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

//Download Android Things Flashing Tool $ ./androidthings-quickstart //Mac and Linux - Windows double click .exe Onboarding tool for Android Things ================================== This tool will help you install Android Things on your board and set up Wifi. What do you want to do? 1 - Install Android Things on a board and optionally set up Wifi 2 - Set up Wifi on an existing Android Things device 1 What is your board? 1 - Raspberry Pi 3 2 - NXP Pico i.MX7D 3 - NXP Pico i.MX6UL 2 You chose NXP Pico i.MX7D.

Slide 49

Slide 49 text

Before downloading Android Things, please consult our terms of service at https://developer.android.com/things/terms/index.html If you agree to our terms of service, press y to download it. Otherwise press n. (y/n) y Downloading Android Things image for NXP Pico i.MX7D... Downloaded image. Connect the board to your host computer with a USB-C cable. Once connected, press [Enter] key to install Android Things onto the board... Looking for devices... An Android device has been detected. Do you want to flash it? (y/n) WARNING: this will remove any existing data from it! y Installing fastboot... Waiting for fastboot to launch... This can take up to 3 minutes. Found fastboot device. Unzipping image... Installing Android Things on your board. This will take a few minutes... *Do not disconnect or interrupt!* ... Successfully flashed your imx7d. Successfully installed Android Things...

Slide 50

Slide 50 text

Would you like to set up Wi-Fi on your board? (y/n) y Please ensure antenna is already attached. If it is not, disconnect your board, attach the antenna and reconnect your board to your computer. When ready, press [Enter]... Enter the Wi-Fi network name: somenetwork Enter the Wi-Fi network password (leave empty if no password): somepassword Connecting to Wi-Fi network somenetwork... Looking for device... This can take up to 3 minutes. Device found. Waiting... Successfully connected to Wifi

Slide 51

Slide 51 text

Project Setup

Slide 52

Slide 52 text

Install Android Studio 3.0

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

//Automatically added in app level build.gradle compileOnly 'com.google.android.things:androidthings:...'

Slide 57

Slide 57 text

//Automatically added in AndroidManifest.xml

Slide 58

Slide 58 text

//Automatically added in AndroidManifest.xml

Slide 59

Slide 59 text

Physical Button

Slide 60

Slide 60 text

//Accessing Sensor Data private lateinit var buttonGpio: Gpio = PeripheralManagerService().openGpio("GPIO6_IO15") fun start() { buttonGpio.setDirection(Gpio.DIRECTION_IN) buttonGpio.setActiveType(Gpio.ACTIVE_HIGH) buttonGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) buttonGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { ledGpio.value = true } else { //do something else } return true } }) }

Slide 61

Slide 61 text

//Accessing Sensor Data private lateinit var buttonGpio: Gpio = PeripheralManagerService().openGpio("GPIO6_IO15") fun start() { buttonGpio.setDirection(Gpio.DIRECTION_IN) buttonGpio.setActiveType(Gpio.ACTIVE_HIGH) buttonGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) buttonGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { ledGpio.value = true } else { //do something else } return true } }) }

Slide 62

Slide 62 text

//Accessing Sensor Data private lateinit var buttonGpio: Gpio = PeripheralManagerService().openGpio("GPIO6_IO15") fun start() { buttonGpio.setDirection(Gpio.DIRECTION_IN) buttonGpio.setActiveType(Gpio.ACTIVE_HIGH) buttonGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) buttonGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { ledGpio.value = true } else { //do something else } return true } }) }

Slide 63

Slide 63 text

Control LED

Slide 64

Slide 64 text

val peripheralManagerService = PeripheralManagerService() ledGpio = peripheralManagerService.openGpio("GPIO6_IO14") ledGpio.setDirection(Gpio.DIRECTION_OUT_INITIALLY_LOW) ledGpio.value = true ledGpio.close()

Slide 65

Slide 65 text

val peripheralManagerService = PeripheralManagerService() ledGpio = peripheralManagerService.openGpio("GPIO6_IO14") ledGpio.setDirection(Gpio.DIRECTION_OUT_INITIALLY_LOW) ledGpio.value = true ledGpio.close()

Slide 66

Slide 66 text

val peripheralManagerService = PeripheralManagerService() ledGpio = peripheralManagerService.openGpio("GPIO6_IO14") ledGpio.setDirection(Gpio.DIRECTION_OUT_INITIALLY_LOW) ledGpio.value = true ledGpio.close()

Slide 67

Slide 67 text

Motion Sensor Code

Slide 68

Slide 68 text

//Accessing Sensor Data private val motionSensorGpio: Gpio = PeripheralManagerService().openGpio("GPIO2_IO03") fun start() { motionSensorGpio.setDirection(Gpio.DIRECTION_IN) motionSensorGpio.setActiveType(Gpio.ACTIVE_HIGH) motionSensorGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) motionSensorGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { motionListener.onMotionDetected() } else { motionListener.onMotionStopped() } return true } }) }

Slide 69

Slide 69 text

//Accessing Sensor Data private val motionSensorGpio: Gpio = PeripheralManagerService().openGpio("GPIO2_IO03") fun start() { motionSensorGpio.setDirection(Gpio.DIRECTION_IN) motionSensorGpio.setActiveType(Gpio.ACTIVE_HIGH) motionSensorGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) motionSensorGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { motionListener.onMotionDetected() } else { motionListener.onMotionStopped() } return true } }) }

Slide 70

Slide 70 text

//Accessing Sensor Data private val motionSensorGpio: Gpio = PeripheralManagerService().openGpio("GPIO2_IO03") fun start() { motionSensorGpio.setDirection(Gpio.DIRECTION_IN) motionSensorGpio.setActiveType(Gpio.ACTIVE_HIGH) motionSensorGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) motionSensorGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { camera.takePhoto() } else { //do something else } return true } }) }

Slide 71

Slide 71 text

Take a Photo

Slide 72

Slide 72 text

CustomCamera bit.ly/custom-camera-android

Slide 73

Slide 73 text

private fun setupCamera() { camera = CustomCamera.getInstance() camera.initializeCamera(this, Handler(), imageAvailableListener) } private val imageAvailableListener = object : CustomCamera.ImageCapturedListener { override fun onImageCaptured(bitmap: Bitmap) { motionImageView.setImageBitmap(bitmap) } } camera.takePicture()

Slide 74

Slide 74 text

private fun setupCamera() { camera = CustomCamera.getInstance() camera.initializeCamera(this, Handler(), imageAvailableListener) } private val imageAvailableListener = object : CustomCamera.ImageCapturedListener { override fun onImageCaptured(bitmap: Bitmap) { motionImageView.setImageBitmap(bitmap) } } camera.takePicture()

Slide 75

Slide 75 text

private fun setupCamera() { camera = CustomCamera.getInstance() camera.initializeCamera(this, Handler(), imageAvailableListener) } private val imageAvailableListener = object : CustomCamera.ImageCapturedListener { override fun onImageCaptured(bitmap: Bitmap) { motionImageView.setImageBitmap(bitmap) } } camera.takePicture()

Slide 76

Slide 76 text

Congrats! You’ve built a motion sensing camera!

Slide 77

Slide 77 text

Advance your app with Firebase + Cloud Functions

Slide 78

Slide 78 text

Firebase - Saving Data Things app

Slide 79

Slide 79 text

//data class for saving data class FirebaseImageLog(val timestamp: Long? = null, val imageRef: String? = null) //upload image val storageRef = FirebaseStorage.getInstance().getReference(FIREBASE_MOTION_REF) val imageStorageRef = storageRef.child(FIREBASE_IMAGE_PREFIX + System.currentTimeMillis() + ".jpg") val uploadTask = imageStorageRef.putBytes(imageStream.toByteArray()) uploadTask.addOnFailureListener { Log.d(TAG, "onFailure uploadMotionImage") }.addOnSuccessListener { // save image log to realtime db val downloadUrl = imageStorageRef.path val ref = FirebaseDatabase.getInstance().getReference(FIREBASE_MOTION_LOGS).push() ref.setValue(FirebaseImageLog(System.currentTimeMillis(), downloadUrl)) }

Slide 80

Slide 80 text

//data class for saving data class FirebaseImageLog(val timestamp: Long? = null, val imageRef: String? = null) //upload image val storageRef = FirebaseStorage.getInstance().getReference(FIREBASE_MOTION_REF) val imageStorageRef = storageRef.child(FIREBASE_IMAGE_PREFIX + System.currentTimeMillis() + ".jpg") val uploadTask = imageStorageRef.putBytes(imageStream.toByteArray()) uploadTask.addOnFailureListener { Log.d(TAG, "onFailure uploadMotionImage") }.addOnSuccessListener { // save image log to realtime db val downloadUrl = imageStorageRef.path val ref = FirebaseDatabase.getInstance().getReference(FIREBASE_MOTION_LOGS).push() ref.setValue(FirebaseImageLog(System.currentTimeMillis(), downloadUrl)) }

Slide 81

Slide 81 text

//data class for saving data class FirebaseImageLog(val timestamp: Long? = null, val imageRef: String? = null) //upload image val storageRef = FirebaseStorage.getInstance().getReference(FIREBASE_MOTION_REF) val imageStorageRef = storageRef.child(FIREBASE_IMAGE_PREFIX + System.currentTimeMillis() + ".jpg") val uploadTask = imageStorageRef.putBytes(imageStream.toByteArray()) uploadTask.addOnFailureListener { Log.d(TAG, "onFailure uploadMotionImage") }.addOnSuccessListener { // save image log to realtime db val downloadUrl = imageStorageRef.path val ref = FirebaseDatabase.getInstance().getReference(FIREBASE_MOTION_LOGS).push() ref.setValue(FirebaseImageLog(System.currentTimeMillis(), downloadUrl)) }

Slide 82

Slide 82 text

//data class for saving data class FirebaseImageLog(val timestamp: Long? = null, val imageRef: String? = null) //upload image val storageRef = FirebaseStorage.getInstance().getReference(FIREBASE_MOTION_REF) val imageStorageRef = storageRef.child(FIREBASE_IMAGE_PREFIX + System.currentTimeMillis() + ".jpg") val uploadTask = imageStorageRef.putBytes(imageStream.toByteArray()) uploadTask.addOnFailureListener { Log.d(TAG, "onFailure uploadMotionImage") }.addOnSuccessListener { // save image log to realtime db val downloadUrl = imageStorageRef.path val ref = FirebaseDatabase.getInstance().getReference(FIREBASE_MOTION_LOGS).push() ref.setValue(FirebaseImageLog(System.currentTimeMillis(), downloadUrl)) }

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

Firebase - Cloud Functions Javascript

Slide 85

Slide 85 text

$ firebase login $ firebase init functions

Slide 86

Slide 86 text

var functions = require('firebase-functions'); var admin = require('firebase-admin'); admin.initializeApp(functions.config().firebase); exports.annotateImage = functions.database.ref('/motion-logs/{id}') .onCreate(event => { const original = event.data.val(); var topic = "/topics/intruders"; var payload = { data: { title: "Intruder Alert!", body: "An intruder has been detected" } }; return admin.messaging().sendToTopic(topic, payload) .then(function (response) { console.log("Successfully sent message:", response); }).catch(function (error) { console.log("Error sending message:", error); }); } );

Slide 87

Slide 87 text

var functions = require('firebase-functions'); var admin = require('firebase-admin'); admin.initializeApp(functions.config().firebase); exports.annotateImage = functions.database.ref('/motion-logs/{id}') .onCreate(event => { const original = event.data.val(); var topic = "/topics/intruders"; var payload = { data: { title: "Intruder Alert!", body: "An intruder has been detected" } }; return admin.messaging().sendToTopic(topic, payload) .then(function (response) { console.log("Successfully sent message:", response); }).catch(function (error) { console.log("Error sending message:", error); }); } );

Slide 88

Slide 88 text

var functions = require('firebase-functions'); var admin = require('firebase-admin'); admin.initializeApp(functions.config().firebase); exports.annotateImage = functions.database.ref('/motion-logs/{id}') .onCreate(event => { const original = event.data.val(); var topic = "/topics/intruders"; var payload = { data: { title: "Intruder Alert!", body: "An intruder has been detected" } }; return admin.messaging().sendToTopic(topic, payload) .then(function (response) { console.log("Successfully sent message:", response); }).catch(function (error) { console.log("Error sending message:", error); }); } );

Slide 89

Slide 89 text

$ firebase deploy --only functions

Slide 90

Slide 90 text

No content

Slide 91

Slide 91 text

Firebase - Retrieving Data Companion Mobile App

Slide 92

Slide 92 text

//Use FirebaseUI dependency api "com.firebaseui:firebase-ui-database:3.0.0" // Get DB Reference val databaseRef = FirebaseDatabase.getInstance().getReference(MOTION_LOGS_FIREBASE_REF) // Create adapter that extends FirebaseRecyclerAdapter class LogsAdapter(ref: DatabaseReference) : FirebaseRecyclerAdapter(FirebaseImageLog::class.java, R.layout.list_item_log, LogsViewHolder::class.java, ref) { //.. Do adapter things } // Set adapter to use reference adapter = LogsAdapter(databaseRef.orderByChild(ORDER_BY_TIMESTAMP).ref)

Slide 93

Slide 93 text

//Use FirebaseUI dependency api "com.firebaseui:firebase-ui-database:3.0.0" // Get DB Reference val databaseRef = FirebaseDatabase.getInstance().getReference(MOTION_LOGS_FIREBASE_REF) // Create adapter that extends FirebaseRecyclerAdapter class LogsAdapter(ref: DatabaseReference) : FirebaseRecyclerAdapter(FirebaseImageLog::class.java, R.layout.list_item_log, LogsViewHolder::class.java, ref) { //.. Do adapter things } // Set adapter to use reference adapter = LogsAdapter(databaseRef.orderByChild(ORDER_BY_TIMESTAMP).ref)

Slide 94

Slide 94 text

//Use FirebaseUI dependency api "com.firebaseui:firebase-ui-database:3.0.0" // Get DB Reference val databaseRef = FirebaseDatabase.getInstance().getReference(MOTION_LOGS_FIREBASE_REF) // Create adapter that extends FirebaseRecyclerAdapter class LogsAdapter(ref: DatabaseReference) : FirebaseRecyclerAdapter(FirebaseImageLog::class.java, R.layout.list_item_log, LogsViewHolder::class.java, ref) { //.. Do adapter things } // Set adapter to use reference adapter = LogsAdapter(databaseRef.orderByChild(ORDER_BY_TIMESTAMP).ref)

Slide 95

Slide 95 text

//Use FirebaseUI dependency api "com.firebaseui:firebase-ui-database:3.0.0" // Get DB Reference val databaseRef = FirebaseDatabase.getInstance().getReference(MOTION_LOGS_FIREBASE_REF) // Create adapter that extends FirebaseRecyclerAdapter class LogsAdapter(ref: DatabaseReference) : FirebaseRecyclerAdapter(FirebaseImageLog::class.java, R.layout.list_item_log, LogsViewHolder::class.java, ref) { //.. Do adapter things } // Set adapter to use reference adapter = LogsAdapter(databaseRef.orderByChild(ORDER_BY_TIMESTAMP).ref)

Slide 96

Slide 96 text

No content

Slide 97

Slide 97 text

Upload your projects to

Slide 98

Slide 98 text

Full Example bit.ly/at-motion-camera Workshop Feedback bit.ly/nyc-at-workshop

Slide 99

Slide 99 text

Thank you! Rebecca Franks Android Engineering Lead at DVT @riggaroo

Slide 100

Slide 100 text

Clear all Android Apps on device: https://gist.github.com/blundell/7c0c3bb17898b28fe8122b0dc230af50 Button Example : https://github.com/androidthings/sample-button Full MainActivity: https://gist.github.com/riggaroo/b4cad83be2f81d54fd854403af029b2d Github Project with Steps as branches: https://github.com/riggaroo/android-things-workshop-camera Links