Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevFest NYC - Android Things Workshop - Build a Motion Sensing Camera using Android Things

DevFest NYC - Android Things Workshop - Build a Motion Sensing Camera using Android Things

In this session, Rebecca walked attendees through building their own Android Things powered motion sensing camera. The slides reveal the different steps required to get to the final result.

The code for this workshop can be found here - https://github.com/riggaroo/android-things-motion-camera

2a37bf1e025cc1523124774c760df91a?s=128

Rebecca Franks

December 03, 2017
Tweet

Transcript

  1. DevFest NYC - Android Things Workshop Build a Motion Sensing

    Camera Rebecca Franks Android Engineering Lead at DVT @riggaroo
  2. Joshua Leibstein GDG Johannesburg Organiser @JoshLiebe Oscar Salguero @OscarSalguero

  3. Slides bit.ly/nyc-at-slides Workshop Feedback bit.ly/nyc-at-workshop

  4. None
  5. What we will build: Intro: 1. Create project and read

    physical button value 2. Read motion values from Motion Sensor 3. Turn LED on/off based on Motion Sensor 4. Take photo and display when Motion is detected Further reading - advanced section: 5. Upload information to Firebase (photo + log) 6. Send Push Notification 7. View list of photos
  6. Android Things Theory

  7. What is Android Things? Android Things is an extension of

    the Android platform for IoT and embedded devices.
  8. Interactive Ads Vending Machines Point of Sale Stock Control Retail

    Cameras Gateways Access Control Smart Meters Business Asset Tracking Fleet Management Driver Assist Predictive Service Logistics Security Systems Smart Doorbells Routers Energy Monitors Home Android Things is ideal for powerful, intelligent devices that need to be secure.
  9. Similarities Any other Android library... Android SDK Play Services Firebase

    Android Studio Cloud Platform
  10. Differences No Play Store Deploy OTAs Subset of APIs Custom

    Hardware Single Purpose Device
  11. Applications Launcher Phone Messaging Contacts Calendar Browser Settings Application Framework

    Activity Manager Window Manager Power Manager Resource Manager XMPP Service Content Providers Wallpapers System UI Package Manager Telephony Manager Location Manager Connectivity Manager View System Runtime Permissions Soft Keyboards Notifications Libraries Surface Manager Media Framework Chromium SSL HAL Audio Manager SQLite Open GL libc Core Libraries Android Runtime (ART) Android Runtime Linux Kernel Display Driver Camera Driver Bluetooth Driver Binder (IPC) Driver USB Driver Audio Driver WiFi Driver Power Management
  12. Applications Launcher Phone Messaging Contacts Calendar Browser Settings Application Framework

    Activity Manager Window Manager Power Manager Resource Manager XMPP Service Content Providers Wallpapers System UI Package Manager Telephony Manager Location Manager Connectivity Manager View System Runtime Permissions Soft Keyboards Notifications Libraries Surface Manager Media Framework Chromium SSL HAL Audio Manager SQLite Open GL libc Core Libraries Android Runtime (ART) Android Runtime Linux Kernel Display Driver Camera Driver Bluetooth Driver Binder (IPC) Driver USB Driver Audio Driver WiFi Driver Power Management
  13. Applications Launcher Phone Messaging Contacts Calendar Browser Settings Application Framework

    Activity Manager Window Manager Power Manager Resource Manager XMPP Service Wallpapers System UI Package Manager Telephony Manager Location Manager Connectivity Manager View System Soft Keyboards Notifications Libraries Surface Manager Media Framework Chromium SSL HAL Audio Manager SQLite Open GL libc Core Libraries Android Runtime (ART) Android Runtime Linux Kernel Display Driver Camera Driver Bluetooth Driver Binder (IPC) Driver USB Driver Audio Driver WiFi Driver Power Management Things Support Library Peripheral I/O Device Management User Drivers Connectivity
  14. SoM Architecture Google Managed BSP

  15. Distribution with Android Things Android Framework Hardware Libraries Linux Kernel

    Managed by Google Apps User Drivers Managed by Developers
  16. Android Things Console • Manage your Android Things IoT Product

    • Download and install the latest Android Things system image • Build factory images that contain OEM applications along with the system image • Push over-the-air (OTA) updates partner.android.com/things/console
  17. None
  18. None
  19. None
  20. Developing for Android Things

  21. Developer Kits NXP I.MX7D Raspberry Pi 3

  22. Hardware Integration Peripheral Driver Library - Github + many more...

    bit.ly/androidthings-github Button GPS PWM Servo RGB LED Strip Temperature Sensor Capacitive Touch Buttons Peripheral I/O - Low Level Access GPIO PWM I2C SPI UART I2S
  23. Build a Motion Sensing Camera

  24. None
  25. things mobile cloud-functions Project Structure Runs on Android Things Device

    Runs on any Android Phone (companion app) Deployed to Firebase Cloud Functions (javascript)
  26. Electronics Basics

  27. Components Breadboard - Construction base for prototyping electronics Jumper Wire

    - Used to interconnect the components of a breadboard
  28. LEDs - Emits visible light when an electric current passes

    through it Push Switches - Allows electricity to flow between its two contacts Components
  29. Resistor -Used to reduce current flow Source: http://www.electronicshub.org/resistor-color-code/ Components and

    many more...
  30. Ohm’s Law The current through a conductor between two points

    is directly proportional to the voltage across the two points. V = I * R V is Voltage (volts) I is Current (amps) R is Resistance (ohms) I = 0.023A V = 5V R = 5V / 0.023A R = ± 217 ohms
  31. Hardware Setup

  32. Pico i.MX7 Pinout * there was a change in pin

    naming in DP6
  33. Red, Red, Brown = 220 ohm Brown, Black, Orange =

    1k ohm
  34. None
  35. None
  36. Working with a Motion Sensor

  37. PIR Sensor

  38. Adjust Sensitivity (Range)

  39. Adjust Timeout

  40. GPIO - General Purpose Input / Output - Programmable way

    to read true / false values (1 or 0) (push button, PIR Sensor) - Write true/false values (LED) - Configurable
  41. Pico Setup: bit.ly/pico-pro-setup

  42. Red, Red, Brown = 220 ohm Brown, Black, Orange =

    1k ohm
  43. Firmware Setup

  44. //Boot into fastboot mode - old method $ adb reboot

    bootloader // Get list of fastboot devices $ fastboot devices 1b2f21d4e1fe0129 fastboot //Locate directory with factory image $ ./flash_all.sh //windows ./flash_all.bat // Get list of devices once flashed $ adb devices List of devices attached 1b2f21d4e1fe0129 device
  45. //Start internet - Old Method $ adb shell am startservice

    -n com.google.wifisetup/.WifiSetupService -a WifiSetupService.Connect -e ssid "Jozihub Events" -e passphrase Jozihub#
  46. None
  47. None
  48. //Download Android Things Flashing Tool $ ./androidthings-quickstart //Mac and Linux

    - Windows double click .exe Onboarding tool for Android Things ================================== This tool will help you install Android Things on your board and set up Wifi. What do you want to do? 1 - Install Android Things on a board and optionally set up Wifi 2 - Set up Wifi on an existing Android Things device 1 What is your board? 1 - Raspberry Pi 3 2 - NXP Pico i.MX7D 3 - NXP Pico i.MX6UL 2 You chose NXP Pico i.MX7D.
  49. Before downloading Android Things, please consult our terms of service

    at https://developer.android.com/things/terms/index.html If you agree to our terms of service, press y to download it. Otherwise press n. (y/n) y Downloading Android Things image for NXP Pico i.MX7D... Downloaded image. Connect the board to your host computer with a USB-C cable. Once connected, press [Enter] key to install Android Things onto the board... Looking for devices... An Android device has been detected. Do you want to flash it? (y/n) WARNING: this will remove any existing data from it! y Installing fastboot... Waiting for fastboot to launch... This can take up to 3 minutes. Found fastboot device. Unzipping image... Installing Android Things on your board. This will take a few minutes... *Do not disconnect or interrupt!* ... Successfully flashed your imx7d. Successfully installed Android Things...
  50. Would you like to set up Wi-Fi on your board?

    (y/n) y Please ensure antenna is already attached. If it is not, disconnect your board, attach the antenna and reconnect your board to your computer. When ready, press [Enter]... Enter the Wi-Fi network name: somenetwork Enter the Wi-Fi network password (leave empty if no password): somepassword Connecting to Wi-Fi network somenetwork... Looking for device... This can take up to 3 minutes. Device found. Waiting... Successfully connected to Wifi
  51. Project Setup

  52. Install Android Studio 3.0

  53. None
  54. None
  55. None
  56. //Automatically added in app level build.gradle compileOnly 'com.google.android.things:androidthings:...'

  57. //Automatically added in AndroidManifest.xml <application ...> <uses-library android:name="com.google.android.things"/> <activity ...>

    <!-- Launch activity automatically on boot --> <intent-filter> <action android:name="android.intent.action.MAIN"/> <category android:name="android.intent.category.IOT_LAUNCHER"/> <category android:name="android.intent.category.DEFAULT"/> </intent-filter> </activity> </application>
  58. //Automatically added in AndroidManifest.xml <application ...> <uses-library android:name="com.google.android.things"/> <activity ...>

    <!-- Launch activity automatically on boot --> <intent-filter> <action android:name="android.intent.action.MAIN"/> <category android:name="android.intent.category.IOT_LAUNCHER"/> <category android:name="android.intent.category.DEFAULT"/> </intent-filter> </activity> </application>
  59. Physical Button

  60. //Accessing Sensor Data private lateinit var buttonGpio: Gpio = PeripheralManagerService().openGpio("GPIO6_IO15")

    fun start() { buttonGpio.setDirection(Gpio.DIRECTION_IN) buttonGpio.setActiveType(Gpio.ACTIVE_HIGH) buttonGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) buttonGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { ledGpio.value = true } else { //do something else } return true } }) }
  61. //Accessing Sensor Data private lateinit var buttonGpio: Gpio = PeripheralManagerService().openGpio("GPIO6_IO15")

    fun start() { buttonGpio.setDirection(Gpio.DIRECTION_IN) buttonGpio.setActiveType(Gpio.ACTIVE_HIGH) buttonGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) buttonGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { ledGpio.value = true } else { //do something else } return true } }) }
  62. //Accessing Sensor Data private lateinit var buttonGpio: Gpio = PeripheralManagerService().openGpio("GPIO6_IO15")

    fun start() { buttonGpio.setDirection(Gpio.DIRECTION_IN) buttonGpio.setActiveType(Gpio.ACTIVE_HIGH) buttonGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) buttonGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { ledGpio.value = true } else { //do something else } return true } }) }
  63. Control LED

  64. val peripheralManagerService = PeripheralManagerService() ledGpio = peripheralManagerService.openGpio("GPIO6_IO14") ledGpio.setDirection(Gpio.DIRECTION_OUT_INITIALLY_LOW) ledGpio.value =

    true ledGpio.close()
  65. val peripheralManagerService = PeripheralManagerService() ledGpio = peripheralManagerService.openGpio("GPIO6_IO14") ledGpio.setDirection(Gpio.DIRECTION_OUT_INITIALLY_LOW) ledGpio.value =

    true ledGpio.close()
  66. val peripheralManagerService = PeripheralManagerService() ledGpio = peripheralManagerService.openGpio("GPIO6_IO14") ledGpio.setDirection(Gpio.DIRECTION_OUT_INITIALLY_LOW) ledGpio.value =

    true ledGpio.close()
  67. Motion Sensor Code

  68. //Accessing Sensor Data private val motionSensorGpio: Gpio = PeripheralManagerService().openGpio("GPIO2_IO03") fun

    start() { motionSensorGpio.setDirection(Gpio.DIRECTION_IN) motionSensorGpio.setActiveType(Gpio.ACTIVE_HIGH) motionSensorGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) motionSensorGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { motionListener.onMotionDetected() } else { motionListener.onMotionStopped() } return true } }) }
  69. //Accessing Sensor Data private val motionSensorGpio: Gpio = PeripheralManagerService().openGpio("GPIO2_IO03") fun

    start() { motionSensorGpio.setDirection(Gpio.DIRECTION_IN) motionSensorGpio.setActiveType(Gpio.ACTIVE_HIGH) motionSensorGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) motionSensorGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { motionListener.onMotionDetected() } else { motionListener.onMotionStopped() } return true } }) }
  70. //Accessing Sensor Data private val motionSensorGpio: Gpio = PeripheralManagerService().openGpio("GPIO2_IO03") fun

    start() { motionSensorGpio.setDirection(Gpio.DIRECTION_IN) motionSensorGpio.setActiveType(Gpio.ACTIVE_HIGH) motionSensorGpio.setEdgeTriggerType(Gpio.EDGE_BOTH) motionSensorGpio.registerGpioCallback(object : GpioCallback() { override fun onGpioEdge(gpio: Gpio): Boolean { if (gpio.value) { camera.takePhoto() } else { //do something else } return true } }) }
  71. Take a Photo

  72. CustomCamera bit.ly/custom-camera-android

  73. private fun setupCamera() { camera = CustomCamera.getInstance() camera.initializeCamera(this, Handler(), imageAvailableListener)

    } private val imageAvailableListener = object : CustomCamera.ImageCapturedListener { override fun onImageCaptured(bitmap: Bitmap) { motionImageView.setImageBitmap(bitmap) } } camera.takePicture()
  74. private fun setupCamera() { camera = CustomCamera.getInstance() camera.initializeCamera(this, Handler(), imageAvailableListener)

    } private val imageAvailableListener = object : CustomCamera.ImageCapturedListener { override fun onImageCaptured(bitmap: Bitmap) { motionImageView.setImageBitmap(bitmap) } } camera.takePicture()
  75. private fun setupCamera() { camera = CustomCamera.getInstance() camera.initializeCamera(this, Handler(), imageAvailableListener)

    } private val imageAvailableListener = object : CustomCamera.ImageCapturedListener { override fun onImageCaptured(bitmap: Bitmap) { motionImageView.setImageBitmap(bitmap) } } camera.takePicture()
  76. Congrats! You’ve built a motion sensing camera!

  77. Advance your app with Firebase + Cloud Functions

  78. Firebase - Saving Data Things app

  79. //data class for saving data class FirebaseImageLog(val timestamp: Long? =

    null, val imageRef: String? = null) //upload image val storageRef = FirebaseStorage.getInstance().getReference(FIREBASE_MOTION_REF) val imageStorageRef = storageRef.child(FIREBASE_IMAGE_PREFIX + System.currentTimeMillis() + ".jpg") val uploadTask = imageStorageRef.putBytes(imageStream.toByteArray()) uploadTask.addOnFailureListener { Log.d(TAG, "onFailure uploadMotionImage") }.addOnSuccessListener { // save image log to realtime db val downloadUrl = imageStorageRef.path val ref = FirebaseDatabase.getInstance().getReference(FIREBASE_MOTION_LOGS).push() ref.setValue(FirebaseImageLog(System.currentTimeMillis(), downloadUrl)) }
  80. //data class for saving data class FirebaseImageLog(val timestamp: Long? =

    null, val imageRef: String? = null) //upload image val storageRef = FirebaseStorage.getInstance().getReference(FIREBASE_MOTION_REF) val imageStorageRef = storageRef.child(FIREBASE_IMAGE_PREFIX + System.currentTimeMillis() + ".jpg") val uploadTask = imageStorageRef.putBytes(imageStream.toByteArray()) uploadTask.addOnFailureListener { Log.d(TAG, "onFailure uploadMotionImage") }.addOnSuccessListener { // save image log to realtime db val downloadUrl = imageStorageRef.path val ref = FirebaseDatabase.getInstance().getReference(FIREBASE_MOTION_LOGS).push() ref.setValue(FirebaseImageLog(System.currentTimeMillis(), downloadUrl)) }
  81. //data class for saving data class FirebaseImageLog(val timestamp: Long? =

    null, val imageRef: String? = null) //upload image val storageRef = FirebaseStorage.getInstance().getReference(FIREBASE_MOTION_REF) val imageStorageRef = storageRef.child(FIREBASE_IMAGE_PREFIX + System.currentTimeMillis() + ".jpg") val uploadTask = imageStorageRef.putBytes(imageStream.toByteArray()) uploadTask.addOnFailureListener { Log.d(TAG, "onFailure uploadMotionImage") }.addOnSuccessListener { // save image log to realtime db val downloadUrl = imageStorageRef.path val ref = FirebaseDatabase.getInstance().getReference(FIREBASE_MOTION_LOGS).push() ref.setValue(FirebaseImageLog(System.currentTimeMillis(), downloadUrl)) }
  82. //data class for saving data class FirebaseImageLog(val timestamp: Long? =

    null, val imageRef: String? = null) //upload image val storageRef = FirebaseStorage.getInstance().getReference(FIREBASE_MOTION_REF) val imageStorageRef = storageRef.child(FIREBASE_IMAGE_PREFIX + System.currentTimeMillis() + ".jpg") val uploadTask = imageStorageRef.putBytes(imageStream.toByteArray()) uploadTask.addOnFailureListener { Log.d(TAG, "onFailure uploadMotionImage") }.addOnSuccessListener { // save image log to realtime db val downloadUrl = imageStorageRef.path val ref = FirebaseDatabase.getInstance().getReference(FIREBASE_MOTION_LOGS).push() ref.setValue(FirebaseImageLog(System.currentTimeMillis(), downloadUrl)) }
  83. None
  84. Firebase - Cloud Functions Javascript

  85. $ firebase login $ firebase init functions

  86. var functions = require('firebase-functions'); var admin = require('firebase-admin'); admin.initializeApp(functions.config().firebase); exports.annotateImage

    = functions.database.ref('/motion-logs/{id}') .onCreate(event => { const original = event.data.val(); var topic = "/topics/intruders"; var payload = { data: { title: "Intruder Alert!", body: "An intruder has been detected" } }; return admin.messaging().sendToTopic(topic, payload) .then(function (response) { console.log("Successfully sent message:", response); }).catch(function (error) { console.log("Error sending message:", error); }); } );
  87. var functions = require('firebase-functions'); var admin = require('firebase-admin'); admin.initializeApp(functions.config().firebase); exports.annotateImage

    = functions.database.ref('/motion-logs/{id}') .onCreate(event => { const original = event.data.val(); var topic = "/topics/intruders"; var payload = { data: { title: "Intruder Alert!", body: "An intruder has been detected" } }; return admin.messaging().sendToTopic(topic, payload) .then(function (response) { console.log("Successfully sent message:", response); }).catch(function (error) { console.log("Error sending message:", error); }); } );
  88. var functions = require('firebase-functions'); var admin = require('firebase-admin'); admin.initializeApp(functions.config().firebase); exports.annotateImage

    = functions.database.ref('/motion-logs/{id}') .onCreate(event => { const original = event.data.val(); var topic = "/topics/intruders"; var payload = { data: { title: "Intruder Alert!", body: "An intruder has been detected" } }; return admin.messaging().sendToTopic(topic, payload) .then(function (response) { console.log("Successfully sent message:", response); }).catch(function (error) { console.log("Error sending message:", error); }); } );
  89. $ firebase deploy --only functions

  90. None
  91. Firebase - Retrieving Data Companion Mobile App

  92. //Use FirebaseUI dependency api "com.firebaseui:firebase-ui-database:3.0.0" // Get DB Reference val

    databaseRef = FirebaseDatabase.getInstance().getReference(MOTION_LOGS_FIREBASE_REF) // Create adapter that extends FirebaseRecyclerAdapter class LogsAdapter(ref: DatabaseReference) : FirebaseRecyclerAdapter<FirebaseImageLog, LogsViewHolder>(FirebaseImageLog::class.java, R.layout.list_item_log, LogsViewHolder::class.java, ref) { //.. Do adapter things } // Set adapter to use reference adapter = LogsAdapter(databaseRef.orderByChild(ORDER_BY_TIMESTAMP).ref)
  93. //Use FirebaseUI dependency api "com.firebaseui:firebase-ui-database:3.0.0" // Get DB Reference val

    databaseRef = FirebaseDatabase.getInstance().getReference(MOTION_LOGS_FIREBASE_REF) // Create adapter that extends FirebaseRecyclerAdapter class LogsAdapter(ref: DatabaseReference) : FirebaseRecyclerAdapter<FirebaseImageLog, LogsViewHolder>(FirebaseImageLog::class.java, R.layout.list_item_log, LogsViewHolder::class.java, ref) { //.. Do adapter things } // Set adapter to use reference adapter = LogsAdapter(databaseRef.orderByChild(ORDER_BY_TIMESTAMP).ref)
  94. //Use FirebaseUI dependency api "com.firebaseui:firebase-ui-database:3.0.0" // Get DB Reference val

    databaseRef = FirebaseDatabase.getInstance().getReference(MOTION_LOGS_FIREBASE_REF) // Create adapter that extends FirebaseRecyclerAdapter class LogsAdapter(ref: DatabaseReference) : FirebaseRecyclerAdapter<FirebaseImageLog, LogsViewHolder>(FirebaseImageLog::class.java, R.layout.list_item_log, LogsViewHolder::class.java, ref) { //.. Do adapter things } // Set adapter to use reference adapter = LogsAdapter(databaseRef.orderByChild(ORDER_BY_TIMESTAMP).ref)
  95. //Use FirebaseUI dependency api "com.firebaseui:firebase-ui-database:3.0.0" // Get DB Reference val

    databaseRef = FirebaseDatabase.getInstance().getReference(MOTION_LOGS_FIREBASE_REF) // Create adapter that extends FirebaseRecyclerAdapter class LogsAdapter(ref: DatabaseReference) : FirebaseRecyclerAdapter<FirebaseImageLog, LogsViewHolder>(FirebaseImageLog::class.java, R.layout.list_item_log, LogsViewHolder::class.java, ref) { //.. Do adapter things } // Set adapter to use reference adapter = LogsAdapter(databaseRef.orderByChild(ORDER_BY_TIMESTAMP).ref)
  96. None
  97. Upload your projects to

  98. Full Example bit.ly/at-motion-camera Workshop Feedback bit.ly/nyc-at-workshop

  99. Thank you! Rebecca Franks Android Engineering Lead at DVT @riggaroo

  100. Clear all Android Apps on device: https://gist.github.com/blundell/7c0c3bb17898b28fe8122b0dc230af50 Button Example :

    https://github.com/androidthings/sample-button Full MainActivity: https://gist.github.com/riggaroo/b4cad83be2f81d54fd854403af029b2d Github Project with Steps as branches: https://github.com/riggaroo/android-things-workshop-camera Links