GCP and IoT

GCP and IoT

A quick talk about getting GCP to talk to IoT devices

6c18ceafef161be26ae441469b29c475?s=128

Terrence Ryan

October 12, 2016
Tweet

Transcript

  1. ‹#› @tpryan Terrence Ryan Developer Advocate IoT and Google Cloud

    Platform
  2. ‹#› @tpryan Who are you?

  3. ‹#› @tpryan 01 Introduction What are we trying to do

  4. • Products • Nest • Fitbit • WeMo • Phillips

    Hue • Lockitron • Building Blocks • Arduino • Raspberry Pi • BeagleBone • etc • etc • Providers • AWS • GCP • Azure • Building Blocks • Storage • Analytics • Processing
  5. • Building Blocks • Raspberry Pi • BeagleBone • Providers

    • GCP • Building Blocks • Cloud Storage • App Engine • Pub/Sub • BigQuery
  6. @tpryan Why did you choose these?

  7. @tpryan These are the things that made sense to use.

  8. @tpryan These are the things I could get working

  9. @tpryan

  10. @tpryan

  11. @tpryan

  12. ‹#› @tpryan Beaglebone to App Engine to Cloud Storage

  13. BeagleBone App Engine Cloud Storage

  14. ‹#› @tpryan App Engine • Platform as a Service •

    Give it code • It gives you a URL running that code • Advantages: • Scales up and down • Simple • Disadvantages • Language • PHP, Java, Python, Go • Restrictions • No writing to local disk • Other restrictions
  15. None
  16. None
  17. ‹#› @tpryan Demo: App Engine

  18. ‹#› @tpryan Cloud Storage • File Storage • Cheap and

    fast storage. • Easy to write • Easy to share • Can send to most other GCP Techs • Advantages: • Scales • Simple • Disadvantages • File based storage not best for analytics
  19. BeagleBone App Engine Cloud Storage

  20. ‹#› @tpryan Source Code - App Engine - PHP use

    google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->remote_host = $_SERVER['REMOTE_HOST']; $instance_data->latlon = $_SERVER['HTTP_X_APPENGINE_CITYLATLONG']; $instance_data->time = time(); $instance_data->data = $_REQUEST; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/simple/${instance}/${request}.json", $instance_json); use google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->remote_host = $_SERVER['REMOTE_HOST']; $instance_data->latlon = $_SERVER['HTTP_X_APPENGINE_CITYLATLONG']; $instance_data->time = time(); $instance_data->data = $_REQUEST; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/simple/${instance}/${request}.json", $instance_json);
  21. ‹#› @tpryan Source Code - Beaglebone - Python import time

    import Adafruit_BBIO.GPIO as GPIO import urllib2 # Note: Use P9_22(UART2_RXD) as GPIO. # Connect the Grove Button to UART Grove port of Beaglebone Green. Button = "P9_22" # GPIO P9_22 GPIO.setup(Button, GPIO.IN) base_url = "https://simple-dot-gcpiotdemo.appspot.com" params = "source=beagleboard&type=button&reading=1" if __name__== '__main__': while True: if GPIO.input(Button): print "Button is pressed." urllib2.urlopen(base_url + "/?" + params).read() time.sleep(.1) else: print "Button is unstuck." time.sleep(1) import time import Adafruit_BBIO.GPIO as GPIO import urllib2 # Note: Use P9_22(UART2_RXD) as GPIO. # Connect the Grove Button to UART Grove port of Beaglebone Green. Button = "P9_22" # GPIO P9_22 GPIO.setup(Button, GPIO.IN) base_url = "https://simple-dot-gcpiotdemo.appspot.com" params = "source=beagleboard&type=button&reading=1" if __name__== '__main__': while True: if GPIO.input(Button): print "Button is pressed." urllib2.urlopen(base_url + "/?" + params).read() time.sleep(.1) else: print "Button is unstuck." time.sleep(1)
  22. ‹#› @tpryan Demo: Connecting from BeagleBoard to App Engine

  23. ‹#› @tpryan Lessons Learned • Maybe go more mainstream kit

    • Raspberry Pi • Arduino • With more updated sensors
  24. ‹#› @tpryan Raspberry Pi to Pub/Sub to App Engine to

    Cloud Storage
  25. App Engine Cloud Storage Raspberry Pi Pub/Sub

  26. ‹#› @tpryan Pub/Sub • Messenger service • Publish and Subscribe

    Model • Advantages: • Unlimited Quota • Acts as a buffer • Disadvantages • Requires security on the device
  27. ‹#› @tpryan Cloud PUB/SUB

  28. ‹#› @tpryan

  29. ‹#› @tpryan Source Code - Raspberry Pi - Python #!/usr/bin/env

    python import PCF8591 as ADC import RPi.GPIO as GPIO import time import math import urllib2 import os from google.cloud import pubsub INTERVAL = .1 DO = 17 GPIO.setmode(GPIO.BCM) os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/home/pi/gcp/creds.json" #!/usr/bin/env python import PCF8591 as ADC import RPi.GPIO as GPIO import time import math import urllib2 import os from google.cloud import pubsub INTERVAL = .1 DO = 17 GPIO.setmode(GPIO.BCM) os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/home/pi/gcp/creds.json"
  30. ‹#› @tpryan

  31. ‹#› @tpryan Source Code - creds.json { "type": "service_account", "project_id":

    "gcpiotdemo", "private_key_id": "a098b68643ae4ccda140d907d5a2394a895e8be4", "private_key": "-----BEGIN PRIVATE KEY——\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBA….Q..\n——END PRIVATE KEY-----\n", "client_email": "raspberrypiclient@gcpiotdemo.iam.gserviceaccount.com", "client_id": "106452013022587438299", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://accounts.google.com/o/oauth2/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/raspberrypiclient%40gcpiotdemo.iam.gserviceaccount.com" }
  32. ‹#› @tpryan Source Code - Raspberry Pi - Python def

    setup(): ADC.setup(0x48) GPIO.setup(DO, GPIO.IN) def loop(): status = 1 tmp = 1 while True: analogVal = ADC.read(0) Vr = 5 * float(analogVal) / 255 Rt = 10000 * Vr / (5 - Vr) temp = 1/(((math.log(Rt / 10000)) / 3950) + (1 / (273.15+25))) temp = temp - 273.15 print 'temperature = ', temp, 'C' publish("iotstream", get_serial() + "," +str(temp)) time.sleep(INTERVAL) if __name__ == '__main__': try: setup() loop() except KeyboardInterrupt: pass def setup(): ADC.setup(0x48) GPIO.setup(DO, GPIO.IN) def loop(): status = 1 tmp = 1 while True: analogVal = ADC.read(0) Vr = 5 * float(analogVal) / 255 Rt = 10000 * Vr / (5 - Vr) temp = 1/(((math.log(Rt / 10000)) / 3950) + (1 / (273.15+25))) temp = temp - 273.15 print 'temperature = ', temp, 'C' publish("iotstream", get_serial() + "," +str(temp)) time.sleep(INTERVAL) if __name__ == '__main__': try: setup() loop() except KeyboardInterrupt: pass
  33. ‹#› @tpryan Source Code - Raspberry Pi - Python def

    publish(topic_name, data): pubsub_client = pubsub.Client("gcpiotdemo") topic = pubsub_client.topic(topic_name) message_id = topic.publish(data) print('Message {} published.'.format(message_id))
  34. ‹#› @tpryan Source Code - Raspberry Pi - Python def

    get_serial(): cpuserial = "0000000000000000" try: f = open('/proc/cpuinfo','r') for line in f: if line[0:6]=='Serial': cpuserial = line[10:26] f.close() except: cpuserial = "ERROR00000000000" return cpuserial
  35. ‹#› @tpryan

  36. ‹#› @tpryan Source Code - App Engine - PHP use

    google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $message = json_decode(file_get_contents('php://input')); $datastr = base64_decode(base64_decode($message->message->data)); $arr = explode(",", $datastr); // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->time = time(); $instance_data->device = $arr[0]; $instance_data->temp = $arr[1]; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/pubsub/${instance}/${request}.json", $instance_json); use google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $message = json_decode(file_get_contents('php://input')); $datastr = base64_decode(base64_decode($message->message->data)); $arr = explode(",", $datastr); // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->time = time(); $instance_data->device = $arr[0]; $instance_data->temp = $arr[1]; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/pubsub/${instance}/${request}.json", $instance_json);
  37. ‹#› @tpryan Demo: Connecting from RaspberryPi

  38. ‹#› @tpryan Lessons Learned • Securing IoT devices is not

    trivial
  39. ‹#› @tpryan Data analysis with BigQuery

  40. @tpryan 10 x a second * 60 * 60 *

    24 864,000 data points a day * 400 sensors 345,600,000 data points a day * 365 days a year 126,144,000,000 data points a year
  41. @tpryan IoT = BigData

  42. ‹#› @tpryan Big Query • Scan Terabytes in seconds •

    Use SQLish Queries • REST, Web UI, ODBC
  43. ‹#› @tpryan Demo: BigQuery

  44. @tpryan Count Shakespeare SELECT count(word) FROM publicdata:samples.shakespeare

  45. @tpryan Count to a Million SELECT sum(requests) as total FROM

    [fh-bigquery:wikipedia.pagecounts_20151109_18]
  46. @tpryan Count to a Billion SELECT sum(requests) as total FROM

    [fh-bigquery:wikipedia.pagecounts_201505]
  47. @tpryan Count to a Trillion SELECT SUM(requests) AS total FROM

    TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_201[3-4][0-9]{2}$")')
  48. @tpryan Run a RegEx on a Hundreds of Billions SELECT

    SUM(requests) AS total FROM TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_201[3-4][0-9]{2}$")') WHERE (REGEXP_MATCH(title, '.*[dD]inosaur.*'))
  49. ‹#› @tpryan Demo: Getting our data into BigQuery

  50. ‹#› @tpryan 06 Conclusion What now

  51. App Engine Cloud Storage Raspberry Pi Pub/Sub Big Query

  52. Raspberry Pi

  53. None
  54. Raspberry Pi

  55. @tpryan ‹#› Google Cloud Platform Compute Connectivity Big Data Storage

    Developer Tools Management Machine Learning
  56. ‹#› @tpryan App Engine • Flexible runtime • Based on

    Docker containers • Very few restrictions • Base images can use GCP resources easily • Custom can still use them with more setup • Scales slower • Scales to 1 not zero
  57. ‹#› @tpryan Data Flow

  58. ‹#› @tpryan Cloud Vision

  59. ‹#› @tpryan Demo: Cloud Vision API

  60. ‹#› @tpryan Cloud ML

  61. ‹#› @tpryan Stackdriver

  62. ‹#› @tpryan Lessons Learned • IoT = Big Data •

    Once you get it to a cloud provider your only limit is creativity • Security doesn’t have a clear path… yet
  63. ‹#› @tpryan Lessons Implied • Maybe reduce your sample size

    • Maybe skip some steps • Maybe batch up data • Maybe do some processing on the device
  64. ‹#› @tpryan Next Steps • https://cloud.google.com/solutions/iot/ • https://cloud.google.com/solutions/iot-overview • https://googlecloudplatform.github.io/

  65. ‹#› @tpryan Thank You terrenceryan.com @tpryan This preso: http://bit.ly/tpryan-iot