GCP and IoT

A quick talk about getting GCP to talk to IoT devices

Transcript

1. ‹#› @tpryan Terrence Ryan Developer Advocate IoT and Google Cloud Platform
2. ‹#› @tpryan Who are you?
3. ‹#› @tpryan 01 Introduction What are we trying to do
4. • Products • Nest • Fitbit • WeMo • Phillips Hue • Lockitron • Building Blocks • Arduino • Raspberry Pi • BeagleBone • etc • etc • Providers • AWS • GCP • Azure • Building Blocks • Storage • Analytics • Processing
5. • Building Blocks • Raspberry Pi • BeagleBone • Providers • GCP • Building Blocks • Cloud Storage • App Engine • Pub/Sub • BigQuery
6. @tpryan Why did you choose these?
7. @tpryan These are the things that made sense to use.
8. @tpryan These are the things I could get working
9. @tpryan
10. @tpryan
11. @tpryan
12. ‹#› @tpryan Beaglebone to App Engine to Cloud Storage
13. BeagleBone App Engine Cloud Storage
14. ‹#› @tpryan App Engine • Platform as a Service • Give it code • It gives you a URL running that code • Advantages: • Scales up and down • Simple • Disadvantages • Language • PHP, Java, Python, Go • Restrictions • No writing to local disk • Other restrictions
17. ‹#› @tpryan Demo: App Engine
18. ‹#› @tpryan Cloud Storage • File Storage • Cheap and fast storage. • Easy to write • Easy to share • Can send to most other GCP Techs • Advantages: • Scales • Simple • Disadvantages • File based storage not best for analytics
19. BeagleBone App Engine Cloud Storage
20. ‹#› @tpryan Source Code - App Engine - PHP use google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->remote_host = $_SERVER['REMOTE_HOST']; $instance_data->latlon = $_SERVER['HTTP_X_APPENGINE_CITYLATLONG']; $instance_data->time = time(); $instance_data->data = $_REQUEST; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/simple/${instance}/${request}.json", $instance_json); use google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->remote_host = $_SERVER['REMOTE_HOST']; $instance_data->latlon = $_SERVER['HTTP_X_APPENGINE_CITYLATLONG']; $instance_data->time = time(); $instance_data->data = $_REQUEST; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/simple/${instance}/${request}.json", $instance_json);
21. ‹#› @tpryan Source Code - Beaglebone - Python import time import Adafruit_BBIO.GPIO as GPIO import urllib2 # Note: Use P9_22(UART2_RXD) as GPIO. # Connect the Grove Button to UART Grove port of Beaglebone Green. Button = "P9_22" # GPIO P9_22 GPIO.setup(Button, GPIO.IN) base_url = "https://simple-dot-gcpiotdemo.appspot.com" params = "source=beagleboard&type=button&reading=1" if __name__== '__main__': while True: if GPIO.input(Button): print "Button is pressed." urllib2.urlopen(base_url + "/?" + params).read() time.sleep(.1) else: print "Button is unstuck." time.sleep(1) import time import Adafruit_BBIO.GPIO as GPIO import urllib2 # Note: Use P9_22(UART2_RXD) as GPIO. # Connect the Grove Button to UART Grove port of Beaglebone Green. Button = "P9_22" # GPIO P9_22 GPIO.setup(Button, GPIO.IN) base_url = "https://simple-dot-gcpiotdemo.appspot.com" params = "source=beagleboard&type=button&reading=1" if __name__== '__main__': while True: if GPIO.input(Button): print "Button is pressed." urllib2.urlopen(base_url + "/?" + params).read() time.sleep(.1) else: print "Button is unstuck." time.sleep(1)
22. ‹#› @tpryan Demo: Connecting from BeagleBoard to App Engine
23. ‹#› @tpryan Lessons Learned • Maybe go more mainstream kit • Raspberry Pi • Arduino • With more updated sensors
24. ‹#› @tpryan Raspberry Pi to Pub/Sub to App Engine to Cloud Storage
25. App Engine Cloud Storage Raspberry Pi Pub/Sub
26. ‹#› @tpryan Pub/Sub • Messenger service • Publish and Subscribe Model • Advantages: • Unlimited Quota • Acts as a buffer • Disadvantages • Requires security on the device
27. ‹#› @tpryan Cloud PUB/SUB
28. ‹#› @tpryan
29. ‹#› @tpryan Source Code - Raspberry Pi - Python #!/usr/bin/env python import PCF8591 as ADC import RPi.GPIO as GPIO import time import math import urllib2 import os from google.cloud import pubsub INTERVAL = .1 DO = 17 GPIO.setmode(GPIO.BCM) os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/home/pi/gcp/creds.json" #!/usr/bin/env python import PCF8591 as ADC import RPi.GPIO as GPIO import time import math import urllib2 import os from google.cloud import pubsub INTERVAL = .1 DO = 17 GPIO.setmode(GPIO.BCM) os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/home/pi/gcp/creds.json"
30. ‹#› @tpryan
31. ‹#› @tpryan Source Code - creds.json { "type": "service_account", "project_id": "gcpiotdemo", "private_key_id": "a098b68643ae4ccda140d907d5a2394a895e8be4", "private_key": "-----BEGIN PRIVATE KEY——\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBA….Q..\n——END PRIVATE KEY-----\n", "client_email": "raspberrypiclient@gcpiotdemo.iam.gserviceaccount.com", "client_id": "106452013022587438299", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://accounts.google.com/o/oauth2/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/raspberrypiclient%40gcpiotdemo.iam.gserviceaccount.com" }
32. ‹#› @tpryan Source Code - Raspberry Pi - Python def setup(): ADC.setup(0x48) GPIO.setup(DO, GPIO.IN) def loop(): status = 1 tmp = 1 while True: analogVal = ADC.read(0) Vr = 5 * float(analogVal) / 255 Rt = 10000 * Vr / (5 - Vr) temp = 1/(((math.log(Rt / 10000)) / 3950) + (1 / (273.15+25))) temp = temp - 273.15 print 'temperature = ', temp, 'C' publish("iotstream", get_serial() + "," +str(temp)) time.sleep(INTERVAL) if __name__ == '__main__': try: setup() loop() except KeyboardInterrupt: pass def setup(): ADC.setup(0x48) GPIO.setup(DO, GPIO.IN) def loop(): status = 1 tmp = 1 while True: analogVal = ADC.read(0) Vr = 5 * float(analogVal) / 255 Rt = 10000 * Vr / (5 - Vr) temp = 1/(((math.log(Rt / 10000)) / 3950) + (1 / (273.15+25))) temp = temp - 273.15 print 'temperature = ', temp, 'C' publish("iotstream", get_serial() + "," +str(temp)) time.sleep(INTERVAL) if __name__ == '__main__': try: setup() loop() except KeyboardInterrupt: pass
33. ‹#› @tpryan Source Code - Raspberry Pi - Python def publish(topic_name, data): pubsub_client = pubsub.Client("gcpiotdemo") topic = pubsub_client.topic(topic_name) message_id = topic.publish(data) print('Message {} published.'.format(message_id))
34. ‹#› @tpryan Source Code - Raspberry Pi - Python def get_serial(): cpuserial = "0000000000000000" try: f = open('/proc/cpuinfo','r') for line in f: if line[0:6]=='Serial': cpuserial = line[10:26] f.close() except: cpuserial = "ERROR00000000000" return cpuserial
35. ‹#› @tpryan
36. ‹#› @tpryan Source Code - App Engine - PHP use google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $message = json_decode(file_get_contents('php://input')); $datastr = base64_decode(base64_decode($message->message->data)); $arr = explode(",", $datastr); // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->time = time(); $instance_data->device = $arr[0]; $instance_data->temp = $arr[1]; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/pubsub/${instance}/${request}.json", $instance_json); use google\appengine\api\cloud_storage\CloudStorageTools; // Get app starting variables. $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName(); $instance = $_SERVER['INSTANCE_ID']; $request = $_SERVER['REQUEST_LOG_ID']; $message = json_decode(file_get_contents('php://input')); $datastr = base64_decode(base64_decode($message->message->data)); $arr = explode(",", $datastr); // Create data structure $instance_data = new stdClass(); $instance_data->instance = $instance; $instance_data->request = $request; $instance_data->remote_address = $_SERVER['REMOTE_ADDR']; $instance_data->time = time(); $instance_data->device = $arr[0]; $instance_data->temp = $arr[1]; $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT); file_put_contents("gs://${default_bucket}/pubsub/${instance}/${request}.json", $instance_json);
37. ‹#› @tpryan Demo: Connecting from RaspberryPi
38. ‹#› @tpryan Lessons Learned • Securing IoT devices is not trivial
39. ‹#› @tpryan Data analysis with BigQuery
40. @tpryan 10 x a second * 60 * 60 * 24 864,000 data points a day * 400 sensors 345,600,000 data points a day * 365 days a year 126,144,000,000 data points a year
41. @tpryan IoT = BigData
42. ‹#› @tpryan Big Query • Scan Terabytes in seconds • Use SQLish Queries • REST, Web UI, ODBC
43. ‹#› @tpryan Demo: BigQuery
44. @tpryan Count Shakespeare SELECT count(word) FROM publicdata:samples.shakespeare
45. @tpryan Count to a Million SELECT sum(requests) as total FROM [fh-bigquery:wikipedia.pagecounts_20151109_18]
46. @tpryan Count to a Billion SELECT sum(requests) as total FROM [fh-bigquery:wikipedia.pagecounts_201505]
47. @tpryan Count to a Trillion SELECT SUM(requests) AS total FROM TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_201[3-4][0-9]{2}$")')
48. @tpryan Run a RegEx on a Hundreds of Billions SELECT SUM(requests) AS total FROM TABLE_QUERY( [fh-bigquery:wikipedia], 'REGEXP_MATCH( table_id, r"pagecounts_201[3-4][0-9]{2}$")') WHERE (REGEXP_MATCH(title, '.*[dD]inosaur.*'))
49. ‹#› @tpryan Demo: Getting our data into BigQuery
50. ‹#› @tpryan 06 Conclusion What now
51. App Engine Cloud Storage Raspberry Pi Pub/Sub Big Query
52. Raspberry Pi
54. Raspberry Pi
55. @tpryan ‹#› Google Cloud Platform Compute Connectivity Big Data Storage Developer Tools Management Machine Learning
56. ‹#› @tpryan App Engine • Flexible runtime • Based on Docker containers • Very few restrictions • Base images can use GCP resources easily • Custom can still use them with more setup • Scales slower • Scales to 1 not zero
57. ‹#› @tpryan Data Flow
58. ‹#› @tpryan Cloud Vision
59. ‹#› @tpryan Demo: Cloud Vision API
60. ‹#› @tpryan Cloud ML
61. ‹#› @tpryan Stackdriver
62. ‹#› @tpryan Lessons Learned • IoT = Big Data • Once you get it to a cloud provider your only limit is creativity • Security doesn’t have a clear path… yet
63. ‹#› @tpryan Lessons Implied • Maybe reduce your sample size • Maybe skip some steps • Maybe batch up data • Maybe do some processing on the device
64. ‹#› @tpryan Next Steps • https://cloud.google.com/solutions/iot/ • https://cloud.google.com/solutions/iot-overview • https://googlecloudplatform.github.io/
65. ‹#› @tpryan Thank You terrenceryan.com @tpryan This preso: http://bit.ly/tpryan-iot