Upgrade to Pro — share decks privately, control downloads, hide ads and more …

GCP and IoT

GCP and IoT

A quick talk about getting GCP to talk to IoT devices

Terrence Ryan

October 12, 2016
Tweet

More Decks by Terrence Ryan

Other Decks in Technology

Transcript

  1. ‹#›
    @tpryan
    Terrence Ryan
    Developer Advocate
    IoT and Google Cloud
    Platform

    View full-size slide

  2. ‹#›
    @tpryan
    Who are you?

    View full-size slide

  3. ‹#›
    @tpryan
    01 Introduction
    What are we trying to do

    View full-size slide

  4. • Products
    • Nest
    • Fitbit
    • WeMo
    • Phillips Hue
    • Lockitron
    • Building Blocks
    • Arduino
    • Raspberry Pi
    • BeagleBone
    • etc
    • etc
    • Providers
    • AWS
    • GCP
    • Azure
    • Building Blocks
    • Storage
    • Analytics
    • Processing

    View full-size slide

  5. • Building Blocks
    • Raspberry Pi
    • BeagleBone
    • Providers
    • GCP
    • Building Blocks
    • Cloud Storage
    • App Engine
    • Pub/Sub
    • BigQuery

    View full-size slide

  6. @tpryan
    Why did you choose these?

    View full-size slide

  7. @tpryan
    These are the things that made
    sense to use.

    View full-size slide

  8. @tpryan
    These are the things I could get
    working

    View full-size slide

  9. ‹#›
    @tpryan
    Beaglebone to
    App Engine to
    Cloud Storage

    View full-size slide

  10. BeagleBone
    App Engine
    Cloud Storage

    View full-size slide

  11. ‹#›
    @tpryan
    App Engine
    • Platform as a Service
    • Give it code
    • It gives you a URL running that code
    • Advantages:
    • Scales up and down
    • Simple
    • Disadvantages
    • Language
    • PHP, Java, Python, Go
    • Restrictions
    • No writing to local disk
    • Other restrictions

    View full-size slide

  12. ‹#›
    @tpryan
    Demo: App Engine

    View full-size slide

  13. ‹#›
    @tpryan
    Cloud Storage
    • File Storage
    • Cheap and fast storage.
    • Easy to write
    • Easy to share
    • Can send to most other GCP Techs
    • Advantages:
    • Scales
    • Simple
    • Disadvantages
    • File based storage not best for analytics

    View full-size slide

  14. BeagleBone
    App Engine
    Cloud Storage

    View full-size slide

  15. ‹#›
    @tpryan
    Source Code - App Engine - PHP
    use google\appengine\api\cloud_storage\CloudStorageTools;
    // Get app starting variables.
    $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName();
    $instance = $_SERVER['INSTANCE_ID'];
    $request = $_SERVER['REQUEST_LOG_ID'];
    // Create data structure
    $instance_data = new stdClass();
    $instance_data->instance = $instance;
    $instance_data->request = $request;
    $instance_data->remote_address = $_SERVER['REMOTE_ADDR'];
    $instance_data->remote_host = $_SERVER['REMOTE_HOST'];
    $instance_data->latlon = $_SERVER['HTTP_X_APPENGINE_CITYLATLONG'];
    $instance_data->time = time();
    $instance_data->data = $_REQUEST;
    $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT);
    file_put_contents("gs://${default_bucket}/simple/${instance}/${request}.json", $instance_json);
    use google\appengine\api\cloud_storage\CloudStorageTools;
    // Get app starting variables.
    $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName();
    $instance = $_SERVER['INSTANCE_ID'];
    $request = $_SERVER['REQUEST_LOG_ID'];
    // Create data structure
    $instance_data = new stdClass();
    $instance_data->instance = $instance;
    $instance_data->request = $request;
    $instance_data->remote_address = $_SERVER['REMOTE_ADDR'];
    $instance_data->remote_host = $_SERVER['REMOTE_HOST'];
    $instance_data->latlon = $_SERVER['HTTP_X_APPENGINE_CITYLATLONG'];
    $instance_data->time = time();
    $instance_data->data = $_REQUEST;
    $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT);
    file_put_contents("gs://${default_bucket}/simple/${instance}/${request}.json", $instance_json);

    View full-size slide

  16. ‹#›
    @tpryan
    Source Code - Beaglebone - Python
    import time
    import Adafruit_BBIO.GPIO as GPIO
    import urllib2
    # Note: Use P9_22(UART2_RXD) as GPIO.
    # Connect the Grove Button to UART Grove port of Beaglebone Green.
    Button = "P9_22" # GPIO P9_22
    GPIO.setup(Button, GPIO.IN)
    base_url = "https://simple-dot-gcpiotdemo.appspot.com"
    params = "source=beagleboard&type=button&reading=1"
    if __name__== '__main__':
    while True:
    if GPIO.input(Button):
    print "Button is pressed."
    urllib2.urlopen(base_url + "/?" + params).read()
    time.sleep(.1)
    else:
    print "Button is unstuck."
    time.sleep(1)
    import time
    import Adafruit_BBIO.GPIO as GPIO
    import urllib2
    # Note: Use P9_22(UART2_RXD) as GPIO.
    # Connect the Grove Button to UART Grove port of Beaglebone Green.
    Button = "P9_22" # GPIO P9_22
    GPIO.setup(Button, GPIO.IN)
    base_url = "https://simple-dot-gcpiotdemo.appspot.com"
    params = "source=beagleboard&type=button&reading=1"
    if __name__== '__main__':
    while True:
    if GPIO.input(Button):
    print "Button is pressed."
    urllib2.urlopen(base_url + "/?" + params).read()
    time.sleep(.1)
    else:
    print "Button is unstuck."
    time.sleep(1)

    View full-size slide

  17. ‹#›
    @tpryan
    Demo: Connecting from BeagleBoard to App Engine

    View full-size slide

  18. ‹#›
    @tpryan
    Lessons Learned
    • Maybe go more mainstream kit
    • Raspberry Pi
    • Arduino
    • With more updated sensors

    View full-size slide

  19. ‹#›
    @tpryan
    Raspberry Pi to
    Pub/Sub to
    App Engine to
    Cloud Storage

    View full-size slide

  20. App Engine
    Cloud Storage
    Raspberry Pi
    Pub/Sub

    View full-size slide

  21. ‹#›
    @tpryan
    Pub/Sub
    • Messenger service
    • Publish and Subscribe Model
    • Advantages:
    • Unlimited Quota
    • Acts as a buffer
    • Disadvantages
    • Requires security on the device

    View full-size slide

  22. ‹#›
    @tpryan
    Cloud PUB/SUB

    View full-size slide

  23. ‹#›
    @tpryan

    View full-size slide

  24. ‹#›
    @tpryan
    Source Code - Raspberry Pi - Python
    #!/usr/bin/env python
    import PCF8591 as ADC
    import RPi.GPIO as GPIO
    import time
    import math
    import urllib2
    import os
    from google.cloud import pubsub
    INTERVAL = .1
    DO = 17
    GPIO.setmode(GPIO.BCM)
    os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/home/pi/gcp/creds.json"
    #!/usr/bin/env python
    import PCF8591 as ADC
    import RPi.GPIO as GPIO
    import time
    import math
    import urllib2
    import os
    from google.cloud import pubsub
    INTERVAL = .1
    DO = 17
    GPIO.setmode(GPIO.BCM)
    os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "/home/pi/gcp/creds.json"

    View full-size slide

  25. ‹#›
    @tpryan

    View full-size slide

  26. ‹#›
    @tpryan
    Source Code - creds.json
    {
    "type": "service_account",
    "project_id": "gcpiotdemo",
    "private_key_id": "a098b68643ae4ccda140d907d5a2394a895e8be4",
    "private_key": "-----BEGIN PRIVATE KEY——\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBA….Q..\n——END PRIVATE KEY-----\n",
    "client_email": "[email protected]",
    "client_id": "106452013022587438299",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://accounts.google.com/o/oauth2/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/raspberrypiclient%40gcpiotdemo.iam.gserviceaccount.com"
    }

    View full-size slide

  27. ‹#›
    @tpryan
    Source Code - Raspberry Pi - Python
    def setup():
    ADC.setup(0x48)
    GPIO.setup(DO, GPIO.IN)
    def loop():
    status = 1
    tmp = 1
    while True:
    analogVal = ADC.read(0)
    Vr = 5 * float(analogVal) / 255
    Rt = 10000 * Vr / (5 - Vr)
    temp = 1/(((math.log(Rt / 10000)) / 3950) + (1 / (273.15+25)))
    temp = temp - 273.15
    print 'temperature = ', temp, 'C'
    publish("iotstream", get_serial() + "," +str(temp))
    time.sleep(INTERVAL)
    if __name__ == '__main__':
    try:
    setup()
    loop()
    except KeyboardInterrupt:
    pass
    def setup():
    ADC.setup(0x48)
    GPIO.setup(DO, GPIO.IN)
    def loop():
    status = 1
    tmp = 1
    while True:
    analogVal = ADC.read(0)
    Vr = 5 * float(analogVal) / 255
    Rt = 10000 * Vr / (5 - Vr)
    temp = 1/(((math.log(Rt / 10000)) / 3950) + (1 / (273.15+25)))
    temp = temp - 273.15
    print 'temperature = ', temp, 'C'
    publish("iotstream", get_serial() + "," +str(temp))
    time.sleep(INTERVAL)
    if __name__ == '__main__':
    try:
    setup()
    loop()
    except KeyboardInterrupt:
    pass

    View full-size slide

  28. ‹#›
    @tpryan
    Source Code - Raspberry Pi - Python
    def publish(topic_name, data):
    pubsub_client = pubsub.Client("gcpiotdemo")
    topic = pubsub_client.topic(topic_name)
    message_id = topic.publish(data)
    print('Message {} published.'.format(message_id))

    View full-size slide

  29. ‹#›
    @tpryan
    Source Code - Raspberry Pi - Python
    def get_serial():
    cpuserial = "0000000000000000"
    try:
    f = open('/proc/cpuinfo','r')
    for line in f:
    if line[0:6]=='Serial':
    cpuserial = line[10:26]
    f.close()
    except:
    cpuserial = "ERROR00000000000"
    return cpuserial

    View full-size slide

  30. ‹#›
    @tpryan

    View full-size slide

  31. ‹#›
    @tpryan
    Source Code - App Engine - PHP
    use google\appengine\api\cloud_storage\CloudStorageTools;
    // Get app starting variables.
    $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName();
    $instance = $_SERVER['INSTANCE_ID'];
    $request = $_SERVER['REQUEST_LOG_ID'];
    $message = json_decode(file_get_contents('php://input'));
    $datastr = base64_decode(base64_decode($message->message->data));
    $arr = explode(",", $datastr);
    // Create data structure
    $instance_data = new stdClass();
    $instance_data->instance = $instance;
    $instance_data->request = $request;
    $instance_data->remote_address = $_SERVER['REMOTE_ADDR'];
    $instance_data->time = time();
    $instance_data->device = $arr[0];
    $instance_data->temp = $arr[1];
    $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT);
    file_put_contents("gs://${default_bucket}/pubsub/${instance}/${request}.json", $instance_json);
    use google\appengine\api\cloud_storage\CloudStorageTools;
    // Get app starting variables.
    $default_bucket = CloudStorageTools::getDefaultGoogleStorageBucketName();
    $instance = $_SERVER['INSTANCE_ID'];
    $request = $_SERVER['REQUEST_LOG_ID'];
    $message = json_decode(file_get_contents('php://input'));
    $datastr = base64_decode(base64_decode($message->message->data));
    $arr = explode(",", $datastr);
    // Create data structure
    $instance_data = new stdClass();
    $instance_data->instance = $instance;
    $instance_data->request = $request;
    $instance_data->remote_address = $_SERVER['REMOTE_ADDR'];
    $instance_data->time = time();
    $instance_data->device = $arr[0];
    $instance_data->temp = $arr[1];
    $instance_json = json_encode($instance_data, JSON_PRETTY_PRINT);
    file_put_contents("gs://${default_bucket}/pubsub/${instance}/${request}.json", $instance_json);

    View full-size slide

  32. ‹#›
    @tpryan
    Demo: Connecting from RaspberryPi

    View full-size slide

  33. ‹#›
    @tpryan
    Lessons Learned
    • Securing IoT devices is not trivial

    View full-size slide

  34. ‹#›
    @tpryan
    Data analysis with
    BigQuery

    View full-size slide

  35. @tpryan
    10 x a second
    * 60 * 60 * 24
    864,000 data points a day
    * 400 sensors
    345,600,000 data points a day
    * 365 days a year
    126,144,000,000 data points a year

    View full-size slide

  36. @tpryan
    IoT = BigData

    View full-size slide

  37. ‹#›
    @tpryan
    Big Query
    • Scan Terabytes in seconds
    • Use SQLish Queries
    • REST, Web UI, ODBC

    View full-size slide

  38. ‹#›
    @tpryan
    Demo: BigQuery

    View full-size slide

  39. @tpryan
    Count Shakespeare
    SELECT count(word)
    FROM publicdata:samples.shakespeare

    View full-size slide

  40. @tpryan
    Count to a Million
    SELECT sum(requests) as total
    FROM [fh-bigquery:wikipedia.pagecounts_20151109_18]

    View full-size slide

  41. @tpryan
    Count to a Billion
    SELECT sum(requests) as total
    FROM [fh-bigquery:wikipedia.pagecounts_201505]

    View full-size slide

  42. @tpryan
    Count to a Trillion
    SELECT
    SUM(requests) AS total
    FROM
    TABLE_QUERY(
    [fh-bigquery:wikipedia],
    'REGEXP_MATCH(
    table_id,
    r"pagecounts_201[3-4][0-9]{2}$")')

    View full-size slide

  43. @tpryan
    Run a RegEx on a Hundreds of Billions
    SELECT
    SUM(requests) AS total
    FROM
    TABLE_QUERY(
    [fh-bigquery:wikipedia],
    'REGEXP_MATCH(
    table_id,
    r"pagecounts_201[3-4][0-9]{2}$")')
    WHERE
    (REGEXP_MATCH(title, '.*[dD]inosaur.*'))

    View full-size slide

  44. ‹#›
    @tpryan
    Demo: Getting our data into BigQuery

    View full-size slide

  45. ‹#›
    @tpryan
    06 Conclusion
    What now

    View full-size slide

  46. App Engine
    Cloud Storage
    Raspberry Pi
    Pub/Sub
    Big Query

    View full-size slide

  47. Raspberry Pi

    View full-size slide

  48. Raspberry Pi

    View full-size slide

  49. @tpryan ‹#›
    Google Cloud Platform
    Compute
    Connectivity
    Big Data
    Storage
    Developer
    Tools
    Management
    Machine Learning

    View full-size slide

  50. ‹#›
    @tpryan
    App Engine
    • Flexible runtime
    • Based on Docker containers
    • Very few restrictions
    • Base images can use GCP resources easily
    • Custom can still use them with more setup
    • Scales slower
    • Scales to 1 not zero

    View full-size slide

  51. ‹#›
    @tpryan
    Data Flow

    View full-size slide

  52. ‹#›
    @tpryan
    Cloud Vision

    View full-size slide

  53. ‹#›
    @tpryan
    Demo: Cloud Vision API

    View full-size slide

  54. ‹#›
    @tpryan
    Cloud ML

    View full-size slide

  55. ‹#›
    @tpryan
    Stackdriver

    View full-size slide

  56. ‹#›
    @tpryan
    Lessons Learned
    • IoT = Big Data
    • Once you get it to a cloud provider your only limit is creativity
    • Security doesn’t have a clear path… yet

    View full-size slide

  57. ‹#›
    @tpryan
    Lessons Implied
    • Maybe reduce your sample size
    • Maybe skip some steps
    • Maybe batch up data
    • Maybe do some processing on the device

    View full-size slide

  58. ‹#›
    @tpryan
    Next Steps
    • https://cloud.google.com/solutions/iot/
    • https://cloud.google.com/solutions/iot-overview
    • https://googlecloudplatform.github.io/

    View full-size slide

  59. ‹#›
    @tpryan
    Thank You
    terrenceryan.com
    @tpryan
    This preso: http://bit.ly/tpryan-iot

    View full-size slide