Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ML and the IoT: Living on the Edge

ML and the IoT: Living on the Edge

Machine Learning and the IoT are a match made in heaven. After all, IoT devices collect mountains of sensor data, what better way to uncover insights and actions than through sophisticated, modern computing methods like ML and AI?

The problem is, leveraging ML with IoT has historically meant backhauling all your sensor data to the Cloud. When the cloud is involved, security is a concern, and in the realm of IoT, security is often a dirty word.

But modern embedded systems, microcontrollers and single-board computers are getting more powerful, and more sophisticated, and its becoming increasingly possible to bring Machine Learning closer to sensors and IoT devices. "Edge ML" enables quicker insights, tighter security, and even true predictive action, and it's going to become the norm in the IoT in the near future.

In this session, we'll explore the state of the art in Edge ML and IoT, and talk about practical ways that developers can get started with both, today.

Brandon Satrom

June 20, 2019
Tweet

More Decks by Brandon Satrom

Other Decks in Technology

Transcript

  1. MACHINE LEARNING & THE IOT: LIVING ON THE EDGE
    BRANDON SATROM | [email protected]

    View full-size slide

  2. To “Zeke”

    View full-size slide

  3. To “Zeke”

    View full-size slide

  4. 2 ESSENTIAL TRUTHS OF MACHINE LEARNING
    1 2
    DATA DOESN’T COME FROM THE CLOUD MACHINE LEARNING IS HUMAN

    View full-size slide

  5. 2 ESSENTIAL TRUTHS OF MACHINE LEARNING
    1 2
    DATA DOESN’T COME FROM THE CLOUD MACHINE LEARNING IS HUMAN

    View full-size slide

  6. 2 ESSENTIAL TRUTHS OF MACHINE LEARNING
    1 2
    DATA DOESN’T COME FROM THE CLOUD MACHINE LEARNING IS HUMAN

    View full-size slide

  7. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT

    View full-size slide

  8. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT

    View full-size slide

  9. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT

    View full-size slide

  10. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT

    View full-size slide

  11. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT

    View full-size slide

  12. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT

    View full-size slide

  13. AI, MACHINE LEARNING & DEEP LEARNING
    AI
    Machine learning
    Representation learning
    Deep learning
    Example:
    Knowledge
    bases
    Example:
    Logistic
    regression
    Example:
    Shallow
    autoencoders
    Example:
    MLPs

    View full-size slide

  14. ALGORITHMS VS. PREDICITONS

    View full-size slide

  15. ALGORITHMS VS. PREDICITONS
    “Learning” Happens Here

    View full-size slide

  16. DEEP LEARNING: FINDING CORRELATION AMONG COMPLEX DATA

    View full-size slide

  17. DEEP LEARNING AS FEATURE EXTRACTION

    View full-size slide

  18. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask

    View full-size slide

  19. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    Some combination of
    these three lights
    tells us when its safe
    to walk or stop

    View full-size slide

  20. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    Some combination of
    these three lights
    tells us when its safe
    to walk or stop

    View full-size slide

  21. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    Some combination of
    these three lights
    tells us when its safe
    to walk or stop

    View full-size slide

  22. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    Some combination of
    these three lights
    tells us when its safe
    to walk or stop

    View full-size slide

  23. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    Some combination of
    these three lights
    tells us when its safe
    to walk or stop
    After a few rounds, a
    human can infer that
    the middle light is the
    key

    View full-size slide

  24. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask

    View full-size slide

  25. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask

    View full-size slide

  26. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask

    View full-size slide

  27. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask

    View full-size slide

  28. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    import numpy as np
    weights = np.array([0.5, 0.48, -0.7])
    alpha = 0.1
    streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1],
    [0, 1, 1],[1, 0, 1]])
    walk_vs_stop = np.array([0, 1, 0, 1, 1, 0])
    input = streetlights[0]
    goal_prediction = walk_vs_stop[0]
    for iteration in range(40):
    error_for_all_lights = 0
    for row_index in range(len(walk_vs_stop)):
    input = streetlights[row_index]
    goal_prediction = walk_vs_stop[row_index]
    prediction = input.dot(weights)
    error = (goal_prediction - prediction) "** 2
    error_for_all_lights += error
    delta = prediction - goal_prediction
    weights = weights - (alpha * (input * delta))
    print("Prediction:" + str(prediction))
    print("Error:" + str(error_for_all_lights) + "\n")

    View full-size slide

  29. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    import numpy as np
    weights = np.array([0.5, 0.48, -0.7])
    alpha = 0.1
    streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1],
    [0, 1, 1],[1, 0, 1]])
    walk_vs_stop = np.array([0, 1, 0, 1, 1, 0])
    input = streetlights[0]
    goal_prediction = walk_vs_stop[0]
    for iteration in range(40):
    error_for_all_lights = 0
    for row_index in range(len(walk_vs_stop)):
    input = streetlights[row_index]
    goal_prediction = walk_vs_stop[row_index]
    prediction = input.dot(weights)
    error = (goal_prediction - prediction) "** 2
    error_for_all_lights += error
    delta = prediction - goal_prediction
    weights = weights - (alpha * (input * delta))
    print("Prediction:" + str(prediction))
    print("Error:" + str(error_for_all_lights) + "\n")

    View full-size slide

  30. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    import numpy as np
    weights = np.array([0.5, 0.48, -0.7])
    alpha = 0.1
    streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1],
    [0, 1, 1],[1, 0, 1]])
    walk_vs_stop = np.array([0, 1, 0, 1, 1, 0])
    input = streetlights[0]
    goal_prediction = walk_vs_stop[0]
    for iteration in range(40):
    error_for_all_lights = 0
    for row_index in range(len(walk_vs_stop)):
    input = streetlights[row_index]
    goal_prediction = walk_vs_stop[row_index]
    prediction = input.dot(weights)
    error = (goal_prediction - prediction) "** 2
    error_for_all_lights += error
    delta = prediction - goal_prediction
    weights = weights - (alpha * (input * delta))
    print("Prediction:" + str(prediction))
    print("Error:" + str(error_for_all_lights) + "\n")
    Prediction: -0.19999999999999996
    Prediction: -0.19999999999999996
    Prediction: -0.5599999999999999
    Prediction: 0.6160000000000001
    Prediction: 0.17279999999999995
    Prediction: 0.17552
    Error: 2.6561231104

    Prediction: -0.0022410273814405524
    Prediction: 0.9978745386023716
    Prediction: -0.016721264429884947
    Prediction: 1.0151127459893812
    Prediction: 0.9969492081270097
    Prediction: -0.0026256193329783125
    Error:0.00053373677328488

    View full-size slide

  31. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    import numpy as np
    weights = np.array([0.5, 0.48, -0.7])
    alpha = 0.1
    streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1],
    [0, 1, 1],[1, 0, 1]])
    walk_vs_stop = np.array([0, 1, 0, 1, 1, 0])
    input = streetlights[0]
    goal_prediction = walk_vs_stop[0]
    for iteration in range(40):
    error_for_all_lights = 0
    for row_index in range(len(walk_vs_stop)):
    input = streetlights[row_index]
    goal_prediction = walk_vs_stop[row_index]
    prediction = input.dot(weights)
    error = (goal_prediction - prediction) "** 2
    error_for_all_lights += error
    delta = prediction - goal_prediction
    weights = weights - (alpha * (input * delta))
    print("Prediction:" + str(prediction))
    print("Error:" + str(error_for_all_lights) + "\n")
    Prediction: -0.19999999999999996
    Prediction: -0.19999999999999996
    Prediction: -0.5599999999999999
    Prediction: 0.6160000000000001
    Prediction: 0.17279999999999995
    Prediction: 0.17552
    Error: 2.6561231104

    Prediction: -0.0022410273814405524
    Prediction: 0.9978745386023716
    Prediction: -0.016721264429884947
    Prediction: 1.0151127459893812
    Prediction: 0.9969492081270097
    Prediction: -0.0026256193329783125
    Error:0.00053373677328488

    View full-size slide

  32. Machine Learning is Really Human Teaching

    View full-size slide

  33. MACHINE LEARNING IS HUMAN TEACHING
    1 2 3
    CREATE INPUTS AND
    OUTPUTS
    DEFINE A MODEL
    ARCHITECTURE
    TRAIN, EVALUATE, TUNE
    CHAPTER 5. MACHINE LEARNING BASICS
    100 101 102 103 104 105
    Number of training examples
    0.0
    0.5
    1.0
    1.5
    2.0
    2.5
    3.0
    3.5
    Error (MSE)
    Bayes error
    Train (quadratic)
    Test (quadratic)
    Test (optimal capacity)
    Train (optimal capacity)
    10
    15
    20
    ty (polynomial degree)

    View full-size slide

  34. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT

    View full-size slide

  35. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT

    View full-size slide

  36. MACHINE LEARNING ISN’T “DONE” IN ANY ONE PLACE

    View full-size slide

  37. ML = DEVELOPMENT, TRAINING, INFERENCE
    1 2 3
    DEVELOPMENT TRAINING INFERENCE

    View full-size slide

  38. MODEL CREATION TOOLS

    View full-size slide

  39. TRAINING, EXPERIMENTS, & TUNING

    View full-size slide

  40. DEMO
    TRAINING AN EMOTION MODEL

    View full-size slide

  41. THE TRAIN, PREDICT, TUNE LOOP

    View full-size slide

  42. THE TRAIN, PREDICT, TUNE LOOP
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model

    View full-size slide

  43. INFERENCE: PREDICTION IN REAL TIME

    View full-size slide

  44. INFERENCE: PREDICTION IN REAL TIME
    During training, the
    network “learns” by
    tuning weights to
    reduce error

    View full-size slide

  45. INFERENCE: PREDICTION IN REAL TIME
    During training, the
    network “learns” by
    tuning weights to
    reduce error
    These weights are then applied
    against data the network has never
    seen before to make a prediction

    View full-size slide

  46. INFERENCE: PREDICTION IN REAL TIME
    During training, the
    network “learns” by
    tuning weights to
    reduce error
    These weights are then applied
    against data the network has never
    seen before to make a prediction

    View full-size slide

  47. DEMO
    RUNNING INFERENCE ON A MODEL

    View full-size slide

  48. MODEL DEPLOYMENT
    +
    +
    +

    View full-size slide

  49. PRE-TRAINED MODELS AS APIS

    View full-size slide

  50. DEMO
    USING THE AZURE FACE API

    View full-size slide

  51. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model

    View full-size slide

  52. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Training & Validation
    Data

    View full-size slide

  53. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Cloud Training &
    Deployment
    Training & Validation
    Data

    View full-size slide

  54. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Cloud Training &
    Deployment
    Training & Validation
    Data
    Real time Inference
    Data
    ?

    View full-size slide

  55. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT

    View full-size slide

  56. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT

    View full-size slide

  57. TRADITIONAL: DEVICES AND CLOUD

    View full-size slide

  58. TRADITIONAL: DEVICES AND CLOUD
    The Cloud
    The Internet
    of things

    View full-size slide

  59. TRADITIONAL: DEVICES AND CLOUD
    The Cloud
    The Internet
    of things
    “Lots of
    power”

    View full-size slide

  60. TRADITIONAL: DEVICES AND CLOUD
    The Cloud
    The Internet
    of things
    “Lots of
    power”
    “Lots of
    Devices”

    View full-size slide

  61. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    “Lots of
    power”
    “Lots of
    Devices”

    View full-size slide

  62. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    The “Edge”
    “Lots of
    power”
    “Lots of
    Devices”

    View full-size slide

  63. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    The “Edge”
    “Lots of
    power”
    “Lots of
    Devices”

    View full-size slide

  64. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    The “Edge”
    “Lots of
    power”
    “Lots of
    Devices”

    View full-size slide

  65. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    The “Edge”
    “Lots of
    power”
    “Lots of
    Devices”

    View full-size slide

  66. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    The “Edge”
    “Lots of
    power”
    “Lots of
    Devices”
    “A Bit of
    both”

    View full-size slide

  67. THE “EDGE” IS THE INTERNET OF THINGS!
    The Cloud
    The Internet
    of things
    The “Edge”

    View full-size slide

  68. THE “EDGE” IS THE INTERNET OF THINGS!
    The Cloud
    The Internet
    of things
    The “Edge”

    View full-size slide

  69. THE IOT IS THE SOURCE OF DATA FOR ML
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Cloud Training &
    Deployment
    Training & Validation
    Data
    Real time Inference
    Data
    ?

    View full-size slide

  70. THE IOT IS THE SOURCE OF DATA FOR ML
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Cloud Training &
    Deployment
    Training & Validation
    Data
    Real time Inference
    Data
    ?

    View full-size slide

  71. THE IOT IS THE SOURCE OF DATA FOR ML
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Cloud Training &
    Deployment
    Training & Validation
    Data
    Real time Inference
    Data
    ?

    View full-size slide

  72. WHY DO ML INFERENCING IN THE IOT?
    1 2 3
    VOLUME SPEED PRIVACY

    View full-size slide

  73. MORE DEVICES == MORE DATA

    View full-size slide

  74. The Cloud may be infinite;
    your pipe to it is not

    View full-size slide

  75. ML LOVES DATA LIKE COOKIE MONSTER LOVES COOKIES

    View full-size slide

  76. ML LOVES DATA LIKE COOKIE MONSTER LOVES COOKIES

    View full-size slide

  77. DOING INFERENCE IN THE IOT MEANS LESS DATA IN THE CLOUD

    View full-size slide

  78. DOING INFERENCE IN THE IOT MEANS LESS DATA IN THE CLOUD

    View full-size slide

  79. INFERENCE FOR PREVENTATIVE MAINTENANCE

    View full-size slide

  80. INFERENCE FOR PREVENTATIVE MAINTENANCE

    View full-size slide

  81. INFERENCE FOR PREVENTATIVE MAINTENANCE

    View full-size slide

  82. INFERENCE FOR PREVENTATIVE MAINTENANCE

    View full-size slide

  83. INFERENCE FOR PREVENTATIVE MAINTENANCE

    View full-size slide

  84. THERE IS NO CLOUD IF THE NETWORK IS OFFLINE

    View full-size slide

  85. THERE IS NO CLOUD IF THE NETWORK IS OFFLINE

    View full-size slide

  86. INFERENCE FOR PREVENTATIVE MAINTENANCE

    View full-size slide

  87. INFERENCE FOR PREVENTATIVE MAINTENANCE

    View full-size slide

  88. INFERENCE FOR PREVENTATIVE MAINTENANCE

    View full-size slide

  89. IoT Inferencing == Built-In Privacy

    View full-size slide

  90. VISION IN THE CLOUD

    View full-size slide

  91. VISION IN THE CLOUD

    View full-size slide

  92. VISION AT THE EDGE

    View full-size slide

  93. VISION AT THE EDGE

    View full-size slide

  94. VISION AT THE EDGE
    [
    {
    "faceId": "b0f2804a-649f-4da3-b6cf-ff459c867ae8",
    "faceRectangle": {
    "width": 63,
    "left": 168,
    "height": 63,
    "top": 110
    },
    "faceAttributes": {
    "smile": 0.0,
    "exposure": {
    "exposureLevel": "overExposure",
    "value": 0.75
    },
    "emotion": {
    "sadness": 0.0,
    "surprise": 0.0,
    "neutral": 0.0,
    "contempt": 0.001,
    "anger": 0.666,
    "disgust": 0.333,
    "fear": 0.0,
    "happiness": 0.0
    }
    "facialHair": {
    "beard": 0.4,
    "moustache": 0.4,
    "sideburns": 0.4
    },
    "gender": "male",
    "age": 33.0,
    "headPose": {
    "yaw": -24.3,
    "pitch": -6.4,
    "roll": -8.1
    }
    }
    }
    ]

    View full-size slide

  95. EDGE ML INFERENCING DEVICES
    1 2
    GOOGLE CORAL NVIDIA JETSON NANO

    View full-size slide

  96. GOOGLE CORAL
    • Edge TPU Module (SOM)
    • NXP i.MX 8M SOC (Quad-core Cortex-A53, plus Cortex-M4F)
    • Google Edge TPU ML accelerator coprocessor
    • Wi-Fi 2x2 MIMO (802.11b/g/n/ac 2.4/5GHz)
    • Bluetooth 4.1
    • 8GB eMMC
    • 1GB LPDDR4
    • USB connections
    • USB Type-C power port (5V DC)
    • USB 3.0 Type-C OTG port
    • USB 3.0 Type-A host port
    • USB 2.0 Micro-B serial console port
    • Audio & Video connections
    • Digital PDM microphone (x2)
    • HDMI 2.0a (full size)
    • 39-pin FFC connector for MIPI DSI display (4-lane)
    • 24-pin FFC connector for MIPI CSI-2 camera (4-lane)
    • MicroSD card slot
    • Gigabit Ethernet port
    • 40-pin GPIO expansion header
    • Supports Mendel Linux (derivative of Debian)

    View full-size slide

  97. NVIDIA JETSON NANO
    • GPU and CPU
    • 128 Core Maxwell (472 GFLOPs)
    • Quad-core Cortex-A57 @ 1.43 GHz
    • 4 GB 64 bit LPDDR4 25.6 GB/s
    • 16GB eMMC
    • Video
    • HDMI 2.0 or DP1.2
    • 4K Encode and Decode
    • 12-pin FFC connector for MIPI CSI-2 camera (1.1-lane)
    • Connections
    • 1x SDIO, 2x SPI, 5x SysIO, 13x GPIOs, 6x I2C
    • USB connections
    • 1 USB 3.0
    • MicroSD card slot
    • Gigabit Ethernet port
    • Supports Ubuntu Linux

    View full-size slide

  98. DEMO
    FACE DETECTION WITH GOOGLE CORAL

    View full-size slide

  99. Where does this leave the Internet of
    Things?

    View full-size slide

  100. THE INCREASING ROLE OF THE IOT IN ML
    1 2
    COLLECTION &
    COORDINATION ON
    DEVICES
    ML INFERENCING ON
    MICROCONTROLLERS!

    View full-size slide

  101. ML data doesn’t come from the cloud, or the
    edge…
    It comes from the IoT

    View full-size slide

  102. IOT DEVICES SENSE AND SHARE

    View full-size slide

  103. IOT DEVICES SENSE AND SHARE

    View full-size slide

  104. IOT DEVICES TAKE ACTION

    View full-size slide

  105. IOT DEVICES TAKE ACTION

    View full-size slide

  106. IOT DEVICES TAKE ACTION

    View full-size slide

  107. IOT DEVICES TAKE ACTION

    View full-size slide

  108. Edge ML + Mesh Networking = Crazy
    Delicious

    View full-size slide

  109. TRADITIONAL IOT: HUB AND SPOKE
    Wi-Fi Router

    View full-size slide

  110. TRADITIONAL IOT: HUB AND SPOKE
    Wi-Fi Router

    View full-size slide

  111. TRADITIONAL IOT: HUB AND SPOKE
    Wi-Fi Router

    View full-size slide

  112. TRADITIONAL IOT: HUB AND SPOKE

    View full-size slide

  113. ENTER MESH NETWORKING
    Gateway

    View full-size slide

  114. ENTER MESH NETWORKING
    Gateway

    View full-size slide

  115. ENTER MESH NETWORKING
    Gateway

    View full-size slide

  116. ENTER MESH NETWORKING
    Gateway

    View full-size slide

  117. ENTER MESH NETWORKING
    Gateway

    View full-size slide

  118. ENTER MESH NETWORKING

    View full-size slide

  119. OPENTHREAD VS. ZIGBEE, ZWAVE AND LORA
    Operating range 100 ft 35 ft
    Max # of devices 232 65,000
    Data rate 9.6-100 kbps 40-250 kbps
    Cost $ $$
    IP-Based Networking No No
    Open Standard? No Yes
    1-4 mi 100 ft
    N/A 300
    27 kbps 250 kbps
    $$$ $
    Yes Yes
    No Yes

    View full-size slide

  120. WHAT IS THREAD?
    ✴IPv6-based mesh
    ✴Wireless Personal Area Network
    ✴No single point of failure
    ✴Tailored to IoT Scenarios
    ✴Can be used in concert with Wi-Fi, Cellular and Bluetooth
    is a low-power networking protocol

    View full-size slide

  121. PARTICLE MESH DEVICES
    Nordic nRF52840 SoC
    ✴ ARM Cortex-M4F 32-bit
    ✴ 1MB flash, 256KB RAM
    ✴ IEEE 802.15.4-2006
    ✴ Bluetooth 5: 2 Mbps, 1 Mbps, 500
    Kbps, 125 Kbps
    ✴ ARM TrustZone Cryptographic
    security module
    ✴ NFC-A tag
    Argon
    » Wi-Fi + BLE +Mesh
    » Wi-Fi endpoint or mesh gateway
    Xenon
    » BLE + Mesh
    » Mesh endpoint
    Boron
    » LTE-M1 + BLE + Mesh
    » Cellular endpoint or mesh gateway

    View full-size slide

  122. BRINGING AN EDGE DEVICE INTO A MESH NETWORK
    The Internet
    of things

    View full-size slide

  123. BRINGING AN EDGE DEVICE INTO A MESH NETWORK
    The Internet
    of things

    View full-size slide

  124. BRINGING AN EDGE DEVICE INTO A MESH NETWORK
    The Internet
    of things
    We can bring an Edge ML
    device into an IoT mesh
    network

    View full-size slide

  125. DEMO
    EMOTION MESH

    View full-size slide

  126. THE FUTURE OF ML AND THE IOT
    1 2
    ML INFERENCING ON
    MICROCONTROLLERS!
    FEDERATED AND
    DISTRIBUTED LEARNING

    View full-size slide

  127. TENSORFLOW LITE MICRO

    View full-size slide

  128. MICROTENSOR = ML INFERENCING ON ARM-BASED DEVICES

    View full-size slide

  129. MICROTENSOR = ML INFERENCING ON ARM-BASED DEVICES

    View full-size slide

  130. TF LITE + MICROTENSOR == A ROBUST, OPEN APPROACH TO MCU ML INFERENCING

    View full-size slide

  131. DEMO
    ML ON MCUS

    View full-size slide

  132. USING FEDERATED LEARNING TO MAXIMIZE IOT COMPUTE RESOURCES

    View full-size slide

  133. USING FEDERATED LEARNING TO MAXIMIZE IOT COMPUTE RESOURCES
    IoT Devices spend most
    of their time idle…

    View full-size slide

  134. USING FEDERATED LEARNING TO MAXIMIZE IOT COMPUTE RESOURCES
    IoT Devices spend most
    of their time idle…
    Federated learning may enable models to be
    distributed between the cloud, edge and devices for
    the most efficient processing

    View full-size slide

  135. OTA MODEL UPDATES!

    View full-size slide

  136. OTA MODEL UPDATES!

    View full-size slide

  137. OTA MODEL UPDATES!

    View full-size slide

  138. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT

    View full-size slide

  139. Explore ML inferencing with SBCs

    View full-size slide

  140. Explore Mesh networking for local network
    intelligence

    View full-size slide

  141. Explore TensorFlow Lite and µTensor for ML
    on MCUs

    View full-size slide

  142. THANK YOU!
    BRANDON SATROM | [email protected]
    GITHUB.COM/BSATROM/EMOTION-MESH

    View full-size slide