ML and the IoT: Living on the Edge

Machine Learning and the IoT are a match made in heaven. After all, IoT devices collect mountains of sensor data, what better way to uncover insights and actions than through sophisticated, modern computing methods like ML and AI?

The problem is, leveraging ML with IoT has historically meant backhauling all your sensor data to the Cloud. When the cloud is involved, security is a concern, and in the realm of IoT, security is often a dirty word.

But modern embedded systems, microcontrollers and single-board computers are getting more powerful, and more sophisticated, and its becoming increasingly possible to bring Machine Learning closer to sensors and IoT devices. "Edge ML" enables quicker insights, tighter security, and even true predictive action, and it's going to become the norm in the IoT in the near future.

In this session, we'll explore the state of the art in Edge ML and IoT, and talk about practical ways that developers can get started with both, today.

Transcript

1. MACHINE LEARNING & THE IOT: LIVING ON THE EDGE
   BRANDON SATROM | BRANDON@PARTICLE.IO
   

   View Slide

2. View Slide

3. Zeke
   

   View Slide

4. Zeke
   

   View Slide

5. To “Zeke”
   

   View Slide

6. To “Zeke”
   

   View Slide

7. Zeked!
   

   View Slide

8. Zeked!
   

   View Slide

9. Zeked!
   

   View Slide

10. Zeked!
    

    View Slide

11. Zeked!
    

    View Slide

12. Zeked!
    

    View Slide

13. Zeked!
    

    View Slide

14. Zeked!
    

    View Slide

15. Zeked!
    

    View Slide

16. View Slide

17. View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. 2 ESSENTIAL TRUTHS OF MACHINE LEARNING
    1 2
    DATA DOESN’T COME FROM THE CLOUD MACHINE LEARNING IS HUMAN
    

    View Slide

31. 2 ESSENTIAL TRUTHS OF MACHINE LEARNING
    1 2
    DATA DOESN’T COME FROM THE CLOUD MACHINE LEARNING IS HUMAN
    

    View Slide

32. 2 ESSENTIAL TRUTHS OF MACHINE LEARNING
    1 2
    DATA DOESN’T COME FROM THE CLOUD MACHINE LEARNING IS HUMAN
    

    View Slide

33. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT
    

    View Slide

34. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT
    

    View Slide

35. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT
    

    View Slide

36. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT
    

    View Slide

37. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT
    

    View Slide

38. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT
    

    View Slide

39. AI, MACHINE LEARNING & DEEP LEARNING
    AI
    Machine learning
    Representation learning
    Deep learning
    Example:
    Knowledge
    bases
    Example:
    Logistic
    regression
    Example:
    Shallow
    autoencoders
    Example:
    MLPs
    

    View Slide

40. ALGORITHMS VS. PREDICITONS
    

    View Slide

41. ALGORITHMS VS. PREDICITONS
    “Learning” Happens Here
    

    View Slide

42. DEEP LEARNING: FINDING CORRELATION AMONG COMPLEX DATA
    

    View Slide

43. DEEP LEARNING AS FEATURE EXTRACTION
    

    View Slide

44. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    

    View Slide

45. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    Some combination of
    these three lights
    tells us when its safe
    to walk or stop
    

    View Slide

46. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    Some combination of
    these three lights
    tells us when its safe
    to walk or stop
    

    View Slide

47. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    Some combination of
    these three lights
    tells us when its safe
    to walk or stop
    

    View Slide

48. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    Some combination of
    these three lights
    tells us when its safe
    to walk or stop
    

    View Slide

49. THE STREETLIGHT PROBLEM
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    Some combination of
    these three lights
    tells us when its safe
    to walk or stop
    After a few rounds, a
    human can infer that
    the middle light is the
    key
    

    View Slide

50. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    

    View Slide

51. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    

    View Slide

52. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    

    View Slide

53. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    

    View Slide

54. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    import numpy as np
    weights = np.array([0.5, 0.48, -0.7])
    alpha = 0.1
    streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1],
    [0, 1, 1],[1, 0, 1]])
    walk_vs_stop = np.array([0, 1, 0, 1, 1, 0])
    input = streetlights[0]
    goal_prediction = walk_vs_stop[0]
    for iteration in range(40):
    error_for_all_lights = 0
    for row_index in range(len(walk_vs_stop)):
    input = streetlights[row_index]
    goal_prediction = walk_vs_stop[row_index]
    prediction = input.dot(weights)
    error = (goal_prediction - prediction) "** 2
    error_for_all_lights += error
    delta = prediction - goal_prediction
    weights = weights - (alpha * (input * delta))
    print("Prediction:" + str(prediction))
    print("Error:" + str(error_for_all_lights) + "\n")
    

    View Slide

55. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    import numpy as np
    weights = np.array([0.5, 0.48, -0.7])
    alpha = 0.1
    streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1],
    [0, 1, 1],[1, 0, 1]])
    walk_vs_stop = np.array([0, 1, 0, 1, 1, 0])
    input = streetlights[0]
    goal_prediction = walk_vs_stop[0]
    for iteration in range(40):
    error_for_all_lights = 0
    for row_index in range(len(walk_vs_stop)):
    input = streetlights[row_index]
    goal_prediction = walk_vs_stop[row_index]
    prediction = input.dot(weights)
    error = (goal_prediction - prediction) "** 2
    error_for_all_lights += error
    delta = prediction - goal_prediction
    weights = weights - (alpha * (input * delta))
    print("Prediction:" + str(prediction))
    print("Error:" + str(error_for_all_lights) + "\n")
    

    View Slide

56. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    import numpy as np
    weights = np.array([0.5, 0.48, -0.7])
    alpha = 0.1
    streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1],
    [0, 1, 1],[1, 0, 1]])
    walk_vs_stop = np.array([0, 1, 0, 1, 1, 0])
    input = streetlights[0]
    goal_prediction = walk_vs_stop[0]
    for iteration in range(40):
    error_for_all_lights = 0
    for row_index in range(len(walk_vs_stop)):
    input = streetlights[row_index]
    goal_prediction = walk_vs_stop[row_index]
    prediction = input.dot(weights)
    error = (goal_prediction - prediction) "** 2
    error_for_all_lights += error
    delta = prediction - goal_prediction
    weights = weights - (alpha * (input * delta))
    print("Prediction:" + str(prediction))
    print("Error:" + str(error_for_all_lights) + "\n")
    Prediction: -0.19999999999999996
    Prediction: -0.19999999999999996
    Prediction: -0.5599999999999999
    Prediction: 0.6160000000000001
    Prediction: 0.17279999999999995
    Prediction: 0.17552
    Error: 2.6561231104
    …
    Prediction: -0.0022410273814405524
    Prediction: 0.9978745386023716
    Prediction: -0.016721264429884947
    Prediction: 1.0151127459893812
    Prediction: 0.9969492081270097
    Prediction: -0.0026256193329783125
    Error:0.00053373677328488
    

    View Slide

57. DEEP LEARNING = TEACHING COMPUTERS TO FIND CORRELATION
    Examples and images from “Grokking Deep Learning” by Andrew Trask
    import numpy as np
    weights = np.array([0.5, 0.48, -0.7])
    alpha = 0.1
    streetlights = np.array([[1, 0, 1],[0, 1, 1],[0, 0, 1],[1, 1, 1],
    [0, 1, 1],[1, 0, 1]])
    walk_vs_stop = np.array([0, 1, 0, 1, 1, 0])
    input = streetlights[0]
    goal_prediction = walk_vs_stop[0]
    for iteration in range(40):
    error_for_all_lights = 0
    for row_index in range(len(walk_vs_stop)):
    input = streetlights[row_index]
    goal_prediction = walk_vs_stop[row_index]
    prediction = input.dot(weights)
    error = (goal_prediction - prediction) "** 2
    error_for_all_lights += error
    delta = prediction - goal_prediction
    weights = weights - (alpha * (input * delta))
    print("Prediction:" + str(prediction))
    print("Error:" + str(error_for_all_lights) + "\n")
    Prediction: -0.19999999999999996
    Prediction: -0.19999999999999996
    Prediction: -0.5599999999999999
    Prediction: 0.6160000000000001
    Prediction: 0.17279999999999995
    Prediction: 0.17552
    Error: 2.6561231104
    …
    Prediction: -0.0022410273814405524
    Prediction: 0.9978745386023716
    Prediction: -0.016721264429884947
    Prediction: 1.0151127459893812
    Prediction: 0.9969492081270097
    Prediction: -0.0026256193329783125
    Error:0.00053373677328488
    

    View Slide

58. Machine Learning is Really Human Teaching
    

    View Slide

59. MACHINE LEARNING IS HUMAN TEACHING
    1 2 3
    CREATE INPUTS AND
    OUTPUTS
    DEFINE A MODEL
    ARCHITECTURE
    TRAIN, EVALUATE, TUNE
    CHAPTER 5. MACHINE LEARNING BASICS
    100 101 102 103 104 105
    Number of training examples
    0.0
    0.5
    1.0
    1.5
    2.0
    2.5
    3.0
    3.5
    Error (MSE)
    Bayes error
    Train (quadratic)
    Test (quadratic)
    Test (optimal capacity)
    Train (optimal capacity)
    10
    15
    20
    ty (polynomial degree)
    

    View Slide

60. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT
    

    View Slide

61. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT
    

    View Slide

62. MACHINE LEARNING ISN’T “DONE” IN ANY ONE PLACE
    

    View Slide

63. ML = DEVELOPMENT, TRAINING, INFERENCE
    1 2 3
    DEVELOPMENT TRAINING INFERENCE
    

    View Slide

64. MODEL CREATION TOOLS
    

    View Slide

65. TRAINING, EXPERIMENTS, & TUNING
    

    View Slide

66. DEMO
    TRAINING AN EMOTION MODEL
    

    View Slide

67. THE TRAIN, PREDICT, TUNE LOOP
    

    View Slide

68. THE TRAIN, PREDICT, TUNE LOOP
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    

    View Slide

69. INFERENCE: PREDICTION IN REAL TIME
    

    View Slide

70. INFERENCE: PREDICTION IN REAL TIME
    During training, the
    network “learns” by
    tuning weights to
    reduce error
    

    View Slide

71. INFERENCE: PREDICTION IN REAL TIME
    During training, the
    network “learns” by
    tuning weights to
    reduce error
    These weights are then applied
    against data the network has never
    seen before to make a prediction
    

    View Slide

72. INFERENCE: PREDICTION IN REAL TIME
    During training, the
    network “learns” by
    tuning weights to
    reduce error
    These weights are then applied
    against data the network has never
    seen before to make a prediction
    

    View Slide

73. DEMO
    RUNNING INFERENCE ON A MODEL
    

    View Slide

74. MODEL DEPLOYMENT
    +
    +
    +
    

    View Slide

75. PRE-TRAINED MODELS AS APIS
    

    View Slide

76. DEMO
    USING THE AZURE FACE API
    

    View Slide

77. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    

    View Slide

78. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Training & Validation
    Data
    

    View Slide

79. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Cloud Training &
    Deployment
    Training & Validation
    Data
    

    View Slide

80. LOCAL DEVELOPMENT, CLOUD TRAINING AND DEPLOYMENT
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Cloud Training &
    Deployment
    Training & Validation
    Data
    Real time Inference
    Data
    ?
    

    View Slide

81. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT
    

    View Slide

82. THREE BIG IDEAS
    1 2 3
    MACHINE LEARNING
    IS HUMAN TEACHING
    TRAINING IS MEANT
    FOR THE CLOUD
    THE REAL WORK OF ML
    IS IN THE IOT
    

    View Slide

83. TRADITIONAL: DEVICES AND CLOUD
    

    View Slide

84. TRADITIONAL: DEVICES AND CLOUD
    The Cloud
    The Internet
    of things
    

    View Slide

85. TRADITIONAL: DEVICES AND CLOUD
    The Cloud
    The Internet
    of things
    “Lots of
    power”
    

    View Slide

86. TRADITIONAL: DEVICES AND CLOUD
    The Cloud
    The Internet
    of things
    “Lots of
    power”
    “Lots of
    Devices”
    

    View Slide

87. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    “Lots of
    power”
    “Lots of
    Devices”
    

    View Slide

88. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    The “Edge”
    “Lots of
    power”
    “Lots of
    Devices”
    

    View Slide

89. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    The “Edge”
    “Lots of
    power”
    “Lots of
    Devices”
    

    View Slide

90. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    The “Edge”
    “Lots of
    power”
    “Lots of
    Devices”
    

    View Slide

91. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    The “Edge”
    “Lots of
    power”
    “Lots of
    Devices”
    

    View Slide

92. THE CLOUD, THE “EDGE,” AND THE IOT
    The Cloud
    The Internet
    of things
    The “Edge”
    “Lots of
    power”
    “Lots of
    Devices”
    “A Bit of
    both”
    

    View Slide

93. THE “EDGE” IS THE INTERNET OF THINGS!
    The Cloud
    The Internet
    of things
    The “Edge”
    

    View Slide

94. THE “EDGE” IS THE INTERNET OF THINGS!
    The Cloud
    The Internet
    of things
    The “Edge”
    

    View Slide

95. THE IOT IS THE SOURCE OF DATA FOR ML
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Cloud Training &
    Deployment
    Training & Validation
    Data
    Real time Inference
    Data
    ?
    

    View Slide

96. THE IOT IS THE SOURCE OF DATA FOR ML
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Cloud Training &
    Deployment
    Training & Validation
    Data
    Real time Inference
    Data
    ?
    

    View Slide

97. THE IOT IS THE SOURCE OF DATA FOR ML
    Define Task
    Collect Data
    Prepare Data
    Test & Evaluate
    Train Model
    Deploy Model
    Maintain, Update &
    Refine
    Tune
    & Refine
    Explore
    Model
    Local Development
    Cloud Training &
    Deployment
    Training & Validation
    Data
    Real time Inference
    Data
    ?
    

    View Slide

98. WHY DO ML INFERENCING IN THE IOT?
    1 2 3
    VOLUME SPEED PRIVACY
    

    View Slide

99. MORE DEVICES == MORE DATA
    

    View Slide

100. View Slide

101. View Slide

102. View Slide

103. The Cloud may be infinite;
     your pipe to it is not
     

     View Slide

104. ML LOVES DATA LIKE COOKIE MONSTER LOVES COOKIES
     

     View Slide

105. ML LOVES DATA LIKE COOKIE MONSTER LOVES COOKIES
     

     View Slide

106. DOING INFERENCE IN THE IOT MEANS LESS DATA IN THE CLOUD
     

     View Slide

107. DOING INFERENCE IN THE IOT MEANS LESS DATA IN THE CLOUD
     

     View Slide

108. INFERENCE FOR PREVENTATIVE MAINTENANCE
     

     View Slide

109. INFERENCE FOR PREVENTATIVE MAINTENANCE
     

     View Slide

110. INFERENCE FOR PREVENTATIVE MAINTENANCE
     

     View Slide

111. INFERENCE FOR PREVENTATIVE MAINTENANCE
     

     View Slide

112. INFERENCE FOR PREVENTATIVE MAINTENANCE
     

     View Slide

113. THERE IS NO CLOUD IF THE NETWORK IS OFFLINE
     

     View Slide

114. THERE IS NO CLOUD IF THE NETWORK IS OFFLINE
     

     View Slide

115. INFERENCE FOR PREVENTATIVE MAINTENANCE
     

     View Slide

116. INFERENCE FOR PREVENTATIVE MAINTENANCE
     

     View Slide

117. INFERENCE FOR PREVENTATIVE MAINTENANCE
     

     View Slide

118. IoT Inferencing == Built-In Privacy
     

     View Slide

119. VISION IN THE CLOUD
     

     View Slide

120. VISION IN THE CLOUD
     

     View Slide

121. VISION AT THE EDGE
     

     View Slide

122. VISION AT THE EDGE
     

     View Slide

123. VISION AT THE EDGE
     [
     {
     "faceId": "b0f2804a-649f-4da3-b6cf-ff459c867ae8",
     "faceRectangle": {
     "width": 63,
     "left": 168,
     "height": 63,
     "top": 110
     },
     "faceAttributes": {
     "smile": 0.0,
     "exposure": {
     "exposureLevel": "overExposure",
     "value": 0.75
     },
     "emotion": {
     "sadness": 0.0,
     "surprise": 0.0,
     "neutral": 0.0,
     "contempt": 0.001,
     "anger": 0.666,
     "disgust": 0.333,
     "fear": 0.0,
     "happiness": 0.0
     }
     "facialHair": {
     "beard": 0.4,
     "moustache": 0.4,
     "sideburns": 0.4
     },
     "gender": "male",
     "age": 33.0,
     "headPose": {
     "yaw": -24.3,
     "pitch": -6.4,
     "roll": -8.1
     }
     }
     }
     ]
     

     View Slide

124. EDGE ML INFERENCING DEVICES
     1 2
     GOOGLE CORAL NVIDIA JETSON NANO
     

     View Slide

125. GOOGLE CORAL
     • Edge TPU Module (SOM)
     • NXP i.MX 8M SOC (Quad-core Cortex-A53, plus Cortex-M4F)
     • Google Edge TPU ML accelerator coprocessor
     • Wi-Fi 2x2 MIMO (802.11b/g/n/ac 2.4/5GHz)
     • Bluetooth 4.1
     • 8GB eMMC
     • 1GB LPDDR4
     • USB connections
     • USB Type-C power port (5V DC)
     • USB 3.0 Type-C OTG port
     • USB 3.0 Type-A host port
     • USB 2.0 Micro-B serial console port
     • Audio & Video connections
     • Digital PDM microphone (x2)
     • HDMI 2.0a (full size)
     • 39-pin FFC connector for MIPI DSI display (4-lane)
     • 24-pin FFC connector for MIPI CSI-2 camera (4-lane)
     • MicroSD card slot
     • Gigabit Ethernet port
     • 40-pin GPIO expansion header
     • Supports Mendel Linux (derivative of Debian)
     

     View Slide

126. NVIDIA JETSON NANO
     • GPU and CPU
     • 128 Core Maxwell (472 GFLOPs)
     • Quad-core Cortex-A57 @ 1.43 GHz
     • 4 GB 64 bit LPDDR4 25.6 GB/s
     • 16GB eMMC
     • Video
     • HDMI 2.0 or DP1.2
     • 4K Encode and Decode
     • 12-pin FFC connector for MIPI CSI-2 camera (1.1-lane)
     • Connections
     • 1x SDIO, 2x SPI, 5x SysIO, 13x GPIOs, 6x I2C
     • USB connections
     • 1 USB 3.0
     • MicroSD card slot
     • Gigabit Ethernet port
     • Supports Ubuntu Linux
     

     View Slide

127. DEMO
     FACE DETECTION WITH GOOGLE CORAL
     

     View Slide

128. Where does this leave the Internet of
     Things?
     

     View Slide

129. THE INCREASING ROLE OF THE IOT IN ML
     1 2
     COLLECTION &
     COORDINATION ON
     DEVICES
     ML INFERENCING ON
     MICROCONTROLLERS!
     

     View Slide

130. View Slide

131. ML data doesn’t come from the cloud, or the
     edge…
     It comes from the IoT
     

     View Slide

132. IOT DEVICES SENSE AND SHARE
     

     View Slide

133. IOT DEVICES SENSE AND SHARE
     

     View Slide

134. IOT DEVICES TAKE ACTION
     

     View Slide

135. IOT DEVICES TAKE ACTION
     

     View Slide

136. IOT DEVICES TAKE ACTION
     

     View Slide

137. IOT DEVICES TAKE ACTION
     

     View Slide

138. Edge ML + Mesh Networking = Crazy
     Delicious
     

     View Slide

139. TRADITIONAL IOT: HUB AND SPOKE
     Wi-Fi Router
     

     View Slide

140. TRADITIONAL IOT: HUB AND SPOKE
     Wi-Fi Router
     

     View Slide

141. TRADITIONAL IOT: HUB AND SPOKE
     Wi-Fi Router
     

     View Slide

142. TRADITIONAL IOT: HUB AND SPOKE
     

     View Slide

143. ENTER MESH NETWORKING
     Gateway
     

     View Slide

144. ENTER MESH NETWORKING
     Gateway
     

     View Slide

145. ENTER MESH NETWORKING
     Gateway
     

     View Slide

146. ENTER MESH NETWORKING
     Gateway
     

     View Slide

147. ENTER MESH NETWORKING
     Gateway
     

     View Slide

148. ENTER MESH NETWORKING
     

     View Slide

149. OPENTHREAD VS. ZIGBEE, ZWAVE AND LORA
     Operating range 100 ft 35 ft
     Max # of devices 232 65,000
     Data rate 9.6-100 kbps 40-250 kbps
     Cost $ $$
     IP-Based Networking No No
     Open Standard? No Yes
     1-4 mi 100 ft
     N/A 300
     27 kbps 250 kbps
     $$$ $
     Yes Yes
     No Yes
     

     View Slide

150. WHAT IS THREAD?
     ✴IPv6-based mesh
     ✴Wireless Personal Area Network
     ✴No single point of failure
     ✴Tailored to IoT Scenarios
     ✴Can be used in concert with Wi-Fi, Cellular and Bluetooth
     is a low-power networking protocol
     

     View Slide

151. PARTICLE MESH DEVICES
     Nordic nRF52840 SoC
     ✴ ARM Cortex-M4F 32-bit
     ✴ 1MB flash, 256KB RAM
     ✴ IEEE 802.15.4-2006
     ✴ Bluetooth 5: 2 Mbps, 1 Mbps, 500
     Kbps, 125 Kbps
     ✴ ARM TrustZone Cryptographic
     security module
     ✴ NFC-A tag
     Argon
     » Wi-Fi + BLE +Mesh
     » Wi-Fi endpoint or mesh gateway
     Xenon
     » BLE + Mesh
     » Mesh endpoint
     Boron
     » LTE-M1 + BLE + Mesh
     » Cellular endpoint or mesh gateway
     

     View Slide

152. BRINGING AN EDGE DEVICE INTO A MESH NETWORK
     The Internet
     of things
     

     View Slide

153. BRINGING AN EDGE DEVICE INTO A MESH NETWORK
     The Internet
     of things
     

     View Slide

154. BRINGING AN EDGE DEVICE INTO A MESH NETWORK
     The Internet
     of things
     We can bring an Edge ML
     device into an IoT mesh
     network
     

     View Slide

155. DEMO
     EMOTION MESH
     

     View Slide

156. THE FUTURE OF ML AND THE IOT
     1 2
     ML INFERENCING ON
     MICROCONTROLLERS!
     FEDERATED AND
     DISTRIBUTED LEARNING
     

     View Slide

157. TENSORFLOW LITE MICRO
     

     View Slide

158. MICROTENSOR = ML INFERENCING ON ARM-BASED DEVICES
     

     View Slide

159. MICROTENSOR = ML INFERENCING ON ARM-BASED DEVICES
     

     View Slide

160. TF LITE + MICROTENSOR == A ROBUST, OPEN APPROACH TO MCU ML INFERENCING
     

     View Slide

161. DEMO
     ML ON MCUS
     

     View Slide

162. USING FEDERATED LEARNING TO MAXIMIZE IOT COMPUTE RESOURCES
     

     View Slide

163. USING FEDERATED LEARNING TO MAXIMIZE IOT COMPUTE RESOURCES
     IoT Devices spend most
     of their time idle…
     

     View Slide

164. USING FEDERATED LEARNING TO MAXIMIZE IOT COMPUTE RESOURCES
     IoT Devices spend most
     of their time idle…
     Federated learning may enable models to be
     distributed between the cloud, edge and devices for
     the most efficient processing
     

     View Slide

165. OTA MODEL UPDATES!
     

     View Slide

166. OTA MODEL UPDATES!
     

     View Slide

167. OTA MODEL UPDATES!
     

     View Slide

168. THREE BIG IDEAS
     1 2 3
     MACHINE LEARNING
     IS HUMAN TEACHING
     TRAINING IS MEANT
     FOR THE CLOUD
     THE REAL WORK OF ML
     IS IN THE IOT
     

     View Slide

169. Explore ML inferencing with SBCs
     

     View Slide

170. Explore Mesh networking for local network
     intelligence
     

     View Slide

171. Explore TensorFlow Lite and µTensor for ML
     on MCUs
     

     View Slide

172. THANK YOU!
     BRANDON SATROM | BRANDON@PARTICLE.IO
     GITHUB.COM/BSATROM/EMOTION-MESH
     

     View Slide