.NET Day 19 - Real Time Analytics in IoT by Marcel Lattmann

.NET Day 19 - Real Time Analytics in IoT by Marcel Lattmann

The number of IoT devices which stream data to the cloud increases daily. In this practical session, we will build an end-to-end architecture for real-time analytics using the latest IoT technologies like IoT edge and data bricks.

E6cffbf3b7a5fbfee4707033ef1636f5?s=128

dotnetday

May 28, 2019
Tweet

Transcript

  1. Real Time Data Analytics in IoT From edge to cloud

    Hi, I am Marcel, technical director of Codit Switzerland 1 marcel.lattmann@codit.swiss Live from Zürich !
  2. Connecting end to end Connect things to the cloud Cloud

    strategy & architecture Automate business processes Expose API’s 2
  3. About Codit 3 2000 Belgium 2004 France 2013 Portugal 2016

    Switzerland 2016 UK 2016 The Netherlands 2017 Malta 180 worldwide Largest Microsoft partner in Europe for integration, API management, IoT and Azure Solutions
  4. 4 The IoT Value Chain

  5. Traffic camera scenario 5

  6. Traffic camera scenario 6

  7. Traffic camera scenario 7 Edge | License plate recognition |

    Speeding alert (display) | Display traffic info Cloud Speeding tickets Detect suspicious cars Detect and predict incidents | | |
  8. 1. Azure IoT Edge : Transmit camera events 8 Azure

    IoT Edge IoT Hub Devices Local Storage Nebulus SerialPort (Container) Functions Runtime Container Management Device Twin Device Twin Azure Stream Analytics (Container) Azure Functions (Container) Cognitive Services (Container) Azure Machine Learning (Container) Module Twin Module Twin Module Twin Module Twin Module Twin Module Twin Module Twin Module Twin Module Twin Module Twin
  9. Architecture 9 IoT Hub Time Series Stream Analytics Blob Storage

    Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
  10. 2. Time Series Insights : Data exploration 10 • Get

    near real-time insights in seconds • Start in seconds, scale in minutes • Create a global view of your IoT-scale data • Leverage the power of Time Series Insights in your Apps and Solutions
  11. Architecture 11 IoT Hub Time Series Stream Analytics Blob Storage

    Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
  12. Architecture 12 IoT Hub Time Series Stream Analytics Blob Storage

    Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
  13. 3. Stream Analytics: in the cloud & on the edge

    13 Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production IoT Hubs Applications Archiving for long term storage/ batch analytics Real-time dashboard Stream Analytics Automation to kick-off workflows Machine Learning Reference Data Event Hubs Blobs Devices & Gateways PowerBI
  14. Stream analytics on the edge 14 Camera simulator Stream Analytics

    Display simulator Cloud hub Edge hub "routes": { "edge": "FROM /messages/modules/camera/outputs/* INTO BrokeredEndpoint(\"/modules/traffic-speeding-detection-edge/inputs/iot-speed-events\")", "camera": "FROM /messages/modules/traffic-speeding-detection-edge/outputs/* INTO BrokeredEndpoint(\"/modules/display/inputs/camera\")", "cloud": "FROM /messages/modules/camera/outputs/* INTO $upstream" }
  15. Architecture 15 IoT Hub Time Series Stream Analytics Blob Storage

    Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
  16. Architecture 16 IoT Hub Time Series Stream Analytics Blob Storage

    Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
  17. 4. Structured data streaming: Azure Data Bricks 17 Optimized Databricks

    Runtime Engine DATABRICKS I/O SERVERLESS Collaborative Workspace Cloud storage Data warehouses Hadoop storage IoT / streaming data Rest APIs Machine learning models BI tools Data exports Data warehouses Azure Databricks Deploy Production Jobs & Workflows APACHE SPARK MULTI-STAGE PIPELINES DATA ENGINEER JOB SCHEDULER NOTIFICATION & LOGS DATA SCIENTIST BUSINESS ANALYST
  18. Streaming pipeline | readStream…load() creates a streaming DataFrame, does not

    start any computation val input = spark.readStream .format("json") .load("source-path") val output = input .select(“clientid“, “querytime”) .where(“querytime > 100") output.writeStream .format("json") .start(“dest-path")
  19. Structured streaming model

  20. Structured streaming model ± ±

  21. Continuous aggregations | Continuously compute average query time across all

    clients input.avg(“querytime") input.groupBy(“devicemake") .avg(“querytime")
  22. Continuous windowed aggregations • Simplifies event-time stream processing (not possible

    in DStreams) • Works in both, streaming and batch jobs input.groupBy( $“devicemake”, window($“event-time”, “10 min”)) .avg(“querytime”)
  23. Joining streams with static data | Join streaming data from

    Kafka with static dataset from JDBC source to enrich streaming data val kafkaDataset = spark.readStream .kafka(“device-updates”) .load() val staticDataset = spark.read .jdbc(“jdbc://”, “device-info”) val joinedDataset = kafkaDataset.join( staticDataset, “devicemake”)
  24. Query management | query: a handle to the running streaming

    computation | Multiple queries can be active at the same time | Each query has unique name to keep track of it’s state val query = result.writeStream .format(“parquet”) .outputMode(“append”) .start(“dest-path”) query.stop() query.awaitTermination() query.exception() query.sourceStatuses() query.sinkStatuses()
  25. Architecture 25 IoT Hub Time Series Stream Analytics Blob Storage

    Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
  26. Architecture 26 IoT Hub Time Series Stream Analytics Blob Storage

    Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
  27. Takeaways 27 | Azure IoT Edge for connectivity & AI

    close to the devices | Azure IoT Hub as secure , high performant service that connects it all | Data analytics options available | Stream Analytics for quick starting and easy query logic | Azure Data Bricks as 1st class citizen for streaming, machine learning and translation | Multiple data integration options available The value of IoT is defined by the data and integration
  28. Thank you. Let’s connect! 28