.NET Day 19 - Real Time Analytics in IoT by Marcel Lattmann

The number of IoT devices which stream data to the cloud increases daily. In this practical session, we will build an end-to-end architecture for real-time analytics using the latest IoT technologies like IoT edge and data bricks.

Transcript

1. Real Time Data Analytics in IoT From edge to cloud Hi, I am Marcel, technical director of Codit Switzerland 1 marcel.lattmann@codit.swiss Live from Zürich !
2. Connecting end to end Connect things to the cloud Cloud strategy & architecture Automate business processes Expose API’s 2
3. About Codit 3 2000 Belgium 2004 France 2013 Portugal 2016 Switzerland 2016 UK 2016 The Netherlands 2017 Malta 180 worldwide Largest Microsoft partner in Europe for integration, API management, IoT and Azure Solutions
4. 4 The IoT Value Chain
5. Traffic camera scenario 5
6. Traffic camera scenario 6
7. Traffic camera scenario 7 Edge | License plate recognition | Speeding alert (display) | Display traffic info Cloud Speeding tickets Detect suspicious cars Detect and predict incidents | | |
8. 1. Azure IoT Edge : Transmit camera events 8 Azure IoT Edge IoT Hub Devices Local Storage Nebulus SerialPort (Container) Functions Runtime Container Management Device Twin Device Twin Azure Stream Analytics (Container) Azure Functions (Container) Cognitive Services (Container) Azure Machine Learning (Container) Module Twin Module Twin Module Twin Module Twin Module Twin Module Twin Module Twin Module Twin Module Twin Module Twin
9. Architecture 9 IoT Hub Time Series Stream Analytics Blob Storage Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
10. 2. Time Series Insights : Data exploration 10 • Get near real-time insights in seconds • Start in seconds, scale in minutes • Create a global view of your IoT-scale data • Leverage the power of Time Series Insights in your Apps and Solutions
11. Architecture 11 IoT Hub Time Series Stream Analytics Blob Storage Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
12. Architecture 12 IoT Hub Time Series Stream Analytics Blob Storage Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
13. 3. Stream Analytics: in the cloud & on the edge 13 Presentation & Action Storage & Batch Analysis Stream Analytics Event Queuing & Stream Ingestion Event production IoT Hubs Applications Archiving for long term storage/ batch analytics Real-time dashboard Stream Analytics Automation to kick-off workflows Machine Learning Reference Data Event Hubs Blobs Devices & Gateways PowerBI
14. Stream analytics on the edge 14 Camera simulator Stream Analytics Display simulator Cloud hub Edge hub "routes": { "edge": "FROM /messages/modules/camera/outputs/* INTO BrokeredEndpoint(\"/modules/traffic-speeding-detection-edge/inputs/iot-speed-events\")", "camera": "FROM /messages/modules/traffic-speeding-detection-edge/outputs/* INTO BrokeredEndpoint(\"/modules/display/inputs/camera\")", "cloud": "FROM /messages/modules/camera/outputs/* INTO $upstream" }
15. Architecture 15 IoT Hub Time Series Stream Analytics Blob Storage Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
16. Architecture 16 IoT Hub Time Series Stream Analytics Blob Storage Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
17. 4. Structured data streaming: Azure Data Bricks 17 Optimized Databricks Runtime Engine DATABRICKS I/O SERVERLESS Collaborative Workspace Cloud storage Data warehouses Hadoop storage IoT / streaming data Rest APIs Machine learning models BI tools Data exports Data warehouses Azure Databricks Deploy Production Jobs & Workflows APACHE SPARK MULTI-STAGE PIPELINES DATA ENGINEER JOB SCHEDULER NOTIFICATION & LOGS DATA SCIENTIST BUSINESS ANALYST
18. Streaming pipeline | readStream…load() creates a streaming DataFrame, does not start any computation val input = spark.readStream .format("json") .load("source-path") val output = input .select(“clientid“, “querytime”) .where(“querytime > 100") output.writeStream .format("json") .start(“dest-path")
19. Structured streaming model
20. Structured streaming model ± ±
21. Continuous aggregations | Continuously compute average query time across all clients input.avg(“querytime") input.groupBy(“devicemake") .avg(“querytime")
22. Continuous windowed aggregations • Simplifies event-time stream processing (not possible in DStreams) • Works in both, streaming and batch jobs input.groupBy( $“devicemake”, window($“event-time”, “10 min”)) .avg(“querytime”)
23. Joining streams with static data | Join streaming data from Kafka with static dataset from JDBC source to enrich streaming data val kafkaDataset = spark.readStream .kafka(“device-updates”) .load() val staticDataset = spark.read .jdbc(“jdbc://”, “device-info”) val joinedDataset = kafkaDataset.join( staticDataset, “devicemake”)
24. Query management | query: a handle to the running streaming computation | Multiple queries can be active at the same time | Each query has unique name to keep track of it’s state val query = result.writeStream .format(“parquet”) .outputMode(“append”) .start(“dest-path”) query.stop() query.awaitTermination() query.exception() query.sourceStatuses() query.sinkStatuses()
25. Architecture 25 IoT Hub Time Series Stream Analytics Blob Storage Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
26. Architecture 26 IoT Hub Time Series Stream Analytics Blob Storage Logic Apps Fine creation Event Grid Detect speeding cars Detect Suspected cars Dashboard Logic Apps Alerting ! IoT Edge
27. Takeaways 27 | Azure IoT Edge for connectivity & AI close to the devices | Azure IoT Hub as secure , high performant service that connects it all | Data analytics options available | Stream Analytics for quick starting and easy query logic | Azure Data Bricks as 1st class citizen for streaming, machine learning and translation | Multiple data integration options available The value of IoT is defined by the data and integration
28. Thank you. Let’s connect! 28