.NET Day 19 - Real Time Analytics in IoT by Marcel Lattmann
The number of IoT devices which stream data to the cloud increases daily. In this practical session, we will build an end-to-end architecture for real-time analytics using the latest IoT technologies like IoT edge and data bricks.
Runtime Engine DATABRICKS I/O SERVERLESS Collaborative Workspace Cloud storage Data warehouses Hadoop storage IoT / streaming data Rest APIs Machine learning models BI tools Data exports Data warehouses Azure Databricks Deploy Production Jobs & Workflows APACHE SPARK MULTI-STAGE PIPELINES DATA ENGINEER JOB SCHEDULER NOTIFICATION & LOGS DATA SCIENTIST BUSINESS ANALYST
Kafka with static dataset from JDBC source to enrich streaming data val kafkaDataset = spark.readStream .kafka(“device-updates”) .load() val staticDataset = spark.read .jdbc(“jdbc://”, “device-info”) val joinedDataset = kafkaDataset.join( staticDataset, “devicemake”)
computation | Multiple queries can be active at the same time | Each query has unique name to keep track of it’s state val query = result.writeStream .format(“parquet”) .outputMode(“append”) .start(“dest-path”) query.stop() query.awaitTermination() query.exception() query.sourceStatuses() query.sinkStatuses()
close to the devices | Azure IoT Hub as secure , high performant service that connects it all | Data analytics options available | Stream Analytics for quick starting and easy query logic | Azure Data Bricks as 1st class citizen for streaming, machine learning and translation | Multiple data integration options available The value of IoT is defined by the data and integration