Azure Synapse & IoT Workshop

Most of the IoT value come from analyzing it's data. However, ingesting analyzing high volume of data ca be a complex task. During this workshop, you'll learn how to use Azure Synapse Analytics - a limitless analytics service - to process and make available IoT Data to developers and data analysts.

Duration: 45 minutes

Transcript

1. Ingest & Analyze IoT Data with Azure Synapse Workshop CHRISTOPHER MANEU @CMANEU
2. What we’ll build today An IoT infrastructure (gateway, simulators, …) A data lake to store all our data Query raw data lake data with Synapse Create a continuous job to integrate IoT data into a structured DB
3. Pre-requisites An Azure Subscription
4. aka.ms/iot-synapse-workshop Follow along on GitHub
5. BEFORE WE TALK Setting up your workshop environment
6. Agenda Why we need a Big Data solution for IoT? Analyzing IoT Data with Synapse
7. Why we need a Big Data solution for IoT?
8. The nature of IoT Data JSON Documents Metadata Unstructured bin DeviceId 14f8gq51gq Customer ContosoCity LocationGps 43.5887 1.4522 DeviceType AirQualityv2 Building Main BuildingType School { "TimeStamp": "2020- 0812T13:58:07.5203217Z", "Co": 20.6515621746199, "Co2": 48.47952265338019, "Ppm10": 257.44680750414, "Ozone": 163.747165663515, "EventProcessedUtcTime": " 2020-08- 12T13:58:13.2928373Z", "PartitionId": 3, "EventEnqueuedUtcTime": "2 020-08-12T13:58:08.08500Z" Weather Sunny Wind (km/h) 28
9. As opposed to relational data, the row, unstructured data to be analyzed IoT generates unstructured (images) or semi-structured (json) data The frequency to which the data is generated IoT generates data faster, up to real time (X events/second) Variety Velocity Volume What is Big Data? The amount of stored data to be analyzed IoT generates a high volume of data
10. Using IoT as an operational Data Source Azure IoT Hub ? Where to store our data ? Where to process our data ? Power BI
11. Data Warehousing 101 in a minute Built for writes and micro-reads. Operational databases OLTP Built for read only Modeled for data analysis & reporting Data Warehouse OLAP
12. How Big Data systems are working? Datawarehouse server Compute | Memory Storage
13. How Big Data systems are working? Datawarehouse server Compute | Memory Storage
14. How Big Data systems are working? Compute Node Control Node Shared Storage Compute Node Compute Node Compute Node Compute Node …
15. Analyzing IoT Data with Synapse
16. Store processed data Using IoT as an operational Data Source Azure IoT Hub Azure Data Lake Gen 2 Azure Stream Analytics Azure Synapse Analytics Azure SQL Power BI Store raw data
17. Workshop
18. Using IoT as an operational Data Source Azure IoT Hub Teams Business services Azure Data Lake Gen 2 Azure Stream Analytics Event Hubs Logic App Azure Synapse Analytics Azure SQL Time Series Insights Power BI Synapse Analytics
19. Complete interactive learning exercises, watch videos, and practice and apply your new skills. /learn alert aka.ms/iot40/learn
20. See Big Data and IoT in action in this demo-heavy, 45 minutes session Watch at your pace aka.ms/iot40/watch
21. @cmaneu Thank you