Ingest & Analyze IoT Data with Azure Synapse Workshop CHRISTOPHER MANEU @CMANEU

What we’ll build today An IoT infrastructure (gateway, simulators, …) A data lake to store all our data Query raw data lake data with Synapse Create a continuous job to integrate IoT data into a structured DB

Pre-requisites An Azure Subscription

aka.ms/iot-synapse-workshop Follow along on GitHub

BEFORE WE TALK Setting up your workshop environment

Agenda Why we need a Big Data solution for IoT? Analyzing IoT Data with Synapse

Why we need a Big Data solution for IoT?

The nature of IoT Data JSON Documents Metadata Unstructured bin DeviceId 14f8gq51gq Customer ContosoCity LocationGps 43.5887 1.4522 DeviceType AirQualityv2 Building Main BuildingType School { "TimeStamp": "2020- 0812T13:58:07.5203217Z", "Co": 20.6515621746199, "Co2": 48.47952265338019, "Ppm10": 257.44680750414, "Ozone": 163.747165663515, "EventProcessedUtcTime": " 2020-08- 12T13:58:13.2928373Z", "PartitionId": 3, "EventEnqueuedUtcTime": "2 020-08-12T13:58:08.08500Z" Weather Sunny Wind (km/h) 28

As opposed to relational data, the row, unstructured data to be analyzed IoT generates unstructured (images) or semi-structured (json) data The frequency to which the data is generated IoT generates data faster, up to real time (X events/second) Variety Velocity Volume What is Big Data? The amount of stored data to be analyzed IoT generates a high volume of data

Using IoT as an operational Data Source Azure IoT Hub ? Where to store our data ? Where to process our data ? Power BI

Data Warehousing 101 in a minute Built for writes and micro-reads. Operational databases OLTP Built for read only Modeled for data analysis & reporting Data Warehouse OLAP

How Big Data systems are working? Datawarehouse server Compute | Memory Storage

How Big Data systems are working? Datawarehouse server Compute | Memory Storage

How Big Data systems are working? Compute Node Control Node Shared Storage Compute Node Compute Node Compute Node Compute Node …

Analyzing IoT Data with Synapse

Store processed data Using IoT as an operational Data Source Azure IoT Hub Azure Data Lake Gen 2 Azure Stream Analytics Azure Synapse Analytics Azure SQL Power BI Store raw data

Workshop

Using IoT as an operational Data Source Azure IoT Hub Teams Business services Azure Data Lake Gen 2 Azure Stream Analytics Event Hubs Logic App Azure Synapse Analytics Azure SQL Time Series Insights Power BI Synapse Analytics

Complete interactive learning exercises, watch videos, and practice and apply your new skills. /learn alert aka.ms/iot40/learn

See Big Data and IoT in action in this demo-heavy, 45 minutes session Watch at your pace aka.ms/iot40/watch

@cmaneu Thank you