Azure Synapse & IoT Workshop

Most of the IoT value come from analyzing it's data. However, ingesting analyzing high volume of data ca be a complex task. During this workshop, you'll learn how to use Azure Synapse Analytics - a limitless analytics service - to process and make available IoT Data to developers and data analysts.

Duration: 45 minutes

Transcript

1. Ingest & Analyze IoT Data
   with Azure Synapse
   Workshop
   CHRISTOPHER MANEU
   @CMANEU
   

   View Slide

2. What we’ll build today
   An IoT infrastructure (gateway, simulators, …)
   A data lake to store all our data
   Query raw data lake data with Synapse
   Create a continuous job to integrate IoT data into a
   structured DB
   

   View Slide

3. Pre-requisites
   An Azure Subscription
   

   View Slide

4. aka.ms/iot-synapse-workshop
   Follow along on GitHub
   

   View Slide

5. BEFORE WE TALK
   Setting up your workshop
   environment
   

   View Slide

6. Agenda
   Why we need a Big Data solution for IoT?
   Analyzing IoT Data with Synapse
   

   View Slide

7. Why we need a Big Data
   solution for IoT?
   

   View Slide

8. The nature of IoT Data
   JSON Documents Metadata Unstructured bin
   DeviceId 14f8gq51gq
   Customer ContosoCity
   LocationGps 43.5887 1.4522
   DeviceType AirQualityv2
   Building Main
   BuildingType School
   {
   "TimeStamp": "2020-
   0812T13:58:07.5203217Z",
   "Co": 20.6515621746199,
   "Co2": 48.47952265338019,
   "Ppm10": 257.44680750414,
   "Ozone": 163.747165663515,
   "EventProcessedUtcTime": "
   2020-08-
   12T13:58:13.2928373Z",
   "PartitionId": 3,
   "EventEnqueuedUtcTime": "2
   020-08-12T13:58:08.08500Z"
   Weather Sunny
   Wind (km/h) 28
   

   View Slide

9. As opposed to relational data, the row, unstructured data to be
   analyzed
   IoT generates unstructured (images) or semi-structured (json) data
   The frequency to which the data is generated
   IoT generates data faster, up to real time (X events/second)
   Variety
   Velocity
   Volume
   What is Big Data?
   The amount of stored data to be analyzed
   IoT generates a high volume of data
   

   View Slide

10. Using IoT as an operational Data Source
    Azure
    IoT Hub
    ? Where to store our data ?
    Where to process our data ? Power BI
    

    View Slide

11. Data Warehousing 101 in a minute
    Built for writes and
    micro-reads.
    Operational
    databases
    OLTP
    Built for read only
    Modeled for data
    analysis & reporting
    Data
    Warehouse
    OLAP
    

    View Slide

12. How Big Data systems are working?
    Datawarehouse
    server
    Compute | Memory
    Storage
    

    View Slide

13. How Big Data systems are working?
    Datawarehouse
    server
    Compute | Memory
    Storage
    

    View Slide

14. How Big Data systems are working?
    Compute
    Node
    Control Node
    Shared Storage
    Compute
    Node
    Compute
    Node
    Compute
    Node
    Compute
    Node
    …
    

    View Slide

15. Analyzing IoT Data with
    Synapse
    

    View Slide

16. Store processed data
    Using IoT as an operational Data Source
    Azure
    IoT Hub
    Azure
    Data Lake Gen 2
    Azure
    Stream Analytics
    Azure Synapse
    Analytics
    Azure
    SQL
    Power BI
    Store raw data
    

    View Slide

17. Workshop
    

    View Slide

18. Using IoT as an operational Data Source
    Azure
    IoT Hub
    Teams
    Business
    services
    Azure
    Data Lake Gen 2
    Azure
    Stream Analytics
    Event
    Hubs
    Logic App
    Azure Synapse
    Analytics
    Azure
    SQL
    Time Series
    Insights
    Power BI
    Synapse
    Analytics
    

    View Slide

19. Complete interactive learning
    exercises, watch videos,
    and practice and apply your
    new skills.
    /learn alert
    aka.ms/iot40/learn
    

    View Slide

20. See Big Data and IoT in
    action in this demo-heavy,
    45 minutes session
    Watch at your pace
    aka.ms/iot40/watch
    

    View Slide

21. @cmaneu
    Thank you
    

    View Slide