Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Azure Synapse & IoT Workshop

Azure Synapse & IoT Workshop

Most of the IoT value come from analyzing it's data. However, ingesting analyzing high volume of data ca be a complex task. During this workshop, you'll learn how to use Azure Synapse Analytics - a limitless analytics service - to process and make available IoT Data to developers and data analysts.

Duration: 45 minutes

Christopher MANEU

December 09, 2020
Tweet

More Decks by Christopher MANEU

Other Decks in Programming

Transcript

  1. Ingest & Analyze IoT Data
    with Azure Synapse
    Workshop
    CHRISTOPHER MANEU
    @CMANEU

    View Slide

  2. What we’ll build today
    An IoT infrastructure (gateway, simulators, …)
    A data lake to store all our data
    Query raw data lake data with Synapse
    Create a continuous job to integrate IoT data into a
    structured DB

    View Slide

  3. Pre-requisites
    An Azure Subscription

    View Slide

  4. aka.ms/iot-synapse-workshop
    Follow along on GitHub

    View Slide

  5. BEFORE WE TALK
    Setting up your workshop
    environment

    View Slide

  6. Agenda
    Why we need a Big Data solution for IoT?
    Analyzing IoT Data with Synapse

    View Slide

  7. Why we need a Big Data
    solution for IoT?

    View Slide

  8. The nature of IoT Data
    JSON Documents Metadata Unstructured bin
    DeviceId 14f8gq51gq
    Customer ContosoCity
    LocationGps 43.5887 1.4522
    DeviceType AirQualityv2
    Building Main
    BuildingType School
    {
    "TimeStamp": "2020-
    0812T13:58:07.5203217Z",
    "Co": 20.6515621746199,
    "Co2": 48.47952265338019,
    "Ppm10": 257.44680750414,
    "Ozone": 163.747165663515,
    "EventProcessedUtcTime": "
    2020-08-
    12T13:58:13.2928373Z",
    "PartitionId": 3,
    "EventEnqueuedUtcTime": "2
    020-08-12T13:58:08.08500Z"
    Weather Sunny
    Wind (km/h) 28

    View Slide

  9. As opposed to relational data, the row, unstructured data to be
    analyzed
    IoT generates unstructured (images) or semi-structured (json) data
    The frequency to which the data is generated
    IoT generates data faster, up to real time (X events/second)
    Variety
    Velocity
    Volume
    What is Big Data?
    The amount of stored data to be analyzed
    IoT generates a high volume of data

    View Slide

  10. Using IoT as an operational Data Source
    Azure
    IoT Hub
    ? Where to store our data ?
    Where to process our data ? Power BI

    View Slide

  11. Data Warehousing 101 in a minute
    Built for writes and
    micro-reads.
    Operational
    databases
    OLTP
    Built for read only
    Modeled for data
    analysis & reporting
    Data
    Warehouse
    OLAP

    View Slide

  12. How Big Data systems are working?
    Datawarehouse
    server
    Compute | Memory
    Storage

    View Slide

  13. How Big Data systems are working?
    Datawarehouse
    server
    Compute | Memory
    Storage

    View Slide

  14. How Big Data systems are working?
    Compute
    Node
    Control Node
    Shared Storage
    Compute
    Node
    Compute
    Node
    Compute
    Node
    Compute
    Node

    View Slide

  15. Analyzing IoT Data with
    Synapse

    View Slide

  16. Store processed data
    Using IoT as an operational Data Source
    Azure
    IoT Hub
    Azure
    Data Lake Gen 2
    Azure
    Stream Analytics
    Azure Synapse
    Analytics
    Azure
    SQL
    Power BI
    Store raw data

    View Slide

  17. Workshop

    View Slide

  18. Using IoT as an operational Data Source
    Azure
    IoT Hub
    Teams
    Business
    services
    Azure
    Data Lake Gen 2
    Azure
    Stream Analytics
    Event
    Hubs
    Logic App
    Azure Synapse
    Analytics
    Azure
    SQL
    Time Series
    Insights
    Power BI
    Synapse
    Analytics

    View Slide

  19. Complete interactive learning
    exercises, watch videos,
    and practice and apply your
    new skills.
    /learn alert
    aka.ms/iot40/learn

    View Slide

  20. See Big Data and IoT in
    action in this demo-heavy,
    45 minutes session
    Watch at your pace
    aka.ms/iot40/watch

    View Slide

  21. @cmaneu
    Thank you

    View Slide