Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Azure Synapse & IoT Workshop

Azure Synapse & IoT Workshop

Most of the IoT value come from analyzing it's data. However, ingesting analyzing high volume of data ca be a complex task. During this workshop, you'll learn how to use Azure Synapse Analytics - a limitless analytics service - to process and make available IoT Data to developers and data analysts.

Duration: 45 minutes

Christopher MANEU

December 09, 2020
Tweet

More Decks by Christopher MANEU

Other Decks in Programming

Transcript

  1. Ingest & Analyze IoT Data
    with Azure Synapse
    Workshop
    CHRISTOPHER MANEU
    @CMANEU

    View full-size slide

  2. What we’ll build today
    An IoT infrastructure (gateway, simulators, …)
    A data lake to store all our data
    Query raw data lake data with Synapse
    Create a continuous job to integrate IoT data into a
    structured DB

    View full-size slide

  3. Pre-requisites
    An Azure Subscription

    View full-size slide

  4. aka.ms/iot-synapse-workshop
    Follow along on GitHub

    View full-size slide

  5. BEFORE WE TALK
    Setting up your workshop
    environment

    View full-size slide

  6. Agenda
    Why we need a Big Data solution for IoT?
    Analyzing IoT Data with Synapse

    View full-size slide

  7. Why we need a Big Data
    solution for IoT?

    View full-size slide

  8. The nature of IoT Data
    JSON Documents Metadata Unstructured bin
    DeviceId 14f8gq51gq
    Customer ContosoCity
    LocationGps 43.5887 1.4522
    DeviceType AirQualityv2
    Building Main
    BuildingType School
    {
    "TimeStamp": "2020-
    0812T13:58:07.5203217Z",
    "Co": 20.6515621746199,
    "Co2": 48.47952265338019,
    "Ppm10": 257.44680750414,
    "Ozone": 163.747165663515,
    "EventProcessedUtcTime": "
    2020-08-
    12T13:58:13.2928373Z",
    "PartitionId": 3,
    "EventEnqueuedUtcTime": "2
    020-08-12T13:58:08.08500Z"
    Weather Sunny
    Wind (km/h) 28

    View full-size slide

  9. As opposed to relational data, the row, unstructured data to be
    analyzed
    IoT generates unstructured (images) or semi-structured (json) data
    The frequency to which the data is generated
    IoT generates data faster, up to real time (X events/second)
    Variety
    Velocity
    Volume
    What is Big Data?
    The amount of stored data to be analyzed
    IoT generates a high volume of data

    View full-size slide

  10. Using IoT as an operational Data Source
    Azure
    IoT Hub
    ? Where to store our data ?
    Where to process our data ? Power BI

    View full-size slide

  11. Data Warehousing 101 in a minute
    Built for writes and
    micro-reads.
    Operational
    databases
    OLTP
    Built for read only
    Modeled for data
    analysis & reporting
    Data
    Warehouse
    OLAP

    View full-size slide

  12. How Big Data systems are working?
    Datawarehouse
    server
    Compute | Memory
    Storage

    View full-size slide

  13. How Big Data systems are working?
    Datawarehouse
    server
    Compute | Memory
    Storage

    View full-size slide

  14. How Big Data systems are working?
    Compute
    Node
    Control Node
    Shared Storage
    Compute
    Node
    Compute
    Node
    Compute
    Node
    Compute
    Node

    View full-size slide

  15. Analyzing IoT Data with
    Synapse

    View full-size slide

  16. Store processed data
    Using IoT as an operational Data Source
    Azure
    IoT Hub
    Azure
    Data Lake Gen 2
    Azure
    Stream Analytics
    Azure Synapse
    Analytics
    Azure
    SQL
    Power BI
    Store raw data

    View full-size slide

  17. Using IoT as an operational Data Source
    Azure
    IoT Hub
    Teams
    Business
    services
    Azure
    Data Lake Gen 2
    Azure
    Stream Analytics
    Event
    Hubs
    Logic App
    Azure Synapse
    Analytics
    Azure
    SQL
    Time Series
    Insights
    Power BI
    Synapse
    Analytics

    View full-size slide

  18. Complete interactive learning
    exercises, watch videos,
    and practice and apply your
    new skills.
    /learn alert
    aka.ms/iot40/learn

    View full-size slide

  19. See Big Data and IoT in
    action in this demo-heavy,
    45 minutes session
    Watch at your pace
    aka.ms/iot40/watch

    View full-size slide

  20. @cmaneu
    Thank you

    View full-size slide