Unlock the Value of Your Data with Apache Pinot and AWS (JP Santana & Wahab Syed, AWS) | RTA Summit 2023

A data-centric culture is critical for innovation and growth. Learn how Apache Pinot works with AWS services to unlock the value of your data and gain insights that can help you make smarter, data-driven decisions, no matter where your data lives.

Transcript

1. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   JP Santana
   Principal Startup SA
   Unlock the value of your data
   with Apache Pinot and AWS
   Wahab Syed
   Startup SA
   

   View Slide

2. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   What we’re going to cover
   • Breadth & Depth of AWS services for RT Analytics
   • Options for running Apache Pinot on AWS
   • Demo – Apache Pinot on Amazon ECS
   

   View Slide

3. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   Breadth & Depth
   of AWS services for RT
   Analytics
   

   View Slide

4. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   Building streaming data pipelines is easier with AWS
   Processing
   Ingestion and storage
   Amazon Kinesis
   Data Streams
   Amazon Managed Streaming
   for Apache Kafka
   Apache Pinot
   Batch
   Sources
   IOT sensors
   Enterprise apps
   Social media
   Logs
   Data lakes
   Amazon S3
   

   View Slide

5. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   Kinesis Data Streams
   Kinesis Data Analytics
   Kinesis Video Streams
   Kinesis Data Firehose
   Amazon
   Kinesis
   COLLECT, PROCESS, AND ANALYZE VIDEO
   AND DATA STREAMS IN REAL TIME
   

   View Slide

6. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   Compatible
   Fully managed
   Highly available
   Secure
   Kafk
   a
   Amazon
   MSK
   FULLY MANAGED, HIGHLY AVAILABLE, AND
   SECURE
   

   View Slide

7. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   Built to store and retrieve
   any amount of data
   Supports Apache Pinot
   Auto Ingest and Batch
   Import
   Unmatched durability,
   availability, and scalability
   Amazon
   S3
   Amazon
   S3
   

   View Slide

8. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   Customer references
   Online stylist processing 100
   million events per day
   Facilitate
   communications between
   100+ microservices
   1 billion events per day
   from connected
   devices
   Migrated data bus to
   Amazon Kinesis
   Near real-time home
   valuation (Zestimates)
   Billions of events per
   day from TVs and
   connected devices
   10 TB per day
   clickstreams from
   250+ sites
   Live clickstream
   dashboards refreshed
   under 10s
   IoT predictive
   analytics
   50 billion daily ad
   impressions, sub-50
   ms responses
   NORDSTRO
   M
   

   View Slide

9. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
   Options for running
   Apache Pinot on AWS
   

   View Slide

10. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Control Plane
    Deployment, Scheduling,
    Scalability & management
    Data Plane
    Where containers run
    Amazon Elastic
    Container Service
    Amazon Elastic
    Container Service
    for Kubernetes
    Amazon EC2 AWS Fargate
    Image Repo
    Container Image Repository
    Amazon Elastic
    Container Registry
    Container Services on AWS
    

    View Slide

11. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Amazon ECS and Amazon EKS
    EKS
    ECS
    Powerful simplicity Open flexibility
    

    View Slide

12. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Powerful simplicity
    AWS-opinionated way to
    run containers at scale
    Reduce decisions without
    sacrificing scale or features
    Reduce time to build, deploy, and
    migrate applications
    ECS
    

    View Slide

13. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Open flexibility
    Gain agility and efficiency
    with AWS-optimized Kubernetes,
    and standardize operations
    everywhere
    Secure, highly available,
    with observability across
    all Kubernetes deployments
    Build with choice of solutions
    from the broader community
    around Kubernetes
    EKS
    

    View Slide

14. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Serverless Containers
    Fargate
    No need to manage compute
    provisioning, management, and
    scalability
    Scale easily and pay only for
    what you use
    Natively integrated with VPC,
    ELB, IAM, CloudWatch, and
    more
    

    View Slide

15. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    Demo: Apache Pinot
    on Amazon EKS
    

    View Slide

16. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    ADD DEMO RECORDING
    

    View Slide

17. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    References for Apache Pinot on AWS
    • Apache Pinot on AWS Quickstart
    https://docs.pinot.apache.org/basics/getting-started/public-cloud
    -examples/aws-quickstart
    • Ingest events from an Amazon Kinesis stream into Pinot
    https://docs.pinot.apache.org/basics/data-import/pinot-stream-in
    gestion/amazon-kinesis
    • JSON Batch Import and Auto Ingest from S3 into Apache Pinot
    https://youtu.be/1EMBx1XeI9o
    https://youtu.be/fXraQygBzxg
    

    View Slide

18. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    !
    Please, don’t forget
    to rate our session
    in the app!
    

    View Slide

19. © 2023, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
    JP Santana
    linkedin.com/in/jpsantana
    Wahab Syed
    linkedin.com/in/wahab-io
    Thank you!
    

    View Slide