Leveraging Pinot’s Plugin Architecture to Build a Unified Stream Decoder (Mike Davis, DoorDash) | RTA Summit 2023

Real-time events produced at DoorDash use an in-house framework, Iguazu, that embeds custom serialization metadata which is used for generic and dynamic stream processing. While this makes consuming these events easy for our microservice applications, this provides a challenge when attempting to consume these events more generally.

In this talk we’ll walk through how we’ve taken advantage of the highly extensible framework within Apache Pinot to develop a custom StreamMessageDecoder implementation incorporating the Iguazu consumer library. By leveraging this approach we are able to seamlessly integrate with the existing real-time ecosystem at DoorDash to power real-time analytics for any team or product at DoorDash.

Transcript

1. 1
   Mike Davis
   RTA Summit
   April 25th, 2023
   Unified Stream Decoder:
   Leveraging Pinot's Plugin
   Architecture
   

   View Slide

2. 2
   Use-Cases for Real-time Analytics
   

   View Slide

3. CONFIDENTIAL
   Monitor ETA Models
   3
   ETAs (estimated time of arrival) are a
   common feature within the DoorDash
   app. They give the customer a general
   idea of when their order would be
   delivered. We’re continuously working
   to improve the accuracy of this
   calculation and need to monitor the
   real vs computed results in real time.
   

   View Slide

4. CONFIDENTIAL
   Track Experimentation
   Rollouts
   4
   DoorDash has thousands of
   experiments running month
   through an internal self-serve
   platform named Curie.
   Users need the ability to monitor
   the rollout of their experiments and
   confirm their reaching the desired
   audience.
   *Meet Dash-AB May 2022
   **Experimentation Platform Sep 2020
   

   View Slide

5. CONFIDENTIAL
   Ads Campaign Reporting
   Ads allows customer to boost a
   vendors visibility within the DoorDash
   app. Customers want to know in
   real-time how their ads are performing.
   5
   

   View Slide

6. 6
   Real-time Stream Consumer
   

   View Slide

7. ● Batch loaded
   ○ Spark, Flink, Minions
   ○ Hourly, daily
   ● Offline process, outside of Pinot
   ○ Convert data files into Segments
   ○ Tell Pinot about the new segments
   ● Write-once, read-many
   7
   Pinot Table Types
   7
   Offline Table
   ● Stream ingested
   ○ Apache Kafka
   ○ Amazon Kinesis
   ○ Apache Pulsar
   ● Pinot Servers are direct consumers
   ○ Convert streams into Segments
   ○ Tell Pinot about the new segments
   ○ In-flight events are also queryable
   ● Continuously writing
   Real-Time Table
   

   View Slide

8. 8
   ● streamType (e.g. Kafka)
   ● stream.kafka.topic.name
   ● stream.kafka.broker.list
   ● stream.kafka.consumer.type (LLC vs HLC)
   ● stream.kafka.consumer.factory.class.name
   ● Addl Consumer dependent configs
   ○ SSL
   ○ Authentication
   Kafka Real-Time Configuration
   8
   Consumer
   ● Stream.kafka.decoder.class.name
   ○ JSONMessageDecoder
   ○ KafkaAvroMessageDecoder
   ○ SimpleAvroMessageDecoder
   ○ KafkaConfluentSchemaRegistryAvroMessageDecoder
   ○ CSVMessageDecoder
   ○ ProtoBufMessageDecoder
   ● Decoder dependent configs:
   ○ stream.kafka.decoder.prop.schema.registry.rest.url
   Decoder
   

   View Slide

9. 9
   Why we need a custom decoder?
   

   View Slide

10. By the end of 2020 DoorDash had mostly transitioned to a microservices architecture
    10
    Protobuf and gRPC is the new standard
    gRPC was widely adopted so Protobuf was the encoding of choice for most systems
    Realtime event processing via Flink also adopted Protobuf encoding
    Custom producer and consumer libraries abstracted out the serialization frameworks
    10
    *How DoorDash Transitioned from a Monolith to Microservices Dec 2020
    **Building Scalable Real Time Event Processing with Kafka and Flink Aug 2022
    

    View Slide

11. 11
    Protobuf Avro
    vs
    Protobuf >> Avro
    

    View Slide

12. CONFIDENTIAL
    Just use the ProtoBufMessageDecoder?
    12
    *Pinot Input Formats: Protocol Buffers
    Sample Configuration
    "streamType": "kafka",
    "stream.kafka.decoder.class.name":
    "org.apache.pinot.plugin.inputformat.protobuf.ProtoBufMessageDecoder",
    "stream.kafka.decoder.prop.descriptorFile": "file:///tmp/Workspace/protobuf/metrics.desc",
    "stream.kafka.decoder.prop.protoClassName": "Metrics"
    Does NOT support Schema Registry (open TODO)
    DoorDash implementation of Protobuf schema registry is not compatible
    

    View Slide

13. CONFIDENTIAL
    Workarounds
    ● Maintaining multiple topics
    ● Operating another stream processing job
    ● Most customers were unaware of the serialization format
    ● Error messages were vague and
    ● Back-and-forth development
    INCREASED ONBOARDING FRICTION
    INCREASED OVERHEAD
    ● Solution needed to work with existing streams
    ● Avoid bespoke integrations
    ● Work with existing Data Platform solutions
    SHOULD JUST WORK
    13
    Require customers to use Avro natively
    Replicate their topic to another topic in Avro
    Use a OFFLINE table instead
    

    View Slide

14. 14
    Walk-thru of the Pinot Stream SPI
    

    View Slide

15. 15
    How it works
    

    View Slide

16. 16
    Pinot Plugins via SPI
    16
    *Apache Pinot: Plugins
    

    View Slide

17. 17
    What is SPI?
    17
    *Service Provider Interface Wikipedia
    

    View Slide

18. StreamConsumerFactory
    18
    Stream Ingestion Plugin
    PartitionLevelConsumer
    StreamLevelConsumer
    StreamMessageDecoder
    18
    *Stream Ingestion Plugin
    StreamMetadataProvider
    

    View Slide

19. CONFIDENTIAL
    Custom Decoder Implementation
    19
    *KafkaConfluentSchemaRegistryAvroMessageDecoder.java
    Started by understanding KafkaConfluentSchemaRegistryAvroMessageDecoder
    Decoder only gets the Kafka Payload :(
    

    View Slide

20. CONFIDENTIAL
    Kafka Record Header
    20
    *Top 5 Things Every Apache Kafka Developer Should Know
    “Record headers give you the ability to add some metadata about the Kafka record, without adding
    any extra information to the key/value pair of the record itself”
    ● Kafka Header introduced in Kafka 0.11.0
    ● Record header consists of a String key and Byte value
    ● Support multiple values per key
    Kafka events at DoorDash leverage the record header to specify the encoding
    

    View Slide

21. StreamConsumerFactory
    21
    Stream Ingestion Plugin
    PartitionLevelConsumer
    StreamLevelConsumer
    StreamMessageDecoder
    21
    *Stream Ingestion Plugin
    StreamMetadataProvider
    

    View Slide

22. CONFIDENTIAL
    Kafka Partitions in Review
    22
    *Consuming and Indexing rows in Realtime
    ● Topics in Kafka are made up of one to many partitions
    ● Number of partitions are defined for each topic and are constant
    ● Each partition is assigned to a Kafka Broker
    ● Example topic with 100 partitions and 10 brokers. 10 partitions per broker
    Partitions allow for horizontal scaling of a topic
    

    View Slide

23. CONFIDENTIAL
    PartitionLevelConsumer vs StreamLevelConsumer
    23
    StreamLevelConsumer aka HighLevel (HLC) - Consume data without control over the partitions
    *Consuming and Indexing rows in Realtime
    

    View Slide

24. CONFIDENTIAL
    PartitionLevelConsumer vs StreamLevelConsumer
    24
    PartitionLevelConsumer aka LowLevel (LLC) - Consume data from each partition with offset management
    *Consuming and Indexing rows in Realtime
    

    View Slide

25. CONFIDENTIAL
    PartitionLevelConsumer Implementation
    25
    *Consuming and Indexing rows in Realtime
    

    View Slide

26. CONFIDENTIAL 26
    DoorDashStreamMessageDecoder
    

    View Slide

27. 27
    Deployment
    

    View Slide

28. CONFIDENTIAL
    Bundling with Docker
    28
    *Consuming and Indexing rows in Realtime
    FROM gradle:jdk11 as builder
    WORKDIR /home/gradle/src
    RUN gradle --no-daemon build
    FROM apachepinot/pinot:release-0.11.0
    COPY --from=builder /home/gradle/src/pinot-plugins/build/libs/pinot-plugins.jar
    /opt/pinot/plugins/doordash/plugins.jar
    1) Build our custom assets
    2) Copy them into the base image
    3) Base image builds classpath under plugins dir
    

    View Slide

29. CONFIDENTIAL
    Deploying via Helm
    29
    image:
    repository: apachepinot/pinot
    tag: latest
    values.yaml: (default)
    image:
    repository: /pinot-deploy
    tag: 1.23.0
    prod-values.yaml:
    

    View Slide

30. 30
    Future Plans
    

    View Slide

31. CONFIDENTIAL
    Dead Message Queue
    31
    ● Kafka partitioned consumers process events in order. (e.g. FIFO)
    ● What happens when a bad message enters the stream?
    ● Fail and block or Discard and continue?
    

    View Slide

32. CONFIDENTIAL
    Dead Message Queue
    32
    ● Instead of blocking…
    ● Skip message at push into another Kafka topic
    ● Kafka topic can be written to datalake for recovery
    ● Consumed by Pinot
    ○ topic_name: String
    ○ timestamp: Timestamp
    ○ error_message: String
    ○ payload: JSON
    

    View Slide

33. CONFIDENTIAL
    Default Transformations
    33
    ● Protobuf#Timestamp
    ○ Seconds: Long
    ○ Nanoseconds: Long
    ● Common transformations
    "transformConfigs": [
    {
    "columnName": "current_time_seconds",
    "transformFunction": "Groovy({current_time.seconds}, current_time)"
    },
    {
    "columnName": "current_time_ms",
    "transformFunction": "Groovy({timestamp.seconds * 1000 + timestamp.nanos.intdiv(1000000)}, timestamp)"
    }
    ]
    

    View Slide

34. 34
    Conclusion
    

    View Slide

35. CONFIDENTIAL 35
    Thank You!
    

    View Slide