Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Leveraging Pinot’s Plugin Architecture to Build a Unified Stream Decoder (Mike Davis, DoorDash) | RTA Summit 2023

Leveraging Pinot’s Plugin Architecture to Build a Unified Stream Decoder (Mike Davis, DoorDash) | RTA Summit 2023

Real-time events produced at DoorDash use an in-house framework, Iguazu, that embeds custom serialization metadata which is used for generic and dynamic stream processing. While this makes consuming these events easy for our microservice applications, this provides a challenge when attempting to consume these events more generally.

In this talk we’ll walk through how we’ve taken advantage of the highly extensible framework within Apache Pinot to develop a custom StreamMessageDecoder implementation incorporating the Iguazu consumer library. By leveraging this approach we are able to seamlessly integrate with the existing real-time ecosystem at DoorDash to power real-time analytics for any team or product at DoorDash.


May 23, 2023

More Decks by StarTree

Other Decks in Technology


  1. 1 Mike Davis RTA Summit April 25th, 2023 Unified Stream

    Decoder: Leveraging Pinot's Plugin Architecture
  2. CONFIDENTIAL Monitor ETA Models 3 ETAs (estimated time of arrival)

    are a common feature within the DoorDash app. They give the customer a general idea of when their order would be delivered. We’re continuously working to improve the accuracy of this calculation and need to monitor the real vs computed results in real time.
  3. CONFIDENTIAL Track Experimentation Rollouts 4 DoorDash has thousands of experiments

    running month through an internal self-serve platform named Curie. Users need the ability to monitor the rollout of their experiments and confirm their reaching the desired audience. *Meet Dash-AB May 2022 **Experimentation Platform Sep 2020
  4. CONFIDENTIAL Ads Campaign Reporting Ads allows customer to boost a

    vendors visibility within the DoorDash app. Customers want to know in real-time how their ads are performing. 5
  5. • Batch loaded ◦ Spark, Flink, Minions ◦ Hourly, daily

    • Offline process, outside of Pinot ◦ Convert data files into Segments ◦ Tell Pinot about the new segments • Write-once, read-many 7 Pinot Table Types 7 Offline Table • Stream ingested ◦ Apache Kafka ◦ Amazon Kinesis ◦ Apache Pulsar • Pinot Servers are direct consumers ◦ Convert streams into Segments ◦ Tell Pinot about the new segments ◦ In-flight events are also queryable • Continuously writing Real-Time Table
  6. 8 • streamType (e.g. Kafka) • stream.kafka.topic.name • stream.kafka.broker.list •

    stream.kafka.consumer.type (LLC vs HLC) • stream.kafka.consumer.factory.class.name • Addl Consumer dependent configs ◦ SSL ◦ Authentication Kafka Real-Time Configuration 8 Consumer • Stream.kafka.decoder.class.name ◦ JSONMessageDecoder ◦ KafkaAvroMessageDecoder ◦ SimpleAvroMessageDecoder ◦ KafkaConfluentSchemaRegistryAvroMessageDecoder ◦ CSVMessageDecoder ◦ ProtoBufMessageDecoder • Decoder dependent configs: ◦ stream.kafka.decoder.prop.schema.registry.rest.url Decoder
  7. By the end of 2020 DoorDash had mostly transitioned to

    a microservices architecture 10 Protobuf and gRPC is the new standard gRPC was widely adopted so Protobuf was the encoding of choice for most systems Realtime event processing via Flink also adopted Protobuf encoding Custom producer and consumer libraries abstracted out the serialization frameworks 10 *How DoorDash Transitioned from a Monolith to Microservices Dec 2020 **Building Scalable Real Time Event Processing with Kafka and Flink Aug 2022
  8. CONFIDENTIAL Just use the ProtoBufMessageDecoder? 12 *Pinot Input Formats: Protocol

    Buffers Sample Configuration "streamType": "kafka", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.ProtoBufMessageDecoder", "stream.kafka.decoder.prop.descriptorFile": "file:///tmp/Workspace/protobuf/metrics.desc", "stream.kafka.decoder.prop.protoClassName": "Metrics" Does NOT support Schema Registry (open TODO) DoorDash implementation of Protobuf schema registry is not compatible
  9. CONFIDENTIAL Workarounds • Maintaining multiple topics • Operating another stream

    processing job • Most customers were unaware of the serialization format • Error messages were vague and • Back-and-forth development INCREASED ONBOARDING FRICTION INCREASED OVERHEAD • Solution needed to work with existing streams • Avoid bespoke integrations • Work with existing Data Platform solutions SHOULD JUST WORK 13 Require customers to use Avro natively Replicate their topic to another topic in Avro Use a OFFLINE table instead
  10. CONFIDENTIAL Kafka Record Header 20 *Top 5 Things Every Apache

    Kafka Developer Should Know “Record headers give you the ability to add some metadata about the Kafka record, without adding any extra information to the key/value pair of the record itself” • Kafka Header introduced in Kafka 0.11.0 • Record header consists of a String key and Byte value • Support multiple values per key Kafka events at DoorDash leverage the record header to specify the encoding
  11. CONFIDENTIAL Kafka Partitions in Review 22 *Consuming and Indexing rows

    in Realtime • Topics in Kafka are made up of one to many partitions • Number of partitions are defined for each topic and are constant • Each partition is assigned to a Kafka Broker • Example topic with 100 partitions and 10 brokers. 10 partitions per broker Partitions allow for horizontal scaling of a topic
  12. CONFIDENTIAL PartitionLevelConsumer vs StreamLevelConsumer 23 StreamLevelConsumer aka HighLevel (HLC) -

    Consume data without control over the partitions *Consuming and Indexing rows in Realtime
  13. CONFIDENTIAL PartitionLevelConsumer vs StreamLevelConsumer 24 PartitionLevelConsumer aka LowLevel (LLC) -

    Consume data from each partition with offset management *Consuming and Indexing rows in Realtime
  14. CONFIDENTIAL Bundling with Docker 28 *Consuming and Indexing rows in

    Realtime FROM gradle:jdk11 as builder WORKDIR /home/gradle/src RUN gradle --no-daemon build FROM apachepinot/pinot:release-0.11.0 COPY --from=builder /home/gradle/src/pinot-plugins/build/libs/pinot-plugins.jar /opt/pinot/plugins/doordash/plugins.jar 1) Build our custom assets 2) Copy them into the base image 3) Base image builds classpath under plugins dir
  15. CONFIDENTIAL Deploying via Helm 29 image: repository: apachepinot/pinot tag: latest

    values.yaml: (default) image: repository: <custom-ecr-respository>/pinot-deploy tag: 1.23.0 prod-values.yaml:
  16. CONFIDENTIAL Dead Message Queue 31 • Kafka partitioned consumers process

    events in order. (e.g. FIFO) • What happens when a bad message enters the stream? • Fail and block or Discard and continue?
  17. CONFIDENTIAL Dead Message Queue 32 • Instead of blocking… •

    Skip message at push into another Kafka topic • Kafka topic can be written to datalake for recovery • Consumed by Pinot ◦ topic_name: String ◦ timestamp: Timestamp ◦ error_message: String ◦ payload: JSON
  18. CONFIDENTIAL Default Transformations 33 • Protobuf#Timestamp ◦ Seconds: Long ◦

    Nanoseconds: Long • Common transformations "transformConfigs": [ { "columnName": "current_time_seconds", "transformFunction": "Groovy({current_time.seconds}, current_time)" }, { "columnName": "current_time_ms", "transformFunction": "Groovy({timestamp.seconds * 1000 + timestamp.nanos.intdiv(1000000)}, timestamp)" } ]