Leveraging Pinot’s Plugin Architecture to Build a Unified Stream Decoder (Mike Davis, DoorDash) | RTA Summit 2023

Real-time events produced at DoorDash use an in-house framework, Iguazu, that embeds custom serialization metadata which is used for generic and dynamic stream processing. While this makes consuming these events easy for our microservice applications, this provides a challenge when attempting to consume these events more generally.

In this talk we’ll walk through how we’ve taken advantage of the highly extensible framework within Apache Pinot to develop a custom StreamMessageDecoder implementation incorporating the Iguazu consumer library. By leveraging this approach we are able to seamlessly integrate with the existing real-time ecosystem at DoorDash to power real-time analytics for any team or product at DoorDash.

Transcript

1. 1 Mike Davis RTA Summit April 25th, 2023 Unified Stream Decoder: Leveraging Pinot's Plugin Architecture
2. 2 Use-Cases for Real-time Analytics
3. CONFIDENTIAL Monitor ETA Models 3 ETAs (estimated time of arrival) are a common feature within the DoorDash app. They give the customer a general idea of when their order would be delivered. We’re continuously working to improve the accuracy of this calculation and need to monitor the real vs computed results in real time.
4. CONFIDENTIAL Track Experimentation Rollouts 4 DoorDash has thousands of experiments running month through an internal self-serve platform named Curie. Users need the ability to monitor the rollout of their experiments and confirm their reaching the desired audience. *Meet Dash-AB May 2022 **Experimentation Platform Sep 2020
5. CONFIDENTIAL Ads Campaign Reporting Ads allows customer to boost a vendors visibility within the DoorDash app. Customers want to know in real-time how their ads are performing. 5
6. 6 Real-time Stream Consumer
7. • Batch loaded ◦ Spark, Flink, Minions ◦ Hourly, daily • Offline process, outside of Pinot ◦ Convert data files into Segments ◦ Tell Pinot about the new segments • Write-once, read-many 7 Pinot Table Types 7 Offline Table • Stream ingested ◦ Apache Kafka ◦ Amazon Kinesis ◦ Apache Pulsar • Pinot Servers are direct consumers ◦ Convert streams into Segments ◦ Tell Pinot about the new segments ◦ In-flight events are also queryable • Continuously writing Real-Time Table
8. 8 • streamType (e.g. Kafka) • stream.kafka.topic.name • stream.kafka.broker.list • stream.kafka.consumer.type (LLC vs HLC) • stream.kafka.consumer.factory.class.name • Addl Consumer dependent configs ◦ SSL ◦ Authentication Kafka Real-Time Configuration 8 Consumer • Stream.kafka.decoder.class.name ◦ JSONMessageDecoder ◦ KafkaAvroMessageDecoder ◦ SimpleAvroMessageDecoder ◦ KafkaConfluentSchemaRegistryAvroMessageDecoder ◦ CSVMessageDecoder ◦ ProtoBufMessageDecoder • Decoder dependent configs: ◦ stream.kafka.decoder.prop.schema.registry.rest.url Decoder
9. 9 Why we need a custom decoder?
10. By the end of 2020 DoorDash had mostly transitioned to a microservices architecture 10 Protobuf and gRPC is the new standard gRPC was widely adopted so Protobuf was the encoding of choice for most systems Realtime event processing via Flink also adopted Protobuf encoding Custom producer and consumer libraries abstracted out the serialization frameworks 10 *How DoorDash Transitioned from a Monolith to Microservices Dec 2020 **Building Scalable Real Time Event Processing with Kafka and Flink Aug 2022
11. 11 Protobuf Avro vs Protobuf >> Avro
12. CONFIDENTIAL Just use the ProtoBufMessageDecoder? 12 *Pinot Input Formats: Protocol Buffers Sample Configuration "streamType": "kafka", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.ProtoBufMessageDecoder", "stream.kafka.decoder.prop.descriptorFile": "file:///tmp/Workspace/protobuf/metrics.desc", "stream.kafka.decoder.prop.protoClassName": "Metrics" Does NOT support Schema Registry (open TODO) DoorDash implementation of Protobuf schema registry is not compatible
13. CONFIDENTIAL Workarounds • Maintaining multiple topics • Operating another stream processing job • Most customers were unaware of the serialization format • Error messages were vague and • Back-and-forth development INCREASED ONBOARDING FRICTION INCREASED OVERHEAD • Solution needed to work with existing streams • Avoid bespoke integrations • Work with existing Data Platform solutions SHOULD JUST WORK 13 Require customers to use Avro natively Replicate their topic to another topic in Avro Use a OFFLINE table instead
14. 14 Walk-thru of the Pinot Stream SPI
15. 15 How it works
16. 16 Pinot Plugins via SPI 16 *Apache Pinot: Plugins
17. 17 What is SPI? 17 *Service Provider Interface Wikipedia
18. StreamConsumerFactory 18 Stream Ingestion Plugin PartitionLevelConsumer StreamLevelConsumer StreamMessageDecoder 18 *Stream Ingestion Plugin StreamMetadataProvider
19. CONFIDENTIAL Custom Decoder Implementation 19 *KafkaConfluentSchemaRegistryAvroMessageDecoder.java Started by understanding KafkaConfluentSchemaRegistryAvroMessageDecoder Decoder only gets the Kafka Payload :(
20. CONFIDENTIAL Kafka Record Header 20 *Top 5 Things Every Apache Kafka Developer Should Know “Record headers give you the ability to add some metadata about the Kafka record, without adding any extra information to the key/value pair of the record itself” • Kafka Header introduced in Kafka 0.11.0 • Record header consists of a String key and Byte value • Support multiple values per key Kafka events at DoorDash leverage the record header to specify the encoding
21. StreamConsumerFactory 21 Stream Ingestion Plugin PartitionLevelConsumer StreamLevelConsumer StreamMessageDecoder 21 *Stream Ingestion Plugin StreamMetadataProvider
22. CONFIDENTIAL Kafka Partitions in Review 22 *Consuming and Indexing rows in Realtime • Topics in Kafka are made up of one to many partitions • Number of partitions are defined for each topic and are constant • Each partition is assigned to a Kafka Broker • Example topic with 100 partitions and 10 brokers. 10 partitions per broker Partitions allow for horizontal scaling of a topic
23. CONFIDENTIAL PartitionLevelConsumer vs StreamLevelConsumer 23 StreamLevelConsumer aka HighLevel (HLC) - Consume data without control over the partitions *Consuming and Indexing rows in Realtime
24. CONFIDENTIAL PartitionLevelConsumer vs StreamLevelConsumer 24 PartitionLevelConsumer aka LowLevel (LLC) - Consume data from each partition with offset management *Consuming and Indexing rows in Realtime
25. CONFIDENTIAL PartitionLevelConsumer Implementation 25 *Consuming and Indexing rows in Realtime
26. CONFIDENTIAL 26 DoorDashStreamMessageDecoder
27. 27 Deployment
28. CONFIDENTIAL Bundling with Docker 28 *Consuming and Indexing rows in Realtime FROM gradle:jdk11 as builder WORKDIR /home/gradle/src RUN gradle --no-daemon build FROM apachepinot/pinot:release-0.11.0 COPY --from=builder /home/gradle/src/pinot-plugins/build/libs/pinot-plugins.jar /opt/pinot/plugins/doordash/plugins.jar 1) Build our custom assets 2) Copy them into the base image 3) Base image builds classpath under plugins dir
29. CONFIDENTIAL Deploying via Helm 29 image: repository: apachepinot/pinot tag: latest values.yaml: (default) image: repository: <custom-ecr-respository>/pinot-deploy tag: 1.23.0 prod-values.yaml:
30. 30 Future Plans
31. CONFIDENTIAL Dead Message Queue 31 • Kafka partitioned consumers process events in order. (e.g. FIFO) • What happens when a bad message enters the stream? • Fail and block or Discard and continue?
32. CONFIDENTIAL Dead Message Queue 32 • Instead of blocking… • Skip message at push into another Kafka topic • Kafka topic can be written to datalake for recovery • Consumed by Pinot ◦ topic_name: String ◦ timestamp: Timestamp ◦ error_message: String ◦ payload: JSON
33. CONFIDENTIAL Default Transformations 33 • Protobuf#Timestamp ◦ Seconds: Long ◦ Nanoseconds: Long • Common transformations "transformConfigs": [ { "columnName": "current_time_seconds", "transformFunction": "Groovy({current_time.seconds}, current_time)" }, { "columnName": "current_time_ms", "transformFunction": "Groovy({timestamp.seconds * 1000 + timestamp.nanos.intdiv(1000000)}, timestamp)" } ]
34. 34 Conclusion
35. CONFIDENTIAL 35 Thank You!