1 Mike Davis RTA Summit April 25th, 2023 Unified Stream Decoder: Leveraging Pinot's Plugin Architecture

2 Use-Cases for Real-time Analytics

CONFIDENTIAL Monitor ETA Models 3 ETAs (estimated time of arrival) are a common feature within the DoorDash app. They give the customer a general idea of when their order would be delivered. We’re continuously working to improve the accuracy of this calculation and need to monitor the real vs computed results in real time.

CONFIDENTIAL Track Experimentation Rollouts 4 DoorDash has thousands of experiments running month through an internal self-serve platform named Curie. Users need the ability to monitor the rollout of their experiments and confirm their reaching the desired audience. *Meet Dash-AB May 2022 **Experimentation Platform Sep 2020

CONFIDENTIAL Ads Campaign Reporting Ads allows customer to boost a vendors visibility within the DoorDash app. Customers want to know in real-time how their ads are performing. 5

6 Real-time Stream Consumer

● Batch loaded ○ Spark, Flink, Minions ○ Hourly, daily ● Offline process, outside of Pinot ○ Convert data files into Segments ○ Tell Pinot about the new segments ● Write-once, read-many 7 Pinot Table Types 7 Offline Table ● Stream ingested ○ Apache Kafka ○ Amazon Kinesis ○ Apache Pulsar ● Pinot Servers are direct consumers ○ Convert streams into Segments ○ Tell Pinot about the new segments ○ In-flight events are also queryable ● Continuously writing Real-Time Table

8 ● streamType (e.g. Kafka) ● stream.kafka.topic.name ● stream.kafka.broker.list ● stream.kafka.consumer.type (LLC vs HLC) ● stream.kafka.consumer.factory.class.name ● Addl Consumer dependent configs ○ SSL ○ Authentication Kafka Real-Time Configuration 8 Consumer ● Stream.kafka.decoder.class.name ○ JSONMessageDecoder ○ KafkaAvroMessageDecoder ○ SimpleAvroMessageDecoder ○ KafkaConfluentSchemaRegistryAvroMessageDecoder ○ CSVMessageDecoder ○ ProtoBufMessageDecoder ● Decoder dependent configs: ○ stream.kafka.decoder.prop.schema.registry.rest.url Decoder

9 Why we need a custom decoder?

By the end of 2020 DoorDash had mostly transitioned to a microservices architecture 10 Protobuf and gRPC is the new standard gRPC was widely adopted so Protobuf was the encoding of choice for most systems Realtime event processing via Flink also adopted Protobuf encoding Custom producer and consumer libraries abstracted out the serialization frameworks 10 *How DoorDash Transitioned from a Monolith to Microservices Dec 2020 **Building Scalable Real Time Event Processing with Kafka and Flink Aug 2022

11 Protobuf Avro vs Protobuf >> Avro

CONFIDENTIAL Just use the ProtoBufMessageDecoder? 12 *Pinot Input Formats: Protocol Buffers Sample Configuration "streamType": "kafka", "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.protobuf.ProtoBufMessageDecoder", "stream.kafka.decoder.prop.descriptorFile": "file:///tmp/Workspace/protobuf/metrics.desc", "stream.kafka.decoder.prop.protoClassName": "Metrics" Does NOT support Schema Registry (open TODO) DoorDash implementation of Protobuf schema registry is not compatible

CONFIDENTIAL Workarounds ● Maintaining multiple topics ● Operating another stream processing job ● Most customers were unaware of the serialization format ● Error messages were vague and ● Back-and-forth development INCREASED ONBOARDING FRICTION INCREASED OVERHEAD ● Solution needed to work with existing streams ● Avoid bespoke integrations ● Work with existing Data Platform solutions SHOULD JUST WORK 13 Require customers to use Avro natively Replicate their topic to another topic in Avro Use a OFFLINE table instead

14 Walk-thru of the Pinot Stream SPI

15 How it works

16 Pinot Plugins via SPI 16 *Apache Pinot: Plugins

17 What is SPI? 17 *Service Provider Interface Wikipedia

StreamConsumerFactory 18 Stream Ingestion Plugin PartitionLevelConsumer StreamLevelConsumer StreamMessageDecoder 18 *Stream Ingestion Plugin StreamMetadataProvider

CONFIDENTIAL Custom Decoder Implementation 19 *KafkaConfluentSchemaRegistryAvroMessageDecoder.java Started by understanding KafkaConfluentSchemaRegistryAvroMessageDecoder Decoder only gets the Kafka Payload :(

CONFIDENTIAL Kafka Record Header 20 *Top 5 Things Every Apache Kafka Developer Should Know “Record headers give you the ability to add some metadata about the Kafka record, without adding any extra information to the key/value pair of the record itself” ● Kafka Header introduced in Kafka 0.11.0 ● Record header consists of a String key and Byte value ● Support multiple values per key Kafka events at DoorDash leverage the record header to specify the encoding

StreamConsumerFactory 21 Stream Ingestion Plugin PartitionLevelConsumer StreamLevelConsumer StreamMessageDecoder 21 *Stream Ingestion Plugin StreamMetadataProvider

CONFIDENTIAL Kafka Partitions in Review 22 *Consuming and Indexing rows in Realtime ● Topics in Kafka are made up of one to many partitions ● Number of partitions are defined for each topic and are constant ● Each partition is assigned to a Kafka Broker ● Example topic with 100 partitions and 10 brokers. 10 partitions per broker Partitions allow for horizontal scaling of a topic

CONFIDENTIAL PartitionLevelConsumer vs StreamLevelConsumer 23 StreamLevelConsumer aka HighLevel (HLC) - Consume data without control over the partitions *Consuming and Indexing rows in Realtime

CONFIDENTIAL PartitionLevelConsumer vs StreamLevelConsumer 24 PartitionLevelConsumer aka LowLevel (LLC) - Consume data from each partition with offset management *Consuming and Indexing rows in Realtime

CONFIDENTIAL PartitionLevelConsumer Implementation 25 *Consuming and Indexing rows in Realtime

CONFIDENTIAL 26 DoorDashStreamMessageDecoder

27 Deployment

CONFIDENTIAL Bundling with Docker 28 *Consuming and Indexing rows in Realtime FROM gradle:jdk11 as builder WORKDIR /home/gradle/src RUN gradle --no-daemon build FROM apachepinot/pinot:release-0.11.0 COPY --from=builder /home/gradle/src/pinot-plugins/build/libs/pinot-plugins.jar /opt/pinot/plugins/doordash/plugins.jar 1) Build our custom assets 2) Copy them into the base image 3) Base image builds classpath under plugins dir

CONFIDENTIAL Deploying via Helm 29 image: repository: apachepinot/pinot tag: latest values.yaml: (default) image: repository: /pinot-deploy tag: 1.23.0 prod-values.yaml:

30 Future Plans

CONFIDENTIAL Dead Message Queue 31 ● Kafka partitioned consumers process events in order. (e.g. FIFO) ● What happens when a bad message enters the stream? ● Fail and block or Discard and continue?

CONFIDENTIAL Dead Message Queue 32 ● Instead of blocking… ● Skip message at push into another Kafka topic ● Kafka topic can be written to datalake for recovery ● Consumed by Pinot ○ topic_name: String ○ timestamp: Timestamp ○ error_message: String ○ payload: JSON

CONFIDENTIAL Default Transformations 33 ● Protobuf#Timestamp ○ Seconds: Long ○ Nanoseconds: Long ● Common transformations "transformConfigs": [ { "columnName": "current_time_seconds", "transformFunction": "Groovy({current_time.seconds}, current_time)" }, { "columnName": "current_time_ms", "transformFunction": "Groovy({timestamp.seconds * 1000 + timestamp.nanos.intdiv(1000000)}, timestamp)" } ]

34 Conclusion

CONFIDENTIAL 35 Thank You!