Slide 1

Slide 1 text

Backfill Upsert Table Via Apache Flink/Pinot™ Connector Yupeng Fu, Uber

Slide 2

Slide 2 text

About Me ● Yupeng Fu (yupeng9@github) ● Principal Engineer @ Uber.Inc ● Real-time Data Infrastructure ● Apache Pinot Committer

Slide 3

Slide 3 text

Agenda ● RTA@Uber ● Upsert overview ● Backfill and its challenges ● Flink/Pinot connector ● Future work

Slide 4

Slide 4 text

RTA @ Uber

Slide 5

Slide 5 text

Real-time Analytics in Uber business 1. Real-time and actionable insights 2. Time-sensitive decisions 3. User engagement growth Restaurant Performance View Demand/Supply Management Fast Access to Fresh Data at Scale Freight Carrier Score Card

Slide 6

Slide 6 text

EVA: Uber's consolidated RTA platform

Slide 7

Slide 7 text

Tier 0 platform, 99.99% uptime Self service Onboarding (via uWorc) Built on top of Apache Pinot SQL API (via Presto / Neutrino) Seconds data freshness < 100ms @P99 query latency EVA: Uber's consolidated RTA platform

Slide 8

Slide 8 text

Why upsert?

Slide 9

Slide 9 text

Why upsert in Pinot? ● Data can be updated or corrected ● Pinot shall deliver an accurate and update-to-date real-time view ● No easy workaround in SQL SELECT current_status, count(*) FROM uberEatsOrders WHERE regionid = 1366 AND MinutesSinceEpoch BETWEEN 25432140 AND 25433580 GROUP BY current_status TOP 10000

Slide 10

Slide 10 text

Aug 2020 Feb 2021 Started building Uber Ads Platform Test Launch Jan 2021 Engagement with data infra on exactly-once processing Apr 2021 Case Study: Uber Ads Platform built in <12 months Challenge 1: revenue cut by the third-party Software Challenge 2: previous platform limited by stale, discrepant data First exactly-once use case enabled by Data Infra Cutting-edge real-time technologies (Apache Kafka™/Flink™/Pinot™)

Slide 11

Slide 11 text

How Upsert Works

Slide 12

Slide 12 text

S1 S3 Apache Pinot Controller S2 3 1 2 2 3 4 Pinot Servers Zookeeper Apache Pinot Broker Apache Pinot’s Real-time Data Flow S4 4 1 Seg1 -> S1 Seg2 -> S2 Seg3 -> S3 Seg4 -> S4 Seg1 -> S1, S4 Seg2 -> S2, S3 Seg3 -> S3, S1 Seg4 -> S4, S2 select count(*) from X where country = us PK=1 PK=1 PK=1 PK=1 segments are immutable segments are distributed segments are replicated

Slide 13

Slide 13 text

Upsert via local coordination ● Key challenge is on tracking the records with the same PK ● Reduce the problem to a local coordination problem by ○ Leverage the partition-by-key feature in Kafka ○ Distribute segments of the primary key to the same server

Slide 14

Slide 14 text

Local coordinator design

Slide 15

Slide 15 text

Backfill Challenge

Slide 16

Slide 16 text

Data Backfill ● Bootstrapping with historical data ● Reprocessing due to ○ Bugs ○ Change in computation logic

Slide 17

Slide 17 text

Upsert limitation ● Real-time table only ● Strong dependency on Kafka ● Low ingestion throughput

Slide 18

Slide 18 text

Backfill via Apache Flink™ Kappa+ ● Higher throughput than Kafka ● Reuse the processing logic ● Ease of use

Slide 19

Slide 19 text

Backfill via Apache Flink™ Kappa+ ● Higher throughput than Kafka ● Reuse the processing logic ● Ease of use

Slide 20

Slide 20 text

Pinot Backfill via Apache Flink™ Kappa+ DataStream> stream = getBackfillInput("hiveSrc"); // Watermarking handled internally stream .partitionCustom( new PinotPartitioner(jobConfiguration.getParallelism()), new PrimaryKeySelector("record_uuid")) .addSink(getOutputs().get("rta_ads_metrics_test")) .name("pinot-sink");

Slide 21

Slide 21 text

Apache Flink/Pinot connector FLIP-166: Pinot Connector

Slide 22

Slide 22 text

Next ● Upsert TTL ● Upsert compaction ● Dedup support ● Spark connector

Slide 23

Slide 23 text

Q&A Apache®, Apache Kafka®, Apache Flink®, Apache Pinot®, Apache Hadoop® Presto® and their logo are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by The Apache Software Foundation is implied by the use of these marks.