Netflix Kafka and Eventing for Content Infrastructure - Kafka Summit 2019

Netflix Kafka and Eventing for Content Infrastructure - Kafka Summit 2019

Transcript

1. Eventing Things - A Netflix Original! Nitin Sharma
2. Let’s Warm up- Netflix Trivia!
3. Speaker Info • Nitin Sharma, Content Finance Infrastructure @ Netflix • Decade of work on Large Scale Distributed Systems • Storage, Search, Messaging, Stream Processing, ML Infrastructure
4. How does Netflix make content? Launch Production Creative Forecast Program Deals Post Production Financial Reporting Content Finance Infrastructure
5. Content Finance Infrastructure is at the Heart of Netflix Studio.
6. We enable innovation for how content is financially planned, produced and accounted for. Music Accounting Content Programming Content Payments & Accounting Production Budgeting Talent Payments Content Forecasting (cash & expense) Production Cost Reporting Production Cashflow Content Forecasting & Programming(CF) Production Finance Content Accounting (CA) Content Finance Infrastructure
7. Production Finance ensures our productions are financially healthy. Cashflow Payment Cost Report Budget Estimated cost? How much do we need on an ongoing basis? Talent Payments Snapshot of costs
8. Production Finance needs data from many other services in the Studio Ecosystem. Cashflow Payment Schedule Productions Talent
9. Synchronous communication cannot withstand failures. Productions Schedule Cost Report Cashflow {launch date} {episodes} {Cashflow}
10. Adding new dependencies is non trivial and error prone Productions Cost Report Schedule Payments Cashflow {episodes} {launch date} {Cashflow}
11. Request driven change communication causes chaos. • Request Driven Communication ◦ Synchronous ◦ Complex workflows • Traceability ◦ Source & metadata • Uniformity ◦ Is state consistent across the universe? ◦ Non uniform reconciliation strategy ◦ Duplicate work
12. Eventing-centric is better than Request-centric message exchanges. • The LOG - Canonical stream of facts • Decoupling • Data change or trigger • Traceability
13. The Kafka ecosystem at Netflix enables teams to easily embrace eventing. • Paved Path • Kafka + Flink - Stream Processing as a Service • Fault tolerant and Multi-Region • Observability out of the box • Easy bootstrapping event listeners
14. Kafka is at the heart of the Netflix Studio Message Bus. Producers Data cleansing Stream Processing Consumers
15. We can easily produce events. ◦ Event data & metadata ◦ Normalized schema ▪ Id of the entity , UUID, ts, type • (optional) payload ▪ Standard across producers ◦ Publisher Client ◦ Event Sources ▪ Application/Services ▪ CDC Events (Source -> Sink) Producers Normalized Schema Kafka Client Producer Event Stream
16. We can process and order events. ◦ Input: ▪ Kafka - Multiple input stream ▪ Unordered ◦ Processing: ▪ Flink ◦ Output: ▪ Ordered & Keyed Kafka Stream ▪ Search Index Unordered Producer Event Streams Ordered & Enriched Streams
17. We enrich, flatten, and order entities. ◦ Delayed Materialization ▪ Circumvent ordering issues ◦ Filter, Transform & Window ◦ Enrich ▪ F(Id, Entity) -> LatestState ▪ Call Entity API for that Id ◦ Config driven ◦ Keyed Kafka ▪ Partition Key ▪ Order within partition Enrich Ordered & Enriched Streams <id> Payload
18. We can easily consume events. ◦ Spring Boot 2 ◦ Spring cloud kafka connector ◦ Stream Name ▪ Entity -> Stream name ◦ At Least Once ▪ Idempotent ◦ Offset ▪ Default vs latest vs earliest
19. Schema Registry helps us discover schema per entity. • Streams & Schema ◦ Avro Schema <> Kafka Topics ◦ Schema Registry ◦ Versioning ◦ Backward compatibility Schema Registry Streams & Schema Discovery findByStream() findBySchema() • Schema discovery (future looking) ◦ Search by schema ◦ Search by topic
20. What if I want to add a new stream? • Integrate new producer • Add enricher in the stream processing • Add a sink • Announce schema in registry Enrich Flatmap Ordered & Enriched Streams
21. Eventing is the communication mechanism in Netflix Studio Finance Ecosystem. Schema Registry Productions Payments Capitalization Cashflow Schedule Full Entity State
22. Failure detection and recovery is a first class citizen in design Productions Cashflow Schedule Full Entity State Live Backfill Slow
23. How do I know who has/hasn’t consumed what data? Schedule Payment Forecast Watchdog • Has an event made it through the entire system? (Unified view) • Has an event been consumed? ◦ Offset Monitoring & Alerting • Recon events ◦ Kappa ◦ Replay all events through Streaming
24. Define Performance SLA based on Operational Insights. ◦ Freshness SLA ▪ Message Consumption Lag ◦ Max Transfer Rate ▪ Payload size - Compress ▪ Message rate - Source, process, sink ◦ Partitioning & Parallelism ◦ Message Retention ◦ X-Region Replication SLA
25. Highly Aligned; Loosely coupled. Request Driven Eventing Decoupling Traceability Consistency Uniform Reconciliation Channels of communication Producers * Consumers Producers + Consumers
26. We are hiring! https://www.linkedin.com/in/knitinsharma/ Nitin Sharma