Running Kafka efficiently: Scaling streaming data pipelines in Hypergrowth FinTech - Confluent Online Talk 2023

Faced with potentially exorbitant costs of long-term data storage, KOR Financial, an Atlanta-based fintech startup focused on trade reporting, needed a way to cost-effectively stay in compliance and keep all data secure with at least four decades’ worth of storage. They initially evaluated MSK for Apache Kafka® as a data streaming solution for real-time trade reporting but quickly ran into limitations. They needed to find a way to reduce their ops burden and TCO while working with a partner whose exclusive focus was Kafka.

Hear from KOR Financial's Chief Technology Officer, Andreas Evers, on how they used Confluent Cloud as a single source of truth and immutable transaction log for four decades’ worth of trade reporting data to stay in compliance with financial regulations. He explains how Confluent Cloud enabled KOR to:

- Lower overall TCO by reducing storage costs with Infinite Storage and operational costs with fully managed features
- Convince end customers (tier 1 and 2 banks) that their data is safe with uncompromising data security
- Partner with the Kafka experts, leveraging 1M+ hours of Kafka experience and committer-led support

Join us on June 21, 2023 and learn why KOR chose Confluent over MSK.

Transcript

1. Running Kafka efficiently: Scaling streaming data pipelines in Hypergrowth FinTech Guest speaker Andreas Evers Chief Technology Officer at KOR Financial Moderated by Hasan Jilani Product Marketing at Confluent CUSTOMER SPOTLIGHT
2. 2 KOR Financial The only cloud native family of global trade repositories and regulatory reporting services ❖ Founded in 2021, KOR Financial’s mission is to avoid another financial crisis such as the one in 2008 ❖ Led by former executives of CME Group, ICE, and London Stock Exchange regulatory reporting businesses ❖ KOR's deep reporting expertise combined with cutting-edge technology delivers intelligent reporting services and new innovations for a smarter way to report trades in all asset classes. ❖ KOR is the first US trade repository to be licensed since 2014
3. 3 What is KOR Financial? ❖ Built by former trade repository heads, SMEs and technology pioneers, the people of KOR eat, sleep and breathe trade reporting. Keeping human support at the front of what we do, delivering fast, knowledgeable assistance every time. ❖ A decade of learning, experience and a clean start design combined with our first in the industry innovations opens a new world of data processing, management, analytics and insights. Trade Reporting just received a major upgrade.
4. Apache Kafka has become the de facto standard for writing, reading, and sharing data streams Kafka is the de facto standard for data streaming and has been tapped to power… >70% of the Fortune 500 estimated to be using Kafka >100,000+ organizations using Kafka >75% of companies say they would lose customers without insights from real-time data >8 in 10 IT leaders say real-time data streams are critical to responsive business processes Initiate an action Instant confirmation ~100 ms of latency can cost you… … 20% of traffic … $400M in revenue ...real-time customer experiences and business operations essential to survival Source: GigaSpaces Source: 2022 State of Data in Motion Report
5. • Architecture planning • Cluster sizing • Cluster provisioning • Broker settings • Zookeeper management • Partition placement & data durability • Source/sink connectors development & maintenance • Monitoring & reporting tools setup • Software patches and upgrades • Security controls and integrations • Failover design & planning • Mirroring & geo-replication • Streaming data governance • Load rebalancing & monitoring • Expansion planning & execution • Utilization optimization & visibility • Cluster migrations • Infrastructure & performance upgrades / enhancements I N V E S T M E N T & T I M E V A L U E 1 2 3 4 5 Experimentation / Early Interest Central Nervous System Mission critical, disparate LOBs Identify a Project Mission-critical, connected LOBs Key challenges: Operational burden & resources Manage and scale platform to support ever-growing demand Security & governance Ensure streaming data is as safe & secure as data-at-rest as Kafka usage scales Real-time connectivity & processing Leverage valuable legacy data to power modern, cloud-based apps & experiences Global availability Maintain high availability across environments with minimal downtime Operationalizing Kafka on your own is difficult Kafka is hard in experimentation. It only gets harder (and riskier) as you add mission-critical data and use cases.
6. 6 MSK offers no SLA for Kafka-related failures Source: https://aws.amazon.com/msk/sla/ “The Service Commitment DOES NOT APPLY to any unavailability, suspension or termination … caused by the underlying Apache Kafka or Apache Zookeeper engine software that leads to request failures …” ✖ CUSTOMER EXPERIENCE RUNNING AMAZON MSK May 29, 2019: AWS MSK customers experience known cluster time-out issue (KAFKA-7697) AWS response without solution: amzn.to/2KdH207 Feb 1, 2022. KAFKA-13636 serious offset deletion issue reported Feb 10, 2022. Issue resolved and shared with community by Confluent team, fixed in AK 3.0.1 June 22, 2022. +5 months later, AWS team supports Kafka 3.0.1+ to fix KAFKA-13636 RECENT SITUATION
7. Why Confluent over MSK We’ve completely re-architected Kafka to be truly Cloud-native 7 Months Minutes Weeks Open Source Apache Kafka In-house development and maintenance without support AWS MSK Manual operations with basic tooling and/or support Confluent Cloud Fully managed, elastic, and automated product capabilities with zero overhead
8. 8 KOR Financial’s Data Streaming Architecture
9. 9 Technical Benefits & Business Impact KOR Financial • Scalable audit trace of events that constitute trades being reported into our trade repository. Enables replaying • Storage all our events not simply in a downstream datalake, but keeping it inside Kafka itself. • The ability to interpret these events in different ways, with different models, and different technologies. • Storage of an infinite amount of data inside Kafka, thanks to the multi-tiered storage feature that stores the internal segment files in S3 buckets in a cost-effective way. • Validation of our designs. We had workshops with hands-on architects at Confluent who managed to guide us in the right direction.
10. Q & A
11. Guest speaker Andreas Evers Chief Technology Officer at KOR Financial Moderated by Hasan Jilani Product Marketing at Confluent Thank you for listening!