Confluent Data Expo 2024

Transcript

1. How to make operational data accessible to the analytical world Maria Berinde-Tâmpănariu Staff Solutions Engineer @ Confluent 12th of September 2024  Data Expo Amsterdam
2. Set Your Data In Motion with Data Streaming 01 How Data Streaming can accelerate your shift towards Data Product thinking. What Does Best In Class Look Like? 02 Examples of companies that have transformed their business with Confluent. Why Confluent 03 How Confluent helps turn data mess to data products for step change improvement in agility, productivity, time to market and lower TCO 2 Agenda
3. Why you need Data Streaming
4. Data is the Fuel for a Modern Business Loyalty Rewards Payments Processing Real-time Trades Fraud Detection Personalized Recommendations
5. Unlocking Data and Analytics is Much Harder than it Should Be
6. 6 There are Two Main Locations for Data OPERATIONAL ESTATE Serve the needs of applications to transact with customers in real-time ANALYTICAL ESTATE Support after-the-fact business analysis and reporting for various stakeholders
7. OPERATIONAL ESTATE Serve the needs of applications to transact with customers in real-time ANALYTICAL ESTATE Support after-the-fact business analysis and reporting for various stakeholders 7 As Data Needs Multiplied, Data Complexity Escalated
8. … creating this giant mess!
9. Your teams are building applications on this messy data architecture Recommendations Real-time payments Branch monitoring Real-time bot detection Real-time alerts Live customer chatbot Omni-channel orders Real-time inventory Mobile banking Fraud detection
10. Data Streaming: A New Paradigm to Set Data in Motion Continuously ingest, process, govern and share streams of data in real-time Real-time Data A Transfer A Login A Trade A Customer Experience Rich Front-End Customer Experiences Real-Time Backend Operations New Products and Services “We need to shift our thinking from everything at rest to everything in motion.”
11. From Giant Mess to Central Nervous System DATA STREAMING PLATFORM “For a typical Fortune 1000 company, just a 10% increase in data accessibility will result in more than $65 million additional net income.”
12. Shift Your Thinking Point-in-time queries on static data Continuous processing at generation time Tightly coupled point-to-point integrations Multi-subscriber decoupled architecture Producer-presented, governed & reusable data Complex data wrangling downstream Batch-based data integration Event-driven continuous data flows
13. Apply a Data Product Thinking FINANCIAL SERVICES RETAIL & E-COMMERCE MANUFACTURING HEALTHCARE TRANSPORTATION GAMING Customers Accounts Web logins Clickstream Transactions Payments Branches Claims Trades Customers Accounts Purchases Web clickstream Inventory Stores Employees Shipments Machine Telemetry Inventory Orders Shipments Suppliers Assets Equipment Patients Accounts Diagnoses Treatments Claims Readings Providers Players Achievements Profiles Games Purchases Vehicle Telemetry Drivers Customers Conditions Deliveries Repairs Insurers Chats High Scores Data products: Real-time, discoverable, contextualized, trustworthy and reusable
14. Use Data Products to Build New Business Applications Companies that treat data like a product can: • Reduce the time it takes to implement new use cases by as much as 90% • Decrease total ownership costs by up to 30% (technology, development, and maintenance) • Reduce their risk and data governance burden Customers Accounts Web Logins Transactions Fraud Detection Customers Accounts Inventory Purchases Web Clickstream Customer Loyalty Vehicle Telemetry Repairs Drivers Customers Delivery Management Data Products Use Cases
15. Build an Effective Data Flywheel Strategy by Reusing Data Products Across Multiple Use Cases Supply chain systems Watch lists Profile mgmt Incident mgmt Customers Threat vector Customer churn Transactions Payments Mainframe data Inventory Weather Telemetry IoT data Notification engine Payroll systems Distribution Order fulfillment CRM systems Mobile application Personalization Clickstreams Customer loyalty Change logs Purchases Recommendation engine Web application Payment service Heatmap service ITSM systems Central log systems (CLS) Fraud & SIEM systems Alerting systems AI/ML engines Visualization apps
16. PROCESS GOVERN CONNECT STREAM Accounts Customer Profile Purchases Shipments Claims Orders Clickstreams From Data Mess To Data Products To Instant Value Everywhere CONNECT Data Systems Inventory Replenishment Forecasting … Custom Apps & Microservices Personalization Recommendation Fraud … Confluent Data Streaming Platform A new paradigm for reusable, real-time and trustworthy data products
17. What does best in class look like?
18. There are specific ways that we can incorporate real-time data into software development. And then there are other parts of our business, like our ads business, that want to move quickly, want to understand user behavior, and try to inspire users with content they value. Confluent will unlock the creativity of a lot of really smart people who really understand their customers, and that can’t be anything but good. — VP OF ENGINEERING
19. We look at events as running our business. Business people within our organization want to be able to react to events—and oftentimes it's a combination of events. — VP OF STREAMING DATA ENGINEERING
20. With Confluent, we are transforming this very batch, very monolithic information system into one in which data is constantly in motion. It really helps us to decouple our monolith into autonomous systems, to help them evolve independently of each other, and therefore, to be more agile for our business to drive real-time, data-driven decisions and operations. It is data in motion serving business agility. — CHIEF ARCHITECT AND CTO
21. Turn Data Mess to Data Products to Supercharge Business Outcomes 21 Reimagine Customer Experiences Build Efficiencies in Backend Operations Deliver New Products and Services
22. Set your data in motion with Data Streaming
23. The Real-time Rise of Apache Kafka Continuously ingesting and continuously sharing streams of data in real time Real-time Data A Sale A shipment A Trade A Customer Experience Rich Front-End Customer Experiences Real-Time Backend Operations New Products and Services “We need to shift our thinking from everything at rest to everything in motion.”
24. Apache Kafka The de facto data streaming standard across all verticals TRAVEL & RETAIL 10 OUT OF 10 BANKING INSURANCE TELECO TECHNOLOGY AUTO & TRANS. HEALTHCARE ENTERTAINMENT 10 OUT OF 10 9 OUT OF 10 10 OUT OF 10 10 OUT OF 10 10 OUT OF 10 10 OUT OF 10 7 OUT OF 8 Fortune 500 Companies Estimated to be Using Kafka >75%
25. What Makes Apache Kafka So Popular A scalable and decoupled architecture as a single source of record for high-quality, self-service access to real-time and reusable data products Source Source Source Source Data Product Data Product Data Product Data Product Source Source Source Source Data Product Data Product Data Product Data Product Combine and enrich data from anywhere to anywhere for real-time data sharing and greater reuse “With Kafka and our schema evolution strategy, we’re able to decouple everything. So you can change either side of the equation. I think that’s something unique in the platform.”
26. Mainframe Integration Cloud DWH Ingestion Log Aggregation Clickstream Real-Time Analytics Ingestion Real-Time Analytics Change Data Capture Complex Events Processing Hybrid / Multi-Cloud Data Flow SaaS App Integration SIEM Pipelines ML Feature Pipelines Unlock Endless Use Cases with Instant Data Access Credit Card Fraud Detection Customer Retention / Loyalty APT Detection IT Inventory Logistics Geofencing Route Optimization Payment Verification Dynamic Pricing Fleet Management De-identification Shipment Tracking / Alerting Personal Recommendations STREAMING ARCHITECTURE PIPELINES APPLICATIONS
27. Why Confluent
28. We are the pioneers and experts in data streaming Created by the founders of Confluent “Confluent has an asset that competitors can't replicate: The founders of Confluent were the original developers of Kafka, making it very hard to find anyone that knows the open-source project better than them.”
29. Confluent Is So Much More Than Apache Kafka Reimagine data streaming everywhere, on-prem and in every major public cloud STREAM Make data in motion self-service, secure, compliant and trustworthy GOVERN Drive greater data reuse with always-on stream processing PROCESS Make it easy to on-ramp and off-ramp data from existing systems and apps CONNECT
30. Operating Apache Kafka on Your Own Is Difficult I N V E S T M E N T & T I M E V A L U E 4 Early Interest / Experimentation Central Nervous System Mission critical, in Production, but disparate LOBs Identify a Project Mission-critical, connected LOBs As the Kafka footprint expands across more use cases, teams, and environments, it creates significant ongoing operational burden… one that only grows over time, requiring more top engineering talent & resources 5 3 2 1 “One of the challenges with Kafka was its operational complexity, especially as the footprint expanded across our organization. It’s a complex, distributed system, so we had to allocate a lot of our valuable technical resources and expertise to babysit it and keep it running. This also required us to incur additional in-house maintenance costs and risks.” ONEROUS OPERATIONS • Architecture planning • Cluster sizing • Cluster provisioning • Broker settings • Zookeeper management EXPENSIVE ENGINEERING, WEAK SECURITY AND AVAILABILITY • Partition placement & data durability • Source/sink connectors development & maintenance • Monitoring & reporting tools setup • Software patches and upgrades • Security controls and integrations • Failover design & planning RISKY GOVERNANCE AND RELIABILITY AT ENTERPRISE SCALE • Mirroring & geo-replication • Streaming data governance • Load rebalancing & monitoring • Expansion planning & execution • Utilization optimization & visibility • Cluster migrations • Infrastructure & performance upgrades / enhancements INCREASED TOTAL COST OF OWNERSHIP STREAM
31. We Transformed Apache Kafka for the Cloud, Ground Up with KORA! NETWORK COMPUTE OBJECT STORAGE CUSTOMERS AZ AZ AZ Cells Cells Cells Multi-Cloud Networking & Routing Tier Metadata Durability Audits Data Balancing Health Checks Real-time feedback data GLOBAL CONTROL PLANE S E C U R I T Y STREAM The Kora Architecture ELASTIC to seamlessly expand and shrink based on usage demands RESILIENT with automated operations to ensure high availability and reliability COST-EFFICIENT with multi-tenancy, intelligent data tiering and cloud optimizations GLOBAL across all major public clouds and on-prem environments
32. Free Up Talent to Move Up the Stack Serverless • Elastic scaling up & down from 0 to GBps • Auto capacity mgmt, load balancing, and upgrades High Availability • 99.99% SLA • Multi-region / AZ availability across cloud providers • Patches deployed in Confluent Cloud before Apache Kafka Infinite Storage • Store data cost- effectively at any scale without growing compute DevOps Automation • API-driven and/or point-and-click ops • Service portability & consistency across cloud providers and on-prem Network Flexibility • Public, VPC, and Private Link • Seamlessly link across clouds and on-prem with Cluster Linking “Before Confluent Cloud, when we had an outage, developers would have to stop their development work and focus on operations until the problem was sorted out, which in some cases took up to three days. (Now) Confluent takes care of everything for us, so our developers can focus on building new features and applications that add value to the business.” STREAM
33. $2.57M Total savings Operate 60%+ more efficiently with reduced infrastructure costs, maintenance demands and overhead, and downtime risk 257% 3-year ROI Launch in months rather than years by reducing the burden on your teams with our fully managed cloud service Save on Costs and Increase your ROI Total Economic Impact of using Confluent • Forrester, March 2022 (Link) STREAM “We are in the business of selling and renting clothes. We are not in the business of managing a data streaming platform… If we had to manage everything ourselves, I would’ve had to hire at least 10 more people to keep the systems up and running.” “Confluent Cloud made it possible for us to meet our tight launch deadline with limited resources. With event streaming as a managed service, we had no costly hires to maintain our clusters and no worries about 24x7 reliability."
34. Connect and Share your Data Everywhere “We are making Confluent the true backbone of BHG, including leveraging over 20 Confluent connectors across both modern, cloud-based technologies and legacy systems, to help integrate our critical apps and data systems together.” 34 Cloud Cloud On-premises Run Anywhere Run Unified Run Consistently Supporting any combination of on-prem, hybrid cloud and multicloud deployments for real-time data interoperability STREAM
35. Unlock Real-time Value From Existing Systems and Applications with Ease Pre-built Connectors Use 120+ self-managed and 70+ fully managed zero-code connectors for instant connectivity from and to existing data systems Custom Connectors Bring your own connector and run it confidently and cost-effectively on our fully managed cloud service Native Integrations Enjoy instant access to streaming data directly within popular platforms, so you never have to leave the tool of your choice “Confluent provides exactly what we dreamed of: an ecosystem of tools to source and sink data from data streams. It’s provided us not only with great data pipeline agility and flexibility but also a highly simplified infrastructure that’s allowed us to reduce costs.” Data Diode CONNECT
36. Process your data once, process your data right Process and Shape Data As It’s Generated For Greater Reuse 36 Lower costs Improve resource utilization and cost-effectiveness by avoiding redundant processing across silos Data reusability Share consistent and reusable data streams widely with downstream applications and systems Data enrichment Curate, filter, and augment data on-the-fly with additional context to improve completeness and accuracy Real-time everywhere Power low-latency applications and pipelines that react to real-time events, as and when they occur PROCESS
37. Discover, Understand, and Trust Your Data Streams Where did data come from? Where is it going? Where, when, and how was it transformed? What’s the common taxonomy? What is the current state of the stream? Stream Catalog Increase collaboration and productivity with self-service data discovery Stream Lineage Understand complex data relationships and uncover more insights Stream Quality Deliver trusted, high-quality event streams to the business “Confluent’s Stream Governance suite will play a major role in our expanded use of data in motion and creation of a central nervous system for the enterprise. With the self-service capabilities in stream catalog and stream lineage, we’ll be able to greatly simplify and accelerate the onboarding of new teams working with our most valuable data." GOVERN
38. Discover Search and understand the data of interest through a self-service user interface Access Request and securely grant access to data streams through simple clicks of buttons Build Start working with your data in real-time to pipe it into existing systems or develop real-time apps Explore and Use Real-time Data Products with a Self-service Portal GOVERN
39. Stream Confidently with Robust Security, Privacy, Compliance and Observability Rich identity, access and confidentiality controls End-to-end encryption, granular role based access controls, BYOK and cloud-native authentication methods Native infrastructure health checks and audit controls Central management of environment health, automated load rebalancing, built-in audit logs and health reports Integrations with market leading observability tools Built-in integrations into tools such as Datadog, Prometheus, Dynatrace, Grafana Cloud and New Relic Compliance with industry regulations Certified for ISO 27001, ISO 27701, SOC I, SOC II and SOC III, PCI, CSA, TISAX, and HITRUST; GDPR, CCPA, HIPAA ready and Financial Service Regulation Compliance “We need authentication, certification, and highly secure systems for our own compliance and obviously our clients. From a security point of view, I don’t know of any other vendor with a cross-cloud offering we would trust—and that our clients would trust—more than Confluent.” GOVERN
40. Targeted Resources and Expertise for Every Stage of the Journey “When we saw the demand for harnessing data in motion growing, working with Confluent and investing in our Center of Excellence enabled us to implement a solution in a way that best benefits the entire enterprise by reducing complexity, costs, and time-to-market.” EVP Chief Data Officer & Divisional CIO KeyBank Central Nervous System Mission critical, in production, but disparate lines of business (LOBs) Mission critical, connected LOBs Identify and launch a project Experimentation / early interest • Fundamentals Classes • How-to Demos • Limited Free Support • PS Starter Packages • Success Planning • Confluent Certifications • Developer Support • Technical Onboarding • Deployment, Governance, and Migration Services • Stream Processing app design • Business / Gold Support • Multi-cloud / Multi-region Architecture Design • Advanced Security Configuration • Disaster Recovery planning • Premier / Platinum Support with architecture guidance • Confluent Operating Model • Data Mesh Architecture Design • Autoscaling & On-demand Cluster Governance • Enterprise Education Subscription World Class Support for every Stage Resident & Success Services Tech Touch for Service at Scale Architectural and Implementation Experts 1 2 3 4 5
41. The Confluent Advantage Create a central nervous system for your data across your enterprise Resilient Infinite Elastic Run Consistently Run Anywhere Stream Governance Stream Processing 120+ Connectors Run Unified Apache Kafka® fully managed, and re-architected to harness the power of the cloud Go way beyond Kafka to build real-time apps quickly, reliably, and securely Flexible deployment options across clouds and on-premises that sync in real-time Cloud Native Complete Everywhere Built and supported by the founders of Apache Kafka® Freedom of Choice Committer-driven Expertise Fully Managed Cloud Service Self-managed Software Training Partners Professional Services Enterprise Support End-to-end Security, Privacy, Compliance and Observability