ABN AMRO Data Expo 2024

Transcript

1. The Value of Data Streaming at ABN AMRO Bank September 2024 Data & Analytics Platform
2. Agenda 1 2 3 4 5 6 AAB: data at the heart of everything Three main pillars in our data strategy Our central data distribution platform High-level overview of DIAL Why Streaming? High-level overview of our Streaming platform Confluent Kafka Streaming Platform Technology overview Key challenges & platform evolution What challenges did we run into, and how did we tackle those? Streaming brings value to AAB But how is value being created in practice? 6-9-2024 | 2
3. High-level overview & context Jelle Jansma
4. ABN AMRO: Data at the heart of everything 6-9-2024 | 4
5. Three main pillars in our data strategy 6-9-2024 | 5
6. DIAL has a central position in ABN AMRO Data landscape 6-9-2024 | 6 Data Distribution Data Marketplace Digital Integration & Access Layer (DIAL) Data Provider Data Consumer Data sharing agreement Data delivery Define datasets Search & request access B. Rapid Access to Authorised Data C. Accelerate Data Driven Decisions A. Making fit for purpose data
7. We have three main integration patterns within DIAL 6-9-2024 | 7 Streaming pattern Batch pattern API pattern
8. Batch vs. Streaming 6-9-2024 Source: confluent.io/learn/batch-vs-real-time-data-processing/ | 8 Batch Data Processing Streaming Data Hardware Most storage and processing resources requirement to process large batches of data. Less storage required to process current data packets. More processing resources required to “stay awake” in order to meet real-time processing guarantees Performance Latency could be minutes, hours, or days Latency must be guaranteed in milliseconds Dataset Large batches of data Continuous streams of data Analysis Complex computation and analysis of a larger time frame Simple reporting or computation
9. Why Streaming? Continuous flow of data 6-9-2024 | 9 Real-time insights Wide variety of use cases Seamless integration Reduced latency
10. Key figures on AAB Kafka Streaming Platform 6-9-2024 | 10 Consumers Producers/Providers Use cases in Production Use cases in lower environments (DTA) 41 use cases 26 use cases 56 use cases 42 use cases
11. Streaming platform: how does it add value to AAB? Shahebaz Shaik
12. Platform overview (DIAL Context) 6-9-2024 | 12 Files Events with Payload Golden Sources Providers Data Lake Streaming Platform Data Governance, Metadata & Access Management Consumers File Transferring API Processing Ad-hoc Querying Event Processing Channels Reporting Advanced Analytics Realtime Applications Federated Domain Data Platforms
13. Supported patterns on AAB Streaming Platform 6-9-2024 | 13 Consumers Producers Overview of the current implemented connection types CDC Sink connector for files Sink connector for Database Kafka Client Kafka Client Use cases in Production Use cases in lower environments (DTA) Kafka Replicator
14. Technology state Kafka client support Java C++ Go .NET Python JMS Kafka connectors Replicator JDBC source/sink Cloud storage Hadoop / SFTP MQ and other brokers Cloud other Schema registry Monitoring Kafka REST API’s Kafka provides various api’s to manage the platform. Producer and Consumer API’s are available Azure Monitor is used for OS, Platform Logs & JMX monitoring kSQLDB and Kafka Streams Build applications and microservices using Kafka Streams and ksqlDB Schema Registry provides a serving layer (RESTful interface) for your metadata. For storing and retrieving your Avro, JSON and Protobuf schemas. New New New
15. Key challenges ✓ Provider data does not completely fit consumers’ requirements & PII Data handling ✓Row/Column Level Filtering ✓Union of multiple datasets ✓Transformation/Aggregation of datasets ✓Masking PII data fields using KSQL ✓Crypto Shedding Technique 6-9-2024 | 15
16. Key challenges (2) Ordering & Data Guarantees ✓ Use 1 partition for global ordering ✓ Use transactional producer/consumer for exactly once semantics ✓ Use Idempotent consumer to handle duplicates 6-9-2024 | 16 One type of governance doesn’t fit all usecases Make the right data for a specific purpose discoverable by consumers Loosely-coupling mindset in a strictly regulated environment Integration with the existing landscape Smooth transition from sharing events locally towards sharing enterprise- wide
17. Evolving our Enterprise Streaming Platform (Business Events) Event Notification 6-9-2024 | 17 Event with Payload Example { "specversion" : "1.0", "type" : "com.example.orderPlaced", "source" : "/order/v1/A001-1234-1234", "id" : "A001-1234-1234", "time" : "2020-12-15T00:00:00Z" } Example: { "specversion" : "1.0", "type" : "com.example.orderPlaced", "datacontenttype" : "application/json", "data" : { "id" : "A001-1234-1234", "time" : "2020-12-15T00:00:00Z", "products" : [{ "id" : "1234321", "name" : "eBook Seven Languages in Seven Weeks", "price" : 25.00, "quantity" : 1 }] } }
18. App 2 App 3 App 1 App 2 App 3 App 1 App 2 App 3 App 1 Evolving our Enterprise Streaming Platform (for ‘private’ use) 6-9-2024 | 18 Streaming Platform Private Events Events with Payload Business Events Governance Governance Governance
19. Example use case: near real-time mainframe offloading 6-9-2024 | 19
20. Example use case: Digital Document Exchange 6-9-2024 | 20
21. Example use case: Track & Trace 6-9-2024 | 21
22. Q&A