Mainframe Modernization: From COBOL to Event Sourcing

This talk will explore modern programming techniques such as reactive programming, event storming, and domain-driven design to demonstrate how leading organizations in banking are modernizing their legacy systems -- including COBOL! -- to solve scaling, cost, and business continuity issues such as recruiting. We'll cover a mix of real-world use cases and our collective experience to help practitioners and technology leaders get started.

Anyone interested in reactive programming or how banks *actually work* should get a lot out of this talk. There are a host of business opportunities waiting to be unleashed through the power of reactive programming! For the most part, companies have stopped short of full modernization: bringing book of record systems into a modern era of technology. Meanwhile, customer adoption of these new platforms has led to an explosion of traffic to underlying legacy systems such as mainframes, resulting in huge cost increases, degraded performance and sub-par customer experiences.

We'll showcase the power of modern technologies such as Play, Akka, and Kafka, and how they are being used to break the chains of the past. We will demonstrate techniques such as event sourcing and CQRS for improving latency from mobile to mainframe from 10+ seconds to sub-second. In this always-on, always-connected era, consumers expect real-time insights. We'll show you a path towards real-time.

Transcript

1. Mainframe Modernization From COBOL to Event Sourcing PRESENTED BY:
2. Topics Part 1: An Introduction to Legacy Systems Part 2: The Old World and New World Collide Part 3: A Path to Full Modernization Part 4: The Read Only Bank Part 5: Conclusion
4. Part 1 An Introduction to Legacy Systems Part 1: An introduction to legacy systems Part 2: The old world and new world collide Part 3: A path to full modernization Part 4: The Read Only Bank Part 5: Conclusion
5. Batches, batches everywhere!
7. Need a new feature? Add a Layer!
13. Let’s go!
14. How long do you think it takes for this call to return?
16. It’s not just the performance... Customer Expectations Risk Cost + +
17. Customer expectations As the market evolves, customer expectations go up They expect higher quality data that is up to date Customers want an instant view of their world Customers compare to all apps on their phone
18. All companies are technology companies The Wealthsimple effect Robo advisors were the first wave of competition Amazon launching chequing accounts The second wave of competition will be fierce Uber is a bank? Uber is one of the largest originators of small business loans in the U.S
19. 2020 and beyond? Core mainframe systems struggling to adapt to a mobile-first, always connected world Legacy systems designed decades ago Near-real time, API-driven usage was unanticipated How long is this sustainable?
20. Ch-ch-changes What happens when a core system needs to change with urgency? Regulators are forcing improvements in many areas ◦ PSD2 Data Requirements (Open Banking) ◦ Real Time Payments ◦ GDPR
21. Technical limitations Different views across systems
22. Batch-based processing Multi-day processing times for simple transactions Technical limitations
23. Limited size for transaction descriptions Means they are ı̖͇̳̰̰͍̲̻͕̩͗̒̊̈́̈́̀͆ ̃̀͘ň̢ ̤͔̙͙̖̙͙͍͉͛̈͋͑̄͒ ̃ ͗̚ ç̧ ̲͍̯̝̻͉͎̦͛͛̒̀̀̎̍̌ ̕ ͠ ǒ̧ ̙̞̭̺̘͈͇̤̀̐̑͗́̃ ͂͂͂͜ m̨ ̩̙̝̹̥̟͚̼̟͌͆̓̏̈́̆͐ ͘̕͝p̛̛̠̲̙̰͚̥͙̳͈̦͐̇̑̈̊̈́̉͠ r̡ ̼͎͕̗͈͍̘̗͚̋͌̈́͋̌̒̓́̈̚ę̢ ̩̜̱̖͍̮̜͖̿̿́̑̓̉̓̀̋̚ ḥ ͇̳̟̪͖̜̙̳̙́͊̆̂͊̈̎̿̓͠ e ̖̙̪͖̹̫͇̱͉̹͆́̽͂̍͑̓̍͝͝ n ̰̞̳̭̭͙̮̥̤̖͌͂̌̂̀̈̍͛͒͠ s̢ ̨̛͖̗̰̤͚̞̙͙̆̆͐̆̈́̃ ̂͆̈́ï̧̟̜̪̩̠̜̥͖̥̂̒̏̂̅̀͑̿ ͘b̧ ̫̙̘̥̯̖̞͚͕̉̃ ͋̀̀̀̎͌ ͘͘l̢̨̢͚͉̜͕̪̙͉͑͐̌̓̆̽͂̂̆̀ë̛ ̟̯ ̣ ̙̫̰̼̭̫͖́̌̎̑̈̅̾̅͝ Technical limitations
24. Technical limitations Deposit systems are running out of account numbers Accounts have balance and transaction size limits due to lack of integer space Support for only one currency
25. Increasing risk SMEs are retiring Technical expertise No Documentation
26. Regulators demand more
27. Batch Risk Batch based systems have failure modes that can quickly cascade and are quite fragile Due to tight time windows, failures can take days to rectify Despite the jobs being rock solid, they are only as good as the scheduling system
28. Getting Already very expensive Mainframes, and most software running on them (OS, databases, etc), are priced on capacity measured in MIPS Testing requires an army of people, and the expense of the underlying platform limits the number of testing environments
29. Getting Already very expensive The business have large back office teams who perform tasks solely to overcome system limitations
30. Recap of goals ➢ Improve customer experience ➢ Create a path for recruiting and retaining top talent ➢ Reduce operational costs ➢ Reduce risk ➢ Increase responsiveness to business needs ➢ What’s in our way?
31. Part 2 The Old World and New World Collide Part 1: An introduction to legacy systems Part 2: The old world and new world collide Part 3: A path to full modernization Part 4: The Read Only Bank Part 5: Conclusion
32. COBOL: The Good Parts COBOL and JCL ◦ Immutable datasets written to disk, read by next step ◦ Can handle massive throughput ◦ Resilient, can be rerun in the event of failure (e.g, bad data) ◦ Referential transparency What do mainframe systems sound similar to?
33. Now is the time for full modernization Modern tools and techniques are ready for mainframe workloads
34. Steps Step 1: Understand current state Step 2: Model future state Step 3: Isolate legacy components Step 4: API enablement Step 5: Conclusion
35. Tactic 1: Understand current state Event storming: Model interesting business processes in your organization Use a well defined set of operations (commands, events)
36. Credit evaluation policy
37. Tactic 2: Model future state Domain Driven Design: ◦ Domains ◦ Subdomains ◦ Cores ◦ Ubiquitous language
38. Domains and events - Domain driven design helps us identify boundaries - Event sourcing helps us define interactions via aggregate events and domain events
39. Tactic 3: Isolate Ensure responsiveness to change: ◦ Isolate business functionality within well defined domains ◦ Model interaction between domains with domain events ◦ Unbounded complexity is what prevents change!
40. Interaction between domains with domain events
41. Tactic 4: Enable Microservices and gateways ◦ Expose data and functionality through service interfaces ◦ REST, pub/sub, or streams: anything with a contract ◦ Bezos’ tenets ▪ No direct linking ▪ No direct reads of another team’s data store ▪ No shared-memory model ▪ No backdoors whatsoever
42. CQRS and Event Sourcing - CQRS within a single domain - Commands, events, queries, and views form the contract - No relational database required
43. Tactic 5: Modernize Slowly replace the old with the new: ◦ Stabilize legacy integrations ◦ Replace legacy where it brings value ◦ Leave legacy where replacement costs outweighs benefits
44. ➢ Named after the stranger vine that grows up and around trees, effectively “replacing” them ➢ We can use a similar pattern to replace functionality in a legacy system with new functionality ➢ Keeps the tree intact until completely replaced “Strangler Pattern”: Decompose without destroying!
45. Decompose without destroying! ◦ The legacy system is never modified ◦ Transaction scripts can be thousands of lines long and have many side effects ◦ Worst case scenario: revert to an unmodified, functioning legacy system
49. Legacy integration with anti- corruption layer
50. Part 3 A Path to Full Modernization Part 1: An introduction to legacy systems Part 2: The old world and new world collide Part 3: A path to full modernization Part 4: The Read Only Bank Part 5: Conclusion
51. Commonwealth Bank of Australia Core Banking Replacement $1.3Bn core banking replacement system and still replacing components ”They have been developing it for 18 months and they are still adding modules.” ”The upgrade started in 2007 and was finished by April of this year [2013] , with 1500 people working on the project full time for six years.”
52. There is a another way...
53. 70% read-only traffic
55. Facts Relevant Facts
56. Change Data Capture (CDC)
59. Part 4 The Read Only Bank Part 1: An introduction to legacy systems Part 2: The old world and new world collide Part 3: A path to full modernization Part 4: The Read Only Bank Part 5: Conclusion
61. Why is this so important?
62. Lower cost & create new possibilities
63. What about speed and innovation?
64. What about organization structure?
65. What about architecture?
66. What about talent?
67. What about DevOps?
68. Part 5 Conclusion Part 1: An introduction to legacy systems Part 2: The old world and new world collide Part 3: A path to full modernization Part 4: The Read Only Bank Part 5: Conclusion
69. Conclusion Current core systems are under immense strain with an explosion in traffic and heightened customer expectations The implementation is new but the concepts are proven Urgency to act - regulators, competitors and talent There is a path forward to break the chains of the past
70. Let’s chat! Kevin Webber, RedElastic Principal Consultant kevin.webber@redelastic.com Dominic Wallace, CGI Director, Emerging Technology Engineering dominic.wallace@cgi.com Adrian Maurer, CGI Director, Emerging Technology Engineering adriano.maurer@cgi.com