$30 off During Our Annual Pro Sale. View Details »

Mainframe Modernization: From COBOL to Event Sourcing

Mainframe Modernization: From COBOL to Event Sourcing

This talk will explore modern programming techniques such as reactive programming, event storming, and domain-driven design to demonstrate how leading organizations in banking are modernizing their legacy systems -- including COBOL! -- to solve scaling, cost, and business continuity issues such as recruiting. We'll cover a mix of real-world use cases and our collective experience to help practitioners and technology leaders get started.

Anyone interested in reactive programming or how banks *actually work* should get a lot out of this talk. There are a host of business opportunities waiting to be unleashed through the power of reactive programming! For the most part, companies have stopped short of full modernization: bringing book of record systems into a modern era of technology. Meanwhile, customer adoption of these new platforms has led to an explosion of traffic to underlying legacy systems such as mainframes, resulting in huge cost increases, degraded performance and sub-par customer experiences.

We'll showcase the power of modern technologies such as Play, Akka, and Kafka, and how they are being used to break the chains of the past. We will demonstrate techniques such as event sourcing and CQRS for improving latency from mobile to mainframe from 10+ seconds to sub-second. In this always-on, always-connected era, consumers expect real-time insights. We'll show you a path towards real-time.

Kevin Webber

May 10, 2018
Tweet

More Decks by Kevin Webber

Other Decks in Programming

Transcript

  1. Mainframe
    Modernization
    From COBOL to Event Sourcing
    PRESENTED BY:

    View Slide

  2. Topics
    Part 1: An Introduction to Legacy Systems
    Part 2: The Old World and New World Collide
    Part 3: A Path to Full Modernization
    Part 4: The Read Only Bank
    Part 5: Conclusion

    View Slide

  3. View Slide

  4. Part 1
    An Introduction to Legacy
    Systems
    Part 1: An introduction to legacy systems
    Part 2: The old world and new world collide
    Part 3: A path to full modernization
    Part 4: The Read Only Bank
    Part 5: Conclusion

    View Slide

  5. Batches, batches everywhere!

    View Slide

  6. View Slide

  7. Need a new feature?
    Add a Layer!

    View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. Let’s go!

    View Slide

  14. How long do you think it takes
    for this call to return?

    View Slide

  15. View Slide

  16. It’s not just the performance...
    Customer
    Expectations
    Risk Cost
    + +

    View Slide

  17. Customer expectations
    As the market evolves, customer expectations go up
    They expect higher quality data that is up to date
    Customers want an instant view of their world
    Customers compare to all apps on their phone

    View Slide

  18. All companies are technology companies
    The Wealthsimple
    effect
    Robo advisors were
    the first wave of
    competition
    Amazon launching
    chequing accounts
    The second wave of
    competition will be
    fierce
    Uber is a bank?
    Uber is one of the
    largest originators of
    small business loans in
    the U.S

    View Slide

  19. 2020 and beyond?
    Core mainframe systems struggling to adapt to a
    mobile-first, always connected world
    Legacy systems designed decades ago
    Near-real time, API-driven usage was unanticipated
    How long is this sustainable?

    View Slide

  20. Ch-ch-changes
    What happens when a core system needs to
    change with urgency?
    Regulators are forcing improvements in many
    areas
    ○ PSD2 Data Requirements (Open Banking)
    ○ Real Time Payments
    ○ GDPR

    View Slide

  21. Technical limitations
    Different views across systems

    View Slide

  22. Batch-based processing
    Multi-day processing times for simple transactions
    Technical limitations

    View Slide

  23. Limited size for transaction descriptions
    Means they are ı̖͇̳̰̰͍̲̻͕̩͗̒̊̈́̈́̀͆ ̃̀͘ň̢
    ̤͔̙͙̖̙͙͍͉͛̈͋͑̄͒ ̃
    ͗̚ ç̧
    ̲͍̯̝̻͉͎̦͛͛̒̀̀̎̍̌ ̕
    ͠ ǒ̧
    ̙̞̭̺̘͈͇̤̀̐̑͗́̃
    ͂͂͂͜ m̨
    ̩̙̝̹̥̟͚̼̟͌͆̓̏̈́̆͐ ͘̕͝p̛̛̠̲̙̰͚̥͙̳͈̦͐̇̑̈̊̈́̉͠ r̡
    ̼͎͕̗͈͍̘̗͚̋͌̈́͋̌̒̓́̈̚ę̢
    ̩̜̱̖͍̮̜͖̿̿́̑̓̉̓̀̋̚ ḥ
    ͇̳̟̪͖̜̙̳̙́͊̆̂͊̈̎̿̓͠ e
    ̖̙̪͖̹̫͇̱͉̹͆́̽͂̍͑̓̍͝͝ n
    ̰̞̳̭̭͙̮̥̤̖͌͂̌̂̀̈̍͛͒͠ s̢
    ̨̛͖̗̰̤͚̞̙͙̆̆͐̆̈́̃
    ̂͆̈́ï̧̟̜̪̩̠̜̥͖̥̂̒̏̂̅̀͑̿ ͘b̧
    ̫̙̘̥̯̖̞͚͕̉̃
    ͋̀̀̀̎͌ ͘͘l̢̨̢͚͉̜͕̪̙͉͑͐̌̓̆̽͂̂̆̀ë̛
    ̟̯ ̣
    ̙̫̰̼̭̫͖́̌̎̑̈̅̾̅͝
    Technical limitations

    View Slide

  24. Technical limitations
    Deposit systems are running out of account numbers
    Accounts have balance and transaction size limits due to
    lack of integer space
    Support for only one currency

    View Slide

  25. Increasing risk
    SMEs are retiring
    Technical expertise
    No Documentation

    View Slide

  26. Regulators demand more

    View Slide

  27. Batch Risk
    Batch based systems have failure modes that can quickly
    cascade and are quite fragile
    Due to tight time windows, failures can take days to
    rectify
    Despite the jobs being rock solid, they are only as good
    as the scheduling system

    View Slide

  28. Getting Already very expensive
    Mainframes, and most software running on them (OS,
    databases, etc), are priced on capacity measured in MIPS
    Testing requires an army of people, and the expense of
    the underlying platform limits the number of testing
    environments

    View Slide

  29. Getting Already very expensive
    The business have large back office teams who perform
    tasks solely to overcome system limitations

    View Slide

  30. Recap of goals
    ➢ Improve customer experience
    ➢ Create a path for recruiting and retaining top talent
    ➢ Reduce operational costs
    ➢ Reduce risk
    ➢ Increase responsiveness to business needs
    ➢ What’s in our way?

    View Slide

  31. Part 2
    The Old World and New
    World Collide
    Part 1: An introduction to legacy systems
    Part 2: The old world and new world collide
    Part 3: A path to full modernization
    Part 4: The Read Only Bank
    Part 5: Conclusion

    View Slide

  32. COBOL: The Good Parts
    COBOL and JCL
    ○ Immutable datasets written to disk, read by next step
    ○ Can handle massive throughput
    ○ Resilient, can be rerun in the event of failure (e.g, bad data)
    ○ Referential transparency
    What do mainframe systems sound similar to?

    View Slide

  33. Now is the time for full modernization
    Modern tools and
    techniques are ready for
    mainframe workloads

    View Slide

  34. Steps
    Step 1: Understand current state
    Step 2: Model future state
    Step 3: Isolate legacy components
    Step 4: API enablement
    Step 5: Conclusion

    View Slide

  35. Tactic 1: Understand current state
    Event storming:
    Model interesting business
    processes in your organization
    Use a well defined set of
    operations (commands,
    events)

    View Slide

  36. Credit
    evaluation
    policy

    View Slide

  37. Tactic 2: Model future state
    Domain Driven Design:
    ○ Domains
    ○ Subdomains
    ○ Cores
    ○ Ubiquitous language

    View Slide

  38. Domains and
    events
    - Domain driven design
    helps us identify boundaries
    - Event sourcing helps us
    define interactions via
    aggregate events and
    domain events

    View Slide

  39. Tactic 3: Isolate
    Ensure responsiveness to change:
    ○ Isolate business functionality within well defined domains
    ○ Model interaction between domains with domain events
    ○ Unbounded complexity is what prevents change!

    View Slide

  40. Interaction
    between
    domains
    with domain
    events

    View Slide

  41. Tactic 4: Enable
    Microservices and gateways
    ○ Expose data and functionality through service interfaces
    ○ REST, pub/sub, or streams: anything with a contract
    ○ Bezos’ tenets
    ■ No direct linking
    ■ No direct reads of another team’s data store
    ■ No shared-memory model
    ■ No backdoors whatsoever

    View Slide

  42. CQRS and
    Event Sourcing
    - CQRS within a single
    domain
    - Commands, events,
    queries, and views
    form the contract
    - No relational
    database required

    View Slide

  43. Tactic 5: Modernize
    Slowly replace the old with the new:
    ○ Stabilize legacy integrations
    ○ Replace legacy where it brings value
    ○ Leave legacy where replacement costs outweighs benefits

    View Slide

  44. ➢ Named after the stranger vine that grows
    up and around trees, effectively
    “replacing” them
    ➢ We can use a similar pattern to replace
    functionality in a legacy system with new
    functionality
    ➢ Keeps the tree intact until completely
    replaced
    “Strangler Pattern”:
    Decompose without destroying!

    View Slide

  45. Decompose without destroying!
    ○ The legacy system is never
    modified
    ○ Transaction scripts can be
    thousands of lines long and have
    many side effects
    ○ Worst case scenario: revert to an
    unmodified, functioning legacy
    system

    View Slide

  46. View Slide

  47. View Slide

  48. View Slide

  49. Legacy
    integration
    with anti-
    corruption
    layer

    View Slide

  50. Part 3
    A Path to Full
    Modernization
    Part 1: An introduction to legacy systems
    Part 2: The old world and new world collide
    Part 3: A path to full modernization
    Part 4: The Read Only Bank
    Part 5: Conclusion

    View Slide

  51. Commonwealth Bank of Australia
    Core Banking Replacement
    $1.3Bn core banking replacement system and still replacing
    components
    ”They have been developing it for 18 months and they are still
    adding modules.”
    ”The upgrade started in 2007 and was finished by April of this year
    [2013] , with 1500 people working on the project full time for six
    years.”

    View Slide

  52. There is a another way...

    View Slide

  53. 70% read-only
    traffic

    View Slide

  54. View Slide

  55. Facts
    Relevant Facts

    View Slide

  56. Change Data
    Capture (CDC)

    View Slide

  57. View Slide

  58. View Slide

  59. Part 4
    The Read Only Bank
    Part 1: An introduction to legacy systems
    Part 2: The old world and new world collide
    Part 3: A path to full modernization
    Part 4: The Read Only Bank
    Part 5: Conclusion

    View Slide

  60. View Slide

  61. Why is this so important?

    View Slide

  62. Lower cost &
    create new
    possibilities

    View Slide

  63. What about speed
    and innovation?

    View Slide

  64. What about organization structure?

    View Slide

  65. What about architecture?

    View Slide

  66. What about talent?

    View Slide

  67. What about
    DevOps?

    View Slide

  68. Part 5
    Conclusion
    Part 1: An introduction to legacy systems
    Part 2: The old world and new world collide
    Part 3: A path to full modernization
    Part 4: The Read Only Bank
    Part 5: Conclusion

    View Slide

  69. Conclusion
    Current core systems are under immense strain with an explosion in traffic
    and heightened customer expectations
    The implementation is new but the concepts are proven
    Urgency to act - regulators, competitors and talent
    There is a path forward to break the chains of the past

    View Slide

  70. Let’s chat!
    Kevin Webber, RedElastic
    Principal Consultant
    [email protected]
    Dominic Wallace, CGI
    Director, Emerging Technology
    Engineering
    [email protected]
    Adrian Maurer, CGI
    Director, Emerging Technology
    Engineering
    [email protected]

    View Slide