DevOpsDays Cuba 2017: Experiences on building a modular reactive architecture CIR-BCC

D5db2dc3cc883df3479797edb63b581b?s=47 DevOpsDays Cuba
October 25, 2017
110

DevOpsDays Cuba 2017: Experiences on building a modular reactive architecture CIR-BCC

Author: Anesto del Toro
Summary: In this talk we’ll share some experiences during the development of the
CIR-BCC (Risk Information Center - Cuban Central Bank). We briefly describe the problem, the main challenges and their impact in key architectural and design decisions. We present some insights of our reactive modular solution, developed with Scala, Play Framework and Akka, and we illustrate how Event Sourcing (ES), Command-Query Responsibility Segregation (CQRS) and Change Data Capture (CDC) have helped us to solve some important challenges. We expose several
of the practices that have allowed us to achieve faster development
and deployment cycles for this product.

D5db2dc3cc883df3479797edb63b581b?s=128

DevOpsDays Cuba

October 25, 2017
Tweet

Transcript

  1. Experiences on building a modular reactive architecture CIR-BCC

  2. About Me @anestodta anesto.deltoro@datys.cu https://www.linkedin.com/in/anesto-del-toro • Head of Production, DATYS

    VC • MSc Computer Sciences • Wearing multiple hats oTechnical Lead oArchitecture oSoftware Development (mostly backend) oProject Management oBusiness Analysis oLean-Agile enthusiast oEager learner • 17 years of experience o~6 Researcher at UCLV (BE, PR & CV) Anesto del Toro Almenares
  3. This talk and Devops…

  4. •CIR-BCC: Scope •Building the solution • Challenges • Architecture/design decisions

    • Dev/IT support services •Conclusions Outline
  5. SCOPE

  6. None
  7. What is the Risk Information Center (CIR)? Free public service

    offered by the Superintendence of the Central Bank of Cuba (BCC). Directed to the National Banking and Financial System and to legal and natural persons who have credit from the country's financial institutions and operate current account.
  8. What is the objective? Provide information to: -financial institutions, in

    order to contribute to the mitigation of their risks -debtors about their credit situation -clients on the measures applied in their accounts for breaches of the banking rules.
  9. Software Delivery System Achieve faster development and deployment cycles of

    quality products, in a reliable and sustainable way Enterprise Agility
  10. Communication matters…

  11. (c4 diagrams)

  12. •What are we building? •Who is using it? (users, actors,

    roles, personas, etc.) •How does it fit to the existing IT environment? Context diagram
  13. Risk Information Center - BCC Supervisor Financial Institution Legal person

    Admin [sign-in user + psw] Users Accesses Apps authorizations Legislations Natural person Credit info Operational info Exposure Level Risk concentration Credit info Operational info Credit info Operational info Credit info Operational info Ext. App 1 … Ext App N [sign-in AppId + token] Analysis requests SIB (legacy system)
  14. SIB (legacy system) Risk Information Center - BCC Supervisor Financial

    Institution Legal person Admin [sign-in user + psw] Users Accesses Apps authorizations Legislations Natural person Credit info Operational info Exposure Level Risk concentration Credit info Operational info Credit info Operational info Credit info Operational info Ext. App 1 … Ext App N [sign-in AppId + token] Analysis requests NTP SMTP Monitoring Register events of interest Email responses Time synch
  15. What are the high-level technology decisions?

  16. Challenges

  17. CONCURRENT ACCESS

  18. DEPLOYMENT FLEXIBILITY

  19. LEGACY SYSTEM INTEGRATION Data lifecycle

  20. Decoupling frontend and backend teams

  21. Frontend WebApp Backend Services • Non blocking / Async •

    Event Sourcing (ES) • Command Query Responsibility Segregation (CQRS) • SPA Stack Clear API contract
  22. STACK SELECTION

  23. Frontend stack

  24. Backend stack

  25. Backend stack • Seamless JAVA interop • Type inference •

    Concurrency and distribution • Traits • Pattern matching • High-order functions • Design patterns
  26. Backend stack • MVC Web development framework • Stateless, asynchronous

    and non- blocking • Deliver all the benefits of RS, including high degree of responsiveness, elasticity and resiliency • Hot reloading, interoperability with DI frameworks, modularity and extensibility,..
  27. Backend stack Toolkit for building highly concurrent, distributed, and resilient

    message-driven applications for Java and Scala • Simpler Concurrent & Distributed Systems • Resilient by Design • High Performance • Elastic & Decentralized • Reactive Streaming Data
  28. Supporting services • CVS • Dependencies repository • Docker repository

    • Continuous Integration Automation scripts • build.sbt • Testing • Code metrics, Check style • gitlab-ci.yml
  29. None
  30. - Reactive stores - Kafka (durable message bus…) - net.cakesolutions.scala-kafka-client

    - net.cakesolutions.scala-kafka-client-akka Storage services - Reactive DB drivers and tools (Scala) - reactive-mongodb - rediscala - play-ws
  31. docker-compose.yml Sandboxing

  32. Modularity & Composability

  33. Modularity

  34. build.sbt mainApp [Play] JWT Auth, Users Management, Credit and Operational

    Analysis Modularity
  35. mainApp/application.conf Modularity

  36. d2AppCOInfo [Play] Detailed Credit and Operational Info d2AppCore [Play] Authentication

    (JWT), User management, Customers, Roles, 3rd party Apps, d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity API Gateway [NGinX ] build.sbt Modularity
  37. .../d2AppRCEL/application.conf .../d2AppCore/application.conf ... Modularity

  38. Messaging System [Kafka] NoSQL Data Store [MongoDB 3.4.x] Cache Data

    Store [Redis 3.2.x] d2AppCOInfo [Play] Detailed Credit and Operational Info d2AppCore [Play] Authentication (JWT), User management, Customers, Roles, 3rd party Apps, d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity API Gateway [NGinX ] Modularity
  39. Messaging System [Kafka] NoSQL Data Store [MongoDB 3.4.x] Cache Data

    Store [Redis 3.2.x] d2AppCOInfo [Play] Detailed Credit and Operational Info d2AppCore [Play] Authentication (JWT), User management, Customers, Roles, 3rd party Apps, d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity API Gateway [NGinX ] NoSQL Data Store [MongoDB 3.4.x] Cache Data Store [Redis 3.2.x] NoSQL Data Store [MongoDB 3.4.x] Cache Data Store [Redis 3.2.x] Modularity
  40. Deploy project

  41. •Modularity / App Composability •Horizontal scalability •Deployment flexibility Different scalability

    levels per App, including data services ad-hoc, incremental Small apps working coordinately (Bounded Contexts) Apps evolving in parallel (different teams) Benefits
  42. The ability to behave correctly under load and to scale

    on demand can not be a last time though
  43. Legacy system integration

  44. SIB (legacy system) Ingestion Aggregation Query/Retrieval Event Journal Snapshots Topics

    subscription Topics publisher Read Model
  45. read model aggr-apps SIB Credit Info AggApp Risk Concentration AggApp

    Exposure Level AggApp M300Event SIB Ingester M301Event M910Event M918Event MCMEvent … M920Event topics web-app mainApp [Play] JWT Auth, Users Management, Credit and Operational Analysis
  46. M300Event SIB Ingester M301Event M910Event M918Event MCMEvent … M920Event SIB

    topics Risk Concentration AggApp Credit Info AggApp Exposure Level AggApp aggr-apps read model web-app d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity d2AppCOInfo [Play] Detailed Credit and Operational Info
  47. - Event Sourcing (ES) ES-CQRS - Command Query Responsibility Segregation

    (CQRS)
  48. "com.hootsuite" %% "akka-persistence-redis" % 0.6.0 "com.github.scullxbones" %% "akka-persistence-mongo-rxmongo" % 1.4.1

    Persistent Actor
  49. SIB (legacy system) Ingestion Aggregation Query/Retrieval Data lifecycle Change Data

    Capture (CDC) to the rescue… Extracting any inserts, updates, and deletes into a stream of data change events
  50. • What is the shape / general appearance of the

    system? • How the main responsibilities are distributed in the system? • How do components communicate? Containers diagram
  51. [https, WS] SIB [External System] Web browser Frontend / Proxy

    [NGinX ] [https] Risk Information Center - BCC
  52. [https, WS] SIB [External System] Web Application [Play, Stateless] JWT

    Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Frontend / Proxy [NGinX ] [https]
  53. [https, WS] SIB [External System] Web Application [Play, Stateless] JWT

    Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Frontend / Proxy [NGinX ] [https]
  54. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES,

    Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Frontend / Proxy [NGinX ] [https]
  55. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES,

    Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Changes Tracking [SQLServer feature] Track in SQL Server changes to records. Frontend / Proxy [NGinX ] [https]
  56. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES,

    Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Changes Tracking [SQLServer feature] Track in SQL Server changes to records. Frontend / Proxy [NGinX ] [https] Schema Registry
  57. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES,

    Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Changes Tracking [SQLServer feature] Track in SQL Server changes to records. Frontend / Proxy [NGinX ] [https] Schema Registry vcMon vcAlert [External System] [External System] Gets metrics from [Http] Metrics Collector [cAdvisor] Gathers metrics from Hosts and Containers Metrics Engine [Prometheus] Stores and serves aggregated metrics and emits alerts
  58. What do we need?

  59. Production Staging Security / NFR Minutes Weeks Each cycle targets

    a different concern Feedback loops UAT Increment deploy Code Reviews Integration tests Acceptance tests Unit Testing
  60. Infrastructure supporting services

  61. Bring Ops tools closer to Developers

  62. Load and Performance tests automation

  63. UAT Deploy Production like Deploy Unit tests Integration tests Code

    & Vulnerabilities Analysis Acceptance tests Load, Smoke, Security tests … Exploratory Production deploy Deployment pipeline Up to 8 deploys/releases* per day
  64. Increment Review Increment Review User Acceptance Testing User Acceptance Testing

    Documentation Documentation Development & Operations Development & Operations Management & Planning Management & Planning Knowledge sharing/retention Knowledge sharing/retention Automation Self-services Automation Self-services Code repository Code repository Platform as a Service Platform as a Service Artifacts and dependencies repositories Artifacts and dependencies repositories Stakeholders Stakeholders Internet Production System
  65. None
  66. None
  67. • Communication/collaboration amplified • Shared mindset/vision • Automation of supporting

    services • Results are not the solely point, but the development of People and Systems that allows us to achieve the results we expect Conclusions
  68. anesto.deltoro@datys.cu @anestodta https://www.linkedin.com/in/anesto-del-toro Anesto del Toro Almenares Experiences on building

    a modular reactive architecture