Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DevOpsDays Cuba 2017: Experiences on building a modular reactive architecture CIR-BCC

DevOpsDays Cuba
October 25, 2017
350

DevOpsDays Cuba 2017: Experiences on building a modular reactive architecture CIR-BCC

Author: Anesto del Toro
Summary: In this talk we’ll share some experiences during the development of the
CIR-BCC (Risk Information Center - Cuban Central Bank). We briefly describe the problem, the main challenges and their impact in key architectural and design decisions. We present some insights of our reactive modular solution, developed with Scala, Play Framework and Akka, and we illustrate how Event Sourcing (ES), Command-Query Responsibility Segregation (CQRS) and Change Data Capture (CDC) have helped us to solve some important challenges. We expose several
of the practices that have allowed us to achieve faster development
and deployment cycles for this product.

DevOpsDays Cuba

October 25, 2017
Tweet

More Decks by DevOpsDays Cuba

Transcript

  1. About Me @anestodta [email protected] https://www.linkedin.com/in/anesto-del-toro • Head of Production, DATYS

    VC • MSc Computer Sciences • Wearing multiple hats oTechnical Lead oArchitecture oSoftware Development (mostly backend) oProject Management oBusiness Analysis oLean-Agile enthusiast oEager learner • 17 years of experience o~6 Researcher at UCLV (BE, PR & CV) Anesto del Toro Almenares
  2. What is the Risk Information Center (CIR)? Free public service

    offered by the Superintendence of the Central Bank of Cuba (BCC). Directed to the National Banking and Financial System and to legal and natural persons who have credit from the country's financial institutions and operate current account.
  3. What is the objective? Provide information to: -financial institutions, in

    order to contribute to the mitigation of their risks -debtors about their credit situation -clients on the measures applied in their accounts for breaches of the banking rules.
  4. Software Delivery System Achieve faster development and deployment cycles of

    quality products, in a reliable and sustainable way Enterprise Agility
  5. •What are we building? •Who is using it? (users, actors,

    roles, personas, etc.) •How does it fit to the existing IT environment? Context diagram
  6. Risk Information Center - BCC Supervisor Financial Institution Legal person

    Admin [sign-in user + psw] Users Accesses Apps authorizations Legislations Natural person Credit info Operational info Exposure Level Risk concentration Credit info Operational info Credit info Operational info Credit info Operational info Ext. App 1 … Ext App N [sign-in AppId + token] Analysis requests SIB (legacy system)
  7. SIB (legacy system) Risk Information Center - BCC Supervisor Financial

    Institution Legal person Admin [sign-in user + psw] Users Accesses Apps authorizations Legislations Natural person Credit info Operational info Exposure Level Risk concentration Credit info Operational info Credit info Operational info Credit info Operational info Ext. App 1 … Ext App N [sign-in AppId + token] Analysis requests NTP SMTP Monitoring Register events of interest Email responses Time synch
  8. Frontend WebApp Backend Services • Non blocking / Async •

    Event Sourcing (ES) • Command Query Responsibility Segregation (CQRS) • SPA Stack Clear API contract
  9. Backend stack • Seamless JAVA interop • Type inference •

    Concurrency and distribution • Traits • Pattern matching • High-order functions • Design patterns
  10. Backend stack • MVC Web development framework • Stateless, asynchronous

    and non- blocking • Deliver all the benefits of RS, including high degree of responsiveness, elasticity and resiliency • Hot reloading, interoperability with DI frameworks, modularity and extensibility,..
  11. Backend stack Toolkit for building highly concurrent, distributed, and resilient

    message-driven applications for Java and Scala • Simpler Concurrent & Distributed Systems • Resilient by Design • High Performance • Elastic & Decentralized • Reactive Streaming Data
  12. Supporting services • CVS • Dependencies repository • Docker repository

    • Continuous Integration Automation scripts • build.sbt • Testing • Code metrics, Check style • gitlab-ci.yml
  13. - Reactive stores - Kafka (durable message bus…) - net.cakesolutions.scala-kafka-client

    - net.cakesolutions.scala-kafka-client-akka Storage services - Reactive DB drivers and tools (Scala) - reactive-mongodb - rediscala - play-ws
  14. d2AppCOInfo [Play] Detailed Credit and Operational Info d2AppCore [Play] Authentication

    (JWT), User management, Customers, Roles, 3rd party Apps, d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity API Gateway [NGinX ] build.sbt Modularity
  15. Messaging System [Kafka] NoSQL Data Store [MongoDB 3.4.x] Cache Data

    Store [Redis 3.2.x] d2AppCOInfo [Play] Detailed Credit and Operational Info d2AppCore [Play] Authentication (JWT), User management, Customers, Roles, 3rd party Apps, d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity API Gateway [NGinX ] Modularity
  16. Messaging System [Kafka] NoSQL Data Store [MongoDB 3.4.x] Cache Data

    Store [Redis 3.2.x] d2AppCOInfo [Play] Detailed Credit and Operational Info d2AppCore [Play] Authentication (JWT), User management, Customers, Roles, 3rd party Apps, d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity API Gateway [NGinX ] NoSQL Data Store [MongoDB 3.4.x] Cache Data Store [Redis 3.2.x] NoSQL Data Store [MongoDB 3.4.x] Cache Data Store [Redis 3.2.x] Modularity
  17. •Modularity / App Composability •Horizontal scalability •Deployment flexibility Different scalability

    levels per App, including data services ad-hoc, incremental Small apps working coordinately (Bounded Contexts) Apps evolving in parallel (different teams) Benefits
  18. The ability to behave correctly under load and to scale

    on demand can not be a last time though
  19. read model aggr-apps SIB Credit Info AggApp Risk Concentration AggApp

    Exposure Level AggApp M300Event SIB Ingester M301Event M910Event M918Event MCMEvent … M920Event topics web-app mainApp [Play] JWT Auth, Users Management, Credit and Operational Analysis
  20. M300Event SIB Ingester M301Event M910Event M918Event MCMEvent … M920Event SIB

    topics Risk Concentration AggApp Credit Info AggApp Exposure Level AggApp aggr-apps read model web-app d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity d2AppCOInfo [Play] Detailed Credit and Operational Info
  21. SIB (legacy system) Ingestion Aggregation Query/Retrieval Data lifecycle Change Data

    Capture (CDC) to the rescue… Extracting any inserts, updates, and deletes into a stream of data change events
  22. • What is the shape / general appearance of the

    system? • How the main responsibilities are distributed in the system? • How do components communicate? Containers diagram
  23. [https, WS] SIB [External System] Web browser Frontend / Proxy

    [NGinX ] [https] Risk Information Center - BCC
  24. [https, WS] SIB [External System] Web Application [Play, Stateless] JWT

    Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Frontend / Proxy [NGinX ] [https]
  25. [https, WS] SIB [External System] Web Application [Play, Stateless] JWT

    Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Frontend / Proxy [NGinX ] [https]
  26. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES,

    Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Frontend / Proxy [NGinX ] [https]
  27. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES,

    Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Changes Tracking [SQLServer feature] Track in SQL Server changes to records. Frontend / Proxy [NGinX ] [https]
  28. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES,

    Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Changes Tracking [SQLServer feature] Track in SQL Server changes to records. Frontend / Proxy [NGinX ] [https] Schema Registry
  29. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES,

    Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Changes Tracking [SQLServer feature] Track in SQL Server changes to records. Frontend / Proxy [NGinX ] [https] Schema Registry vcMon vcAlert [External System] [External System] Gets metrics from [Http] Metrics Collector [cAdvisor] Gathers metrics from Hosts and Containers Metrics Engine [Prometheus] Stores and serves aggregated metrics and emits alerts
  30. Production Staging Security / NFR Minutes Weeks Each cycle targets

    a different concern Feedback loops UAT Increment deploy Code Reviews Integration tests Acceptance tests Unit Testing
  31. UAT Deploy Production like Deploy Unit tests Integration tests Code

    & Vulnerabilities Analysis Acceptance tests Load, Smoke, Security tests … Exploratory Production deploy Deployment pipeline Up to 8 deploys/releases* per day
  32. Increment Review Increment Review User Acceptance Testing User Acceptance Testing

    Documentation Documentation Development & Operations Development & Operations Management & Planning Management & Planning Knowledge sharing/retention Knowledge sharing/retention Automation Self-services Automation Self-services Code repository Code repository Platform as a Service Platform as a Service Artifacts and dependencies repositories Artifacts and dependencies repositories Stakeholders Stakeholders Internet Production System
  33. • Communication/collaboration amplified • Shared mindset/vision • Automation of supporting

    services • Results are not the solely point, but the development of People and Systems that allows us to achieve the results we expect Conclusions