DevOpsDays Cuba 2017: Experiences on building a modular reactive architecture CIR-BCC

Author: Anesto del Toro
Summary: In this talk we’ll share some experiences during the development of the
CIR-BCC (Risk Information Center - Cuban Central Bank). We briefly describe the problem, the main challenges and their impact in key architectural and design decisions. We present some insights of our reactive modular solution, developed with Scala, Play Framework and Akka, and we illustrate how Event Sourcing (ES), Command-Query Responsibility Segregation (CQRS) and Change Data Capture (CDC) have helped us to solve some important challenges. We expose several
of the practices that have allowed us to achieve faster development
and deployment cycles for this product.

Transcript

1. Experiences on building a modular reactive architecture CIR-BCC
2. About Me @anestodta anesto.deltoro@datys.cu https://www.linkedin.com/in/anesto-del-toro • Head of Production, DATYS VC • MSc Computer Sciences • Wearing multiple hats oTechnical Lead oArchitecture oSoftware Development (mostly backend) oProject Management oBusiness Analysis oLean-Agile enthusiast oEager learner • 17 years of experience o~6 Researcher at UCLV (BE, PR & CV) Anesto del Toro Almenares
3. This talk and Devops…
4. •CIR-BCC: Scope •Building the solution • Challenges • Architecture/design decisions • Dev/IT support services •Conclusions Outline
5. SCOPE
7. What is the Risk Information Center (CIR)? Free public service offered by the Superintendence of the Central Bank of Cuba (BCC). Directed to the National Banking and Financial System and to legal and natural persons who have credit from the country's financial institutions and operate current account.
8. What is the objective? Provide information to: -financial institutions, in order to contribute to the mitigation of their risks -debtors about their credit situation -clients on the measures applied in their accounts for breaches of the banking rules.
9. Software Delivery System Achieve faster development and deployment cycles of quality products, in a reliable and sustainable way Enterprise Agility
10. Communication matters…
11. (c4 diagrams)
12. •What are we building? •Who is using it? (users, actors, roles, personas, etc.) •How does it fit to the existing IT environment? Context diagram
13. Risk Information Center - BCC Supervisor Financial Institution Legal person Admin [sign-in user + psw] Users Accesses Apps authorizations Legislations Natural person Credit info Operational info Exposure Level Risk concentration Credit info Operational info Credit info Operational info Credit info Operational info Ext. App 1 … Ext App N [sign-in AppId + token] Analysis requests SIB (legacy system)
14. SIB (legacy system) Risk Information Center - BCC Supervisor Financial Institution Legal person Admin [sign-in user + psw] Users Accesses Apps authorizations Legislations Natural person Credit info Operational info Exposure Level Risk concentration Credit info Operational info Credit info Operational info Credit info Operational info Ext. App 1 … Ext App N [sign-in AppId + token] Analysis requests NTP SMTP Monitoring Register events of interest Email responses Time synch
15. What are the high-level technology decisions?
16. Challenges
17. CONCURRENT ACCESS
18. DEPLOYMENT FLEXIBILITY
19. LEGACY SYSTEM INTEGRATION Data lifecycle
20. Decoupling frontend and backend teams
21. Frontend WebApp Backend Services • Non blocking / Async • Event Sourcing (ES) • Command Query Responsibility Segregation (CQRS) • SPA Stack Clear API contract
22. STACK SELECTION
23. Frontend stack
24. Backend stack
25. Backend stack • Seamless JAVA interop • Type inference • Concurrency and distribution • Traits • Pattern matching • High-order functions • Design patterns
26. Backend stack • MVC Web development framework • Stateless, asynchronous and non- blocking • Deliver all the benefits of RS, including high degree of responsiveness, elasticity and resiliency • Hot reloading, interoperability with DI frameworks, modularity and extensibility,..
27. Backend stack Toolkit for building highly concurrent, distributed, and resilient message-driven applications for Java and Scala • Simpler Concurrent & Distributed Systems • Resilient by Design • High Performance • Elastic & Decentralized • Reactive Streaming Data
28. Supporting services • CVS • Dependencies repository • Docker repository • Continuous Integration Automation scripts • build.sbt • Testing • Code metrics, Check style • gitlab-ci.yml
30. - Reactive stores - Kafka (durable message bus…) - net.cakesolutions.scala-kafka-client - net.cakesolutions.scala-kafka-client-akka Storage services - Reactive DB drivers and tools (Scala) - reactive-mongodb - rediscala - play-ws
31. docker-compose.yml Sandboxing
32. Modularity & Composability
33. Modularity
34. build.sbt mainApp [Play] JWT Auth, Users Management, Credit and Operational Analysis Modularity
35. mainApp/application.conf Modularity
36. d2AppCOInfo [Play] Detailed Credit and Operational Info d2AppCore [Play] Authentication (JWT), User management, Customers, Roles, 3rd party Apps, d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity API Gateway [NGinX ] build.sbt Modularity
37. .../d2AppRCEL/application.conf .../d2AppCore/application.conf ... Modularity
38. Messaging System [Kafka] NoSQL Data Store [MongoDB 3.4.x] Cache Data Store [Redis 3.2.x] d2AppCOInfo [Play] Detailed Credit and Operational Info d2AppCore [Play] Authentication (JWT), User management, Customers, Roles, 3rd party Apps, d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity API Gateway [NGinX ] Modularity
39. Messaging System [Kafka] NoSQL Data Store [MongoDB 3.4.x] Cache Data Store [Redis 3.2.x] d2AppCOInfo [Play] Detailed Credit and Operational Info d2AppCore [Play] Authentication (JWT), User management, Customers, Roles, 3rd party Apps, d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity API Gateway [NGinX ] NoSQL Data Store [MongoDB 3.4.x] Cache Data Store [Redis 3.2.x] NoSQL Data Store [MongoDB 3.4.x] Cache Data Store [Redis 3.2.x] Modularity
40. Deploy project
41. •Modularity / App Composability •Horizontal scalability •Deployment flexibility Different scalability levels per App, including data services ad-hoc, incremental Small apps working coordinately (Bounded Contexts) Apps evolving in parallel (different teams) Benefits
42. The ability to behave correctly under load and to scale on demand can not be a last time though
43. Legacy system integration
44. SIB (legacy system) Ingestion Aggregation Query/Retrieval Event Journal Snapshots Topics subscription Topics publisher Read Model
45. read model aggr-apps SIB Credit Info AggApp Risk Concentration AggApp Exposure Level AggApp M300Event SIB Ingester M301Event M910Event M918Event MCMEvent … M920Event topics web-app mainApp [Play] JWT Auth, Users Management, Credit and Operational Analysis
46. M300Event SIB Ingester M301Event M910Event M918Event MCMEvent … M920Event SIB topics Risk Concentration AggApp Credit Info AggApp Exposure Level AggApp aggr-apps read model web-app d2AppRCEL [Play] Exposure Level Risk Concentration Gross Loss Stockholders' Equity d2AppCOInfo [Play] Detailed Credit and Operational Info
47. - Event Sourcing (ES) ES-CQRS - Command Query Responsibility Segregation (CQRS)
48. "com.hootsuite" %% "akka-persistence-redis" % 0.6.0 "com.github.scullxbones" %% "akka-persistence-mongo-rxmongo" % 1.4.1 Persistent Actor
49. SIB (legacy system) Ingestion Aggregation Query/Retrieval Data lifecycle Change Data Capture (CDC) to the rescue… Extracting any inserts, updates, and deletes into a stream of data change events
50. • What is the shape / general appearance of the system? • How the main responsibilities are distributed in the system? • How do components communicate? Containers diagram
51. [https, WS] SIB [External System] Web browser Frontend / Proxy [NGinX ] [https] Risk Information Center - BCC
52. [https, WS] SIB [External System] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Frontend / Proxy [NGinX ] [https]
53. [https, WS] SIB [External System] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Frontend / Proxy [NGinX ] [https]
54. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES, Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Frontend / Proxy [NGinX ] [https]
55. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES, Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Changes Tracking [SQLServer feature] Track in SQL Server changes to records. Frontend / Proxy [NGinX ] [https]
56. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES, Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Changes Tracking [SQLServer feature] Track in SQL Server changes to records. Frontend / Proxy [NGinX ] [https] Schema Registry
57. [https, WS] SIB [External System] Content Updater [Standalone, Akka] ES, Updates Models, Customers, Financial Institutions, etc. Messaging System [Kafka] Durable message bus, integration, etc. Publish messages [Avro] Web Application [Play, Stateless] JWT Auth, Management of Users & Legislations, Credit and Operational Analysis Rest Services. UI (SPA) [NGinX] Reads and write messages [Avro] NoSQL Data Store [MongoDB 3.4.x] Store CI preaggregated docs (CQRS) Cache Data Store [Redis 3.2.x] Caches IFs, Entities, NE, CB, CC, pre- computes, etc. API Gateway [NGinX ] Authentication (JWT), Request throttling, Load balancing. Web browser Changes Tracking [SQLServer feature] Track in SQL Server changes to records. Frontend / Proxy [NGinX ] [https] Schema Registry vcMon vcAlert [External System] [External System] Gets metrics from [Http] Metrics Collector [cAdvisor] Gathers metrics from Hosts and Containers Metrics Engine [Prometheus] Stores and serves aggregated metrics and emits alerts
58. What do we need?
59. Production Staging Security / NFR Minutes Weeks Each cycle targets a different concern Feedback loops UAT Increment deploy Code Reviews Integration tests Acceptance tests Unit Testing
60. Infrastructure supporting services
61. Bring Ops tools closer to Developers
62. Load and Performance tests automation
63. UAT Deploy Production like Deploy Unit tests Integration tests Code & Vulnerabilities Analysis Acceptance tests Load, Smoke, Security tests … Exploratory Production deploy Deployment pipeline Up to 8 deploys/releases* per day
64. Increment Review Increment Review User Acceptance Testing User Acceptance Testing Documentation Documentation Development & Operations Development & Operations Management & Planning Management & Planning Knowledge sharing/retention Knowledge sharing/retention Automation Self-services Automation Self-services Code repository Code repository Platform as a Service Platform as a Service Artifacts and dependencies repositories Artifacts and dependencies repositories Stakeholders Stakeholders Internet Production System
67. • Communication/collaboration amplified • Shared mindset/vision • Automation of supporting services • Results are not the solely point, but the development of People and Systems that allows us to achieve the results we expect Conclusions
68. anesto.deltoro@datys.cu @anestodta https://www.linkedin.com/in/anesto-del-toro Anesto del Toro Almenares Experiences on building a modular reactive architecture