DevOpsDays Cuba 2017: Experiences on building a modular reactive architecture CIR-BCC

Author: Anesto del Toro
Summary: In this talk we’ll share some experiences during the development of the
CIR-BCC (Risk Information Center - Cuban Central Bank). We briefly describe the problem, the main challenges and their impact in key architectural and design decisions. We present some insights of our reactive modular solution, developed with Scala, Play Framework and Akka, and we illustrate how Event Sourcing (ES), Command-Query Responsibility Segregation (CQRS) and Change Data Capture (CDC) have helped us to solve some important challenges. We expose several
of the practices that have allowed us to achieve faster development
and deployment cycles for this product.

Transcript

1. Experiences on
   building a modular
   reactive architecture
   CIR-BCC
   

   View Slide

2. About Me
   @anestodta
   anesto.deltoro@datys.cu
   https://www.linkedin.com/in/anesto-del-toro
   • Head of Production, DATYS VC
   • MSc Computer Sciences
   • Wearing multiple hats
   oTechnical Lead
   oArchitecture
   oSoftware Development (mostly backend)
   oProject Management
   oBusiness Analysis
   oLean-Agile enthusiast
   oEager learner
   • 17 years of experience
   o~6 Researcher at UCLV (BE, PR & CV)
   Anesto del Toro Almenares
   

   View Slide

3. This talk and Devops…
   

   View Slide

4. •CIR-BCC: Scope
   •Building the solution
   • Challenges
   • Architecture/design decisions
   • Dev/IT support services
   •Conclusions
   Outline
   

   View Slide

5. SCOPE
   

   View Slide

6. View Slide

7. What is the Risk Information Center (CIR)?
   Free public service offered by the Superintendence of the Central
   Bank of Cuba (BCC). Directed to the National Banking and Financial
   System and to legal and natural persons who have credit from the
   country's financial institutions and operate current account.
   

   View Slide

8. What is the objective?
   Provide information to:
   -financial institutions, in order to contribute to the mitigation of
   their risks
   -debtors about their credit situation
   -clients on the measures applied in their accounts for breaches of
   the banking rules.
   

   View Slide

9. Software Delivery System
   Achieve faster development and deployment cycles of
   quality products, in a reliable and sustainable way
   Enterprise Agility
   

   View Slide

10. Communication matters…
    

    View Slide

11. (c4 diagrams)
    

    View Slide

12. •What are we building?
    •Who is using it? (users, actors, roles, personas, etc.)
    •How does it fit to the existing IT environment?
    Context diagram
    

    View Slide

13. Risk
    Information
    Center - BCC
    Supervisor Financial Institution Legal person
    Admin
    [sign-in user + psw]
    Users
    Accesses
    Apps authorizations
    Legislations
    Natural person
    Credit info
    Operational info
    Exposure Level
    Risk concentration
    Credit info
    Operational info
    Credit info
    Operational info
    Credit info
    Operational info
    Ext. App 1
    …
    Ext App N
    [sign-in AppId + token]
    Analysis requests
    SIB
    (legacy system)
    

    View Slide

14. SIB
    (legacy system)
    Risk
    Information
    Center - BCC
    Supervisor Financial Institution Legal person
    Admin
    [sign-in user + psw]
    Users
    Accesses
    Apps authorizations
    Legislations
    Natural person
    Credit info
    Operational info
    Exposure Level
    Risk concentration
    Credit info
    Operational info
    Credit info
    Operational info
    Credit info
    Operational info
    Ext. App 1
    …
    Ext App N
    [sign-in AppId + token]
    Analysis requests
    NTP
    SMTP
    Monitoring
    Register events of
    interest
    Email responses
    Time synch
    

    View Slide

15. What are the high-level technology decisions?
    

    View Slide

16. Challenges
    

    View Slide

17. CONCURRENT ACCESS
    

    View Slide

18. DEPLOYMENT FLEXIBILITY
    

    View Slide

19. LEGACY SYSTEM INTEGRATION
    Data lifecycle
    

    View Slide

20. Decoupling frontend and backend teams
    

    View Slide

21. Frontend
    WebApp
    Backend
    Services
    • Non blocking / Async
    • Event Sourcing (ES)
    • Command Query Responsibility
    Segregation (CQRS)
    • SPA Stack
    Clear API contract
    

    View Slide

22. STACK SELECTION
    

    View Slide

23. Frontend stack
    

    View Slide

24. Backend stack
    

    View Slide

25. Backend stack
    • Seamless JAVA interop
    • Type inference
    • Concurrency and distribution
    • Traits
    • Pattern matching
    • High-order functions
    • Design patterns
    

    View Slide

26. Backend stack
    • MVC Web development framework
    • Stateless, asynchronous and non-
    blocking
    • Deliver all the benefits of RS, including
    high degree of responsiveness,
    elasticity and resiliency
    • Hot reloading, interoperability with DI
    frameworks, modularity and
    extensibility,..
    

    View Slide

27. Backend stack
    Toolkit for building highly concurrent,
    distributed, and resilient message-driven
    applications for Java and Scala
    • Simpler Concurrent & Distributed
    Systems
    • Resilient by Design
    • High Performance
    • Elastic & Decentralized
    • Reactive Streaming Data
    

    View Slide

28. Supporting services
    • CVS
    • Dependencies repository
    • Docker repository
    • Continuous Integration
    Automation scripts
    • build.sbt
    • Testing
    • Code metrics, Check style
    • gitlab-ci.yml
    

    View Slide

29. View Slide

30. - Reactive stores
    - Kafka (durable message bus…)
    - net.cakesolutions.scala-kafka-client
    - net.cakesolutions.scala-kafka-client-akka
    Storage services
    - Reactive DB drivers and tools (Scala)
    - reactive-mongodb
    - rediscala
    - play-ws
    

    View Slide

31. docker-compose.yml
    Sandboxing
    

    View Slide

32. Modularity & Composability
    

    View Slide

33. Modularity
    

    View Slide

34. build.sbt
    mainApp
    [Play]
    JWT Auth, Users
    Management, Credit
    and Operational
    Analysis
    Modularity
    

    View Slide

35. mainApp/application.conf
    Modularity
    

    View Slide

36. d2AppCOInfo
    [Play]
    Detailed Credit and
    Operational Info
    d2AppCore
    [Play]
    Authentication (JWT),
    User management,
    Customers, Roles, 3rd
    party Apps,
    d2AppRCEL
    [Play]
    Exposure Level
    Risk Concentration
    Gross Loss
    Stockholders' Equity
    API Gateway
    [NGinX ]
    build.sbt
    Modularity
    

    View Slide

37. .../d2AppRCEL/application.conf
    .../d2AppCore/application.conf
    ...
    Modularity
    

    View Slide

38. Messaging System
    [Kafka]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Cache Data Store
    [Redis 3.2.x]
    d2AppCOInfo
    [Play]
    Detailed Credit and
    Operational Info
    d2AppCore
    [Play]
    Authentication (JWT),
    User management,
    Customers, Roles, 3rd
    party Apps, d2AppRCEL
    [Play]
    Exposure Level
    Risk Concentration
    Gross Loss
    Stockholders' Equity
    API Gateway
    [NGinX ]
    Modularity
    

    View Slide

39. Messaging System
    [Kafka]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Cache Data Store
    [Redis 3.2.x]
    d2AppCOInfo
    [Play]
    Detailed Credit and
    Operational Info
    d2AppCore
    [Play]
    Authentication (JWT),
    User management,
    Customers, Roles, 3rd
    party Apps, d2AppRCEL
    [Play]
    Exposure Level
    Risk Concentration
    Gross Loss
    Stockholders' Equity
    API Gateway
    [NGinX ]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Cache Data Store
    [Redis 3.2.x]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Cache Data Store
    [Redis 3.2.x]
    Modularity
    

    View Slide

40. Deploy project
    

    View Slide

41. •Modularity / App Composability
    •Horizontal scalability
    •Deployment flexibility
    Different scalability levels per App, including data services
    ad-hoc, incremental
    Small apps working coordinately (Bounded Contexts)
    Apps evolving in parallel (different teams)
    Benefits
    

    View Slide

42. The ability to behave correctly under load and to scale
    on demand can not be a last time though
    

    View Slide

43. Legacy system integration
    

    View Slide

44. SIB
    (legacy system)
    Ingestion Aggregation Query/Retrieval
    Event Journal Snapshots
    Topics
    subscription
    Topics
    publisher
    Read Model
    

    View Slide

45. read model
    aggr-apps
    SIB
    Credit
    Info
    AggApp
    Risk
    Concentration
    AggApp
    Exposure
    Level
    AggApp
    M300Event
    SIB
    Ingester
    M301Event
    M910Event
    M918Event
    MCMEvent
    …
    M920Event
    topics web-app
    mainApp
    [Play]
    JWT Auth, Users
    Management, Credit
    and Operational
    Analysis
    

    View Slide

46. M300Event
    SIB
    Ingester
    M301Event
    M910Event
    M918Event
    MCMEvent
    …
    M920Event
    SIB
    topics
    Risk
    Concentration
    AggApp
    Credit
    Info
    AggApp
    Exposure
    Level
    AggApp
    aggr-apps read model web-app
    d2AppRCEL
    [Play]
    Exposure Level
    Risk Concentration
    Gross Loss
    Stockholders' Equity
    d2AppCOInfo
    [Play]
    Detailed Credit and
    Operational Info
    

    View Slide

47. - Event Sourcing (ES)
    ES-CQRS
    - Command Query Responsibility Segregation (CQRS)
    

    View Slide

48. "com.hootsuite" %% "akka-persistence-redis" % 0.6.0
    "com.github.scullxbones" %% "akka-persistence-mongo-rxmongo" % 1.4.1
    Persistent
    Actor
    

    View Slide

49. SIB
    (legacy system)
    Ingestion Aggregation Query/Retrieval
    Data lifecycle
    Change Data Capture (CDC) to the rescue…
    Extracting any inserts, updates,
    and deletes into a stream of data
    change events
    

    View Slide

50. • What is the shape / general appearance of the system?
    • How the main responsibilities are distributed in the
    system?
    • How do components communicate?
    Containers diagram
    

    View Slide

51. [https, WS]
    SIB
    [External System]
    Web browser
    Frontend
    / Proxy
    [NGinX ]
    [https]
    Risk
    Information
    Center - BCC
    

    View Slide

52. [https, WS]
    SIB
    [External System]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Frontend
    / Proxy
    [NGinX ]
    [https]
    

    View Slide

53. [https, WS]
    SIB
    [External System]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Store CI preaggregated
    docs (CQRS)
    Cache Data Store
    [Redis 3.2.x]
    Caches IFs, Entities,
    NE, CB, CC, pre-
    computes, etc.
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Frontend
    / Proxy
    [NGinX ]
    [https]
    

    View Slide

54. [https, WS]
    SIB
    [External System]
    Content Updater
    [Standalone, Akka]
    ES, Updates Models,
    Customers, Financial
    Institutions, etc.
    Messaging System
    [Kafka]
    Durable message bus,
    integration, etc.
    Publish messages
    [Avro]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    Reads and write
    messages
    [Avro]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Store CI preaggregated
    docs (CQRS)
    Cache Data Store
    [Redis 3.2.x]
    Caches IFs, Entities,
    NE, CB, CC, pre-
    computes, etc.
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Frontend
    / Proxy
    [NGinX ]
    [https]
    

    View Slide

55. [https, WS]
    SIB
    [External System]
    Content Updater
    [Standalone, Akka]
    ES, Updates Models,
    Customers, Financial
    Institutions, etc.
    Messaging System
    [Kafka]
    Durable message bus,
    integration, etc.
    Publish messages
    [Avro]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    Reads and write
    messages
    [Avro]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Store CI preaggregated
    docs (CQRS)
    Cache Data Store
    [Redis 3.2.x]
    Caches IFs, Entities,
    NE, CB, CC, pre-
    computes, etc.
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Changes Tracking
    [SQLServer feature]
    Track in SQL Server
    changes to records.
    Frontend
    / Proxy
    [NGinX ]
    [https]
    

    View Slide

56. [https, WS]
    SIB
    [External System]
    Content Updater
    [Standalone, Akka]
    ES, Updates Models,
    Customers, Financial
    Institutions, etc.
    Messaging System
    [Kafka]
    Durable message bus,
    integration, etc.
    Publish messages
    [Avro]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    Reads and write
    messages
    [Avro]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Store CI preaggregated
    docs (CQRS)
    Cache Data Store
    [Redis 3.2.x]
    Caches IFs, Entities,
    NE, CB, CC, pre-
    computes, etc.
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Changes Tracking
    [SQLServer feature]
    Track in SQL Server
    changes to records.
    Frontend
    / Proxy
    [NGinX ]
    [https]
    Schema
    Registry
    

    View Slide

57. [https, WS]
    SIB
    [External System]
    Content Updater
    [Standalone, Akka]
    ES, Updates Models,
    Customers, Financial
    Institutions, etc.
    Messaging System
    [Kafka]
    Durable message bus,
    integration, etc.
    Publish messages
    [Avro]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    Reads and write
    messages
    [Avro]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Store CI preaggregated
    docs (CQRS)
    Cache Data Store
    [Redis 3.2.x]
    Caches IFs, Entities,
    NE, CB, CC, pre-
    computes, etc.
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Changes Tracking
    [SQLServer feature]
    Track in SQL Server
    changes to records.
    Frontend
    / Proxy
    [NGinX ]
    [https]
    Schema
    Registry
    vcMon
    vcAlert
    [External System]
    [External System]
    Gets metrics from
    [Http]
    Metrics Collector
    [cAdvisor]
    Gathers metrics from
    Hosts and Containers
    Metrics Engine
    [Prometheus]
    Stores and serves
    aggregated metrics
    and emits alerts
    

    View Slide

58. What do we need?
    

    View Slide

59. Production
    Staging
    Security / NFR
    Minutes
    Weeks
    Each cycle targets
    a different concern
    Feedback loops
    UAT
    Increment
    deploy
    Code Reviews
    Integration tests
    Acceptance
    tests
    Unit
    Testing
    

    View Slide

60. Infrastructure supporting services
    

    View Slide

61. Bring Ops tools
    closer to
    Developers
    

    View Slide

62. Load and
    Performance tests
    automation
    

    View Slide

63. UAT
    Deploy
    Production
    like Deploy
    Unit
    tests
    Integration
    tests
    Code &
    Vulnerabilities
    Analysis
    Acceptance
    tests Load,
    Smoke,
    Security
    tests
    …
    Exploratory
    Production
    deploy
    Deployment pipeline
    Up to 8 deploys/releases* per day
    

    View Slide

64. Increment
    Review
    Increment
    Review
    User Acceptance
    Testing
    User Acceptance
    Testing Documentation
    Documentation
    Development &
    Operations
    Development &
    Operations
    Management &
    Planning
    Management &
    Planning
    Knowledge sharing/retention
    Knowledge sharing/retention
    Automation
    Self-services
    Automation
    Self-services
    Code
    repository
    Code
    repository
    Platform as a
    Service
    Platform as a
    Service
    Artifacts and dependencies repositories
    Artifacts and dependencies repositories
    Stakeholders
    Stakeholders
    Internet
    Production System
    

    View Slide

65. View Slide

66. View Slide

67. • Communication/collaboration amplified
    • Shared mindset/vision
    • Automation of supporting services
    • Results are not the solely point, but the development
    of People and Systems that allows us to achieve the
    results we expect
    Conclusions
    

    View Slide

68. anesto.deltoro@datys.cu @anestodta
    https://www.linkedin.com/in/anesto-del-toro
    Anesto
    del Toro Almenares
    Experiences on
    building a modular
    reactive architecture
    

    View Slide