$30 off During Our Annual Pro Sale. View Details »

DevOpsDays Cuba 2017: Experiences on building a modular reactive architecture CIR-BCC

DevOpsDays Cuba
October 25, 2017
300

DevOpsDays Cuba 2017: Experiences on building a modular reactive architecture CIR-BCC

Author: Anesto del Toro
Summary: In this talk we’ll share some experiences during the development of the
CIR-BCC (Risk Information Center - Cuban Central Bank). We briefly describe the problem, the main challenges and their impact in key architectural and design decisions. We present some insights of our reactive modular solution, developed with Scala, Play Framework and Akka, and we illustrate how Event Sourcing (ES), Command-Query Responsibility Segregation (CQRS) and Change Data Capture (CDC) have helped us to solve some important challenges. We expose several
of the practices that have allowed us to achieve faster development
and deployment cycles for this product.

DevOpsDays Cuba

October 25, 2017
Tweet

More Decks by DevOpsDays Cuba

Transcript

  1. Experiences on
    building a modular
    reactive architecture
    CIR-BCC

    View Slide

  2. About Me
    @anestodta
    [email protected]
    https://www.linkedin.com/in/anesto-del-toro
    • Head of Production, DATYS VC
    • MSc Computer Sciences
    • Wearing multiple hats
    oTechnical Lead
    oArchitecture
    oSoftware Development (mostly backend)
    oProject Management
    oBusiness Analysis
    oLean-Agile enthusiast
    oEager learner
    • 17 years of experience
    o~6 Researcher at UCLV (BE, PR & CV)
    Anesto del Toro Almenares

    View Slide

  3. This talk and Devops…

    View Slide

  4. •CIR-BCC: Scope
    •Building the solution
    • Challenges
    • Architecture/design decisions
    • Dev/IT support services
    •Conclusions
    Outline

    View Slide

  5. SCOPE

    View Slide

  6. View Slide

  7. What is the Risk Information Center (CIR)?
    Free public service offered by the Superintendence of the Central
    Bank of Cuba (BCC). Directed to the National Banking and Financial
    System and to legal and natural persons who have credit from the
    country's financial institutions and operate current account.

    View Slide

  8. What is the objective?
    Provide information to:
    -financial institutions, in order to contribute to the mitigation of
    their risks
    -debtors about their credit situation
    -clients on the measures applied in their accounts for breaches of
    the banking rules.

    View Slide

  9. Software Delivery System
    Achieve faster development and deployment cycles of
    quality products, in a reliable and sustainable way
    Enterprise Agility

    View Slide

  10. Communication matters…

    View Slide

  11. (c4 diagrams)

    View Slide

  12. •What are we building?
    •Who is using it? (users, actors, roles, personas, etc.)
    •How does it fit to the existing IT environment?
    Context diagram

    View Slide

  13. Risk
    Information
    Center - BCC
    Supervisor Financial Institution Legal person
    Admin
    [sign-in user + psw]
    Users
    Accesses
    Apps authorizations
    Legislations
    Natural person
    Credit info
    Operational info
    Exposure Level
    Risk concentration
    Credit info
    Operational info
    Credit info
    Operational info
    Credit info
    Operational info
    Ext. App 1

    Ext App N
    [sign-in AppId + token]
    Analysis requests
    SIB
    (legacy system)

    View Slide

  14. SIB
    (legacy system)
    Risk
    Information
    Center - BCC
    Supervisor Financial Institution Legal person
    Admin
    [sign-in user + psw]
    Users
    Accesses
    Apps authorizations
    Legislations
    Natural person
    Credit info
    Operational info
    Exposure Level
    Risk concentration
    Credit info
    Operational info
    Credit info
    Operational info
    Credit info
    Operational info
    Ext. App 1

    Ext App N
    [sign-in AppId + token]
    Analysis requests
    NTP
    SMTP
    Monitoring
    Register events of
    interest
    Email responses
    Time synch

    View Slide

  15. What are the high-level technology decisions?

    View Slide

  16. Challenges

    View Slide

  17. CONCURRENT ACCESS

    View Slide

  18. DEPLOYMENT FLEXIBILITY

    View Slide

  19. LEGACY SYSTEM INTEGRATION
    Data lifecycle

    View Slide

  20. Decoupling frontend and backend teams

    View Slide

  21. Frontend
    WebApp
    Backend
    Services
    • Non blocking / Async
    • Event Sourcing (ES)
    • Command Query Responsibility
    Segregation (CQRS)
    • SPA Stack
    Clear API contract

    View Slide

  22. STACK SELECTION

    View Slide

  23. Frontend stack

    View Slide

  24. Backend stack

    View Slide

  25. Backend stack
    • Seamless JAVA interop
    • Type inference
    • Concurrency and distribution
    • Traits
    • Pattern matching
    • High-order functions
    • Design patterns

    View Slide

  26. Backend stack
    • MVC Web development framework
    • Stateless, asynchronous and non-
    blocking
    • Deliver all the benefits of RS, including
    high degree of responsiveness,
    elasticity and resiliency
    • Hot reloading, interoperability with DI
    frameworks, modularity and
    extensibility,..

    View Slide

  27. Backend stack
    Toolkit for building highly concurrent,
    distributed, and resilient message-driven
    applications for Java and Scala
    • Simpler Concurrent & Distributed
    Systems
    • Resilient by Design
    • High Performance
    • Elastic & Decentralized
    • Reactive Streaming Data

    View Slide

  28. Supporting services
    • CVS
    • Dependencies repository
    • Docker repository
    • Continuous Integration
    Automation scripts
    • build.sbt
    • Testing
    • Code metrics, Check style
    • gitlab-ci.yml

    View Slide

  29. View Slide

  30. - Reactive stores
    - Kafka (durable message bus…)
    - net.cakesolutions.scala-kafka-client
    - net.cakesolutions.scala-kafka-client-akka
    Storage services
    - Reactive DB drivers and tools (Scala)
    - reactive-mongodb
    - rediscala
    - play-ws

    View Slide

  31. docker-compose.yml
    Sandboxing

    View Slide

  32. Modularity & Composability

    View Slide

  33. Modularity

    View Slide

  34. build.sbt
    mainApp
    [Play]
    JWT Auth, Users
    Management, Credit
    and Operational
    Analysis
    Modularity

    View Slide

  35. mainApp/application.conf
    Modularity

    View Slide

  36. d2AppCOInfo
    [Play]
    Detailed Credit and
    Operational Info
    d2AppCore
    [Play]
    Authentication (JWT),
    User management,
    Customers, Roles, 3rd
    party Apps,
    d2AppRCEL
    [Play]
    Exposure Level
    Risk Concentration
    Gross Loss
    Stockholders' Equity
    API Gateway
    [NGinX ]
    build.sbt
    Modularity

    View Slide

  37. .../d2AppRCEL/application.conf
    .../d2AppCore/application.conf
    ...
    Modularity

    View Slide

  38. Messaging System
    [Kafka]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Cache Data Store
    [Redis 3.2.x]
    d2AppCOInfo
    [Play]
    Detailed Credit and
    Operational Info
    d2AppCore
    [Play]
    Authentication (JWT),
    User management,
    Customers, Roles, 3rd
    party Apps, d2AppRCEL
    [Play]
    Exposure Level
    Risk Concentration
    Gross Loss
    Stockholders' Equity
    API Gateway
    [NGinX ]
    Modularity

    View Slide

  39. Messaging System
    [Kafka]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Cache Data Store
    [Redis 3.2.x]
    d2AppCOInfo
    [Play]
    Detailed Credit and
    Operational Info
    d2AppCore
    [Play]
    Authentication (JWT),
    User management,
    Customers, Roles, 3rd
    party Apps, d2AppRCEL
    [Play]
    Exposure Level
    Risk Concentration
    Gross Loss
    Stockholders' Equity
    API Gateway
    [NGinX ]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Cache Data Store
    [Redis 3.2.x]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Cache Data Store
    [Redis 3.2.x]
    Modularity

    View Slide

  40. Deploy project

    View Slide

  41. •Modularity / App Composability
    •Horizontal scalability
    •Deployment flexibility
    Different scalability levels per App, including data services
    ad-hoc, incremental
    Small apps working coordinately (Bounded Contexts)
    Apps evolving in parallel (different teams)
    Benefits

    View Slide

  42. The ability to behave correctly under load and to scale
    on demand can not be a last time though

    View Slide

  43. Legacy system integration

    View Slide

  44. SIB
    (legacy system)
    Ingestion Aggregation Query/Retrieval
    Event Journal Snapshots
    Topics
    subscription
    Topics
    publisher
    Read Model

    View Slide

  45. read model
    aggr-apps
    SIB
    Credit
    Info
    AggApp
    Risk
    Concentration
    AggApp
    Exposure
    Level
    AggApp
    M300Event
    SIB
    Ingester
    M301Event
    M910Event
    M918Event
    MCMEvent

    M920Event
    topics web-app
    mainApp
    [Play]
    JWT Auth, Users
    Management, Credit
    and Operational
    Analysis

    View Slide

  46. M300Event
    SIB
    Ingester
    M301Event
    M910Event
    M918Event
    MCMEvent

    M920Event
    SIB
    topics
    Risk
    Concentration
    AggApp
    Credit
    Info
    AggApp
    Exposure
    Level
    AggApp
    aggr-apps read model web-app
    d2AppRCEL
    [Play]
    Exposure Level
    Risk Concentration
    Gross Loss
    Stockholders' Equity
    d2AppCOInfo
    [Play]
    Detailed Credit and
    Operational Info

    View Slide

  47. - Event Sourcing (ES)
    ES-CQRS
    - Command Query Responsibility Segregation (CQRS)

    View Slide

  48. "com.hootsuite" %% "akka-persistence-redis" % 0.6.0
    "com.github.scullxbones" %% "akka-persistence-mongo-rxmongo" % 1.4.1
    Persistent
    Actor

    View Slide

  49. SIB
    (legacy system)
    Ingestion Aggregation Query/Retrieval
    Data lifecycle
    Change Data Capture (CDC) to the rescue…
    Extracting any inserts, updates,
    and deletes into a stream of data
    change events

    View Slide

  50. • What is the shape / general appearance of the system?
    • How the main responsibilities are distributed in the
    system?
    • How do components communicate?
    Containers diagram

    View Slide

  51. [https, WS]
    SIB
    [External System]
    Web browser
    Frontend
    / Proxy
    [NGinX ]
    [https]
    Risk
    Information
    Center - BCC

    View Slide

  52. [https, WS]
    SIB
    [External System]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Frontend
    / Proxy
    [NGinX ]
    [https]

    View Slide

  53. [https, WS]
    SIB
    [External System]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Store CI preaggregated
    docs (CQRS)
    Cache Data Store
    [Redis 3.2.x]
    Caches IFs, Entities,
    NE, CB, CC, pre-
    computes, etc.
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Frontend
    / Proxy
    [NGinX ]
    [https]

    View Slide

  54. [https, WS]
    SIB
    [External System]
    Content Updater
    [Standalone, Akka]
    ES, Updates Models,
    Customers, Financial
    Institutions, etc.
    Messaging System
    [Kafka]
    Durable message bus,
    integration, etc.
    Publish messages
    [Avro]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    Reads and write
    messages
    [Avro]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Store CI preaggregated
    docs (CQRS)
    Cache Data Store
    [Redis 3.2.x]
    Caches IFs, Entities,
    NE, CB, CC, pre-
    computes, etc.
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Frontend
    / Proxy
    [NGinX ]
    [https]

    View Slide

  55. [https, WS]
    SIB
    [External System]
    Content Updater
    [Standalone, Akka]
    ES, Updates Models,
    Customers, Financial
    Institutions, etc.
    Messaging System
    [Kafka]
    Durable message bus,
    integration, etc.
    Publish messages
    [Avro]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    Reads and write
    messages
    [Avro]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Store CI preaggregated
    docs (CQRS)
    Cache Data Store
    [Redis 3.2.x]
    Caches IFs, Entities,
    NE, CB, CC, pre-
    computes, etc.
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Changes Tracking
    [SQLServer feature]
    Track in SQL Server
    changes to records.
    Frontend
    / Proxy
    [NGinX ]
    [https]

    View Slide

  56. [https, WS]
    SIB
    [External System]
    Content Updater
    [Standalone, Akka]
    ES, Updates Models,
    Customers, Financial
    Institutions, etc.
    Messaging System
    [Kafka]
    Durable message bus,
    integration, etc.
    Publish messages
    [Avro]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    Reads and write
    messages
    [Avro]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Store CI preaggregated
    docs (CQRS)
    Cache Data Store
    [Redis 3.2.x]
    Caches IFs, Entities,
    NE, CB, CC, pre-
    computes, etc.
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Changes Tracking
    [SQLServer feature]
    Track in SQL Server
    changes to records.
    Frontend
    / Proxy
    [NGinX ]
    [https]
    Schema
    Registry

    View Slide

  57. [https, WS]
    SIB
    [External System]
    Content Updater
    [Standalone, Akka]
    ES, Updates Models,
    Customers, Financial
    Institutions, etc.
    Messaging System
    [Kafka]
    Durable message bus,
    integration, etc.
    Publish messages
    [Avro]
    Web Application
    [Play, Stateless]
    JWT Auth, Management
    of Users & Legislations,
    Credit and Operational
    Analysis Rest Services.
    UI (SPA)
    [NGinX]
    Reads and write
    messages
    [Avro]
    NoSQL Data Store
    [MongoDB 3.4.x]
    Store CI preaggregated
    docs (CQRS)
    Cache Data Store
    [Redis 3.2.x]
    Caches IFs, Entities,
    NE, CB, CC, pre-
    computes, etc.
    API Gateway
    [NGinX ]
    Authentication (JWT),
    Request throttling,
    Load balancing.
    Web browser
    Changes Tracking
    [SQLServer feature]
    Track in SQL Server
    changes to records.
    Frontend
    / Proxy
    [NGinX ]
    [https]
    Schema
    Registry
    vcMon
    vcAlert
    [External System]
    [External System]
    Gets metrics from
    [Http]
    Metrics Collector
    [cAdvisor]
    Gathers metrics from
    Hosts and Containers
    Metrics Engine
    [Prometheus]
    Stores and serves
    aggregated metrics
    and emits alerts

    View Slide

  58. What do we need?

    View Slide

  59. Production
    Staging
    Security / NFR
    Minutes
    Weeks
    Each cycle targets
    a different concern
    Feedback loops
    UAT
    Increment
    deploy
    Code Reviews
    Integration tests
    Acceptance
    tests
    Unit
    Testing

    View Slide

  60. Infrastructure supporting services

    View Slide

  61. Bring Ops tools
    closer to
    Developers

    View Slide

  62. Load and
    Performance tests
    automation

    View Slide

  63. UAT
    Deploy
    Production
    like Deploy
    Unit
    tests
    Integration
    tests
    Code &
    Vulnerabilities
    Analysis
    Acceptance
    tests Load,
    Smoke,
    Security
    tests

    Exploratory
    Production
    deploy
    Deployment pipeline
    Up to 8 deploys/releases* per day

    View Slide

  64. Increment
    Review
    Increment
    Review
    User Acceptance
    Testing
    User Acceptance
    Testing Documentation
    Documentation
    Development &
    Operations
    Development &
    Operations
    Management &
    Planning
    Management &
    Planning
    Knowledge sharing/retention
    Knowledge sharing/retention
    Automation
    Self-services
    Automation
    Self-services
    Code
    repository
    Code
    repository
    Platform as a
    Service
    Platform as a
    Service
    Artifacts and dependencies repositories
    Artifacts and dependencies repositories
    Stakeholders
    Stakeholders
    Internet
    Production System

    View Slide

  65. View Slide

  66. View Slide

  67. • Communication/collaboration amplified
    • Shared mindset/vision
    • Automation of supporting services
    • Results are not the solely point, but the development
    of People and Systems that allows us to achieve the
    results we expect
    Conclusions

    View Slide

  68. [email protected] @anestodta
    https://www.linkedin.com/in/anesto-del-toro
    Anesto
    del Toro Almenares
    Experiences on
    building a modular
    reactive architecture

    View Slide