Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017

Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017

2gether is a financial platform based on Blockchain, Big Data and Artificial Intelligence that allows interaction between users and third-party services in a single interface.

https://www.bigdataspain.org/2017/talk/scaling-a-backend-for-a-big-data-and-blockchain-environment

Big Data Spain 2017
November 16th - 17th Kinépolis Madrid

Big Data Spain

December 05, 2017
Tweet

More Decks by Big Data Spain

Other Decks in Technology

Transcript

  1. Scaling a backend for big data
    and blockchain environment

    View full-size slide

  2. 1. Introduction (The companies, project and me)
    2. Backend challenge
    3. Big Data Integration
    4. Blockchain Integration
    ·············· P. 3
    ···················································
    ················ P. 7
    ···················································
    ·············· P. 17
    ···················································
    ········ P. 21

    View full-size slide

  3. @ganchix
    Head of Blockchain / Backend
    Me
    ganchix

    View full-size slide

  4. The companies

    View full-size slide

  5. Fecha
    The project
    • Financial platform
    • Marketplace
    • Tokenization
    • Operate with cryptocurrency
    • Liquidity predictive models
    • Credit scoring
    • Product recommendation

    View full-size slide

  6. Backend challenge - The evolution
    Dapp
    Smart Contract
    Ethereum

    View full-size slide

  7. Backend challenge - The evolution

    View full-size slide

  8. Backend challenge - The evolution
    Dapps problem:
    • Noncryptocurrency users
    • Problem with some integrations
    • Legal problem

    View full-size slide

  9. Backend challenge - The evolution

    View full-size slide

  10. Backend challenge - The evolution

    View full-size slide

  11. Fecha
    Backend challenge - Why microservices?
    • Migration of Dapp easier
    • Easy to scale
    • Polyglot Database and Languages

    View full-size slide

  12. Why don't use exclusively blockchain
    with a database?

    View full-size slide

  13. 1. Spring Cloud
    Netflix and Kubernetes
    • Easy to learn.
    • Nice integrations
    • Spring 5 reactive
    2. Docker
    • Most adopted vendor technology for containers
    • Well supported
    3. Kubernetes
    • Multi-cloud provider and on-premises data centers
    • Self-repair and health check capabilities
    • Auto-scale
    Backend challenge - Microservice Architecture Stack

    View full-size slide

  14. Backend challenge - Microservice Architecture Stack

    View full-size slide

  15. Backend challenge - Deployment

    View full-size slide

  16. PFM values generation from user data.
    Apache Spark + Cassandra
    Forecast prediction and regeneration of this models
    Apache Spark + Cassandra
    Product recommendations based on the economic profile of the user and his real needs.
    Apache Spark + Cassandra + Neo4j
    Credit scoring calculation
    Apache Spark + Cassandra
    Big Data Integration - Tasks

    View full-size slide

  17. • Tasks are hard, needs:
    • Time
    • Resources
    • Not Real Time is needed.
    • Event Driven Architecture.
    Big Data Integration - Events

    View full-size slide

  18. Big Data Integration - How?

    View full-size slide

  19. Big Data Integration - RabbitMQ vs Kafka
    Kafka
    RabbitMq
    • RabbitMQ is designed as a general
    purpose message broker
    • Support existing protocols like
    AMQP, STOMP, MQTT.
    • Finer-grained consistency
    control/guarantees on a peer-
    message.
    • Complex routing.
    • Apache Kafka is designed for high
    volume publish-subscribe
    messages and streams, meant to
    be durable, fast, and scalable.
    • Event Sourcing
    • Your application needs access to
    stream history.
    • No complex routing.
    https://content.pivotal.io/blog/understanding-when-to-use-rabbitmq-or-apache-kafka

    View full-size slide

  20. • Deployed in Kubernetes.
    • Only accessible by NodeJS API.
    • All keys are stored in secrets vaults.
    • Used for:
    • Tokenization
    • Transactions of users
    Blockchain Integration - Private Ethereum

    View full-size slide

  21. Blockchain Integration - Private Ethereum

    View full-size slide

  22. Blockchain Integration - Ethereum Main Net

    View full-size slide

  23. Blockchain Integration - Ethereum Main Net

    View full-size slide

  24. Blockchain Integration - Ethereum Main Net
    • We are the owner of the wallets
    • We use Infura to connect blockchain
    • Used for:
    • Payment
    • Transfers

    View full-size slide

  25. @ganchix
    @2getherbank
    @bigeeksoftwa
    re
    Thanks!
    ICO is coming!

    View full-size slide