Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017

2gether is a financial platform based on Blockchain, Big Data and Artificial Intelligence that allows interaction between users and third-party services in a single interface.

https://www.bigdataspain.org/2017/talk/scaling-a-backend-for-a-big-data-and-blockchain-environment

Big Data Spain 2017
November 16th - 17th Kinépolis Madrid

Transcript

1. View Slide

2. Scaling a backend for big data
   and blockchain environment
   

   View Slide

3. 1. Introduction (The companies, project and me)
   2. Backend challenge
   3. Big Data Integration
   4. Blockchain Integration
   ·············· P. 3
   ···················································
   ················ P. 7
   ···················································
   ·············· P. 17
   ···················································
   ········ P. 21
   

   View Slide

4. @ganchix
   Head of Blockchain / Backend
   Me
   ganchix
   

   View Slide

5. The companies
   

   View Slide

6. View Slide

7. Fecha
   The project
   • Financial platform
   • Marketplace
   • Tokenization
   • Operate with cryptocurrency
   • Liquidity predictive models
   • Credit scoring
   • Product recommendation
   

   View Slide

8. Backend challenge - The evolution
   Dapp
   Smart Contract
   Ethereum
   

   View Slide

9. Backend challenge - The evolution
   

   View Slide

10. Backend challenge - The evolution
    Dapps problem:
    • Noncryptocurrency users
    • Problem with some integrations
    • Legal problem
    

    View Slide

11. Backend challenge - The evolution
    

    View Slide

12. Backend challenge - The evolution
    

    View Slide

13. Fecha
    Backend challenge - Why microservices?
    • Migration of Dapp easier
    • Easy to scale
    • Polyglot Database and Languages
    

    View Slide

14. Why don't use exclusively blockchain
    with a database?
    

    View Slide

15. 1. Spring Cloud
    Netflix and Kubernetes
    • Easy to learn.
    • Nice integrations
    • Spring 5 reactive
    2. Docker
    • Most adopted vendor technology for containers
    • Well supported
    3. Kubernetes
    • Multi-cloud provider and on-premises data centers
    • Self-repair and health check capabilities
    • Auto-scale
    Backend challenge - Microservice Architecture Stack
    

    View Slide

16. Backend challenge - Microservice Architecture Stack
    

    View Slide

17. Backend challenge - Deployment
    

    View Slide

18. PFM values generation from user data.
    Apache Spark + Cassandra
    Forecast prediction and regeneration of this models
    Apache Spark + Cassandra
    Product recommendations based on the economic profile of the user and his real needs.
    Apache Spark + Cassandra + Neo4j
    Credit scoring calculation
    Apache Spark + Cassandra
    Big Data Integration - Tasks
    

    View Slide

19. • Tasks are hard, needs:
    • Time
    • Resources
    • Not Real Time is needed.
    • Event Driven Architecture.
    Big Data Integration - Events
    

    View Slide

20. Big Data Integration - How?
    

    View Slide

21. Big Data Integration - RabbitMQ vs Kafka
    Kafka
    RabbitMq
    • RabbitMQ is designed as a general
    purpose message broker
    • Support existing protocols like
    AMQP, STOMP, MQTT.
    • Finer-grained consistency
    control/guarantees on a peer-
    message.
    • Complex routing.
    • Apache Kafka is designed for high
    volume publish-subscribe
    messages and streams, meant to
    be durable, fast, and scalable.
    • Event Sourcing
    • Your application needs access to
    stream history.
    • No complex routing.
    https://content.pivotal.io/blog/understanding-when-to-use-rabbitmq-or-apache-kafka
    

    View Slide

22. • Deployed in Kubernetes.
    • Only accessible by NodeJS API.
    • All keys are stored in secrets vaults.
    • Used for:
    • Tokenization
    • Transactions of users
    Blockchain Integration - Private Ethereum
    

    View Slide

23. Blockchain Integration - Private Ethereum
    

    View Slide

24. Blockchain Integration - Ethereum Main Net
    

    View Slide

25. Blockchain Integration - Ethereum Main Net
    

    View Slide

26. Blockchain Integration - Ethereum Main Net
    • We are the owner of the wallets
    • We use Infura to connect blockchain
    • Used for:
    • Payment
    • Transfers
    

    View Slide

27. @ganchix
    @2getherbank
    @bigeeksoftwa
    re
    Thanks!
    ICO is coming!
    

    View Slide