$30 off During Our Annual Pro Sale. View Details »

Scaling a backend for a big data and blockchain...

Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017

2gether is a financial platform based on Blockchain, Big Data and Artificial Intelligence that allows interaction between users and third-party services in a single interface.

https://www.bigdataspain.org/2017/talk/scaling-a-backend-for-a-big-data-and-blockchain-environment

Big Data Spain 2017
November 16th - 17th Kinépolis Madrid

Big Data Spain

December 05, 2017
Tweet

More Decks by Big Data Spain

Other Decks in Technology

Transcript

  1. 1. Introduction (The companies, project and me) 2. Backend challenge

    3. Big Data Integration 4. Blockchain Integration ·············· P. 3 ··················································· ················ P. 7 ··················································· ·············· P. 17 ··················································· ········ P. 21
  2. Fecha The project • Financial platform • Marketplace • Tokenization

    • Operate with cryptocurrency • Liquidity predictive models • Credit scoring • Product recommendation
  3. Backend challenge - The evolution Dapps problem: • Noncryptocurrency users

    • Problem with some integrations • Legal problem
  4. Fecha Backend challenge - Why microservices? • Migration of Dapp

    easier • Easy to scale • Polyglot Database and Languages
  5. 1. Spring Cloud Netflix and Kubernetes • Easy to learn.

    • Nice integrations • Spring 5 reactive 2. Docker • Most adopted vendor technology for containers • Well supported 3. Kubernetes • Multi-cloud provider and on-premises data centers • Self-repair and health check capabilities • Auto-scale Backend challenge - Microservice Architecture Stack
  6. PFM values generation from user data. Apache Spark + Cassandra

    Forecast prediction and regeneration of this models Apache Spark + Cassandra Product recommendations based on the economic profile of the user and his real needs. Apache Spark + Cassandra + Neo4j Credit scoring calculation Apache Spark + Cassandra Big Data Integration - Tasks
  7. • Tasks are hard, needs: • Time • Resources •

    Not Real Time is needed. • Event Driven Architecture. Big Data Integration - Events
  8. Big Data Integration - RabbitMQ vs Kafka Kafka RabbitMq •

    RabbitMQ is designed as a general purpose message broker • Support existing protocols like AMQP, STOMP, MQTT. • Finer-grained consistency control/guarantees on a peer- message. • Complex routing. • Apache Kafka is designed for high volume publish-subscribe messages and streams, meant to be durable, fast, and scalable. • Event Sourcing • Your application needs access to stream history. • No complex routing. https://content.pivotal.io/blog/understanding-when-to-use-rabbitmq-or-apache-kafka
  9. • Deployed in Kubernetes. • Only accessible by NodeJS API.

    • All keys are stored in secrets vaults. • Used for: • Tokenization • Transactions of users Blockchain Integration - Private Ethereum
  10. Blockchain Integration - Ethereum Main Net • We are the

    owner of the wallets • We use Infura to connect blockchain • Used for: • Payment • Transfers