Scaling a backend for a big data and blockchain environment by Rafael Ríos at Big Data Spain 2017

2gether is a financial platform based on Blockchain, Big Data and Artificial Intelligence that allows interaction between users and third-party services in a single interface.

https://www.bigdataspain.org/2017/talk/scaling-a-backend-for-a-big-data-and-blockchain-environment

Big Data Spain 2017
November 16th - 17th Kinépolis Madrid

Transcript

2. Scaling a backend for big data and blockchain environment
3. 1. Introduction (The companies, project and me) 2. Backend challenge 3. Big Data Integration 4. Blockchain Integration ·············· P. 3 ··················································· ················ P. 7 ··················································· ·············· P. 17 ··················································· ········ P. 21
4. @ganchix Head of Blockchain / Backend Me ganchix
5. The companies
7. Fecha The project • Financial platform • Marketplace • Tokenization • Operate with cryptocurrency • Liquidity predictive models • Credit scoring • Product recommendation
8. Backend challenge - The evolution Dapp Smart Contract Ethereum
9. Backend challenge - The evolution
10. Backend challenge - The evolution Dapps problem: • Noncryptocurrency users • Problem with some integrations • Legal problem
11. Backend challenge - The evolution
12. Backend challenge - The evolution
13. Fecha Backend challenge - Why microservices? • Migration of Dapp easier • Easy to scale • Polyglot Database and Languages
14. Why don't use exclusively blockchain with a database?
15. 1. Spring Cloud Netflix and Kubernetes • Easy to learn. • Nice integrations • Spring 5 reactive 2. Docker • Most adopted vendor technology for containers • Well supported 3. Kubernetes • Multi-cloud provider and on-premises data centers • Self-repair and health check capabilities • Auto-scale Backend challenge - Microservice Architecture Stack
16. Backend challenge - Microservice Architecture Stack
17. Backend challenge - Deployment
18. PFM values generation from user data. Apache Spark + Cassandra Forecast prediction and regeneration of this models Apache Spark + Cassandra Product recommendations based on the economic profile of the user and his real needs. Apache Spark + Cassandra + Neo4j Credit scoring calculation Apache Spark + Cassandra Big Data Integration - Tasks
19. • Tasks are hard, needs: • Time • Resources • Not Real Time is needed. • Event Driven Architecture. Big Data Integration - Events
20. Big Data Integration - How?
21. Big Data Integration - RabbitMQ vs Kafka Kafka RabbitMq • RabbitMQ is designed as a general purpose message broker • Support existing protocols like AMQP, STOMP, MQTT. • Finer-grained consistency control/guarantees on a peer- message. • Complex routing. • Apache Kafka is designed for high volume publish-subscribe messages and streams, meant to be durable, fast, and scalable. • Event Sourcing • Your application needs access to stream history. • No complex routing. https://content.pivotal.io/blog/understanding-when-to-use-rabbitmq-or-apache-kafka
22. • Deployed in Kubernetes. • Only accessible by NodeJS API. • All keys are stored in secrets vaults. • Used for: • Tokenization • Transactions of users Blockchain Integration - Private Ethereum
23. Blockchain Integration - Private Ethereum
24. Blockchain Integration - Ethereum Main Net
25. Blockchain Integration - Ethereum Main Net
26. Blockchain Integration - Ethereum Main Net • We are the owner of the wallets • We use Infura to connect blockchain • Used for: • Payment • Transfers
27. @ganchix @2getherbank @bigeeksoftwa re Thanks! ICO is coming!