Backend Architecture Design for Analyzing and Processing Massive Blockchain Traffic

Toshimasa Nasu
LINE Blockchain Engineering Team Manager
https://linedevday.linecorp.com/jp/2019/sessions/E1-7

Transcript

1. 2019 DevDay Backend Architecture Design for Analyzing and Processing Massive Blockchain Traffic > Toshimasa Nasu > LINE Blockchain Engineering Team Manager
2. Agenda > Blockchain Data Structure > Blockchain Visualization > LINK SCAN > Data Pipeline
3. Blockchain Data Structure
4. Bank Passbook Data Structure PK Date Withdraw Deposit Balance 1 2019-11-20 1 1 2 2019-11-21 10,000,000 10,000,001 3 2019-11-22 10,000,000 1 4 2019-11-23 100,000 100,001 5 2019-11-24 100,000 200,001
5. Bank Passbook Data Structure PK Date Withdraw Deposit Balance 1 2019-11-20 1 1 2 2019-11-22 10,000,000 1 3 2019-11-21 10,000,000 10,000,001 4 2019-11-23 100,000 100,001 5 2019-11-24 100,000 200,001
6. Bank Passbook Data Structure PK Date Withdraw Deposit Balance 1 2019-11-20 1 1 2 2019-11-21 10,000,000 -9,999,999 3 2019-11-22 10,000,000 1 4 2019-11-23 100,000 100,001 5 2019-11-24 100,000 200,001
7. Block Data Structure Characteristics PK Date Withdraw Deposit Balance Hash Previous Hash 1 2019-11-20 1 1 HashA 2 2019-11-21 10,000,000 10,000,001 HashB HashA 3 2019-11-22 10,000,000 1 HashC HashB 4 2019-11-23 100,000 100,001 HashD HashC 5 2019-11-24 100,000 200,001 HashE HashD
8. Characteristics Hash Chain h(h(h(…h(x)))) e.g. Hash = hash(Data + Previous Hash) HashB = hash(Data B + HashA) Blockchain Data Structure PK Data Hash Previous Hash 1 Data A HashA 2 Data B HashB HashA 3 Data C HashC HashB 4 Data D HashD HashC 5 Data E HashE HashD
9. PK Data Hash Previous Hash 1 Data A HashA 2 Data B HashB HashA 3 Data C HashC HashB 4 Data D HashD HashC 5 Data E HashE HashD HashBB HashCC HashBB HashCC HashDD HashDD HashEE Characteristics If the Data Changes… If “Data B” changes to “Data BB”, the hash will change to a new hash. Then the hash and the previous hash of the next record must also change accordingly. And so on… Blockchain Data Structure Data BB
10. one record = one block Data Transaction Hash Previous Hash PK ɾ ɾ ɾ Block Blockchain Data Structure one record = one block Data Transaction Transaction Hash Previous Hash PK one record = one block Data Transaction Hash Previous Hash PK
11. Transaction Blockchain Data Structure Hash From To Amount 1 Address 1 Address 2 1,000,000 2 Address 1 Address 3 2,000,000 3 Address 1 Address 4 3,000,000 4 Address 3 Address 2 4,000,000 5 Address 4 Address 2 1,000,000
12. User State Blockchain Data Structure User Balance Address 1 1,000,000 Address 2 2,000,000 Address 3 3,000,000 Address 4 4,000,000 Address 5 5,000,000
13. Important Things Blockchain Data Structure Required Block (Option) User State Required Transaction
14. Blockchain Visualization
15. Blockchain Visualization If You Are an Engineer
16. Blockchain Visualization If You Are NOT an Engineer
17. Blockchain Visualization If You Are NOT an Engineer
18. Block Explorer Scan Blocks/Transactions Etherscan https://etherscan.io/ Blockchain Explorer https://www.blockchain.com/explorer
19. Bitcoin > Blockchain Explorer > chainFlyer > Bitcoin Block Explorer > Etc. Ethereum > Etherscan > Blockchain Explorer > Ethplorer > Etc.
20. LINE Token Economy LINK Chain
21. Visualize LINK Chain LINK SCAN https://scan.link.network/
22. Ƃ Blocks > Block height (= PK) > Age > Transaction count > Block hash
23. Ƃ Transactions > Tx hash > Block height > Age > From > To > Amount > Status
24. Block > Block size > Transactions > Etc. Transaction > Data type > Input data & detail > Event > Etc.
25. Ƃ User State/History > Address > Balance > Transactions > Etc.
26. LINK SCAN
27. Important Things Blockchain Data Structure Required Block (Option) User State Required Transaction
28. Visualize LINK Chain LINK SCAN Block Transaction User State User History ʴ Required
29. For LINK SCAN Let's Update LINK Chain!
30. For LINK SCAN Let's Update LINK Chain!
31. For LINK SCAN Consideration ?
32. For LINK SCAN Consideration Separate LINK Chain and LINK SCAN Build data pipeline Use LINK Chain API
33. System Design LINK Chain To LINK SCAN API API LINK SCAN Data Pipeline Storage LINK Chain Storage
34. Data Pipeline
35. System Design LINK Chain To LINK SCAN API API LINK SCAN Data pipeline Storage LINK Chain Storage How Where
36. Ƃ Apache Kafka > Publish & Subscribe > Process > Store How
37. Ƃ Apache Kafka > Publish & Subscribe > Process > Store How Used/Supported by LINE
38. Ƃ Apache HBase > Linear and modular scalability > Strictly consistent reads and writes > Automatic and configurable sharding of tables Where
39. Ƃ Apache HBase > Linear and modular scalability > Strictly consistent reads and writes > Automatic and configurable sharding of tables Where Used/Supported by LINE
40. System Design LINK Chain To LINK SCAN API API LINK SCAN Data pipeline Storage LINK Chain Storage Kafka HBase Producer Consumer
41. LINK Chain To LINK SCAN How To Produce
42. Block Blockchain Data Structure one record = one block Data Transaction Hash Previous Hash PK ɾ ɾ one record = one block Data Transaction Transaction Hash Previous Hash PK one record = one block Data Transaction Hash Previous Hash PK 00:00:01 00:00:02 00:00:03 00:00:00
43. How To Produce Requirements > Re-produce block/transaction > Manage producing application > Atomically produce on a single thread for each chain
44. Ƃ Apache Airflow > Airflow is a platform to programmatically author, schedule and monitor workflows. Producer
45. LINK Chain To LINK SCAN How To Consume
46. How To Consume Requirements > Easy add consumers > Manage consumer application > Easy consume
47. System Design LINK Chain To LINK SCAN API API LINK SCAN Data pipeline Storage LINK Chain Storage Kafka (How) HBase (Where) Airflow (Producer) App (Consumer)
48. Monitoring System Design API API LINK SCAN Data pipeline Storage LINK Chain Storage Producer Consumer ElasticSearch Monitoring
49. Multi Chain System Design API API LINK SCAN Data pipeline Storage LINK Chain Storage API Storage LINK Chain Producer Producer
50. Multi Storage System Design API API LINK SCAN Data pipeline Storage LINK Chain Storage HBase Redis Consumer Consumer MongoDB
51. Issues LINK Chain To LINK SCAN
52. Create Ranking Data Structure Issue Sorted Hash on Redis Unlimited integer not supported https://redis.io/commands/zadd Balance Unlimited integer
53. Create Ranking Data Structure Issue Manual Sorting Redis Hash NOT Used Redis SortedHash
54. Create Ranking Data Structure Issue Add (hset) Hash 1 2 3 5 4 4 6
55. On memory Create Ranking Data Structure Issue Add (hset) Hash 1 2 3 5 4 4 1 2 3 4 5 6 6
56. On memory Create Ranking Data Structure Issue Add (hset) Hash 1 2 3 5 4 Hash Del (hdel) 4 6 1 2 3 4 5 6 6 1 2 3 5 4 6
57. On memory On memory Create Ranking Data Structure Issue Add (hset) Hash 1 2 3 5 4 Hash Del (hdel) 4 6 Get (hgetall) 1 2 3 4 5 1 2 3 4 6 6 1 2 3 5 4 6 5
58. Issue Avoid Double-Counting Transactions Count = 4 Consumer Consumer Tx1 Tx2 Tx3 Tx4 Tx5 Tx6 Kafka Count = 2
59. Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4 Tx5 Tx5 Tx6 Kafka Producer Producer Count = 4 Count = 3
60. Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4 Tx5 Tx5 Tx6 Kafka Producer Producer Count = 4 Count = 4
61. Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4 Tx5 Tx5 Tx6 Kafka Tx1 Tx2 Tx3 Tx4 Tx5 Tx6 Redis Producer Producer
62. Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4 Tx5 Tx5 Tx6 Consumer Kafka Tx1 Tx2 Tx3 Tx4 Tx5 Tx6 Counting Time Range Redis HBase Producer Producer
63. LINK Chain To LINK SCAN Next Architecture Easily attach/detach a feature without impacting the existing system Scale-out data processing Index the data with various requirements
64. Thank You