Backend Architecture Design for Analyzing and Processing Massive Blockchain Traffic

Backend Architecture Design for Analyzing and Processing Massive Blockchain Traffic

Toshimasa Nasu
LINE Blockchain Engineering Team Manager
https://linedevday.linecorp.com/jp/2019/sessions/E1-7

Be4518b119b8eb017625e0ead20f8fe7?s=128

LINE DevDay 2019

November 20, 2019
Tweet

Transcript

  1. 2019 DevDay Backend Architecture Design for Analyzing and Processing Massive

    Blockchain Traffic > Toshimasa Nasu > LINE Blockchain Engineering Team Manager
  2. Agenda > Blockchain Data Structure > Blockchain Visualization > LINK

    SCAN > Data Pipeline
  3. Blockchain Data Structure

  4. Bank Passbook Data Structure PK Date Withdraw Deposit Balance 1

    2019-11-20 1 1 2 2019-11-21 10,000,000 10,000,001 3 2019-11-22 10,000,000 1 4 2019-11-23 100,000 100,001 5 2019-11-24 100,000 200,001
  5. Bank Passbook Data Structure PK Date Withdraw Deposit Balance 1

    2019-11-20 1 1 2 2019-11-22 10,000,000 1 3 2019-11-21 10,000,000 10,000,001 4 2019-11-23 100,000 100,001 5 2019-11-24 100,000 200,001
  6. Bank Passbook Data Structure PK Date Withdraw Deposit Balance 1

    2019-11-20 1 1 2 2019-11-21 10,000,000 -9,999,999 3 2019-11-22 10,000,000 1 4 2019-11-23 100,000 100,001 5 2019-11-24 100,000 200,001
  7. Block Data Structure Characteristics PK Date Withdraw Deposit Balance Hash

    Previous Hash 1 2019-11-20 1 1 HashA 2 2019-11-21 10,000,000 10,000,001 HashB HashA 3 2019-11-22 10,000,000 1 HashC HashB 4 2019-11-23 100,000 100,001 HashD HashC 5 2019-11-24 100,000 200,001 HashE HashD
  8. Characteristics Hash Chain h(h(h(…h(x)))) e.g. Hash = hash(Data + Previous

    Hash) HashB = hash(Data B + HashA) Blockchain Data Structure PK Data Hash Previous Hash 1 Data A HashA 2 Data B HashB HashA 3 Data C HashC HashB 4 Data D HashD HashC 5 Data E HashE HashD
  9. PK Data Hash Previous Hash 1 Data A HashA 2

    Data B HashB HashA 3 Data C HashC HashB 4 Data D HashD HashC 5 Data E HashE HashD HashBB HashCC HashBB HashCC HashDD HashDD HashEE Characteristics If the Data Changes… If “Data B” changes to “Data BB”, the hash will change to a new hash. Then the hash and the previous hash of the next record must also change accordingly. And so on… Blockchain Data Structure Data BB
  10. one record = one block Data Transaction Hash Previous Hash

    PK ɾ ɾ ɾ Block Blockchain Data Structure one record = one block Data Transaction Transaction Hash Previous Hash PK one record = one block Data Transaction Hash Previous Hash PK
  11. Transaction Blockchain Data Structure Hash From To Amount 1 Address

    1 Address 2 1,000,000 2 Address 1 Address 3 2,000,000 3 Address 1 Address 4 3,000,000 4 Address 3 Address 2 4,000,000 5 Address 4 Address 2 1,000,000
  12. User State Blockchain Data Structure User Balance Address 1 1,000,000

    Address 2 2,000,000 Address 3 3,000,000 Address 4 4,000,000 Address 5 5,000,000
  13. Important Things Blockchain Data Structure Required Block (Option) User State

    Required Transaction
  14. Blockchain Visualization

  15. Blockchain Visualization If You Are an Engineer

  16. Blockchain Visualization If You Are NOT an Engineer

  17. Blockchain Visualization If You Are NOT an Engineer

  18. Block Explorer Scan Blocks/Transactions Etherscan https://etherscan.io/ Blockchain Explorer https://www.blockchain.com/explorer

  19. Bitcoin > Blockchain Explorer > chainFlyer > Bitcoin Block Explorer

    > Etc. Ethereum > Etherscan > Blockchain Explorer > Ethplorer > Etc.
  20. LINE Token Economy LINK Chain

  21. Visualize LINK Chain LINK SCAN https://scan.link.network/

  22. Ƃ Blocks > Block height (= PK) > Age >

    Transaction count > Block hash
  23. Ƃ Transactions > Tx hash > Block height > Age

    > From > To > Amount > Status
  24. Block > Block size > Transactions > Etc. Transaction >

    Data type > Input data & detail > Event > Etc.
  25. Ƃ User State/History > Address > Balance > Transactions >

    Etc.
  26. LINK SCAN

  27. Important Things Blockchain Data Structure Required Block (Option) User State

    Required Transaction
  28. Visualize LINK Chain LINK SCAN Block Transaction User State User

    History ʴ Required
  29. For LINK SCAN Let's Update LINK Chain!

  30. For LINK SCAN Let's Update LINK Chain!

  31. For LINK SCAN Consideration ?

  32. For LINK SCAN Consideration Separate LINK Chain and LINK SCAN

    Build data pipeline Use LINK Chain API
  33. System Design LINK Chain To LINK SCAN API API LINK

    SCAN Data Pipeline Storage LINK Chain Storage
  34. Data Pipeline

  35. System Design LINK Chain To LINK SCAN API API LINK

    SCAN Data pipeline Storage LINK Chain Storage How Where
  36. Ƃ Apache Kafka > Publish & Subscribe > Process >

    Store How
  37. Ƃ Apache Kafka > Publish & Subscribe > Process >

    Store How Used/Supported by LINE
  38. Ƃ Apache HBase > Linear and modular scalability > Strictly

    consistent reads and writes > Automatic and configurable sharding of tables Where
  39. Ƃ Apache HBase > Linear and modular scalability > Strictly

    consistent reads and writes > Automatic and configurable sharding of tables Where Used/Supported by LINE
  40. System Design LINK Chain To LINK SCAN API API LINK

    SCAN Data pipeline Storage LINK Chain Storage Kafka HBase Producer Consumer
  41. LINK Chain To LINK SCAN How To Produce

  42. Block Blockchain Data Structure one record = one block Data

    Transaction Hash Previous Hash PK ɾ ɾ one record = one block Data Transaction Transaction Hash Previous Hash PK one record = one block Data Transaction Hash Previous Hash PK 00:00:01 00:00:02 00:00:03 00:00:00
  43. How To Produce Requirements > Re-produce block/transaction > Manage producing

    application > Atomically produce on a single thread for each chain
  44. Ƃ Apache Airflow > Airflow is a platform to programmatically

    author, schedule and monitor workflows. Producer
  45. LINK Chain To LINK SCAN How To Consume

  46. How To Consume Requirements > Easy add consumers > Manage

    consumer application > Easy consume
  47. System Design LINK Chain To LINK SCAN API API LINK

    SCAN Data pipeline Storage LINK Chain Storage Kafka (How) HBase (Where) Airflow (Producer) App (Consumer)
  48. Monitoring System Design API API LINK SCAN Data pipeline Storage

    LINK Chain Storage Producer Consumer ElasticSearch Monitoring
  49. Multi Chain System Design API API LINK SCAN Data pipeline

    Storage LINK Chain Storage API Storage LINK Chain Producer Producer
  50. Multi Storage System Design API API LINK SCAN Data pipeline

    Storage LINK Chain Storage HBase Redis Consumer Consumer MongoDB
  51. Issues LINK Chain To LINK SCAN

  52. Create Ranking Data Structure Issue Sorted Hash on Redis Unlimited

    integer not supported https://redis.io/commands/zadd Balance Unlimited integer
  53. Create Ranking Data Structure Issue Manual Sorting Redis Hash NOT

    Used Redis SortedHash
  54. Create Ranking Data Structure Issue Add (hset) Hash 1 2

    3 5 4 4 6
  55. On memory Create Ranking Data Structure Issue Add (hset) Hash

    1 2 3 5 4 4 1 2 3 4 5 6 6
  56. On memory Create Ranking Data Structure Issue Add (hset) Hash

    1 2 3 5 4 Hash Del (hdel) 4 6 1 2 3 4 5 6 6 1 2 3 5 4 6
  57. On memory On memory Create Ranking Data Structure Issue Add

    (hset) Hash 1 2 3 5 4 Hash Del (hdel) 4 6 Get (hgetall) 1 2 3 4 5 1 2 3 4 6 6 1 2 3 5 4 6 5
  58. Issue Avoid Double-Counting Transactions Count = 4 Consumer Consumer Tx1

    Tx2 Tx3 Tx4 Tx5 Tx6 Kafka Count = 2
  59. Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4

    Tx5 Tx5 Tx6 Kafka Producer Producer Count = 4 Count = 3
  60. Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4

    Tx5 Tx5 Tx6 Kafka Producer Producer Count = 4 Count = 4
  61. Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4

    Tx5 Tx5 Tx6 Kafka Tx1 Tx2 Tx3 Tx4 Tx5 Tx6 Redis Producer Producer
  62. Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4

    Tx5 Tx5 Tx6 Consumer Kafka Tx1 Tx2 Tx3 Tx4 Tx5 Tx6 Counting Time Range Redis HBase Producer Producer
  63. LINK Chain To LINK SCAN Next Architecture Easily attach/detach a

    feature without impacting the existing system Scale-out data processing Index the data with various requirements
  64. Thank You