Backend Architecture Design for Analyzing and Processing Massive Blockchain Traffic

by LINE DevDay 2019

Slide 1

Slide 1 text

2019 DevDay Backend Architecture Design for Analyzing and Processing Massive Blockchain Traffic > Toshimasa Nasu > LINE Blockchain Engineering Team Manager

Slide 2

Slide 2 text

Agenda > Blockchain Data Structure > Blockchain Visualization > LINK SCAN > Data Pipeline

Slide 3

Slide 3 text

Blockchain Data Structure

Slide 4

Slide 4 text

Bank Passbook Data Structure PK Date Withdraw Deposit Balance 1 2019-11-20 1 1 2 2019-11-21 10,000,000 10,000,001 3 2019-11-22 10,000,000 1 4 2019-11-23 100,000 100,001 5 2019-11-24 100,000 200,001

Slide 5

Slide 5 text

Bank Passbook Data Structure PK Date Withdraw Deposit Balance 1 2019-11-20 1 1 2 2019-11-22 10,000,000 1 3 2019-11-21 10,000,000 10,000,001 4 2019-11-23 100,000 100,001 5 2019-11-24 100,000 200,001

Slide 6

Slide 6 text

Bank Passbook Data Structure PK Date Withdraw Deposit Balance 1 2019-11-20 1 1 2 2019-11-21 10,000,000 -9,999,999 3 2019-11-22 10,000,000 1 4 2019-11-23 100,000 100,001 5 2019-11-24 100,000 200,001

Slide 7

Slide 7 text

Block Data Structure Characteristics PK Date Withdraw Deposit Balance Hash Previous Hash 1 2019-11-20 1 1 HashA 2 2019-11-21 10,000,000 10,000,001 HashB HashA 3 2019-11-22 10,000,000 1 HashC HashB 4 2019-11-23 100,000 100,001 HashD HashC 5 2019-11-24 100,000 200,001 HashE HashD

Slide 8

Slide 8 text

Characteristics Hash Chain h(h(h(…h(x)))) e.g. Hash = hash(Data + Previous Hash) HashB = hash(Data B + HashA) Blockchain Data Structure PK Data Hash Previous Hash 1 Data A HashA 2 Data B HashB HashA 3 Data C HashC HashB 4 Data D HashD HashC 5 Data E HashE HashD

Slide 9

Slide 9 text

PK Data Hash Previous Hash 1 Data A HashA 2 Data B HashB HashA 3 Data C HashC HashB 4 Data D HashD HashC 5 Data E HashE HashD HashBB HashCC HashBB HashCC HashDD HashDD HashEE Characteristics If the Data Changes… If “Data B” changes to “Data BB”, the hash will change to a new hash. Then the hash and the previous hash of the next record must also change accordingly. And so on… Blockchain Data Structure Data BB

Slide 10

Slide 10 text

one record = one block Data Transaction Hash Previous Hash PK ɾ ɾ ɾ Block Blockchain Data Structure one record = one block Data Transaction Transaction Hash Previous Hash PK one record = one block Data Transaction Hash Previous Hash PK

Slide 11

Slide 11 text

Transaction Blockchain Data Structure Hash From To Amount 1 Address 1 Address 2 1,000,000 2 Address 1 Address 3 2,000,000 3 Address 1 Address 4 3,000,000 4 Address 3 Address 2 4,000,000 5 Address 4 Address 2 1,000,000

Slide 12

Slide 12 text

User State Blockchain Data Structure User Balance Address 1 1,000,000 Address 2 2,000,000 Address 3 3,000,000 Address 4 4,000,000 Address 5 5,000,000

Slide 13

Slide 13 text

Important Things Blockchain Data Structure Required Block (Option) User State Required Transaction

Slide 14

Slide 14 text

Blockchain Visualization

Slide 15

Slide 15 text

Blockchain Visualization If You Are an Engineer

Slide 16

Slide 16 text

Blockchain Visualization If You Are NOT an Engineer

Slide 17

Slide 17 text

Blockchain Visualization If You Are NOT an Engineer

Slide 18

Slide 18 text

Block Explorer Scan Blocks/Transactions Etherscan https://etherscan.io/ Blockchain Explorer https://www.blockchain.com/explorer

Slide 19

Slide 19 text

Bitcoin > Blockchain Explorer > chainFlyer > Bitcoin Block Explorer > Etc. Ethereum > Etherscan > Blockchain Explorer > Ethplorer > Etc.

Slide 20

Slide 20 text

LINE Token Economy LINK Chain

Slide 21

Slide 21 text

Visualize LINK Chain LINK SCAN https://scan.link.network/

Slide 22

Slide 22 text

Ƃ Blocks > Block height (= PK) > Age > Transaction count > Block hash

Slide 23

Slide 23 text

Ƃ Transactions > Tx hash > Block height > Age > From > To > Amount > Status

Slide 24

Slide 24 text

Block > Block size > Transactions > Etc. Transaction > Data type > Input data & detail > Event > Etc.

Slide 25

Slide 25 text

Ƃ User State/History > Address > Balance > Transactions > Etc.

Slide 26

Slide 26 text

LINK SCAN

Slide 27

Slide 27 text

Important Things Blockchain Data Structure Required Block (Option) User State Required Transaction

Slide 28

Slide 28 text

Visualize LINK Chain LINK SCAN Block Transaction User State User History ʴ Required

Slide 29

Slide 29 text

For LINK SCAN Let's Update LINK Chain!

Slide 30

Slide 30 text

For LINK SCAN Let's Update LINK Chain!

Slide 31

Slide 31 text

For LINK SCAN Consideration ?

Slide 32

Slide 32 text

For LINK SCAN Consideration Separate LINK Chain and LINK SCAN Build data pipeline Use LINK Chain API

Slide 33

Slide 33 text

System Design LINK Chain To LINK SCAN API API LINK SCAN Data Pipeline Storage LINK Chain Storage

Slide 34

Slide 34 text

Data Pipeline

Slide 35

Slide 35 text

System Design LINK Chain To LINK SCAN API API LINK SCAN Data pipeline Storage LINK Chain Storage How Where

Slide 36

Slide 36 text

Ƃ Apache Kafka > Publish & Subscribe > Process > Store How

Slide 37

Slide 37 text

Ƃ Apache Kafka > Publish & Subscribe > Process > Store How Used/Supported by LINE

Slide 38

Slide 38 text

Ƃ Apache HBase > Linear and modular scalability > Strictly consistent reads and writes > Automatic and configurable sharding of tables Where

Slide 39

Slide 39 text

Ƃ Apache HBase > Linear and modular scalability > Strictly consistent reads and writes > Automatic and configurable sharding of tables Where Used/Supported by LINE

Slide 40

Slide 40 text

System Design LINK Chain To LINK SCAN API API LINK SCAN Data pipeline Storage LINK Chain Storage Kafka HBase Producer Consumer

Slide 41

Slide 41 text

LINK Chain To LINK SCAN How To Produce

Slide 42

Slide 42 text

Block Blockchain Data Structure one record = one block Data Transaction Hash Previous Hash PK ɾ ɾ one record = one block Data Transaction Transaction Hash Previous Hash PK one record = one block Data Transaction Hash Previous Hash PK 00:00:01 00:00:02 00:00:03 00:00:00

Slide 43

Slide 43 text

How To Produce Requirements > Re-produce block/transaction > Manage producing application > Atomically produce on a single thread for each chain

Slide 44

Slide 44 text

Ƃ Apache Airflow > Airflow is a platform to programmatically author, schedule and monitor workflows. Producer

Slide 45

Slide 45 text

LINK Chain To LINK SCAN How To Consume

Slide 46

Slide 46 text

How To Consume Requirements > Easy add consumers > Manage consumer application > Easy consume

Slide 47

Slide 47 text

System Design LINK Chain To LINK SCAN API API LINK SCAN Data pipeline Storage LINK Chain Storage Kafka (How) HBase (Where) Airflow (Producer) App (Consumer)

Slide 48

Slide 48 text

Monitoring System Design API API LINK SCAN Data pipeline Storage LINK Chain Storage Producer Consumer ElasticSearch Monitoring

Slide 49

Slide 49 text

Multi Chain System Design API API LINK SCAN Data pipeline Storage LINK Chain Storage API Storage LINK Chain Producer Producer

Slide 50

Slide 50 text

Multi Storage System Design API API LINK SCAN Data pipeline Storage LINK Chain Storage HBase Redis Consumer Consumer MongoDB

Slide 51

Slide 51 text

Issues LINK Chain To LINK SCAN

Slide 52

Slide 52 text

Create Ranking Data Structure Issue Sorted Hash on Redis Unlimited integer not supported https://redis.io/commands/zadd Balance Unlimited integer

Slide 53

Slide 53 text

Create Ranking Data Structure Issue Manual Sorting Redis Hash NOT Used Redis SortedHash

Slide 54

Slide 54 text

Create Ranking Data Structure Issue Add (hset) Hash 1 2 3 5 4 4 6

Slide 55

Slide 55 text

On memory Create Ranking Data Structure Issue Add (hset) Hash 1 2 3 5 4 4 1 2 3 4 5 6 6

Slide 56

Slide 56 text

On memory Create Ranking Data Structure Issue Add (hset) Hash 1 2 3 5 4 Hash Del (hdel) 4 6 1 2 3 4 5 6 6 1 2 3 5 4 6

Slide 57

Slide 57 text

On memory On memory Create Ranking Data Structure Issue Add (hset) Hash 1 2 3 5 4 Hash Del (hdel) 4 6 Get (hgetall) 1 2 3 4 5 1 2 3 4 6 6 1 2 3 5 4 6 5

Slide 58

Slide 58 text

Issue Avoid Double-Counting Transactions Count = 4 Consumer Consumer Tx1 Tx2 Tx3 Tx4 Tx5 Tx6 Kafka Count = 2

Slide 59

Slide 59 text

Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4 Tx5 Tx5 Tx6 Kafka Producer Producer Count = 4 Count = 3

Slide 60

Slide 60 text

Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4 Tx5 Tx5 Tx6 Kafka Producer Producer Count = 4 Count = 4

Slide 61

Slide 61 text

Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4 Tx5 Tx5 Tx6 Kafka Tx1 Tx2 Tx3 Tx4 Tx5 Tx6 Redis Producer Producer

Slide 62

Slide 62 text

Issue Avoid Double-Counting Transactions Consumer Consumer Tx1 Tx2 Tx3 Tx4 Tx5 Tx5 Tx6 Consumer Kafka Tx1 Tx2 Tx3 Tx4 Tx5 Tx6 Counting Time Range Redis HBase Producer Producer

Slide 63

Slide 63 text

LINK Chain To LINK SCAN Next Architecture Easily attach/detach a feature without impacting the existing system Scale-out data processing Index the data with various requirements

Slide 64

Slide 64 text

Thank You