everything is log - Speaker Deck

Slide 1

Slide 1 text

萬事萬物皆是 LOG Poga @ COSCUP 2016.08.20

Slide 2

Slide 2 text

LOG If you want maintainable, understandable software system

Slide 3

Slide 3 text

什麼是 LOG

Slide 4

Slide 4 text

LOG 的性質 ● 按照時間出現 ● 出現過的就不會改變（append-only） ● 目的：記錄（某個時間）發生了（某件事） ○ 常常用來作為 debug 推理的依據

Slide 5

Slide 5 text

Things based on LOG ● Database ● Version Control ● Distributed System and Distributed Consensus ● Synchronization ● Replication ● Messaging ● UI Framework ● ...And almost EVERYTHING

Slide 6

Slide 6 text

聽完就理解了軟體開發的真理（誤）

Slide 7

Slide 7 text

POGA Software Developer, g0v.tw, KKTIX(前)

Slide 8

Slide 8 text

資料庫 ID Username Balance 231 poga 100 7212 moon 12314 834 et 927 ... ... ... Database

Slide 9

Slide 9 text

● 大量讀寫資料，長期儲存 ● 確保資料正確性（relation, validation, schema, ACID） ● 支援多種高效查詢（SQL, index）如果你從沒用過資料庫...

Slide 10

Slide 10 text

寫入資料時當機？

Slide 11

Slide 11 text

如何避免寫入資料時當機導致資料消失？寫入： ID Username Balance 231 poga 100 ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入： Validation, Update Index… etc.

Slide 12

Slide 12 text

... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 時間發生了什麼事

Slide 13

Slide 13 text

WRITE-AHEAD LOG

Slide 14

Slide 14 text

LOG ● Fast and simple RECAP

Slide 15

Slide 15 text

資料庫 ID Username Balance 231 poga 100 7212 moon 12314 834 et 927 ... ... ... Database

Slide 16

Slide 16 text

● 單一 Database 擴充性有其極限 ● 多台 Database，分散負載 ○ 需要確保多台 Database 之間資料同步 Replication

Slide 17

Slide 17 text

確保多台資料庫間資料同步？

Slide 18

Slide 18 text

寫入一台資料庫寫入： ID Username Balance 231 poga 100 ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入： Validation, Update Index… etc.

Slide 19

Slide 19 text

一次寫入多台資料庫寫入： ID Username Balance 231 poga 100 ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入： Validation, Update Index… etc. ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入： Validation, Update Index… etc. DB1 DB2

Slide 20

Slide 20 text

LOG-SHIPPING

Slide 21

Slide 21 text

LOG ● Fast and simple ● Can be used to share “current state” RECAP

Slide 22

Slide 22 text

微服務 Microservice

Slide 23

Slide 23 text

Monolith v.s. Microservice

Slide 24

Slide 24 text

Monolith ● 所有邏輯集中在單一系統中，共享儲存運算資源 ● 容易開發 ● 不易擴充

Slide 25

Slide 25 text

Microservice ● 將邏輯按照 Domain 切割，有獨立的儲存與運算資源 ● 容易擴充，團隊間分責明確 ● 門檻較高：不易維護，需要進階系統維護能力

Slide 26

Slide 26 text

微服務間怎麼互相溝通？

Slide 27

Slide 27 text

Event Stream

Slide 28

Slide 28 text

EVENT STREAM = LOG

Slide 29

Slide 29 text

LOG ● Fast and simple ● Can be used to share “current state” ● LOG are easy to parse/understand, Universal Interface RECAP

Slide 30

Slide 30 text

前端架構 Flux/Redux

Slide 31

Slide 31 text

● Many, many states ● Constantly changing state ○ User Interaction, Animation, Network ● Hard to reproduce bug and debug UI State

Slide 32

Slide 32 text

前端介面狀態多變，如何維護？

Slide 33

Slide 33 text

Flux/Redux ● Unidirectional Data Flow ○ Append only ○ Handle action one by one, reasonable ● Immutable State ○ Produce new state based on previous state and action ● Everything else can be stateless(pure)

Slide 34

Slide 34 text

UNIDIRECTIONAL = LOG

Slide 35

Slide 35 text

LOG ● Fast and simple ● Can be used to share “current state” ● Text are easy to parse/understand, Universal Interface ● Append-only, easy to reason, everything else can be stateless RECAP

Slide 36

Slide 36 text

區塊鏈 Blockchain

Slide 37

Slide 37 text

● 2016.08.20 13:31:23, A 給 B 100 bitcoin ● State: A 跟 B 身上剩下多少 bitcoin ● Double Spend: ○ A 把 bitcoin 交給 B 後，不能再把同樣的 bitcoin 交給 C Transaction

Slide 38

Slide 38 text

如何讓全世界都認知交易的成立？ a.k.a. Avoid Double-Spending

Slide 39

Slide 39 text

● 如果 ○ 所有參與者都知道/可以驗證所有人的交易記錄 ○ 既有的交易紀錄無法被竄改 ○ 便沒有偽造空間 ● Blockchain ○ 將所有的交易記錄用 block 封裝，每個 block 替之前的交易記錄加密驗證 ○ 想要偽造一筆交易記錄 = 必須偽造整個 blockchain 上的交易記錄 ○ 所有人都基於相同的 blockchain 驗證交易 Distributed Consensus

Slide 40

Slide 40 text

LOG AGREEMENT = CONSENSUS Raft: A Consensus Algorithm for Replicated Logs Apache Mesos - The Mesos Replicated Log (PAXOS)

Slide 41

Slide 41 text

LOG ● Fast and simple ● Can be used to share “current state” ● Text are easy to parse/understand, Universal Interface ● Append-only, Immutable, easy to reason, everything else can be stateless ● Replicate log = consensus RECAP

Slide 42

Slide 42 text

大數據 Data Pipeline

Slide 43

Slide 43 text

● > PB ● Data coming from everywhere ○ User Input ○ Business Data ○ Tracking ○ External Data ● ETL(Extract-Transform-Load) ● Analysis, Machine Learning… etc BIG Data

Slide 44

Slide 44 text

如何處理雜亂、無序、大量的資料？

Slide 45

Slide 45 text

Data Pipeline

Slide 46

Slide 46 text

APPEND ONLY LOG = HIGH SCALABILITY

Slide 47

Slide 47 text

Slide 48

Slide 48 text

LOG = DETERMINISM

Slide 49

Slide 49 text

相同的過程 = 相同的結果 DETERMINISM

Slide 50

Slide 50 text

Make your system deterministic ● 容易理解 ● 容易DEBUG ● 容易同步 ● 容易儲存 ● 容易擴展