Upgrade to Pro — share decks privately, control downloads, hide ads and more …

everything is log

94575b73a05d48e567423d7d6ca8273c?s=47 Poga Po
August 20, 2016
1.6k

everything is log

請搭配逐字稿服用 https://medium.com/@poga/caf3029359f8

94575b73a05d48e567423d7d6ca8273c?s=128

Poga Po

August 20, 2016
Tweet

Transcript

  1. 萬事萬物皆是 LOG Poga @ COSCUP 2016.08.20

  2. LOG If you want maintainable, understandable software system

  3. 什麼是 LOG

  4. LOG 的性質 • 按照時間出現 • 出現過的就不會改變(append-only) • 目的:記錄(某個時間)發生了(某件事) ◦ 常常用來作為

    debug 推理的依據
  5. Things based on LOG • Database • Version Control •

    Distributed System and Distributed Consensus • Synchronization • Replication • Messaging • UI Framework • ...And almost EVERYTHING
  6. 聽完就理解了軟體開發的真理(誤)

  7. POGA Software Developer, g0v.tw, KKTIX(前)

  8. 資料庫 ID Username Balance 231 poga 100 7212 moon 12314

    834 et 927 ... ... ... Database
  9. • 大量讀寫資料,長期儲存 • 確保資料正確性(relation, validation, schema, ACID) • 支援多種高效查詢(SQL, index)

    如果你從沒用過資料庫...
  10. 寫入資料時當機?

  11. 如何避免寫入資料時當機導致資料消失? 寫入: ID Username Balance 231 poga 100 ... 2016.08.20

    13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入: Validation, Update Index… etc.
  12. ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 時間

    發生了什麼事
  13. WRITE-AHEAD LOG

  14. LOG • Fast and simple RECAP

  15. 資料庫 ID Username Balance 231 poga 100 7212 moon 12314

    834 et 927 ... ... ... Database
  16. • 單一 Database 擴充性有其極限 • 多台 Database,分散負載 ◦ 需要確保多台 Database

    之間資料同步 Replication
  17. 確保多台資料庫間資料同步?

  18. 寫入一台資料庫 寫入: ID Username Balance 231 poga 100 ... 2016.08.20

    13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入: Validation, Update Index… etc.
  19. 一次寫入多台資料庫 寫入: ID Username Balance 231 poga 100 ... 2016.08.20

    13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入: Validation, Update Index… etc. ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入: Validation, Update Index… etc. DB1 DB2
  20. LOG-SHIPPING

  21. LOG • Fast and simple • Can be used to

    share “current state” RECAP
  22. 微服務 Microservice

  23. Monolith v.s. Microservice

  24. Monolith • 所有邏輯集中在單一系統中,共 享儲存運算資源 • 容易開發 • 不易擴充

  25. Microservice • 將邏輯按照 Domain 切割,有 獨立的儲存與運算資源 • 容易擴充,團隊間分責明確 • 門檻較高:不易維護,需要進階

    系統維護能力
  26. 微服務間怎麼互相溝通?

  27. Event Stream

  28. EVENT STREAM = LOG

  29. LOG • Fast and simple • Can be used to

    share “current state” • LOG are easy to parse/understand, Universal Interface RECAP
  30. 前端架構 Flux/Redux

  31. • Many, many states • Constantly changing state ◦ User

    Interaction, Animation, Network • Hard to reproduce bug and debug UI State
  32. 前端介面狀態多變,如何維護?

  33. Flux/Redux • Unidirectional Data Flow ◦ Append only ◦ Handle

    action one by one, reasonable • Immutable State ◦ Produce new state based on previous state and action • Everything else can be stateless(pure)
  34. UNIDIRECTIONAL = LOG

  35. LOG • Fast and simple • Can be used to

    share “current state” • Text are easy to parse/understand, Universal Interface • Append-only, easy to reason, everything else can be stateless RECAP
  36. 區塊鏈 Blockchain

  37. • 2016.08.20 13:31:23, A 給 B 100 bitcoin • State:

    A 跟 B 身上剩下多少 bitcoin • Double Spend: ◦ A 把 bitcoin 交給 B 後,不能再把同樣的 bitcoin 交給 C Transaction
  38. 如何讓全世界都認知交易的成立? a.k.a. Avoid Double-Spending

  39. • 如果 ◦ 所有參與者都知道/可以驗證所有人的交易記錄 ◦ 既有的交易紀錄無法被竄改 ◦ 便沒有偽造空間 • Blockchain

    ◦ 將所有的交易記錄用 block 封裝,每個 block 替之前的交易記錄加密驗證 ◦ 想要偽造一筆交易記錄 = 必須偽造整個 blockchain 上的交易記錄 ◦ 所有人都基於相同的 blockchain 驗證交易 Distributed Consensus
  40. LOG AGREEMENT = CONSENSUS Raft: A Consensus Algorithm for Replicated

    Logs Apache Mesos - The Mesos Replicated Log (PAXOS)
  41. LOG • Fast and simple • Can be used to

    share “current state” • Text are easy to parse/understand, Universal Interface • Append-only, Immutable, easy to reason, everything else can be stateless • Replicate log = consensus RECAP
  42. 大數據 Data Pipeline

  43. • > PB • Data coming from everywhere ◦ User

    Input ◦ Business Data ◦ Tracking ◦ External Data • ETL(Extract-Transform-Load) • Analysis, Machine Learning… etc BIG Data
  44. 如何處理雜亂、無序、大量的資料?

  45. Data Pipeline

  46. APPEND ONLY LOG = HIGH SCALABILITY

  47. LOG • Fast and simple • Can be used to

    share “current state” • Text are easy to parse/understand, Universal Interface • Append-only, Immutable, easy to reason, everything else can be stateless • Replicate log = consensus • High Scalability RECAP
  48. LOG = DETERMINISM

  49. 相同的過程 = 相同的結果 DETERMINISM

  50. Make your system deterministic • 容易理解 • 容易DEBUG • 容易同步

    • 容易儲存 • 容易擴展
  51. LOG = DETERMINISM = 穩定的系統

  52. 設計系統架構時,先試著用 LOG 表示你的系統

  53. Thanks!