Tweet
Tweet

Slide 1

Slide 1 text

flowstatd - 那五年 Kudo Chien

Slide 2

Slide 2 text

Kudo Chien CCUCSIE 2002-2008 BS+MS (GAIS Lab)! CNA ! 曾⼯工作於 Trend Micro、︑Waveface! 現任 biideal CTO

Slide 3

Slide 3 text

Kudo Chien 打雜! UN*X system programming! Windows programming! Network programming! Cloud/Web backend! Web frontend! Browser extension development! DevOps! Hacking! iOS! Android! Debug

Slide 4

Slide 4 text

在 biideal 我們沒有辦不到的事 *誤*

Slide 5

Slide 5 text

flowstatd 是?

Slide 6

Slide 6 text

這樣的系統需要什麼樣的機器來跑 Image source: https://www.flickr.com/photos/horiavarlan/4273913966

Slide 7

Slide 7 text

memory/disk 使⽤用量多⼤大 Image source: https://www.flickr.com/photos/horiavarlan/4273913966

Slide 8

Slide 8 text

The difference between genius and stupidity is that genius has its limits.

Slide 9

Slide 9 text

因為有限制
 才得以出類拔萃

Slide 10

Slide 10 text

Netflow introduction From Cisco! Analyze traffic! SRC/DST IP! SRC/DST Port! TOC! IP Protocol

Slide 11

Slide 11 text

宿網流量統計 v1 來⾃自交⼤大 open source 的版本! flow-tools + Perl script! 每⼩小時”重頭”算⼀一次統計! 是網管的災難，使⽤用者的福⾳音

Slide 12

Slide 12 text

宿網流量統計 v2 由⽉月光⼩小俠 Eintisy 學長⽤用 PHP 重寫的版本! “累加”流量解決了第⼀一版的問題! 慢慢還是撐不住全校的流量! 兩⼩小時跑⼀一次，網路速度越來越快，兩⼩小時可 以衝很多 GB

Slide 13

Slide 13 text

年少輕狂的 MySQL 時代 不管 3721，往 MySQL 丟就對了 *誤*! MySQL ⼤大神會幫你管理⼀一切事務! Malicious Detection

Slide 14

Slide 14 text

年少輕狂的 MySQL 時代 以 CCU 全校流量來說，倒進 MySQL 平均每 ⼩小時佔⽤用 Disk 1xx MB

Slide 15

Slide 15 text

吳昇⽼老師的教誨 Data Structure! 對資料本質的掌握與計算! Hash Hash Hash

Slide 16

Slide 16 text

重視統計流量的本質 累加流量! IP address hash table - ⼀一個蘿蔔⼀一個坑

Slide 17

Slide 17 text

那五年 2007~2012 2009 才開始在這個 project ⽤用 git *冏*

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

統計全宿網⼀一天的流量只需要 3.1 MB

Slide 20

Slide 20 text

全中正 Class B 的流量只需要
 25.7 MB

Slide 21

Slide 21 text

Flow daemon! All in memory! Real time

Slide 22

Slide 22 text

Hash function v1

Slide 23

Slide 23 text

Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/

Slide 24

Slide 24 text

Hash function v2

Slide 25

Slide 25 text

Architecture v1 collector! process! 
 listen port 1025 query! process! 
 named pipe! socket shm Command! topN! over 5G

Slide 26

Slide 26 text

query! process! 
 named pipe! socket @WanCW
 <(_ _)>

Slide 27

Slide 27 text

Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/

Slide 28

Slide 28 text

Architecture v2 Single process multiplexing! kqueue / select UDP! collector port TCP! command port 那⼀一年，我還不知道 libevent / libev

Slide 29

Slide 29 text

Object Oriented Programming Trained from Trend Micro ! 常⾒見的好習慣是把 shared code 拆成 functions
 OOP 則更進⼀一步把 shared behaviors 詮釋成 共同的 interfaces! 以上是本⼈人不負責任亂掰的說法 *誤*

Slide 30

Slide 30 text

–Butler Lampson “All problems in computer science can be solved by another level of indirection”

Slide 31

Slide 31 text

Object Oriented C Abstract + struct + function pointer! select()/kqueue multiplexer! Netflow v5/v9 handlers

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

Usage in caller is simple

Slide 35

Slide 35 text

Over design 之 container_of 直接 cast 就好啦 冏

Slide 36

Slide 36 text

Multiple subnets 全校⽤用⼀一個 hash table 相對 簡單! 宿網 30 個 subnets 反⽽而麻煩 subnet 1! hash table subnet 2! hash table subnet N! hash table Binary Search

Slide 37

Slide 37 text

Netflow version 9 template! multiple source + multiple source id Image source: http://www.lancope.com/blog/netflow-v5-vs-netflow-v9/

Slide 38

Slide 38 text

Netflow version 9 多個 source ip 下⾯面又可以有多個 source id! 又想⽤用 hash 又不想⽤用太多 memory! two hash tables! source table! template table

Slide 39

Slide 39 text

Hash from two factors source! table template! table template! table template! table source! table template table

Slide 40

Slide 40 text

Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/

Slide 41

Slide 41 text

Netflow version 9 debugging 時好時壞的 bug 真難抓! tcpdump -> pcap! 對照組! pcap -> wireshark! pcap -> tcpreplay

Slide 42

Slide 42 text

Misc JSON input command! JSON output! cmake! logger! gzopen() / gz*()

Slide 43

Slide 43 text

flowstatd-frontend 圖像化才有感，偏偏我 UI 實作能⼒力很差
 ⼀一年半載又過去了! Open flash chart -> Google Chart API! PHP -> Rails (純練習）

Slide 44

Slide 44 text

Release 每每回頭來看，⼀一點都不滿意，但是，好像是 該讓他出⾨門的時候了! global variable! clean code! Many TODO

Slide 45

Slide 45 text

fork me please https://github.com/Kudo/flowstatd! https://github.com/Kudo/flowstatd-frontend

Slide 46

Slide 46 text

因為有限制
 才得以出類拔萃 不只是設計系統，run startup 也是

Slide 47

Slide 47 text

Image source: https://www.flickr.com/photos/vernhart/1574355240/