Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
flowstatd
Search
Kudo Chien
July 12, 2014
Programming
1
320
flowstatd
Kudo Chien
July 12, 2014
Tweet
Share
More Decks by Kudo Chien
See All by Kudo Chien
阿迪仔, 你為什麼不寫 Unit Test @ TestCorner#14
kudochien
1
130
程式設計師的自我修養
kudochien
3
480
工程師的生涯規劃,從 React Native 開始
kudochien
11
1.2k
React Ecosystem
kudochien
0
180
如果下半輩子只想 DEBUG 怎麼辦?
kudochien
12
2.6k
如果下半輩子只想成功怎麼辦?
kudochien
2
270
Other Decks in Programming
See All in Programming
Piniaの現状と今後
waka292
5
1.5k
飲食業界向けマルチプロダクトを実現させる開発体制とリアルな現状
hiroya0601
1
400
Vue3の一歩踏み込んだパフォーマンスチューニング2024
hal_spidernight
3
3.1k
Go言語でターミナルフレンドリーなAIコマンド、afaを作った/fukuokago20_afa
monochromegane
2
140
Kubernetes for Data Engineers: Building Scalable, Reliable Data Pipelines
sucitw
1
200
Jakarta Concurrencyによる並行処理プログラミングの始め方 (JJUG CCC 2024 Fall)
tnagao7
1
230
Vue SFCのtemplateでTypeScriptの型を活用しよう
tsukkee
3
1.5k
Why Spring Matters to Jakarta EE - and Vice Versa
ivargrimstad
0
1k
Sidekiqで実現する 長時間非同期処理の中断と再開 / Pausing and Resuming Long-Running Asynchronous Jobs with Sidekiq
hypermkt
6
2.7k
弊社の「意識チョット低いアーキテクチャ」10選
texmeijin
5
23k
CSC305 Lecture 13
javiergs
PRO
0
130
PLoP 2024: The evolution of the microservice architecture pattern language
cer
PRO
0
1.7k
Featured
See All Featured
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
37
1.8k
Done Done
chrislema
181
16k
Side Projects
sachag
452
42k
A better future with KSS
kneath
238
17k
Practical Orchestrator
shlominoach
186
10k
The Art of Programming - Codeland 2020
erikaheidi
51
13k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
32
1.8k
GraphQLとの向き合い方2022年版
quramy
43
13k
Designing for humans not robots
tammielis
249
25k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
26
2.1k
4 Signs Your Business is Dying
shpigford
180
21k
Transcript
flowstatd - 那五年 Kudo Chien
Kudo Chien CCUCSIE 2002-2008 BS+MS (GAIS Lab)! CNA ! 曾⼯工作於
Trend Micro、︑Waveface! 現任 biideal CTO
Kudo Chien 打雜! UN*X system programming! Windows programming! Network programming!
Cloud/Web backend! Web frontend! Browser extension development! DevOps! Hacking! iOS! Android! Debug
在 biideal 我們沒有辦不到的事 *誤*
flowstatd 是?
這樣的系統需要什麼樣的機器來跑 Image source: https://www.flickr.com/photos/horiavarlan/4273913966
memory/disk 使⽤用量多⼤大 Image source: https://www.flickr.com/photos/horiavarlan/4273913966
The difference between genius and stupidity is that genius has
its limits.
因為有限制 才得以出類拔萃
Netflow introduction From Cisco! Analyze traffic! SRC/DST IP! SRC/DST Port!
TOC! IP Protocol
宿網流量統計 v1 來⾃自交⼤大 open source 的版本! flow-tools + Perl script!
每⼩小時”重頭”算⼀一次統計! 是網管的災難,使⽤用者的福⾳音
宿網流量統計 v2 由⽉月光⼩小俠 Eintisy 學長⽤用 PHP 重寫的版本! “累加”流量解決了第⼀一版的問題! 慢慢還是撐不住全校的流量! 兩⼩小時跑⼀一次,網路速度越來越快,兩⼩小時可
以衝很多 GB
年少輕狂的 MySQL 時代 不管 3721,往 MySQL 丟就對了 *誤*! MySQL ⼤大神會幫你管理⼀一切事務!
Malicious Detection
年少輕狂的 MySQL 時代 以 CCU 全校流量來說,倒進 MySQL 平均每 ⼩小時佔⽤用 Disk
1xx MB
吳昇⽼老師的教誨 Data Structure! 對資料本質的掌握與計算! Hash Hash Hash
重視統計流量的本質 累加流量! IP address hash table - ⼀一個蘿蔔⼀一個坑
那五年 2007~2012 2009 才開始在這個 project ⽤用 git *冏*
None
統計全宿網⼀一天的流量只需要 3.1 MB
全中正 Class B 的流量只需要 25.7 MB
Flow daemon! All in memory! Real time
Hash function v1
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Hash function v2
Architecture v1 collector! process! listen port 1025 query! process!
named pipe! socket shm Command! topN! over 5G
query! process! named pipe! socket @WanCW <(_ _)>
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Architecture v2 Single process multiplexing! kqueue / select UDP! collector
port TCP! command port 那⼀一年,我還不知道 libevent / libev
Object Oriented Programming Trained from Trend Micro ! 常⾒見的好習慣是把 shared
code 拆成 functions OOP 則更進⼀一步把 shared behaviors 詮釋成 共同的 interfaces! 以上是本⼈人不負責任亂掰的說法 *誤*
–Butler Lampson “All problems in computer science can be solved
by another level of indirection”
Object Oriented C Abstract + struct + function pointer! select()/kqueue
multiplexer! Netflow v5/v9 handlers
None
None
Usage in caller is simple
Over design 之 container_of 直接 cast 就好啦 冏
Multiple subnets 全校⽤用⼀一個 hash table 相對 簡單! 宿網 30 個
subnets 反⽽而麻煩 subnet 1! hash table subnet 2! hash table subnet N! hash table Binary Search
Netflow version 9 template! multiple source + multiple source id
Image source: http://www.lancope.com/blog/netflow-v5-vs-netflow-v9/
Netflow version 9 多個 source ip 下⾯面又可以有多個 source id! 又想⽤用
hash 又不想⽤用太多 memory! two hash tables! source table! template table
Hash from two factors source! table template! table template! table
template! table source! table template table
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Netflow version 9 debugging 時好時壞的 bug 真難抓! tcpdump -> pcap!
對照組! pcap -> wireshark! pcap -> tcpreplay
Misc JSON input command! JSON output! cmake! logger! gzopen() /
gz*()
flowstatd-frontend 圖像化才有感,偏偏我 UI 實作能⼒力很差 ⼀一年半載又過去了! Open flash chart -> Google
Chart API! PHP -> Rails (純練習)
Release 每每回頭來看,⼀一點都不滿意,但是,好像是 該讓他出⾨門的時候了! global variable! clean code! Many TODO
fork me please https://github.com/Kudo/flowstatd! https://github.com/Kudo/flowstatd-frontend
因為有限制 才得以出類拔萃 不只是設計系統,run startup 也是
Image source: https://www.flickr.com/photos/vernhart/1574355240/