Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
flowstatd
Search
Kudo Chien
July 12, 2014
Programming
1
320
flowstatd
Kudo Chien
July 12, 2014
Tweet
Share
More Decks by Kudo Chien
See All by Kudo Chien
阿迪仔, 你為什麼不寫 Unit Test @ TestCorner#14
kudochien
1
130
程式設計師的自我修養
kudochien
3
480
工程師的生涯規劃,從 React Native 開始
kudochien
11
1.2k
React Ecosystem
kudochien
0
180
如果下半輩子只想 DEBUG 怎麼辦?
kudochien
12
2.6k
如果下半輩子只想成功怎麼辦?
kudochien
2
270
Other Decks in Programming
See All in Programming
ActiveSupport::Notifications supporting instrumentation of Rails apps with OpenTelemetry
ymtdzzz
1
230
CSC509 Lecture 09
javiergs
PRO
0
140
ECS Service Connectのこれまでのアップデートと今後のRoadmapを見てみる
tkikuc
2
250
TypeScriptでライブラリとの依存を限定的にする方法
tutinoko
3
690
聞き手から登壇者へ: RubyKaigi2024 LTでの初挑戦が 教えてくれた、可能性の星
mikik0
1
130
Tauriでネイティブアプリを作りたい
tsucchinoko
0
370
Jakarta EE meets AI
ivargrimstad
0
190
型付き API リクエストを実現するいくつかの手法とその選択 / Typed API Request
euxn23
8
2.2k
距離関数を極める! / SESSIONS 2024
gam0022
0
280
Compose 1.7のTextFieldはPOBox Plusで日本語変換できない
tomoya0x00
0
190
LLM生成文章の精度評価自動化とプロンプトチューニングの効率化について
layerx
PRO
2
190
TypeScript Graph でコードレビューの心理的障壁を乗り越える
ysk8hori
2
1.1k
Featured
See All Featured
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
126
18k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
250
21k
Building Applications with DynamoDB
mza
90
6.1k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
27
840
Intergalactic Javascript Robots from Outer Space
tanoku
269
27k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
109
49k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
33k
Fontdeck: Realign not Redesign
paulrobertlloyd
82
5.2k
Practical Orchestrator
shlominoach
186
10k
How GitHub (no longer) Works
holman
310
140k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
246
1.3M
Large-scale JavaScript Application Architecture
addyosmani
510
110k
Transcript
flowstatd - 那五年 Kudo Chien
Kudo Chien CCUCSIE 2002-2008 BS+MS (GAIS Lab)! CNA ! 曾⼯工作於
Trend Micro、︑Waveface! 現任 biideal CTO
Kudo Chien 打雜! UN*X system programming! Windows programming! Network programming!
Cloud/Web backend! Web frontend! Browser extension development! DevOps! Hacking! iOS! Android! Debug
在 biideal 我們沒有辦不到的事 *誤*
flowstatd 是?
這樣的系統需要什麼樣的機器來跑 Image source: https://www.flickr.com/photos/horiavarlan/4273913966
memory/disk 使⽤用量多⼤大 Image source: https://www.flickr.com/photos/horiavarlan/4273913966
The difference between genius and stupidity is that genius has
its limits.
因為有限制 才得以出類拔萃
Netflow introduction From Cisco! Analyze traffic! SRC/DST IP! SRC/DST Port!
TOC! IP Protocol
宿網流量統計 v1 來⾃自交⼤大 open source 的版本! flow-tools + Perl script!
每⼩小時”重頭”算⼀一次統計! 是網管的災難,使⽤用者的福⾳音
宿網流量統計 v2 由⽉月光⼩小俠 Eintisy 學長⽤用 PHP 重寫的版本! “累加”流量解決了第⼀一版的問題! 慢慢還是撐不住全校的流量! 兩⼩小時跑⼀一次,網路速度越來越快,兩⼩小時可
以衝很多 GB
年少輕狂的 MySQL 時代 不管 3721,往 MySQL 丟就對了 *誤*! MySQL ⼤大神會幫你管理⼀一切事務!
Malicious Detection
年少輕狂的 MySQL 時代 以 CCU 全校流量來說,倒進 MySQL 平均每 ⼩小時佔⽤用 Disk
1xx MB
吳昇⽼老師的教誨 Data Structure! 對資料本質的掌握與計算! Hash Hash Hash
重視統計流量的本質 累加流量! IP address hash table - ⼀一個蘿蔔⼀一個坑
那五年 2007~2012 2009 才開始在這個 project ⽤用 git *冏*
None
統計全宿網⼀一天的流量只需要 3.1 MB
全中正 Class B 的流量只需要 25.7 MB
Flow daemon! All in memory! Real time
Hash function v1
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Hash function v2
Architecture v1 collector! process! listen port 1025 query! process!
named pipe! socket shm Command! topN! over 5G
query! process! named pipe! socket @WanCW <(_ _)>
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Architecture v2 Single process multiplexing! kqueue / select UDP! collector
port TCP! command port 那⼀一年,我還不知道 libevent / libev
Object Oriented Programming Trained from Trend Micro ! 常⾒見的好習慣是把 shared
code 拆成 functions OOP 則更進⼀一步把 shared behaviors 詮釋成 共同的 interfaces! 以上是本⼈人不負責任亂掰的說法 *誤*
–Butler Lampson “All problems in computer science can be solved
by another level of indirection”
Object Oriented C Abstract + struct + function pointer! select()/kqueue
multiplexer! Netflow v5/v9 handlers
None
None
Usage in caller is simple
Over design 之 container_of 直接 cast 就好啦 冏
Multiple subnets 全校⽤用⼀一個 hash table 相對 簡單! 宿網 30 個
subnets 反⽽而麻煩 subnet 1! hash table subnet 2! hash table subnet N! hash table Binary Search
Netflow version 9 template! multiple source + multiple source id
Image source: http://www.lancope.com/blog/netflow-v5-vs-netflow-v9/
Netflow version 9 多個 source ip 下⾯面又可以有多個 source id! 又想⽤用
hash 又不想⽤用太多 memory! two hash tables! source table! template table
Hash from two factors source! table template! table template! table
template! table source! table template table
Over Design Image source: https://www.flickr.com/photos/sixybeast/8690039773/
Netflow version 9 debugging 時好時壞的 bug 真難抓! tcpdump -> pcap!
對照組! pcap -> wireshark! pcap -> tcpreplay
Misc JSON input command! JSON output! cmake! logger! gzopen() /
gz*()
flowstatd-frontend 圖像化才有感,偏偏我 UI 實作能⼒力很差 ⼀一年半載又過去了! Open flash chart -> Google
Chart API! PHP -> Rails (純練習)
Release 每每回頭來看,⼀一點都不滿意,但是,好像是 該讓他出⾨門的時候了! global variable! clean code! Many TODO
fork me please https://github.com/Kudo/flowstatd! https://github.com/Kudo/flowstatd-frontend
因為有限制 才得以出類拔萃 不只是設計系統,run startup 也是
Image source: https://www.flickr.com/photos/vernhart/1574355240/