自動化監控伺服器工具 - Gatus

1. Why not Prometheus Alert manager, CloudWatch, or event Splunk?
2. 為什麼我選擇 Gatus?
3. Gatus 運作流程
4. Gatus 系統架構
5. Scalability (Distributed Approach)

Transcript

1. ⾃動化監控伺服器 ⼯具 - Gatus 2022/07/26 Bo-Yi Wu
2. About me • Software Engineer in Mediatek (AIDE) • Member of Drone CI/CD Platform • Member of Gitea Platform • Member of Gin Golang Framework • Maintain Some GitHub Actions Plugins.
3. Why not Prometheus Alert manager, CloudWatch or event Splunk?
4. 內部監控指標 主要依賴現有流量 客⼾遇到問題 團隊才會收到通知
5. 假設 Load Balancer 掛掉 或付款流程中間出現錯誤 團隊是否會收到通知?
6. 該如何在客⼾發現錯誤之前 就提前知道，並且修正完畢
7. 如何實現 • 建立系統服務狀態⾴⾯ • 主動監控系統狀態⼯具
8. 建立系統服務狀態介⾯
9. IUUQTXXXHJUIVCTUBUVTDPN (JU)VC
   ܥ౷෰຿㐫ଶ
11. 主動監控系統狀態⼯具
12. 客製化監控協定 • HTTP • DNS • ICMP/PING • TCP
13. 為什麼我選擇 Gatus IUUQTHJUIVCDPNJWCFHBXFTPNFTUBUVTQBHFT
14. Gatus 優勢 • 監控條件 (客製化回應) • 開源專案 (Go 語⾔) • 簡易⾴⾯ (Status Page)
15. 監控條件
16. Conditions • [STATUS] == 200 • [STATUS] < 300 • [RESPONSE_TIME] < 500 • [BODY].user.name == John • len([BODY].data) < 10
18. ⽀援監控 GraphQL
20. 客製化監控協定 • HTTP (REST API, GraphQL) • ICMP (ping) • DNS (A, AAAA, CNAME, MX, NS) • TCP (Database) • TLS (LDAP, HTTPS, mail servers) • STARTTLS (mail servers)
21. ⽀援多種 Alert 機制 • Discord • Email • Google Chat • Matrix • Mattermost • Slack • Teams • Telegram • Twilio • PagerDuty • Opsgenie • Custom
22. 簡易 監控⾴⾯
25. 開源專案 使⽤ Go 語⾔
26. $POUSJCVUF
    UP
    PQFO
    TPVSDF
    QSPKFU
27. Gatus 系統架構 流程圖
30. endpoints: - name: monitoring group: internal url: "https://example.org/" interval: 5m conditions: - "[STATUS] == 200" - name: example-dns-query url: "1.1.1.1" interval: 5m dns: query-name: "example.com" query-type: "A" conditions: - "[BODY] == 93.184.216.34" - "[DNS_RCODE] == NOERROR" - name: icmp-ping url: "icmp://example.org" interval: 1m conditions: - "[CONNECTED] == true" .FNPSZ 42-JUF 1PTUHSFT ৽⃧ࢿྉ
31. endpoints: - name: monitoring group: internal url: "https://example.org/" interval: 5m conditions: - "[STATUS] == 200" - name: example-dns-query url: "1.1.1.1" interval: 5m dns: query-name: "example.com" query-type: "A" conditions: - "[BODY] == 93.184.216.34" - "[DNS_RCODE] == NOERROR" .FNPSZ 42-JUF 1PTUHSFT Ҡআࢿྉ
32. err := store.Initialize(cfg.Storage) if err != nil { panic(err) } var keys []string for _, endpoint := range cfg.Endpoints { keys = append(keys, endpoint.Key()) } numberOfDeleted := store.Get().DeleteAllEndpointStatusesNotInKeys(keys) if numberOfDeleted > 0 { }
34. WatchDog 流程
35. endpoints: - name: front-end group: core url: "https://twin.sh/health" interval: 5m conditions: - "[STATUS] == 200" - "[BODY].status == UP" - "[RESPONSE_TIME] < 150" - name: back-end group: core url: "https://example.org/" interval: 5m conditions: - "[STATUS] == 200" - "[CERTIFICATE_EXPIRATION] > 48h"
36. endpoints: - name: front-end group: core url: "https://twin.sh/health" interval: 5m conditions: - "[STATUS] == 200" - "[BODY].status == UP" - "[RESPONSE_TIME] < 150" - name: back-end group: core url: "https://example.org/" interval: 5m conditions: - "[STATUS] == 200" - "[CERTIFICATE_EXPIRATION] > 48h"
37. for _, endpoint := range cfg.Endpoints { if endpoint.IsEnabled() { go monitor(endpoint, cfg.Alerting, cfg.Maintenance, …) } } ᩇऔॴ༗
    &OEQPJOU
    ࢿྉ എܠ႔ཧࢿྉ
38. // Run it immediately on start execute(endpoint, alertingConfig …) // Loop for the next executions for { select { case <-ctx.Done(): return case <-time.After(endpoint.Interval): execute(endpoint, alertingConfig …) } }
39. 如何避免服務 初始啟動同時 發出⼤量請求
40. for _, endpoint := range cfg.Endpoints { if endpoint.IsEnabled() { time.Sleep(777 * time.Millisecond) go monitor(endpoint, cfg.Alerting, cfg.Maintenance, …) } }
41. 經過⼀段時間 有⼀定的機率 會在同時間發送請求 ⽽影響回應時間 (Response Time)
42. Using sync.Mutex
44. Endpoint A 調整 Interval 時間為 5min
45. When to disable Monitoring Lock
46. 取消 Lock 機制 • 壓⼒測試 • 有⼤量的 Endpoints 需要監控 • 多個 Endpoints 的 intervals < 5s
47. Gatus 系統架構
49. var router http.Handler = handler.CreateRouter(ui.StaticFolder, securityConfig …) server = &http.Server{ Addr: fmt.Sprintf("%s:%d", webConfig.Address, webConfig.Port), Handler: router, ReadTimeout: 15 * time.Second, WriteTimeout: 15 * time.Second, IdleTimeout: 15 * time.Second, } log.Println("[controller][Handle] Listening on " + webConfig.SocketAddress()) log.Println("[controller][Handle]", server.ListenAndServe())
51. for { time.Sleep(30 * time.Second) if cfg.HasLoadedConfigurationFileBeenModified() { stop() time.Sleep(time.Second) save() updatedConfig, err := loadConfiguration() if err != nil { if cfg.SkipInvalidConfigUpdate { cfg.UpdateLastFileModTime() continue } else { panic(err) } } initializeStorage(updatedConfig) start(updatedConfig) return } }   
52.   
53. 重新啟動 Graceful Shutdown
55. if server != nil { _ = server.Shutdown(context.TODO()) server = nil } 4IVUEPXO
    8FC
    4FSWJDF
56. ctx, cancel = context.WithCancel(context.Background()) for _, endpoint := range cfg.Endpoints { if endpoint.IsEnabled() { time.Sleep(777 * time.Millisecond) go monitor(endpoint, cfg.Alerting, ctx) } } 4IVUEPXO
    8BUDI%PH
57. // Loop for the next executions for { select { case <-ctx.Done(): return case <-time.After(endpoint.Interval): execute(endpoint, alertingConfig …) } } 4IVUEPXO
    8BUDI%PH
58. Scalability Distributed Approach IUUQTHJUIVCDPN5XJ/HBUVTJTTVFT
60. IUUQTHJUIVCDPN5XJ/HBUVTJTTVFT
61. 4FSWFS
    
    "HFOU
    QSPQPTBM
62.   NBLF
    SFRVFTU BDDFQU
    SFRVFTU 4FSWFS
    
    "HFOU
    QSPQPTBM
63. 4FSWFS
    
    "HFOU
    QSPQPTBM  TFOE
    NFUSJD
    EBUB
64.  ֬อ㑌୆
    "HFOU
    ෆ။፤౸ॏෳ
    &OEQPJOU  ᙛ
    &OEQPJOU
    ༗ᏓԽ࣌ɼ೗Կ௨஌
    "HFOU
    ᮫ด  ႔ཧ
    4FSWFS
    ٴ
    "HFOU
    HSBDFGVM
    TIVUEPXO
    ػ੍ 4FSWFS
    
    "HFOU
    QSPQPTBM
65. Gatus Online Version (為什麼我要付錢?)
66. 付錢理由? • 開源版本只能透過 YAML 設定 • 開源版本需要有⼈管理主機 • 假如您的基礎設施遇到問題，那 Gatus 也起不 了任何作⽤了。 • 給我錢，我會花更多時間在 Gatus
68. Thanks