Upgrade to Pro — share decks privately, control downloads, hide ads and more …

自動化監控伺服器工具 - Gatus

Bo-Yi Wu
July 26, 2022

自動化監控伺服器工具 - Gatus

1. Why not Prometheus Alert manager, CloudWatch, or event Splunk?
2. 為什麼我選擇 Gatus?
3. Gatus 運作流程
4. Gatus 系統架構
5. Scalability (Distributed Approach)

Bo-Yi Wu

July 26, 2022
Tweet

More Decks by Bo-Yi Wu

Other Decks in Technology

Transcript

  1. ⾃動化監控伺服器
    ⼯具 - Gatus
    2022/07/26


    Bo-Yi Wu

    View full-size slide

  2. About me
    • Software Engineer in Mediatek (AIDE)


    • Member of Drone CI/CD Platform


    • Member of Gitea Platform


    • Member of Gin Golang Framework


    • Maintain Some GitHub Actions Plugins.

    View full-size slide

  3. Why not


    Prometheus Alert
    manager, CloudWatch
    or event Splunk?

    View full-size slide

  4. 內部監控指標


    主要依賴現有流量


    客⼾遇到問題


    團隊才會收到通知

    View full-size slide

  5. 假設 Load Balancer 掛掉


    或付款流程中間出現錯誤


    團隊是否會收到通知?

    View full-size slide

  6. 該如何在客⼾發現錯誤之前


    就提前知道,並且修正完畢

    View full-size slide

  7. 如何實現
    • 建立系統服務狀態⾴⾯


    • 主動監控系統狀態⼯具

    View full-size slide

  8. 建立系統服務狀態介⾯

    View full-size slide

  9. IUUQTXXXHJUIVCTUBUVTDPN
    (JU)VCܥ౷෰຿㐫ଶ

    View full-size slide

  10. 主動監控系統狀態⼯具

    View full-size slide

  11. 客製化監控協定
    • HTTP


    • DNS


    • ICMP/PING


    • TCP

    View full-size slide

  12. 為什麼我選擇


    Gatus
    IUUQTHJUIVCDPNJWCFHBXFTPNFTUBUVTQBHFT

    View full-size slide

  13. Gatus 優勢
    • 監控條件 (客製化回應)


    • 開源專案 (Go 語⾔)


    • 簡易⾴⾯ (Status Page)

    View full-size slide

  14. 監控條件

    View full-size slide

  15. Conditions
    • [STATUS] == 200


    • [STATUS] < 300


    • [RESPONSE_TIME] < 500


    • [BODY].user.name == John


    • len([BODY].data) < 10

    View full-size slide

  16. ⽀援監控


    GraphQL

    View full-size slide

  17. 客製化監控協定
    • HTTP (REST API, GraphQL)


    • ICMP (ping)


    • DNS (A, AAAA, CNAME, MX, NS)


    • TCP (Database)


    • TLS (LDAP, HTTPS, mail servers)


    • STARTTLS (mail servers)

    View full-size slide

  18. ⽀援多種 Alert 機制
    • Discord


    • Email


    • Google Chat


    • Matrix


    • Mattermost


    • Slack
    • Teams


    • Telegram


    • Twilio


    • PagerDuty


    • Opsgenie


    • Custom

    View full-size slide

  19. 簡易


    監控⾴⾯

    View full-size slide

  20. 開源專案


    使⽤ Go 語⾔

    View full-size slide

  21. $POUSJCVUFUPPQFOTPVSDFQSPKFU

    View full-size slide

  22. Gatus


    系統架構


    流程圖

    View full-size slide

  23. endpoints:
    - name: monitoring
    group: internal
    url: "https://example.org/"
    interval: 5m
    conditions:
    - "[STATUS] == 200"
    - name: example-dns-query
    url: "1.1.1.1"
    interval: 5m
    dns:
    query-name: "example.com"
    query-type: "A"
    conditions:
    - "[BODY] == 93.184.216.34"
    - "[DNS_RCODE] == NOERROR"
    - name: icmp-ping
    url: "icmp://example.org"
    interval: 1m
    conditions:
    - "[CONNECTED] == true"
    .FNPSZ
    42-JUF
    1PTUHSFT
    ৽⃧ࢿྉ

    View full-size slide

  24. endpoints:
    - name: monitoring
    group: internal
    url: "https://example.org/"
    interval: 5m
    conditions:
    - "[STATUS] == 200"
    - name: example-dns-query
    url: "1.1.1.1"
    interval: 5m
    dns:
    query-name: "example.com"
    query-type: "A"
    conditions:
    - "[BODY] == 93.184.216.34"
    - "[DNS_RCODE] == NOERROR"
    .FNPSZ
    42-JUF
    1PTUHSFT
    Ҡআࢿྉ

    View full-size slide

  25. err := store.Initialize(cfg.Storage)
    if err != nil {
    panic(err)
    }
    var keys []string
    for _, endpoint := range cfg.Endpoints {
    keys = append(keys, endpoint.Key())
    }
    numberOfDeleted := store.Get().DeleteAllEndpointStatusesNotInKeys(keys)
    if numberOfDeleted > 0 {
    }

    View full-size slide

  26. WatchDog


    流程

    View full-size slide

  27. endpoints:
    - name: front-end
    group: core
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
    - "[STATUS] == 200"
    - "[BODY].status == UP"
    - "[RESPONSE_TIME] < 150"
    - name: back-end
    group: core
    url: "https://example.org/"
    interval: 5m
    conditions:
    - "[STATUS] == 200"
    - "[CERTIFICATE_EXPIRATION] > 48h"

    View full-size slide

  28. endpoints:
    - name: front-end
    group: core
    url: "https://twin.sh/health"
    interval: 5m
    conditions:
    - "[STATUS] == 200"
    - "[BODY].status == UP"
    - "[RESPONSE_TIME] < 150"
    - name: back-end
    group: core
    url: "https://example.org/"
    interval: 5m
    conditions:
    - "[STATUS] == 200"
    - "[CERTIFICATE_EXPIRATION] > 48h"

    View full-size slide

  29. for _, endpoint := range cfg.Endpoints {
    if endpoint.IsEnabled() {
    go monitor(endpoint, cfg.Alerting, cfg.Maintenance, …)
    }
    }
    ᩇऔॴ༗&OEQPJOUࢿྉ
    എܠ႔ཧࢿྉ

    View full-size slide

  30. // Run it immediately on start
    execute(endpoint, alertingConfig …)
    // Loop for the next executions
    for {
    select {
    case <-ctx.Done():
    return
    case <-time.After(endpoint.Interval):
    execute(endpoint, alertingConfig …)
    }
    }

    View full-size slide

  31. 如何避免服務


    初始啟動同時


    發出⼤量請求

    View full-size slide

  32. for _, endpoint := range cfg.Endpoints {
    if endpoint.IsEnabled() {
    time.Sleep(777 * time.Millisecond)
    go monitor(endpoint, cfg.Alerting, cfg.Maintenance, …)
    }
    }

    View full-size slide

  33. 經過⼀段時間


    有⼀定的機率


    會在同時間發送請求


    ⽽影響回應時間


    (Response Time)

    View full-size slide

  34. Using


    sync.Mutex

    View full-size slide

  35. Endpoint A


    調整 Interval 時間為


    5min

    View full-size slide

  36. When to disable


    Monitoring Lock

    View full-size slide

  37. 取消 Lock 機制
    • 壓⼒測試


    • 有⼤量的 Endpoints 需要監控


    • 多個 Endpoints 的 intervals < 5s

    View full-size slide

  38. Gatus


    系統架構

    View full-size slide

  39. var router http.Handler = handler.CreateRouter(ui.StaticFolder, securityConfig …)
    server = &http.Server{
    Addr: fmt.Sprintf("%s:%d", webConfig.Address, webConfig.Port),
    Handler: router,
    ReadTimeout: 15 * time.Second,
    WriteTimeout: 15 * time.Second,
    IdleTimeout: 15 * time.Second,
    }
    log.Println("[controller][Handle] Listening on " + webConfig.SocketAddress())
    log.Println("[controller][Handle]", server.ListenAndServe())

    View full-size slide

  40. for {
    time.Sleep(30 * time.Second)
    if cfg.HasLoadedConfigurationFileBeenModified() {
    stop()
    time.Sleep(time.Second)
    save()
    updatedConfig, err := loadConfiguration()
    if err != nil {
    if cfg.SkipInvalidConfigUpdate {
    cfg.UpdateLastFileModTime()
    continue
    } else {
    panic(err)
    }
    }
    initializeStorage(updatedConfig)
    start(updatedConfig)
    return
    }
    }



    View full-size slide

  41. 重新啟動


    Graceful Shutdown

    View full-size slide

  42. if server != nil {
    _ = server.Shutdown(context.TODO())
    server = nil
    }
    4IVUEPXO8FC4FSWJDF

    View full-size slide

  43. ctx, cancel = context.WithCancel(context.Background())
    for _, endpoint := range cfg.Endpoints {
    if endpoint.IsEnabled() {
    time.Sleep(777 * time.Millisecond)
    go monitor(endpoint, cfg.Alerting, ctx)
    }
    }
    4IVUEPXO8BUDI%PH

    View full-size slide

  44. // Loop for the next executions
    for {
    select {
    case <-ctx.Done():
    return
    case <-time.After(endpoint.Interval):
    execute(endpoint, alertingConfig …)
    }
    }
    4IVUEPXO8BUDI%PH

    View full-size slide

  45. Scalability


    Distributed Approach
    IUUQTHJUIVCDPN5XJ/HBUVTJTTVFT

    View full-size slide

  46. IUUQTHJUIVCDPN5XJ/HBUVTJTTVFT

    View full-size slide

  47. 4FSWFS"HFOUQSPQPTBM

    View full-size slide



  48. NBLFSFRVFTU
    BDDFQUSFRVFTU
    4FSWFS"HFOUQSPQPTBM

    View full-size slide

  49. 4FSWFS"HFOUQSPQPTBM
    TFOENFUSJDEBUB

    View full-size slide

  50. ֬อ㑌୆"HFOUෆ။፤౸ॏෳ&OEQPJOU
    ᙛ&OEQPJOU༗ᏓԽ࣌ɼ೗Կ௨஌"HFOU᮫ด
    ႔ཧ4FSWFSٴ"HFOUHSBDFGVMTIVUEPXOػ੍
    4FSWFS"HFOUQSPQPTBM

    View full-size slide

  51. Gatus


    Online Version


    (為什麼我要付錢?)

    View full-size slide

  52. 付錢理由?
    • 開源版本只能透過 YAML 設定


    • 開源版本需要有⼈管理主機


    • 假如您的基礎設施遇到問題,那 Gatus 也起不
    了任何作⽤了。


    • 給我錢,我會花更多時間在 Gatus

    View full-size slide