This paper aims to get you quickly started with different kinds of observability you might need to work within the cloud-native world. 既に取り組んでいる企業で得た知見をもとに、 各Signalsの収集方法と使い方がまとめられています • Observabilityに取り組めるようになるためのドキュメント
as the "primary signals" instead of "three pillars" for two reasons • 今はPrimary Signalsと呼びます – Three pillars は 2019年頃に KubeConのKeynoteなどで使われていました • 理由1 – 柱だと3つ全て揃ってないとダメなイメージがあるが 用途によっては1つだけで大丈夫なケースもある • 理由2 – ProfilesやDumpsなどが対象から漏れることが多 い
# TYPE http_requests_total counter http_requests_total{method="post",code="200"} 1027 1395066363000 http_requests_total{method="post",code="400"} 3 1395066363000 metrics name value label • OpenMetrics – 右図のような形式のデータを収集する 参考 : Prometheus and OpenMetrics, with Richard Hartmann < https://kubernetespodcast.com/episode/037-prometheus-and-openmetrics/ > ADAM GLICK: Given all the work that you're doing with Prometheus, how did that lead you into creating the OpenMetrics Project? RICHARD HARTMANN: Politics. It's really hard for other projects, and especially for other companies, to support something with a different name on it. Even though Prometheus itself doesn't have a profit motive, so we don't have to have sales or anything, it was hard for others to accept stuff which is named after Prometheus in their own product. And this is basically why I decided to do that.
SuccessfulCreate ReplicaSet/kube-state-metrics-69fc8b7bb7 Created pod: kube-state-metrics-69fc8b7bb7-8n89q 109s Normal Scheduled Pod/kube-state-metrics-69fc8b7bb7-8n89q Successfully assigned kube-system/kube-state-metrics-69fc8b7bb7-8n89q to demo-yosshi-123-w-default-a5f36c1c-ldssp 109s Normal Started Pod/kube-state-metrics-69fc8b7bb7-8n89q Started container kube-state-metrics 109s Normal Created Pod/kube-state-metrics-69fc8b7bb7-8n89q Created container kube-state-metrics 109s Normal Pulled Pod/kube-state-metrics-69fc8b7bb7-8n89q Container image "k8s.gcr.io/kube-state-metrics-kube-state-metrics:v2.4.1" already present on machine • kubectl alpha events/ get events 等のコマンドで取得 – eventが発生してから1時間以内(デフォルト値)のイベントが取得できる – デバック等の時に必要なログなのでeventrouterでContainer Logsに出力しておくケースが多い
# Don't generate audit events for all requests in RequestReceived stage. omitStages: - "RequestReceived" rules: # Log pod changes at RequestResponse level - level: RequestResponse resources: - group: "" # Resource "pods" doesn't match requests to any subresource of pods, # which is consistent with the RBAC policy. resources: ["pods"] – 以下のようなaudit-policy.yamlを設定することで指定したKubernetesの操作ログを取得できる – LogはAPI-Serverのhostpath上に出力されるので、Logging agentで収集することで永続化可能