How to monitor Cosmos validator by Prometheus

Meetup video: https://youtu.be/iT2gtgx3H0A?t=300
Slide source: https://docs.google.com/presentation/d/1TJR-2RnnRCrO0uEMvEwwXJmPgoM15DIjx_MAo9zeUQM/edit?usp=sharing

Transcript

1. Monitor Cosmos SDK chain Rico Chen, DevOps Engineer stakefish
2. About me 2 • Translated Google - The Site Reliability Workbook for Taiwan community • DevOps Taiwan volunteer • AWS Taiwan User Group volunteer • Starbugs Weekly DevOps recommend article writer • Gamer • Disney Frozen lover
3. Agenda 1. Blockchain Basic Mechanism 2. Cosmos-SDK Basic Mechanism 3. How to monitor? 4. How to monitor remotely? 5. Our architecture 6. Reference 3
4. 4 Blockchain Basic Mechanism
5. Common usage 5
6. Blockchain layers 6 • Networking layer - Makes sure that each node receives transactions • Consensus layer - Makes sure that each node agrees on the same transactions to modify their local state ◦ PoW (Proof of work) ◦ PoS (Proof of stake) • Application layer - Input a transaction and a state will return a new state ◦ Bitcoin - Account balance ◦ Ethereum - EVM ◦ Chainlink - Provide external data Proof of Work (PoW) vs. Proof of Stake (PoS)
7. Cosmos SDK • The goal of the Cosmos SDK is to allow developers to easily create custom blockchains from scratch that can natively interoperate with other blockchains • Blockchains built with the Cosmos SDK are generally referred to as application-specific blockchains ◦ ICO ◦ DeFi ◦ GameFi • Golang programming language 7 gopher!!
8. Cosmos SDK chains • Cosmos Hub • Osmosis • Juno 8
9. 9 Cosmos-SDK Basic Mechanism
10. Cosmos-SDK layers 10
11. Cosmos-SDK usage example 11 • Cosmos-SDK includes Networking, Consensus and covers some Application layers. ◦ Account, balance, governance…etc. • Developers only need to focus on business logic. ◦ For example Defi, GameFi, ICO...etc
12. 12 How to monitor?
13. Prometheus • Prometheus is the Cosmos-SDK monitoring solution • We can leverage on Prometheus Cloud Native ecosystem ◦ node-exporter and process-exporter ◦ Grafana ◦ Alertmanager ◦ Kubernetes 13
14. Cosmos-SDK layers 14
15. Tendermint metrics • Make sure our nodes keep syncing • The P2P count matches our expectations • Does our validator miss the blocks? • We only set up alert rules in these metrics • The P2P count matches our expectations ◦ Sentry P2P peers > 20 ◦ Validator P2P peers > 2 15 Tendermint metrics doc
16. Cosmos-SDK build-in feat metrics • These metrics focus on the application layer. Most of the time, only developers would observe metrics on the dashboards 16 Cosmos-SDK metrics
17. Custom metrics • These metrics focus on the application layer. Most of the time, only developers would observe metrics on the dashboards • Not all the Cosmos-SDK chains create Prometheus metrics for their custom features ◦ For example: Agoric custom metrics 17
18. 18 Description, note, or comment (no more than 1 row) How to monitor remotely?
19. Telemetry monitoring 19
20. Telemetry monitoring • The mechanism is similar to the blockchain scanner website • No need to access the production environment • Be careful with the huge request by exporter itself ◦ To reach the public data node rate limit ◦ DoS (denial-of-service) your own data node 20
21. Prometheus exporters • solarlabsteam/cosmos-exporter ◦ Pros: ▪ Basically, it can use on all Cosmos-SDK chain ▪ Included consensus and application layer metrics ◦ Cons: ▪ Need gRPC and Tendermint RPC (Most of the public nodes don't support gRPC) 21
22. 22 Our architecture
23. Monitoring architecture 23
24. Dashboards 24
25. Dashboards 25
26. Dashboards 26
27. 27 Reference
28. Reference • Understanding the value proposition of Cosmos • Tendermint document • Cosmos SDK document 28
29. stakefish Twitter If you would like to learn more about stakefish or staking in general, please don’t hesitate to reach out.