Application monitoring with Prometheus

A talk entitled "Application monitoring with Prometheus" for Prometheus Tokyo Meetup #1

Transcript

1. Application monitoring with Prometheus Takuya Noguchi @tnir / Tw: @tn961ir Prometheus Tokyo Meetup #1 2017‑04‑10
2. @tnir / @tn961ir Takuya Noguchi Software Engineer (Digital Transformation for 20yrs) GitLab contributor Organizer of Kamiyacho.k8s/GitLab Tokyo/Prometheus Tokyo Loves CNCF SRE/DevOps/Dev at iRidge, Inc.
3. Prometheus Casual Talk (2016) # p r o m e t h e u s c a s u a l Casual Talk #1 6/14 (97 partipants) Casual Talk #2 11/10 w/Brian Brazil
4. Prometheus Developed by SoundCloud since 2012 Inspired by Borgmon (Google's internal monitoring) Written in Go Accepted as 2nd project of CNCF (May 2016)
5. Prometheus Architecture pull metrics HDD / SSD Pushgateway Short-lived jobs Jobs / Exporters Storage Retrieval PromQL Prometheus Server Node Service Discovery find targets Prometheus Server Alertmanager push alerts Web UI Grafana API clients PagerDuty Email DNS Kubernetes Consul ... Custom integration notify ... https://prometheus.io/docs/introduction/overview/
6. Installation https://prometheus.io/docs/introduction/install/ d o c k e r r u n - v / t m p / p r o m e t h e u s . y m l : / e t c / p r o m e t h e u s / p r o m e t h e u s . y m - p 9 0 9 0 : 9 0 9 0 p r o m / p r o m e t h e u s
7. Cloud‑native monitoring "Let's go to SaaS!!!" DataDog https://www.quora.com/Is‑DataDog‑any‑good one for t2.micro? (50% monitoring cost) Stackdriver Monitoring Only for GCP/AWS AWS CloudWatch Only for AWS
8. How can we monitor our system? Our apps Fully‑containerized apps (Docker) Deployed on Kubernetes Code‑hosted on GitLab Answer Prometheus
9. GitLab and Prometheus Review Apps (inspired by Heroku) Each PR (MR) can be reviewed in CI environments We can see performance degradation in each commit.
10. Architecture GitLab ‑‑> Prometheus ‑‑> Kubernetes SD k u b e r n e t e s _ s d _ c o n f i g ‑‑> Kubernetes ‑‑> containers
11. node_exporter https://github.com/prometheus/node_exporter n o d e _ e x p o r t e r provides machine metrics (OS, CPU, Mem)
12. Custom exporter You can create exporter in any language (Go, Python, Ruby, bash etc.)
13. https://about.gitlab.com/2017/03/22/gitlab‑9‑0‑released/
14. Monitoring applications with Prometheus GitLab now has Prometheus integration. Requirements GitLab (9.0+) Deploy your app on Kubernetes Prometheus must be configured to collect Kubernetes metrics Each metric must be have a label to indicate the environment Network: GitLab must have network connectivity to the Prometheus sever
15. Kubernetes Service Discovery Kubernetes Service Discovery is still in beta Configuration: k u b e r n e t e s _ s d _ c o n f i g s : [ a p i _ s e r v e r : k u b e r n e t e s - c l u s t e r . e x a m p l e . c o m ] - r o l e : k u b e r n e t e s b a s i c _ a u t h : [ u s e r n a m e : " u s e r n a m e " ] [ p a s s w o r d : " p a s s w o r d " ]
16. Prometheus integrated to GitLab https://gitlab.com/gitlab‑org/gitlab‑ce/issues/26910
17. TODOs for production readiness App‑specific monitoring Application Performance Monitoring (APM) KPI monitoring Alertmanager & incident management
18. Summary GitLab: Easier way to monitor your apps on Kubernetes with Prometheus but still under development APM will be archived with Prometheus. Let us know use cases in Tokyo & Japan.