Kubernetes Operators Principle and Practice

with Jesus Carillo, TicketMaster - Cloud Native Conf - Berlin, Germany - http://sched.co/9Td4

Transcript

1. Kubernetes Operators: Managing Complex Software with Software Josh Wood DocOps at CoreOS @joshixisjosh9 Jesus Carrillo Sr Systems Engineer at Ticketmaster
2. Overview Scaling Stateless Apps
3. $ kubectl scale replicas=3 ... ReplicaSet
4. ReplicaSet
5. ReplicaSet
7. Overview What about apps that…store data
8. $ kubectl run db --image=quay.io/my/db Creating a Database is Easy
9. • Resize/Upgrade - coordination for availability • Reconfigure - tedious generation / templating • Backup - requires coordination on instances • Healing - restore backups, rejoin Managing a Distributed Database is Harder
10. If only k8s knew...: Extend Kubernetes
11. $ cat my-db-cluster.yaml spec: clusterSize: 3 readReplicas: 2 version: v4.0.1 The Goal
12. Introducing Operators
15. etcd Overview • Distributed key-value store • Primary datastore of Kubernetes • Auto-leader election for availability
16. Operator Construction • Operators build on Kubernetes concepts • Resources: who what where ; desired state • Controllers: Observe, Analyze, Act to reconcile resources
17. Third Party Resources • TPRs extend the Kubernetes API with new API object types • Akin to a database table’s schema - the data model • Designed with custom automation mechanisms in mind • https://kubernetes.io/docs/user-guide/thirdpartyresources/
18. $ kubectl create -f https://coreos.com/operators/etcd/latest/deployment.yaml etcd Operator
19. $ cat deployment.yaml spec: clusterSize: 3 version: v3.1.0 etcd Operator Resource
20. etcd Operator
21. etcd Operator Open-source github.com/coreos/etcd-operator
22. Kubernetes self-hosting etcd Easy HA Setups on Kubernetes (Tectonic 1.5.5-t.2) Automated backup to object store etcd Operator - Current Work
23. Prometheus Operator • Operates Prometheus on k8s • Handles common tasks: ◦ Create/Destroy ◦ Monitor Configuration ◦ Services Targets via Labels • Configured by resources
24. Prometheus Operator Open-source github.com/coreos/prometheus-operator
25. • Read more at coreos.com/blog • Test and Extend the open source Operators • Build and Discuss other Operators (redis, postgres, MySQL) Next Steps
26. https://coreos.com/blog/introducing-operators.html https://coreos.com/blog/introducing-the-etcd-operator.html https://github.com/coreos/etcd-operator https://coreos.com/blog/the-prometheus-operator.html https://github.com/coreos/prometheus-operator https://github.com/kubernetes/minikube URLs
27. CoreOS runs the world’s containers We’re hiring: careers@coreos.com sales@coreos.com 90+ Projects on GitHub, 1,000+ Contributors coreos.com Support plans, training and more OPEN SOURCE ENTERPRISE
28. coreos.com/fest @coreosfest May 31 - June 1, 2017 San Francisco
29. j@coreos.com @joshixisjosh9 joshix.com QUESTIONS? Thanks! We’re hiring: coreos.com/careers Let’s talk! CoreOS-User google group More events: coreos.com/community LONGER CHAT?
31. Ticketmaster - KubeCon 2016 KUBECON MAR 29-30, 2017 Prometheus @ Ticketmaster
32. How Ticketmaster uses Prometheus? • As we transition to a DevOps model: ◦ Replace OpenTSDB ◦ Replace legacy alerting systems ◦ Each team manages it’s own monitoring and alerting
33. Prometheus POC
34. POC • Closed to: ◦ Teams with already instrumented apps. ◦ Teams in the process of migration to AWS. • Architecture ◦ Prometheus and alertmanager running on EC2 instances. ◦ Shared between teams.
35. POC architecture
36. Problems & lessons learned Problems: • Federation scrape timeouts. • Bad configurations can disrupt the service. • Tweaking the storage parameters takes time. • Network acls. Lessons learned: • Each team should have it’s own prometheus stack. • Divide and conquer.
37. Prometheus As a Service
38. Prometheus as a service • Must: ◦ Allow the teams to quickly provision a dedicated stack. ◦ Don’t represent any additional burden to the teams. ◦ Provide pre configured EC2 and k8s service discovery. ◦ Helm based deployment. • Ticketmaster exporter database ◦ Provides a well known port range for the exporters. ◦ Managed Network ACLs. ◦ Scrape jobs are generated from this list.
39. The Prometheus Operator
40. Prometheus Operator • Allows us to easily model complex configuration. • Storage configuration is auto tuned. • Alertmanager HA by default. Looking forward: • Federation and sharding. • Grafana integration. Company adoption rate: • Everyone loves it!.
41. Prometheus as a service architecture
42. Ticketmaster - KubeCon 2016 Questions? jesus.carrillo@ticketmaster.com