Consul @ PagerDuty

How we use consul at PagerDuty across containerized microservices, legacy applications and some custom workflows

Transcript

1. Consul at PagerDuty Hashicorp User Group-SF, November 16th 2016
2. • Scope • Deployment Architecture • Use cases • Q & A Agenda Hashicorp User Group-SF, November 16th 2016
3. • SRE@PagerDuty • All things Automation • Chef, Containers (OpenVZ/LXC/docker/rkt/ns pawn..) • FOSS Enthusiast • @RanjibDey About me Hashicorp User Group-SF, November 16th 2016
4. Scope Hashicorp User Group-SF, November 16th 2016
5. • Introduced early October, 2015 • Agent is present in entire infrastructure • Different features used in different parts of the system • Almost 12-15% of the current infra is using consul in one or other way (mostly around containerized workload) • Incremental deployment across mission critical services Hashicorp User Group-SF, November 16th 2016
6. Deployment Architecture Hashicorp User Group-SF, November 16th 2016
7. dc-1 dc-2 dc-3 consul-masters consul-masters consul-masters fleet fleet fleet Hashicorp User Group-SF, November 16th 2016 consul-replicate consul-replicate
8. Metrics Hashicorp User Group-SF, November 16th 2016
9. Hashicorp User Group-SF, November 16th 2016
10. Use Cases Hashicorp User Group-SF, November 16th 2016
11. Service discovery via DNS lookup Hashicorp User Group-SF, November 16th 2016
12. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th 2016 health check consumer service host consul-agent service process DNS lookup
13. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th 2016 consumer service host consul-agent service process
14. • Less frequent or one-off communication • Example: Obtaining configuration endpoint or vault end point on application startup • Bad for load balancing Hashicorp User Group-SF, November 16th 2016
15. Service discovery via HTTP API Hashicorp User Group-SF, November 16th 2016
16. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th 2016 health check consumer service host consul-agent service process HTTP API Helper library
17. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th 2016 consumer service host consul-agent service process Helper library
18. • Communication in between backends • Can be of any protocol (binary or plain text alike) • Actual performance and user experience will vary among the client libraries • Most dynamic language based libraries are not very good in this. C, Java, go has some decent options, but limited to certain tech stacks Hashicorp User Group-SF, November 16th 2016
19. Client side load balancing Hashicorp User Group-SF, November 16th 2016
20. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th 2016 health check consumer service host consul-template haproxy
21. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th 2016 consumer service host consul-template haproxy producer service host consul-agent producer service host consul-agent
22. • Communication from app servers to DB • Analytics stack (Haddop, Tez) • Almost all low latency, binary protocol based stuff • TCP based communications Hashicorp User Group-SF, November 16th 2016
23. Load balancing under Nomad Hashicorp User Group-SF, November 16th 2016
24. Hashicorp User Group-SF, November 16th 2016 Nomad cluster Nginx+ & consul-template Nginx+ & consul-template Container-1 Container-1 Container-1 Container-1 Container-1 Container-1 Container-1 Container-1 consul-master Nomad-master
25. • Front end load balancing • Clients, customers, partners, channels to your software end points • HTTP, websocket, spdy • Fancier endpoint or host header based routing Hashicorp User Group-SF, November 16th 2016
26. Runtime Application Configuration Hashicorp User Group-SF, November 16th 2016
27. Configuration repository consul-master Hashicorp User Group-SF, November 16th 2016 consumer service host consul-template config file Developers PR workflow GoCD git2consul
28. Network Topology Management Hashicorp User Group-SF, November 16th 2016
29. dc-1 dc-2 dc-3 consul-master Hashicorp User Group-SF, November 16th 2016 consul-replicate consul-replicate Entire infra register consul-master consul-master host1 host1 host1 host1 host1 host1 host1 host1 host1 host1 runner runner runner runner runner runner runner runner runner runner
30. Thank You Hashicorp User Group-SF, November 16th 2016