Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Consul @ PagerDuty

7090d58d804c96911a37c84e4e90a9cf?s=47 Ranjib Dey
November 17, 2016

Consul @ PagerDuty

How we use consul at PagerDuty across containerized microservices, legacy applications and some custom workflows

7090d58d804c96911a37c84e4e90a9cf?s=128

Ranjib Dey

November 17, 2016
Tweet

Transcript

  1. Consul at PagerDuty Hashicorp User Group-SF, November 16th 2016

  2. • Scope • Deployment Architecture • Use cases • Q

    & A Agenda Hashicorp User Group-SF, November 16th 2016
  3. • SRE@PagerDuty • All things Automation • Chef, Containers (OpenVZ/LXC/docker/rkt/ns

    pawn..) • FOSS Enthusiast • @RanjibDey About me Hashicorp User Group-SF, November 16th 2016
  4. Scope Hashicorp User Group-SF, November 16th 2016

  5. • Introduced early October, 2015 • Agent is present in

    entire infrastructure • Different features used in different parts of the system • Almost 12-15% of the current infra is using consul in one or other way (mostly around containerized workload) • Incremental deployment across mission critical services Hashicorp User Group-SF, November 16th 2016
  6. Deployment Architecture Hashicorp User Group-SF, November 16th 2016

  7. dc-1 dc-2 dc-3 consul-masters consul-masters consul-masters fleet fleet fleet Hashicorp

    User Group-SF, November 16th 2016 consul-replicate consul-replicate
  8. Metrics Hashicorp User Group-SF, November 16th 2016

  9. Hashicorp User Group-SF, November 16th 2016

  10. Use Cases Hashicorp User Group-SF, November 16th 2016

  11. Service discovery via DNS lookup Hashicorp User Group-SF, November 16th

    2016
  12. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th

    2016 health check consumer service host consul-agent service process DNS lookup
  13. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th

    2016 consumer service host consul-agent service process
  14. • Less frequent or one-off communication • Example: Obtaining configuration

    endpoint or vault end point on application startup • Bad for load balancing Hashicorp User Group-SF, November 16th 2016
  15. Service discovery via HTTP API Hashicorp User Group-SF, November 16th

    2016
  16. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th

    2016 health check consumer service host consul-agent service process HTTP API Helper library
  17. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th

    2016 consumer service host consul-agent service process Helper library
  18. • Communication in between backends • Can be of any

    protocol (binary or plain text alike) • Actual performance and user experience will vary among the client libraries • Most dynamic language based libraries are not very good in this. C, Java, go has some decent options, but limited to certain tech stacks Hashicorp User Group-SF, November 16th 2016
  19. Client side load balancing Hashicorp User Group-SF, November 16th 2016

  20. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th

    2016 health check consumer service host consul-template haproxy
  21. producer service host consul-agent consul-master Hashicorp User Group-SF, November 16th

    2016 consumer service host consul-template haproxy producer service host consul-agent producer service host consul-agent
  22. • Communication from app servers to DB • Analytics stack

    (Haddop, Tez) • Almost all low latency, binary protocol based stuff • TCP based communications Hashicorp User Group-SF, November 16th 2016
  23. Load balancing under Nomad Hashicorp User Group-SF, November 16th 2016

  24. Hashicorp User Group-SF, November 16th 2016 Nomad cluster Nginx+ &

    consul-template Nginx+ & consul-template Container-1 Container-1 Container-1 Container-1 Container-1 Container-1 Container-1 Container-1 consul-master Nomad-master
  25. • Front end load balancing • Clients, customers, partners, channels

    to your software end points • HTTP, websocket, spdy • Fancier endpoint or host header based routing Hashicorp User Group-SF, November 16th 2016
  26. Runtime Application Configuration Hashicorp User Group-SF, November 16th 2016

  27. Configuration repository consul-master Hashicorp User Group-SF, November 16th 2016 consumer

    service host consul-template config file Developers PR workflow GoCD git2consul
  28. Network Topology Management Hashicorp User Group-SF, November 16th 2016

  29. dc-1 dc-2 dc-3 consul-master Hashicorp User Group-SF, November 16th 2016

    consul-replicate consul-replicate Entire infra register consul-master consul-master host1 host1 host1 host1 host1 host1 host1 host1 host1 host1 runner runner runner runner runner runner runner runner runner runner
  30. Thank You Hashicorp User Group-SF, November 16th 2016