Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Consul @ PagerDuty

Ranjib Dey
November 17, 2016

Consul @ PagerDuty

How we use consul at PagerDuty across containerized microservices, legacy applications and some custom workflows

Ranjib Dey

November 17, 2016
Tweet

More Decks by Ranjib Dey

Other Decks in Programming

Transcript

  1. Consul
    at PagerDuty
    Hashicorp User Group-SF, November 16th 2016

    View Slide


  2. Scope

    Deployment Architecture

    Use cases

    Q & A
    Agenda
    Hashicorp User Group-SF, November 16th 2016

    View Slide


  3. [email protected]

    All things Automation

    Chef, Containers
    (OpenVZ/LXC/docker/rkt/ns
    pawn..)

    FOSS Enthusiast

    @RanjibDey
    About me
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  4. Scope
    Hashicorp User Group-SF, November 16th 2016

    View Slide


  5. Introduced early October, 2015

    Agent is present in entire infrastructure

    Different features used in different parts of the system

    Almost 12-15% of the current infra is using consul in one
    or other way (mostly around containerized workload)

    Incremental deployment across mission critical services
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  6. Deployment Architecture
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  7. dc-1 dc-2 dc-3
    consul-masters
    consul-masters
    consul-masters
    fleet
    fleet
    fleet
    Hashicorp User Group-SF, November 16th 2016
    consul-replicate
    consul-replicate

    View Slide

  8. Metrics
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  9. Hashicorp User Group-SF, November 16th 2016

    View Slide

  10. Use Cases
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  11. Service discovery via DNS lookup
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  12. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    health check
    consumer service host
    consul-agent service process
    DNS lookup

    View Slide

  13. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    consumer service host
    consul-agent service process

    View Slide


  14. Less frequent or one-off communication

    Example: Obtaining configuration endpoint or vault end
    point on application startup

    Bad for load balancing
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  15. Service discovery via HTTP API
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  16. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    health check
    consumer service host
    consul-agent
    service process
    HTTP API
    Helper library

    View Slide

  17. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    consumer service host
    consul-agent
    service process
    Helper library

    View Slide


  18. Communication in between backends

    Can be of any protocol (binary or plain text alike)

    Actual performance and user experience will vary among
    the client libraries

    Most dynamic language based libraries are not very good
    in this. C, Java, go has some decent options, but limited
    to certain tech stacks
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  19. Client side load balancing
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  20. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    health check
    consumer service host
    consul-template haproxy

    View Slide

  21. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    consumer service host
    consul-template haproxy
    producer service host
    consul-agent
    producer service host
    consul-agent

    View Slide


  22. Communication from app servers to DB

    Analytics stack (Haddop, Tez)

    Almost all low latency, binary protocol based stuff

    TCP based communications
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  23. Load balancing under Nomad
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  24. Hashicorp User Group-SF, November 16th 2016
    Nomad cluster
    Nginx+
    &
    consul-template
    Nginx+
    &
    consul-template
    Container-1
    Container-1
    Container-1
    Container-1
    Container-1
    Container-1
    Container-1
    Container-1
    consul-master
    Nomad-master

    View Slide


  25. Front end load balancing

    Clients, customers, partners, channels to your software
    end points

    HTTP, websocket, spdy

    Fancier endpoint or host header based routing
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  26. Runtime Application
    Configuration
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  27. Configuration
    repository consul-master
    Hashicorp User Group-SF, November 16th 2016
    consumer service host
    consul-template
    config file
    Developers
    PR workflow
    GoCD
    git2consul

    View Slide

  28. Network Topology
    Management
    Hashicorp User Group-SF, November 16th 2016

    View Slide

  29. dc-1 dc-2 dc-3
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    consul-replicate
    consul-replicate
    Entire infra
    register
    consul-master
    consul-master
    host1
    host1
    host1
    host1
    host1
    host1
    host1
    host1
    host1
    host1
    runner
    runner
    runner
    runner
    runner
    runner
    runner
    runner
    runner
    runner

    View Slide

  30. Thank You
    Hashicorp User Group-SF, November 16th 2016

    View Slide