Consul @ PagerDuty

How we use consul at PagerDuty across containerized microservices, legacy applications and some custom workflows

Transcript

1. Consul
   at PagerDuty
   Hashicorp User Group-SF, November 16th 2016
   

   View Slide

2. ●
   Scope
   ●
   Deployment Architecture
   ●
   Use cases
   ●
   Q & A
   Agenda
   Hashicorp User Group-SF, November 16th 2016
   

   View Slide

3. ●
   SRE@PagerDuty
   ●
   All things Automation
   ●
   Chef, Containers
   (OpenVZ/LXC/docker/rkt/ns
   pawn..)
   ●
   FOSS Enthusiast
   ●
   @RanjibDey
   About me
   Hashicorp User Group-SF, November 16th 2016
   

   View Slide

4. Scope
   Hashicorp User Group-SF, November 16th 2016
   

   View Slide

5. ●
   Introduced early October, 2015
   ●
   Agent is present in entire infrastructure
   ●
   Different features used in different parts of the system
   ●
   Almost 12-15% of the current infra is using consul in one
   or other way (mostly around containerized workload)
   ●
   Incremental deployment across mission critical services
   Hashicorp User Group-SF, November 16th 2016
   

   View Slide

6. Deployment Architecture
   Hashicorp User Group-SF, November 16th 2016
   

   View Slide

7. dc-1 dc-2 dc-3
   consul-masters
   consul-masters
   consul-masters
   fleet
   fleet
   fleet
   Hashicorp User Group-SF, November 16th 2016
   consul-replicate
   consul-replicate
   

   View Slide

8. Metrics
   Hashicorp User Group-SF, November 16th 2016
   

   View Slide

9. Hashicorp User Group-SF, November 16th 2016
   

   View Slide

10. Use Cases
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide

11. Service discovery via DNS lookup
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide

12. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    health check
    consumer service host
    consul-agent service process
    DNS lookup
    

    View Slide

13. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    consumer service host
    consul-agent service process
    

    View Slide

14. ●
    Less frequent or one-off communication
    ●
    Example: Obtaining configuration endpoint or vault end
    point on application startup
    ●
    Bad for load balancing
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide

15. Service discovery via HTTP API
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide

16. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    health check
    consumer service host
    consul-agent
    service process
    HTTP API
    Helper library
    

    View Slide

17. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    consumer service host
    consul-agent
    service process
    Helper library
    

    View Slide

18. ●
    Communication in between backends
    ●
    Can be of any protocol (binary or plain text alike)
    ●
    Actual performance and user experience will vary among
    the client libraries
    ●
    Most dynamic language based libraries are not very good
    in this. C, Java, go has some decent options, but limited
    to certain tech stacks
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide

19. Client side load balancing
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide

20. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    health check
    consumer service host
    consul-template haproxy
    

    View Slide

21. producer service host
    consul-agent
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    consumer service host
    consul-template haproxy
    producer service host
    consul-agent
    producer service host
    consul-agent
    

    View Slide

22. ●
    Communication from app servers to DB
    ●
    Analytics stack (Haddop, Tez)
    ●
    Almost all low latency, binary protocol based stuff
    ●
    TCP based communications
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide

23. Load balancing under Nomad
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide

24. Hashicorp User Group-SF, November 16th 2016
    Nomad cluster
    Nginx+
    &
    consul-template
    Nginx+
    &
    consul-template
    Container-1
    Container-1
    Container-1
    Container-1
    Container-1
    Container-1
    Container-1
    Container-1
    consul-master
    Nomad-master
    

    View Slide

25. ●
    Front end load balancing
    ●
    Clients, customers, partners, channels to your software
    end points
    ●
    HTTP, websocket, spdy
    ●
    Fancier endpoint or host header based routing
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide

26. Runtime Application
    Configuration
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide

27. Configuration
    repository consul-master
    Hashicorp User Group-SF, November 16th 2016
    consumer service host
    consul-template
    config file
    Developers
    PR workflow
    GoCD
    git2consul
    

    View Slide

28. Network Topology
    Management
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide

29. dc-1 dc-2 dc-3
    consul-master
    Hashicorp User Group-SF, November 16th 2016
    consul-replicate
    consul-replicate
    Entire infra
    register
    consul-master
    consul-master
    host1
    host1
    host1
    host1
    host1
    host1
    host1
    host1
    host1
    host1
    runner
    runner
    runner
    runner
    runner
    runner
    runner
    runner
    runner
    runner
    

    View Slide

30. Thank You
    Hashicorp User Group-SF, November 16th 2016
    

    View Slide