From VMs to Containers: A DevOps Journey

Explore how Comcast moved large-scale, multi-data-center services from an architecture deployed on virtual machines to one based on containers with Apache Mesos

Transcript

1. From VMs to Containers: A DevOps Journey David Grizzanti Principal Engineer, Comcast Cable @dgrizzanti
2. Elements 2 APNS HTTP/S Publish Elements HTTP/S GCM/FCM Topic Subscribers 1215286302 6 Topic Topic
3. Usage •Hundreds of millions of publishes per day •Delivered in < 100ms 3
4. 4 "Cordless Phone" flickr photo by Clearwater1967 https://flickr.com/photos/twalmsley/7132322233 shared under a Creative Commons (BY) license
8. 8 Software Stack •Microservices Protocol Buffers
10. 10 Publish HTTP PubSub PubSub PubSub Persistent Connections
11. 11 PubSub PubSub PubSub Hooks Hooks Persistent Connections
12. Time for a change?
13. What could moving to containers help?
14. 14 Client Isolation
15. 15 Immutable Infrastructure
16. 16 Continuous Deployment
17. 17 Resiliency
18. 18 Scaling
19. 19 Operations Debateable
20. Where are we starting from?
21. 21 Ticket to deploy/destroy an instance "Tickets" flickr photo by thenoodleator https://flickr.com/photos/noodle/3474464038 shared under a Creative Commons (BY-NC-ND) license
22. 22 Manual build & push of an RPM CI/CD?
23. 23 Hardware Load Balancing + VIPs
24. 24 No easy way of isolating client traffic
25. 25 Clients assume static set of PubSub servers “Service Discovery”
26. 26 Logging & Metrics “Log every possible event and figure out what it means later”
27. 27 "Cattle" flickr photo by Alcalaina https://flickr.com/photos/alcalaina/3657650450 shared under a Creative Commons (BY-NC-SA) license "Cat" flickr photo by laurinha_lii https://flickr.com/photos/mundoworldmonde/5621803163 shared under a Creative Commons (BY-SA) license
28. Team Structure
29. 29 Separate dev and Ops teams Dev and QA siloes
30. Early Decisions
33. It’s about the journey not the destination, right?
34. 34 Let’s start with the good
38. 38 The struggle is real
39. 39 "Four Storms And A Twister" flickr photo by JD Hancock https://flickr.com/photos/jdhancock/3842546304 shared under a Creative Commons (BY) license
40. 40 Visibility into the containers
42. 42 Significant decrease in performance
43. 43 Significant decrease in performance Weave? Docker? HAProxy? Graylog GELF driver!
44. 44 Application teardown SIGKILL vs SIGTERM
45. The expected
46. 46 PubSub Service Discovery
47. Core Routing Protocol 47
48. 48 Zookeeper
49. 49 Zookeeper Zookeeper Zookeeper Zookeeper Zookeeper PubSub Client Adapter Register PubSub
50. Zookeeper Zookeeper Zookeeper Zookeeper Zookeeper PubSub Client Adapter Register Return Servers List* PubSub 50
51. Zookeeper Zookeeper Zookeeper Zookeeper Zookeeper PubSub Client Adapter Register Return Servers List* Connect PubSub 51
52. Zookeeper Zookeeper Zookeeper Zookeeper Zookeeper PubSub Client Adapter Register Return Servers List* Connect Servers Message* PubSub 52
53. Zookeeper Zookeeper Zookeeper Zookeeper Zookeeper PubSub Client Adapter PubSub Connect Servers Message* 53
54. 54 Publish Mesos Agent Publish 172.x.x.1:5000 96.x.x.x:31612 Containers $HOST $PORT 172.x.x.2:5000 96.x.x.x:31613
55. 55 Consul
56. Consul Consul Consul Consul Consul PubSub Client Adapter Get list of servers and ports by service name PubSub 56
57. Consul Consul Consul Consul Consul PubSub Client Adapter Get list of servers and ports by service name PubSub Get list of servers and ports by service name 57
58. Consul Consul Consul Consul Consul PubSub Client Adapter Get list of servers and ports by service name PubSub Connect Get list of servers and ports by service name Connect 58
59. Consul Consul Consul Consul Consul PubSub Client Adapter Get list of servers and ports by service name PubSub Servers Message* Connect Get list of servers and ports by service name Connect Servers Message* 59
60. 60 Load Balancing
61. 61 GSLB HTTP Publish VIP Publish VIP Publish Publish
62. 62 HAProxy & Bamboo
63. 63 GSLB HTTP Publish Publish Publish Publish
65. 65 Client/Tenant Isolation
66. 66 GSLB HTTP Publish VIP Publish VIP Publish Publish
67. 67 GSLB HTTP Publish Publish /publish/baz /publish/… /publish/foo Publish Publish
68. 68 Publish Publish /publish/foo /publish/baz PubSub Hooks Hooks /foo subscription /baz subscription
69. 69 Publish Publish /publish/foo /publish/baz PubSub Hooks Hooks /foo subscription /baz subscription PubSub
70. 70 Metrics
71. 71 collectd
72. 72 Log collection
73. 73 gelf driver
74. Where we settled (for now)
79. 79 PubSub protocol better served w/o persistent connections to each client?
80. The “new” team
81. 81 Small team (~7) w/ mixed skill sets
82. 82 Pair “programming”
83. Migration
84. 84 Publish PubSub PubSub Hooks legacy new Bridge Publish PubSub PubSub Hooks /foo
85. Takeaways
86. From VMs to Containers: A DevOps Journey •Containers aren’t a silver bullet •Logging and metrics from the beginning •Declarative configuration management •True CI/CD takes time and effort •Small, agile teams make a difference •Willingness to adapt your application @dgrizzanti