Code.Talks 2016 - How To Hot-swap Your Container Platform

In the age of Continuous Delivery, Containerized applications, and Cloud-based infrastructure, having a platform that is never down is as important as ever. Just as important is that your platform is easy to maintain. Yet this is often not the case. So how can we create a container platform, in the cloud, that is never down, and offers ease of maintenance? The answer is to create a container platform where every component can be hot-swapped, and in this talk I will explain the general principles behind these hot-swappable container platforms, and show you how you can create a container platform that you can completely replace without your users noticing.

Transcript

1. Hot-swappable Container Platforms Benny Cornelissen Infrastructure Architect @ Xebia @whieee
2. I am a Infrastructure Architect at Xebia. I have been building infrastructure for over a decade, with focus on automation, resiliency and maintainability. I once built mobile datacenters that were built into shipping containers, and that were located in the desert. I also enjoy Belgian beer, coffee, travelling, and cycling.
3. The evolution of hot-swapping
4. Hot-swap your Hardware Server Application Server Application
5. Hot-swap your Hypervisor Server Server VM Server VM VM VM
6. Hot-swap your Cloud? VM ? VM VM VM
7. Container-based Infrastructure App IaaS App App Container Platform Scheduling Service Discovery Routing & Proxy App App App App Storage Logging Metrics Linux-based OS
8. We need a container platform where every component can be automatically hot-swapped
9. We can sleep at night
10. We had a minor failure last night at 01:25. Etcd broke on one of the nodes, so I replaced it at 01:26. Everything was back to normal at 01:30. Love, Platform
11. We can do maintenance at any time FREE! 100% REBATE
12. We build platforms to enable product teams to deliver awesome products
13. A platform that is hard to maintain will become a very expensive piece of tech debt
14. We can do maintenance at any time
15. Reference Stack App AWS App App Container Platform Fleet Consul Nginx App App App App S3 / RDS ELK Sysdig CoreOS
16. How do you build a hot- swappable container platform?
17. Infrastructure as Code Static Host OS High Availability By Default Use Autoscaling Externalize Data Automated Repeatable Bootstrapping
18. Infrastructure as Code
19. Terraform resource "aws_instance" "myinstance" { instance_type = "t2.small" ami = "ami-cda312be" root_block_device { delete_on_termination = true volume_size = 20 } key_name = "${aws_key_pair.data-team.key_name}" security_groups = ["${aws_security_group.data- team.name}"] } myinstance.tf
20. Basic Workflow $ vim myplatform.tf $ terraform plan $ terraform apply $ terraform destroy
21. Production Staging Platform Dev $ cd infra-prod $ terraform apply
22. Avoid using provisioners, build images instead https://www.packer.io/
23. Static Minimal Host OS
25. Externalize Data
26. Don't store anything locally unless you're prepared to lose it
28. Handling Failure
29. High Availability By Default
30. Losing an instance of a platform component should never result in a failed state
31. Recovering from Failure
32. Autoscaling
33. Autoscaling, even if we don't want to scale
34. Health Checks
35. Health Checks Recovery Times
36. Scaling Metrics Health Checks Recovery Times
38. Example Recovery of failed CoreOS node
39. CoreOS CoreOS CoreOS CoreOS Auto Scaling Group App App App Proxy Proxy Proxy Proxy Back end Back end Back end Elastic Load Balancer
40. CoreOS CoreOS CoreOS Auto Scaling Group App App Proxy Proxy Proxy Back end Back end Elastic Load Balancer
41. CoreOS CoreOS CoreOS Auto Scaling Group App App Proxy Proxy Proxy Back end Back end Elastic Load Balancer App Back end
42. CoreOS CoreOS CoreOS Auto Scaling Group App App Proxy Proxy Proxy Back end Back end Elastic Load Balancer App Back end CoreOS Proxy
43. minutes 5 humans needed 0
44. minutes 5 humans needed 0
45. minutes 15 humans needed 1
46. minutes 30 humans needed 1
47. minutes ? humans needed 1
48. Automated Repeatable Bootstrapping
49. Cluster all the things!
50. Example Bootstrapping Etcd
51. Use Public Discovery Service Bootstrap Etcd?
52. Use Public Discovery Service Use Private Discovery Service Bootstrap Etcd?
53. Use Public Discovery Service Use Private Discovery Service Use DNS-based Discovery Bootstrap Etcd?
54. Use Public Discovery Service Use Private Discovery Service Use DNS-based Discovery Use Manual Bootstrapping Bootstrap Etcd?
55. Production-ready Documentation
56. Bootstrapping Etcd • Determine if a cluster exists • If not, start a cluster • Otherwise, join cluster CoreOS CoreOS CoreOS Autoscaling Group Etcd Etcd Etcd
57. Infrastructure as Code Static Host OS High Availability By Default Use Autoscaling Externalize Data Automated Repeatable Bootstrapping
58. Using these key principles, you can create a hot-swappable container platform
59. At any time?
60. Let's hot-swap the entire platform during the bi-weekly Sprint demo of the Product Teams
61. Nobody noticed…
62. Peace of Mind Ease of Maintenance
63. A platform that is hard to maintain will become a very expensive piece of tech debt
64. Better. Easier. Cheaper. And yes, you can have all three…
65. Infrastructure as Code Static Host OS High Availability By Default Use Autoscaling Externalize Data Automated Repeatable Bootstrapping
66. Thank You! mail: bcornelissen@xebia.com twitter: @whieee blog: https://blog.bennycornelissen.nl/ github: bennycornelissen