Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Code.Talks 2016 - How To Hot-swap Your Container Platform

Code.Talks 2016 - How To Hot-swap Your Container Platform

In the age of Continuous Delivery, Containerized applications, and Cloud-based infrastructure, having a platform that is never down is as important as ever. Just as important is that your platform is easy to maintain. Yet this is often not the case. So how can we create a container platform, in the cloud, that is never down, and offers ease of maintenance? The answer is to create a container platform where every component can be hot-swapped, and in this talk I will explain the general principles behind these hot-swappable container platforms, and show you how you can create a container platform that you can completely replace without your users noticing.

4ea7c8c1ddc6e1c95c06d30b0b4ee57e?s=128

Benny Cornelissen

September 30, 2016
Tweet

More Decks by Benny Cornelissen

Other Decks in Technology

Transcript

  1. Hot-swappable Container Platforms Benny Cornelissen Infrastructure Architect @ Xebia @whieee

  2. I am a Infrastructure Architect at Xebia. I have been

    building infrastructure for over a decade, with focus on automation, resiliency and maintainability. I once built mobile datacenters that were built into shipping containers, and that were located in the desert. I also enjoy Belgian beer, coffee, travelling, and cycling.
  3. The evolution of hot-swapping

  4. Hot-swap your Hardware Server Application Server Application

  5. Hot-swap your Hypervisor Server Server VM Server VM VM VM

  6. Hot-swap your Cloud? VM ? VM VM VM

  7. Container-based Infrastructure App IaaS App App Container Platform Scheduling Service

    Discovery Routing & Proxy App App App App Storage Logging Metrics Linux-based OS
  8. We need a container platform where every component can be

    automatically hot-swapped
  9. We can sleep at night

  10. We had a minor failure last night at 01:25. Etcd

    broke on one of the nodes, so I replaced it at 01:26. Everything was back to normal at 01:30. Love, Platform
  11. We can do maintenance at any time FREE! 100% REBATE

  12. We build platforms to enable product teams to deliver awesome

    products
  13. A platform that is hard to maintain will become a

    very expensive piece of tech debt
  14. We can do maintenance at any time

  15. Reference Stack App AWS App App Container Platform Fleet Consul

    Nginx App App App App S3 / RDS ELK Sysdig CoreOS
  16. How do you build a hot- swappable container platform?

  17. Infrastructure as Code Static Host OS High Availability By Default

    Use Autoscaling Externalize Data Automated Repeatable Bootstrapping
  18. Infrastructure as Code

  19. Terraform resource "aws_instance" "myinstance" { instance_type = "t2.small" ami =

    "ami-cda312be" root_block_device { delete_on_termination = true volume_size = 20 } key_name = "${aws_key_pair.data-team.key_name}" security_groups = ["${aws_security_group.data- team.name}"] } myinstance.tf
  20. Basic Workflow $ vim myplatform.tf $ terraform plan $ terraform

    apply $ terraform destroy
  21. Production Staging Platform Dev $ cd infra-prod $ terraform apply

  22. Avoid using provisioners, build images instead https://www.packer.io/

  23. Static Minimal Host OS

  24. None
  25. Externalize Data

  26. Don't store anything locally unless you're prepared to lose it

  27. None
  28. Handling Failure

  29. High Availability By Default

  30. Losing an instance of a platform component should never result

    in a failed state
  31. Recovering from Failure

  32. Autoscaling

  33. Autoscaling, even if we don't want to scale

  34. Health Checks

  35. Health Checks Recovery Times

  36. Scaling Metrics Health Checks Recovery Times

  37. None
  38. Example Recovery of failed CoreOS node

  39. CoreOS CoreOS CoreOS CoreOS Auto Scaling Group App App App

    Proxy Proxy Proxy Proxy Back end Back end Back end Elastic Load Balancer
  40. CoreOS CoreOS CoreOS Auto Scaling Group App App Proxy Proxy

    Proxy Back end Back end Elastic Load Balancer
  41. CoreOS CoreOS CoreOS Auto Scaling Group App App Proxy Proxy

    Proxy Back end Back end Elastic Load Balancer App Back end
  42. CoreOS CoreOS CoreOS Auto Scaling Group App App Proxy Proxy

    Proxy Back end Back end Elastic Load Balancer App Back end CoreOS Proxy
  43. minutes 5 humans needed 0

  44. minutes 5 humans needed 0

  45. minutes 15 humans needed 1

  46. minutes 30 humans needed 1

  47. minutes ? humans needed 1

  48. Automated Repeatable Bootstrapping

  49. Cluster all the things!

  50. Example Bootstrapping Etcd

  51. Use Public Discovery Service Bootstrap Etcd?

  52. Use Public Discovery Service Use Private Discovery Service Bootstrap Etcd?

  53. Use Public Discovery Service Use Private Discovery Service Use DNS-based

    Discovery Bootstrap Etcd?
  54. Use Public Discovery Service Use Private Discovery Service Use DNS-based

    Discovery Use Manual Bootstrapping Bootstrap Etcd?
  55. Production-ready Documentation

  56. Bootstrapping Etcd • Determine if a cluster exists • If

    not, start a cluster • Otherwise, join cluster CoreOS CoreOS CoreOS Autoscaling Group Etcd Etcd Etcd
  57. Infrastructure as Code Static Host OS High Availability By Default

    Use Autoscaling Externalize Data Automated Repeatable Bootstrapping
  58. Using these key principles, you can create a hot-swappable container

    platform
  59. At any time?

  60. Let's hot-swap the entire platform during the bi-weekly Sprint demo

    of the Product Teams
  61. Nobody noticed…

  62. Peace of Mind Ease of Maintenance

  63. A platform that is hard to maintain will become a

    very expensive piece of tech debt
  64. Better. Easier. Cheaper. And yes, you can have all three…

  65. Infrastructure as Code Static Host OS High Availability By Default

    Use Autoscaling Externalize Data Automated Repeatable Bootstrapping
  66. Thank You! mail: bcornelissen@xebia.com twitter: @whieee blog: https://blog.bennycornelissen.nl/ github: bennycornelissen