Taming the Modern Data Center

Taming the Modern Data Center

Today we are plagued by hundreds of choices when architecting a modern data center. Should our machines be virtual or physical? Should we use containers or Docker? Should we use a public cloud provider or a private cloud provider? Which configuration management tool is best to use? What about IaaS, PaaS, and SaaS? It would be manageable if these were binary choices; however, we often find ourselves in a hybrid environment.

As more operations choices are added to your data center, whether through company acquisitions, a growing development team, or general technical debt, managing complexity between legacy and new systems becomes a nightmare. Yet the end goal is still the same — safely deploy your application to your infrastructure. We need to tame our data centers by managing change across systems, enforcing policies, and by establishing a workflow for both developers and operations engineers to build in a collaborative environment.

This talk will discuss the problems faced in the modern data center, and how a set of innovative open source tooling can be used to tame the rising complexity curve. Join me on an adventure with Packer, Consul, and Terraform as we take your data center from chaos to control.

502828deee7e3b38ca1e527dded8a1a9?s=128

Seth Vargo

May 17, 2017
Tweet

Transcript

  1. TAMING THE MODERN DATA CENTER A Hybrid Talk for a

    Hybrid World @sethvargo 
  2. @sethvargo  Seth Vargo Director of Technical Advocacy HashiCorp

  3. @sethvargo 

  4. @sethvargo DC EVOLUTION How did we get here?

  5. @sethvargo  RISING DATACENTER COMPLEXITY DC

  6. @sethvargo  RISING DATACENTER COMPLEXITY DC

  7. @sethvargo  RISING DATACENTER COMPLEXITY DC VM VM VM VM

    VM VM VM VM VM VM VM VM VM VM VM VM
  8. @sethvargo  RISING DATACENTER COMPLEXITY DC VM VM VM VM

    VM VM VM VM VM VM VM VM VM VM VM VM C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C
  9. @sethvargo  RISING DATACENTER COMPLEXITY DC DNS Database CDN

  10. @sethvargo  RISING DATACENTER COMPLEXITY DC-01 DC-02

  11. @sethvargo  RISING DATACENTER COMPLEXITY DC-01 DC-02 VM VM VM

    VM VM VM VM VM C C C C C C C C C C C C C C C C C C C C C C C C
  12. @sethvargo  RISING DATACENTER COMPLEXITY IaaS PaaS SaaS

  13. @sethvargo TAMING THE DC Deployment + Maintenance

  14. @sethvargo PREVIOUSLY The APUD cycle

  15. ACQUIRE PROVISION UPDATE DESTROY

  16. ACQUIRE PROVISION UPDATE DESTROY G ’ U VENDOR

  17. ACQUIRE PROVISION UPDATE DESTROY G U ’ U ’ U

    VENDOR DC OPS
  18. ACQUIRE PROVISION UPDATE DESTROY G U ’ U ’ U

    U ’ U VENDOR DC OPS SYSADMIN
  19. ACQUIRE PROVISION UPDATE DESTROY G U ’ U ’ U

    U ’ U U ’ U VENDOR DC OPS SYSADMIN DC OPS
  20. ACQUIRE PROVISION UPDATE DESTROY VENDOR DC OPS SYSADMIN DC OPS

    WEEKS DAYS DAYS DAYS c c c c
  21. @sethvargo PRESENTLY The elastic compute and _aaS era

  22. ACQUIRE PROVISION UPDATE DESTROY WEEKS DAYS DAYS DAYS c c

    c c Elastic Compute
  23. ACQUIRE PROVISION UPDATE DESTROY WEEKS DAYS DAYS DAYS c c

    c c Elastic Compute
  24. ACQUIRE PROVISION UPDATE DESTROY MINUTES DAYS DAYS SECONDS c c

    c c Elastic Compute
  25. CapEx # OpEx #

  26. _aaS

  27. ACQUIRE PROVISION UPDATE DESTROY DAYS DAYS c c Configuration Management

    MINUTES SECONDS c c
  28. ACQUIRE PROVISION UPDATE DESTROY DAYS DAYS c c Configuration Management

    MINUTES SECONDS c c
  29. ACQUIRE PROVISION UPDATE DESTROY MINUTES SECONDS c c Configuration Management

    MINUTES SECONDS c c
  30. ACQUIRE PROVISION UPDATE DESTROY SaaS Proliferation ACQUIRE PROVISION UPDATE DESTROY

    https://specialized.com
  31. @sethvargo  RISING DATACENTER COMPLEXITY DC DNS Database CDN VM

    VM VM VM C C C C C C
  32. None
  33. None
  34. @sethvargo WHY? What was our original goal?

  35. @sethvargo  EFFECTIVELY DELIVER AND MAINTAIN APPLICATIONS

  36. @sethvargo  MOVE FAST AND DON’T BREAK THINGS

  37. RUN Applications, Services, Jobs SECURE Applications, Infrastructure PROVISION Infrastructure, Code,

    Images
  38. RUN Applications, Services, Jobs SECURE Applications, Infrastructure PROVISION Infrastructure, Code,

    Images
  39. None
  40. @sethvargo MOTIVATION Why Terraform?

  41. @sethvargo How do I provision resources? compute? storage? network?

  42. @sethvargo How do I manage resource lifecycles?

  43. @sethvargo How do I balance service providers providing core technology

    for my datacenter?
  44. @sethvargo How do I enforce policy across all these resources?

  45. @sethvargo How do I automate and share those configurations?

  46. @sethvargo  TERRAFORM'S GOAL

  47. @sethvargo PROVIDE A SINGLE WORKFLOW

  48. @sethvargo WITH A UNIFIED VIEW

  49. @sethvargo USING INFRASTRUCTURE AS CODE

  50. @sethvargo THAT CAN BE ITERATED AND CHANGED SAFELY

  51. @sethvargo CAPABLE OF COMPLEX N-TIER APPLICATIONS

  52. @sethvargo resource "digitalocean_droplet" "web" { name = "tf-web" size =

    "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" } main.tf
  53. @sethvargo resource "digitalocean_droplet" "web" { name = "tf-web" size =

    "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" } main.tf
  54. @sethvargo resource "digitalocean_droplet" "web" { name = "tf-web" size =

    "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" } main.tf
  55. @sethvargo resource "digitalocean_droplet" "web" { name = "tf-web" size =

    "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" } main.tf
  56. @sethvargo HUMAN-FRIENDLY CONFIGURATION*

  57. @sethvargo VCS-FRIENDLY FORMAT

  58. @sethvargo ENTIRE INFRASTRUCTURE... CAPTURED TEXT FILES

  59. @sethvargo  TERRAFORM PROVIDERS

  60. @sethvargo SINGLE INTEGRATION POINT

  61. @sethvargo EXPOSE ("PROVIDE") A RESOURCE

  62. @sethvargo CRUD API

  63. @sethvargo PLUGGABLE FOR INTEGRATIONS

  64. @sethvargo MANAGE ANYTHING WITH AN API

  65. @sethvargo $ terraform apply

  66. @sethvargo OVER 65 BUILT-IN PROVIDERS AND COUNTING...

  67. @sethvargo  TERRAFORM PLAN

  68. @sethvargo + digitalocean_droplet.web backups: "" => "<computed>" image: "" =>

    "centos-5-8-x32" ipv4_address: "" => "<computed>" ipv4_address_private: "" => "<computed>" name: "" => "tf-web" private_networking: "" => "<computed>" region: "" => "sfo1" size: "" => "512mb" status: "" => "<computed>" + dnsimple_record.hello domain: "" => "example.com" Terminal
  69. @sethvargo + digitalocean_droplet.web backups: "" => "<computed>" image: "" =>

    "centos-5-8-x32" ipv4_address: "" => "<computed>" ipv4_address_private: "" => "<computed>" name: "" => "tf-web" private_networking: "" => "<computed>" region: "" => "sfo1" size: "" => "512mb" status: "" => "<computed>" + dnsimple_record.hello domain: "" => "example.com" Terminal
  70. @sethvargo + digitalocean_droplet.web backups: "" => "<computed>" image: "" =>

    "centos-5-8-x32" ipv4_address: "" => "<computed>" ipv4_address_private: "" => "<computed>" name: "" => "tf-web" private_networking: "" => "<computed>" region: "" => "sfo1" size: "" => "512mb" status: "" => "<computed>" + dnsimple_record.hello domain: "" => "example.com" Terminal
  71. @sethvargo + digitalocean_droplet.web backups: "" => "<computed>" image: "" =>

    "centos-5-8-x32" ipv4_address: "" => "<computed>" ipv4_address_private: "" => "<computed>" name: "" => "tf-web" private_networking: "" => "<computed>" region: "" => "sfo1" size: "" => "512mb" status: "" => "<computed>" + dnsimple_record.hello domain: "" => "example.com" Terminal
  72. @sethvargo size: "" => "512mb" status: "" => "<computed>" +

    dnsimple_record.hello domain: "" => "example.com" domain_id: "" => "<computed>" hostname: "" => "<computed>" name: "" => "test" priority: "" => "<computed>" ttl: "" => "<computed>" type: "" => "A" value: "" => "${digitalocean_droplet.web.ipv4_address}" Terminal
  73. @sethvargo SHOWS YOU WHAT WILL HAPPEN

  74. @sethvargo EXPLAINS CERTAIN ACTIONS

  75. @sethvargo PREVIOUSLY?

  76. @sethvargo STILL UNCERTAINTY…

  77. @sethvargo FUTURE OPS Managing Tomorrow's Infrastructure

  78. @sethvargo  DEPLOY IMMUTABLE INFRASTRUCTURE

  79. @sethvargo  CHANGES CONFIDENCE Mutable Infrastructure

  80. @sethvargo  ITERATIONS CONSISTENCY Mutable Infrastructure

  81. @sethvargo  ITERATIONS CONSISTENCY Immutable Infrastructure

  82. @sethvargo  IMMUTABLE INFRASTRUCTURE IS FASTER

  83. @sethvargo  IMMUTABLE INFRASTRUCTURE ALLOWS FOR GREATER PARITY

  84. @sethvargo  IMMUTABLE INFRASTRUCTURE NEEDS AUTOMATION

  85. None
  86. @sethvargo MACHINE IMAGES

  87. @sethvargo YUCK... IMAGES?

  88. @sethvargo WHY HAVE WE BEEN GENERALLY AGAINST MACHINE IMAGES?

  89. @sethvargo GOLDEN IMAGES USED TO BE THE WAY

  90. @sethvargo QUARTERLY, UNCHANGED, AND BLESSED IMAGES

  91. @sethvargo CHANGES WERE SLOW AND FRUSTRATING

  92. @sethvargo TOOLING WAS NOT MATURE COMPARED TO TODAY

  93. @sethvargo MODERN CONFIG MANAGEMENT CHANGED THAT

  94. @sethvargo OPS WITHOUT MACHINE IMAGES IS LIKE APPLICATIONS WITHOUT BINARIES

  95. @sethvargo  APPLICATION LIFECYCLE

  96. @sethvargo  APPLICATION LIFECYCLE Source Code Binary

  97. @sethvargo  APPLICATION LIFECYCLE Source Code Binary libA 1.0 libB

    1.0 libC 1.0
  98. @sethvargo  APPLICATION LIFECYCLE Source Code Binary libA 1.0 libB

    1.0 libC 1.0
  99. @sethvargo  APPLICATION LIFECYCLE Source Code Binary libA 1.0 libB

    1.0 libC 1.0
  100. @sethvargo  MUTABLE SERVER LIFECYCLE

  101. @sethvargo  APPLICATION LIFECYCLE Base Server Ready Server

  102. @sethvargo  APPLICATION LIFECYCLE Base Server Ready Server Packages Network

    CM
  103. @sethvargo  APPLICATION LIFECYCLE Base Server Ready Server Packages Network

    CM
  104. @sethvargo  APPLICATION LIFECYCLE Base Server Ready Server Packages Network

    CM
  105. @sethvargo  APPLICATION LIFECYCLE Base Server Ready Server Packages Network

    CM
  106. @sethvargo  APPLICATION LIFECYCLE Base Server Ready Server Packages Network

    CM
  107. @sethvargo  APPLICATION LIFECYCLE IN THE PATH OF DOWNTIME

  108. @sethvargo  MACHINE IMAGE LIFECYCLE

  109. @sethvargo  MACHINE IMAGE LIFECYCLE Base Server Ready Server

  110. @sethvargo  MACHINE IMAGE LIFECYCLE Base Server Ready Server

  111. @sethvargo  PACKER BUILD

  112. @sethvargo EMBRACES CONFIG MANAGEMENT

  113. @sethvargo TRANSITIONS FAILURES FROM RUNTIME TO BUILD-TIME

  114. @sethvargo ENFORCES PARITY WITH STAGING

  115. @sethvargo … AND EVEN DEVELOPMENT

  116. @sethvargo  NEW CHALLENGES

  117. @sethvargo IT DIDN'T BELONG THERE IN THE FIRST PLACE

  118. @sethvargo LIKE TRYING TO USE LS TO CREATE A FILE

  119. None
  120. @sethvargo Consul Features Service Discovery Health Checking KV Store Multi

    Datacenter
  121. @sethvargo Service Discovery

  122. @sethvargo Service Discovery DNS interface is zero-touch - no application

    changes are required HTTP API for modern applications returns rich metadata Allows discovery of both internal and external services
  123. @sethvargo $ host web.service.consul 10.0.3.83 10.0.1.109 10.0.4.21 Terminal

  124. @sethvargo $ curl $CONSUL_ADDR/v1/health/services/web [ { # ... } ]

    Terminal
  125. @sethvargo Health Checking

  126. @sethvargo Health Checking Integrates with the service discovery layer DNS

    does not return results for unhealthy services or nodes HTTP endpoints can list health and query by health
  127. @sethvargo KV Store

  128. @sethvargo KV Store Highly available storage for configuration and feature

    flags Feature flags without big CM processes Supports blocking queries for "pushing" changes Optional ACLs to protect sensitive information at paths
  129. @sethvargo $ consul kv put foo bar Success! Data written

    to: foo Terminal
  130. @sethvargo $ consul kv get foo bar Terminal

  131. @sethvargo Multi-Datacenter

  132. @sethvargo Multi-Datacenter Usually query the local datacenter Can query other

    datacenters however you may need to Can view all datacenters within one OSS UI
  133. @sethvargo $ dig web-frontend.singapore.service.consul. +short 10.3.3.33 10.3.1.18 $ dig web-frontend.germany.service.consul.

    +short 10.7.3.41 10.7.1.76 Terminal
  134. @sethvargo $ curl http://localhost:8500/v1/kv/foo?raw&dc=asia true $ curl http://localhost:8500/v1/kv/foo?raw&dc=eu false Terminal

  135. @sethvargo ... And More!

  136. @sethvargo Events, Exec, and Watches Build powerful orchestration tools Implement

    client-side leader election Distributed locking and event system All approaches proven to scale to thousands of agents
  137. @sethvargo $ consul event deploy 6DF7FE # ... $ consul

    watch -type event -name deploy /usr/bin/deploy.sh # ... $ consul exec -service web /usr/bin/deploy.sh # ... Terminal
  138. @sethvargo Security Encrypt gossip traffic with shared key or keyring

    (UDP) Encrypt HTTP traffic with TLS (TCP) Advanced ACLs and token-based system allows for massive scale
  139. @sethvargo

  140. @sethvargo Completely Open Source

  141. @sethvargo Completely "Dog Fooded"

  142. @sethvargo  Seth Vargo Director of Technical Advocacy HashiCorp Questions?