Tweet
Tweet

Slide 1

Slide 1 text

TAMING THE MODERN DATA CENTER A Hybrid Talk for a Hybrid World @sethvargo 

Slide 2

Slide 2 text

@sethvargo  Seth Vargo Director of Technical Advocacy HashiCorp

Slide 3

Slide 3 text

@sethvargo 

Slide 4

Slide 4 text

@sethvargo DC EVOLUTION How did we get here?

Slide 5

Slide 5 text

@sethvargo  RISING DATACENTER COMPLEXITY DC

Slide 6

Slide 6 text

@sethvargo  RISING DATACENTER COMPLEXITY DC

Slide 7

Slide 7 text

@sethvargo  RISING DATACENTER COMPLEXITY DC VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM

Slide 8

Slide 8 text

@sethvargo  RISING DATACENTER COMPLEXITY DC VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C C

Slide 9

Slide 9 text

@sethvargo  RISING DATACENTER COMPLEXITY DC DNS Database CDN

Slide 10

Slide 10 text

@sethvargo  RISING DATACENTER COMPLEXITY DC-01 DC-02

Slide 11

Slide 11 text

@sethvargo  RISING DATACENTER COMPLEXITY DC-01 DC-02 VM VM VM VM VM VM VM VM C C C C C C C C C C C C C C C C C C C C C C C C

Slide 12

Slide 12 text

@sethvargo  RISING DATACENTER COMPLEXITY IaaS PaaS SaaS

Slide 13

Slide 13 text

@sethvargo TAMING THE DC Deployment + Maintenance

Slide 14

Slide 14 text

@sethvargo PREVIOUSLY The APUD cycle

Slide 15

Slide 15 text

ACQUIRE PROVISION UPDATE DESTROY

Slide 16

Slide 16 text

ACQUIRE PROVISION UPDATE DESTROY G ’ U VENDOR

Slide 17

Slide 17 text

ACQUIRE PROVISION UPDATE DESTROY G U ’ U ’ U VENDOR DC OPS

Slide 18

Slide 18 text

ACQUIRE PROVISION UPDATE DESTROY G U ’ U ’ U U ’ U VENDOR DC OPS SYSADMIN

Slide 19

Slide 19 text

ACQUIRE PROVISION UPDATE DESTROY G U ’ U ’ U U ’ U U ’ U VENDOR DC OPS SYSADMIN DC OPS

Slide 20

Slide 20 text

ACQUIRE PROVISION UPDATE DESTROY VENDOR DC OPS SYSADMIN DC OPS WEEKS DAYS DAYS DAYS c c c c

Slide 21

Slide 21 text

@sethvargo PRESENTLY The elastic compute and _aaS era

Slide 22

Slide 22 text

ACQUIRE PROVISION UPDATE DESTROY WEEKS DAYS DAYS DAYS c c c c Elastic Compute

Slide 23

Slide 23 text

ACQUIRE PROVISION UPDATE DESTROY WEEKS DAYS DAYS DAYS c c c c Elastic Compute

Slide 24

Slide 24 text

ACQUIRE PROVISION UPDATE DESTROY MINUTES DAYS DAYS SECONDS c c c c Elastic Compute

Slide 25

Slide 25 text

CapEx # OpEx #

Slide 26

Slide 26 text

_aaS

Slide 27

Slide 27 text

ACQUIRE PROVISION UPDATE DESTROY DAYS DAYS c c Configuration Management MINUTES SECONDS c c

Slide 28

Slide 28 text

ACQUIRE PROVISION UPDATE DESTROY DAYS DAYS c c Configuration Management MINUTES SECONDS c c

Slide 29

Slide 29 text

ACQUIRE PROVISION UPDATE DESTROY MINUTES SECONDS c c Configuration Management MINUTES SECONDS c c

Slide 30

Slide 30 text

ACQUIRE PROVISION UPDATE DESTROY SaaS Proliferation ACQUIRE PROVISION UPDATE DESTROY https://specialized.com

Slide 31

Slide 31 text

@sethvargo  RISING DATACENTER COMPLEXITY DC DNS Database CDN VM VM VM VM C C C C C C

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

@sethvargo WHY? What was our original goal?

Slide 35

Slide 35 text

@sethvargo  EFFECTIVELY DELIVER AND MAINTAIN APPLICATIONS

Slide 36

Slide 36 text

@sethvargo  MOVE FAST AND DON’T BREAK THINGS

Slide 37

Slide 37 text

RUN Applications, Services, Jobs SECURE Applications, Infrastructure PROVISION Infrastructure, Code, Images

Slide 38

Slide 38 text

RUN Applications, Services, Jobs SECURE Applications, Infrastructure PROVISION Infrastructure, Code, Images

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

@sethvargo MOTIVATION Why Terraform?

Slide 41

Slide 41 text

@sethvargo How do I provision resources? compute? storage? network?

Slide 42

Slide 42 text

@sethvargo How do I manage resource lifecycles?

Slide 43

Slide 43 text

@sethvargo How do I balance service providers providing core technology for my datacenter?

Slide 44

Slide 44 text

@sethvargo How do I enforce policy across all these resources?

Slide 45

Slide 45 text

@sethvargo How do I automate and share those configurations?

Slide 46

Slide 46 text

@sethvargo  TERRAFORM'S GOAL

Slide 47

Slide 47 text

@sethvargo PROVIDE A SINGLE WORKFLOW

Slide 48

Slide 48 text

@sethvargo WITH A UNIFIED VIEW

Slide 49

Slide 49 text

@sethvargo USING INFRASTRUCTURE AS CODE

Slide 50

Slide 50 text

@sethvargo THAT CAN BE ITERATED AND CHANGED SAFELY

Slide 51

Slide 51 text

@sethvargo CAPABLE OF COMPLEX N-TIER APPLICATIONS

Slide 52

Slide 52 text

@sethvargo resource "digitalocean_droplet" "web" { name = "tf-web" size = "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" } main.tf

Slide 53

Slide 53 text

@sethvargo resource "digitalocean_droplet" "web" { name = "tf-web" size = "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" } main.tf

Slide 54

Slide 54 text

@sethvargo resource "digitalocean_droplet" "web" { name = "tf-web" size = "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" } main.tf

Slide 55

Slide 55 text

@sethvargo resource "digitalocean_droplet" "web" { name = "tf-web" size = "512mb" image = "centos-5-8-x32" region = "sfo1" } resource "dnsimple_record" "hello" { domain = "example.com" name = "test" value = "${digitalocean_droplet.web.ipv4_address}" type = "A" } main.tf

Slide 56

Slide 56 text

@sethvargo HUMAN-FRIENDLY CONFIGURATION*

Slide 57

Slide 57 text

@sethvargo VCS-FRIENDLY FORMAT

Slide 58

Slide 58 text

@sethvargo ENTIRE INFRASTRUCTURE... CAPTURED TEXT FILES

Slide 59

Slide 59 text

@sethvargo  TERRAFORM PROVIDERS

Slide 60

Slide 60 text

@sethvargo SINGLE INTEGRATION POINT

Slide 61

Slide 61 text

@sethvargo EXPOSE ("PROVIDE") A RESOURCE

Slide 62

Slide 62 text

@sethvargo CRUD API

Slide 63

Slide 63 text

@sethvargo PLUGGABLE FOR INTEGRATIONS

Slide 64

Slide 64 text

@sethvargo MANAGE ANYTHING WITH AN API

Slide 65

Slide 65 text

@sethvargo $ terraform apply

Slide 66

Slide 66 text

@sethvargo OVER 65 BUILT-IN PROVIDERS AND COUNTING...

Slide 67

Slide 67 text

@sethvargo  TERRAFORM PLAN

Slide 68

Slide 68 text

@sethvargo + digitalocean_droplet.web backups: "" => "" image: "" => "centos-5-8-x32" ipv4_address: "" => "" ipv4_address_private: "" => "" name: "" => "tf-web" private_networking: "" => "" region: "" => "sfo1" size: "" => "512mb" status: "" => "" + dnsimple_record.hello domain: "" => "example.com" Terminal

Slide 69

Slide 69 text

@sethvargo + digitalocean_droplet.web backups: "" => "" image: "" => "centos-5-8-x32" ipv4_address: "" => "" ipv4_address_private: "" => "" name: "" => "tf-web" private_networking: "" => "" region: "" => "sfo1" size: "" => "512mb" status: "" => "" + dnsimple_record.hello domain: "" => "example.com" Terminal

Slide 70

Slide 70 text

@sethvargo + digitalocean_droplet.web backups: "" => "" image: "" => "centos-5-8-x32" ipv4_address: "" => "" ipv4_address_private: "" => "" name: "" => "tf-web" private_networking: "" => "" region: "" => "sfo1" size: "" => "512mb" status: "" => "" + dnsimple_record.hello domain: "" => "example.com" Terminal

Slide 71

Slide 71 text

@sethvargo + digitalocean_droplet.web backups: "" => "" image: "" => "centos-5-8-x32" ipv4_address: "" => "" ipv4_address_private: "" => "" name: "" => "tf-web" private_networking: "" => "" region: "" => "sfo1" size: "" => "512mb" status: "" => "" + dnsimple_record.hello domain: "" => "example.com" Terminal

Slide 72

Slide 72 text

@sethvargo size: "" => "512mb" status: "" => "" + dnsimple_record.hello domain: "" => "example.com" domain_id: "" => "" hostname: "" => "" name: "" => "test" priority: "" => "" ttl: "" => "" type: "" => "A" value: "" => "${digitalocean_droplet.web.ipv4_address}" Terminal

Slide 73

Slide 73 text

@sethvargo SHOWS YOU WHAT WILL HAPPEN

Slide 74

Slide 74 text

@sethvargo EXPLAINS CERTAIN ACTIONS

Slide 75

Slide 75 text

@sethvargo PREVIOUSLY?

Slide 76

Slide 76 text

@sethvargo STILL UNCERTAINTY…

Slide 77

Slide 77 text

@sethvargo FUTURE OPS Managing Tomorrow's Infrastructure

Slide 78

Slide 78 text

@sethvargo  DEPLOY IMMUTABLE INFRASTRUCTURE

Slide 79

Slide 79 text

@sethvargo  CHANGES CONFIDENCE Mutable Infrastructure

Slide 80

Slide 80 text

@sethvargo  ITERATIONS CONSISTENCY Mutable Infrastructure

Slide 81

Slide 81 text

@sethvargo  ITERATIONS CONSISTENCY Immutable Infrastructure

Slide 82

Slide 82 text

@sethvargo  IMMUTABLE INFRASTRUCTURE IS FASTER

Slide 83

Slide 83 text

@sethvargo  IMMUTABLE INFRASTRUCTURE ALLOWS FOR GREATER PARITY

Slide 84

Slide 84 text

@sethvargo  IMMUTABLE INFRASTRUCTURE NEEDS AUTOMATION

Slide 85

Slide 85 text

No content

Slide 86

Slide 86 text

@sethvargo MACHINE IMAGES

Slide 87

Slide 87 text

@sethvargo YUCK... IMAGES?

Slide 88

Slide 88 text

@sethvargo WHY HAVE WE BEEN GENERALLY AGAINST MACHINE IMAGES?

Slide 89

Slide 89 text

@sethvargo GOLDEN IMAGES USED TO BE THE WAY

Slide 90

Slide 90 text

@sethvargo QUARTERLY, UNCHANGED, AND BLESSED IMAGES

Slide 91

Slide 91 text

@sethvargo CHANGES WERE SLOW AND FRUSTRATING

Slide 92

Slide 92 text

@sethvargo TOOLING WAS NOT MATURE COMPARED TO TODAY

Slide 93

Slide 93 text

@sethvargo MODERN CONFIG MANAGEMENT CHANGED THAT

Slide 94

Slide 94 text

@sethvargo OPS WITHOUT MACHINE IMAGES IS LIKE APPLICATIONS WITHOUT BINARIES

Slide 95

Slide 95 text

@sethvargo  APPLICATION LIFECYCLE

Slide 96

Slide 96 text

@sethvargo  APPLICATION LIFECYCLE Source Code Binary

Slide 97

Slide 97 text

@sethvargo  APPLICATION LIFECYCLE Source Code Binary libA 1.0 libB 1.0 libC 1.0

Slide 98

Slide 98 text

@sethvargo  APPLICATION LIFECYCLE Source Code Binary libA 1.0 libB 1.0 libC 1.0

Slide 99

Slide 99 text

@sethvargo  APPLICATION LIFECYCLE Source Code Binary libA 1.0 libB 1.0 libC 1.0

Slide 100

Slide 100 text

@sethvargo  MUTABLE SERVER LIFECYCLE

Slide 101

Slide 101 text

@sethvargo  APPLICATION LIFECYCLE Base Server Ready Server

Slide 102

Slide 102 text

@sethvargo  APPLICATION LIFECYCLE Base Server Ready Server Packages Network CM

Slide 103

Slide 103 text

@sethvargo  APPLICATION LIFECYCLE Base Server Ready Server Packages Network CM

Slide 104

Slide 104 text

@sethvargo  APPLICATION LIFECYCLE Base Server Ready Server Packages Network CM

Slide 105

Slide 105 text

@sethvargo  APPLICATION LIFECYCLE Base Server Ready Server Packages Network CM

Slide 106

Slide 106 text

@sethvargo  APPLICATION LIFECYCLE Base Server Ready Server Packages Network CM

Slide 107

Slide 107 text

@sethvargo  APPLICATION LIFECYCLE IN THE PATH OF DOWNTIME

Slide 108

Slide 108 text

@sethvargo  MACHINE IMAGE LIFECYCLE

Slide 109

Slide 109 text

@sethvargo  MACHINE IMAGE LIFECYCLE Base Server Ready Server

Slide 110

Slide 110 text

@sethvargo  MACHINE IMAGE LIFECYCLE Base Server Ready Server

Slide 111

Slide 111 text

@sethvargo  PACKER BUILD

Slide 112

Slide 112 text

@sethvargo EMBRACES CONFIG MANAGEMENT

Slide 113

Slide 113 text

@sethvargo TRANSITIONS FAILURES FROM RUNTIME TO BUILD-TIME

Slide 114

Slide 114 text

@sethvargo ENFORCES PARITY WITH STAGING

Slide 115

Slide 115 text

@sethvargo … AND EVEN DEVELOPMENT

Slide 116

Slide 116 text

@sethvargo  NEW CHALLENGES

Slide 117

Slide 117 text

@sethvargo IT DIDN'T BELONG THERE IN THE FIRST PLACE

Slide 118

Slide 118 text

@sethvargo LIKE TRYING TO USE LS TO CREATE A FILE

Slide 119

Slide 119 text

No content

Slide 120

Slide 120 text

@sethvargo Consul Features Service Discovery Health Checking KV Store Multi Datacenter

Slide 121

Slide 121 text

@sethvargo Service Discovery

Slide 122

Slide 122 text

@sethvargo Service Discovery DNS interface is zero-touch - no application changes are required HTTP API for modern applications returns rich metadata Allows discovery of both internal and external services

Slide 123

Slide 123 text

@sethvargo $ host web.service.consul 10.0.3.83 10.0.1.109 10.0.4.21 Terminal

Slide 124

Slide 124 text

@sethvargo $ curl $CONSUL_ADDR/v1/health/services/web [ { # ... } ] Terminal

Slide 125

Slide 125 text

@sethvargo Health Checking

Slide 126

Slide 126 text

@sethvargo Health Checking Integrates with the service discovery layer DNS does not return results for unhealthy services or nodes HTTP endpoints can list health and query by health

Slide 127

Slide 127 text

@sethvargo KV Store

Slide 128

Slide 128 text

@sethvargo KV Store Highly available storage for configuration and feature flags Feature flags without big CM processes Supports blocking queries for "pushing" changes Optional ACLs to protect sensitive information at paths

Slide 129

Slide 129 text

@sethvargo $ consul kv put foo bar Success! Data written to: foo Terminal

Slide 130

Slide 130 text

@sethvargo $ consul kv get foo bar Terminal

Slide 131

Slide 131 text

@sethvargo Multi-Datacenter

Slide 132

Slide 132 text

@sethvargo Multi-Datacenter Usually query the local datacenter Can query other datacenters however you may need to Can view all datacenters within one OSS UI

Slide 133

Slide 133 text

@sethvargo $ dig web-frontend.singapore.service.consul. +short 10.3.3.33 10.3.1.18 $ dig web-frontend.germany.service.consul. +short 10.7.3.41 10.7.1.76 Terminal

Slide 134

Slide 134 text

@sethvargo $ curl http://localhost:8500/v1/kv/foo?raw&dc=asia true $ curl http://localhost:8500/v1/kv/foo?raw&dc=eu false Terminal

Slide 135

Slide 135 text

@sethvargo ... And More!

Slide 136

Slide 136 text

@sethvargo Events, Exec, and Watches Build powerful orchestration tools Implement client-side leader election Distributed locking and event system All approaches proven to scale to thousands of agents

Slide 137

Slide 137 text

@sethvargo $ consul event deploy 6DF7FE # ... $ consul watch -type event -name deploy /usr/bin/deploy.sh # ... $ consul exec -service web /usr/bin/deploy.sh # ... Terminal

Slide 138

Slide 138 text

@sethvargo Security Encrypt gossip traffic with shared key or keyring (UDP) Encrypt HTTP traffic with TLS (TCP) Advanced ACLs and token-based system allows for massive scale

Slide 139

Slide 139 text

@sethvargo

Slide 140

Slide 140 text

@sethvargo Completely Open Source

Slide 141

Slide 141 text

@sethvargo Completely "Dog Fooded"

Slide 142

Slide 142 text

@sethvargo  Seth Vargo Director of Technical Advocacy HashiCorp Questions?