CfgMgmtCamp - Infrastructure as code should contain code

Gianluca Arbezzano - @gianarb - SRE at InfluxData
Infrastructure as code
should contain code

View Slide

Gianluca Arbezzano
Site Reliability Engineer @InfluxData
● https://gianarb.it
● @gianarb
What I like:
● I make dirty hacks that look awesome
● I grow my vegetables
● Travel for fun and work

View Slide

@gianarb - [email protected]

View Slide

View Slide

© 2018 InfluxData. All rights reserved.
7 @gianarb - [email protected]
github.com/influxdata

View Slide

I am not against YAML/JSON
but against unmaintainable
YAML/JSON.

View Slide


View Slide

The automation’s goals
¨ The ability to play repeatable tasks with a
single click or continuously.
¨ The possibility to build pipelines using
products like Kubernetes, AWS, Jenkins
¨ We would like to have a friendly UX to
manage “ops”.
Image credit: Pixabay

View Slide

Why Kubernetes
is so powerful, complex
and widely adopted?
widely

View Slide

Why AWS
is so expensive?
widely

View Slide

What do you do
to justify these costs?

View Slide


View Slide

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ template "drone.fullname" . }}-agent
labels:
app: {{ template "drone.name" . }}
chart: "{{ .Chart.Name }}-{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
component: agent
spec:
replicas: {{ .Values.agent.replicas }}
template:
metadata:
annotations:
checksum/secrets: {{ include (print $.Template.BasePath "/secrets.yaml") . | sha256sum }}
{{- if .Values.agent.annotations }
{{ toYaml .Values.agent.annotations | indent 8 }
{{- end }}
labels:
app: {{ template "drone.name" . }}
release: "{{ .Release.Name }}"
component: agent

View Slide

API are
the keys for
your success!
Image credit: Pixabay

View Slide

What do I mean
with “code”

View Slide

Short TTL Long TTL
don’t
change very
often
They never
change
they have a
controllable
“state”
Change
frequently
They have a
state
controllable
by outside
Depend
from the
external

View Slide


View Slide

YAML PRO:
You don’t need to know a
programming language to use it

View Slide

So you are not like me always with the k8s’s docs or AWS’s doc
open to look at YAML formats?

View Slide

Code PRO:
A Go struct brings compile time
validation and inline docs
(it means no unknown fields)

View Slide

Write friendly utility to manipulate the resources
// Create the pull of services you need to deploy
apps := []runtime.Object{}
apps = append(apps, etcd.New()...)
apps = append(apps, monitor.New()...)
apps = append(apps, kafka.New()...)
// Declare the number of transformation you need to apply to the declared resources
ops = append(ops,
WithNoLimits(),
WithMaxReplicas(1),
WithReplicas(map[string]int{
"etcd": 3,
}),
WithNoAffinity(),
WithNoNodePorts())
// Apply Transformation
apps = Transform(ops...)(apps)
// Deploy the transformation using client-sdk

View Slide

Test what you expect avoids mistakes
prod := []runtime.Object{}
prod = append(prod, service1.New()...)
prod = append(prod, service2.New()...)
prod = append(prod, mysql.New()...)
prod = append(prod, kafka.New()...)
ops = append(ops,
WithMaxReplicas(1),
WithReplicas(map[string]int{
"service1": 3,
"service2": 21,
}))
apps = Transform(ops...)(prod)
// You can write a unit test to enforce what you need in the
// prod environment or to avoid to change something that //
should not be changed.

View Slide

// WithReplicas matches the serviceReplicas key to the statefulset service name
// and sets the number of replicas to the value.
func WithReplicas(serviceReplicas map[string]int) Op {
return func(objs []runtime.Object) []runtime.Object {
for _, o := range objs {
switch app := o.(type) {
case *appsv1.StatefulSet:
for name, replicas := range serviceReplicas {
if app.Spec.ServiceName == name {
r := int32(replicas)
app.Spec.Replicas = &r
}
}
}
}
return objs
}
}

View Slide

Develop your pipeline
¨ Monitorable
¨ Reproducible
¨ Easy to expand
¨ Documentative

View Slide

Automatically set labels or taints
to K8S from EC2 tags

View Slide

Kubernetes cluster in AWS / Labels and taints from EC2 tags
Every autoscaling group can passthrough
tags to the underline EC2.
We wrote a Golang application that uses a
shared informer to listen for new registered
node.

View Slide

When a new EC2 joins the cluster we
catched the event and if the EC2
contains specific tags we apply them as
tants or labels on the kubelet

View Slide

EC2 tag: kubernetes/aws-labeler/label/role=ci
K8S label: awslabeler.influxdata.com/role=ci
EC2 tag: kubernetes/aws-labeler/taint/role=ci:NoSchedule
K8S Taint: awslabeler.influxdata.com/role=ci:NoSchedule

View Slide

Kubernetes cluster in AWS / Long TTL
1. Security groups
2. VPC
3. Subnets
4. Route53 zones
5. AMI

View Slide

AWS Autoscaling Group / Short TTL
1. AWS Autoscaling Groups to guarantee the expected number of workers
2. Every Autoscaling Group can run different Kubernetes versions, it can have different
labels, instance type, security groups, AMI and so on.
3. Autoscaling group are provided as a service from AWS. No maintenance at all

View Slide

Kubernetes cluster in AWS

View Slide

Kubernetes cluster in AWS / Master
1. The master is manually provisioned
2. We use kubernetes cronjob to backup the etcd cluster
3. We are practical folks. We know we can do better but for now we are happy with what
we have for now

View Slide

UserData and immutable
infrastructure reduces the need
for a config management tool
like Ansible or Chef.

View Slide

Spit your provisioning by layers
using Packer or LinuxKit to have
a base image and UserData to
specialize it

View Slide

CoreOS supports ignition (ok this is JSON.. but it is not too much JSON!)
{
"ignition": {"version": "2.1.0"},
"storage": {
"directories": [{"filesystem": "root", "path": "/etc/kubernetes/manifests"},
"files": [{
"filesystem": "root",
"path": "/etc/ssl/AmazonRootCA1.pem",
"contents": {"source": "https://www.amazontrust.com/repository/AmazonRootCA1.pem"}
}]
},
"systemd": { … }
}

View Slide

Shorter is the ttl for a single
server and more secure it is

View Slide

Wrap up
¨ JSON and YAML are not the problem
¨ Long TTL vs Short TTL
¨ Use the API!
¨ Patterns and methodology to write good
infra as code
¨ Testing to enforce safety
¨ Inline documentation
¨ Code Review
¨ Immutable Infrastructure

View Slide

Credits and links
¨ https://gianarb.it/blog/infrastructure-as-real-code
¨ https://twitter.com/danveloper/status/1078870433246662656
¨ https://blog.couchbase.com/kubernetes-operators-game-changer/
¨ https://gianarb.it/blog/reactive-planning-is-a-cloud-native-pattern
¨ https://engineering.bitnami.com/articles/a-deep-dive-into-kubernetes-controllers.html

View Slide

Reach out:
@gianarb
[email protected]
https://gianarb.it
Any question?

View Slide

CfgMgmtCamp - Infrastructure as code should contain code

CfgMgmtCamp - Infrastructure as code should contain code

Gianluca Arbezzano

More Decks by Gianluca Arbezzano

Other Decks in Technology

Featured

Transcript