Slide 1

Slide 1 text

Cloud-Init: The Good Parts James Nugent Event Store Ltd @jen20 jen20

Slide 2

Slide 2 text

Why Cloud-Init? • The de-facto industry standard for early-stage initialisation of virtual machines in the cloud running Unix-derived operating systems. • Used to specialise a generic operating system image at runtime to by provisioning a given set of configuration. • Originally developed by Canonical for configuring Ubuntu Linux running in Amazon EC2. • Now prevalent across all major clouds and most Unix-based operating systems.

Slide 3

Slide 3 text

Why Cloud-Init?

Slide 4

Slide 4 text

Why Cloud-Init?

Slide 5

Slide 5 text

Why Cloud-Init? • Building a new machine image for each role a virtual machine must play in your infrastructure can be costly in terms of time: • The cycle of booting a machine, customising it, and imaging can take anywhere from 5 minutes to over an hour. • For rapidly evolving software, building an image for each version of a program increases the amount of time it takes to get that version to production. • Correctly constrained, runtime customisation can provide many of the benefits of an image-based workflow, with different trade-offs. • Image-based workflow can still make use of CloudInit in the image building process!

Slide 6

Slide 6 text

Configuring Cloud-Init

Slide 7

Slide 7 text

Configuring Cloud-Init • The cloud-init package is installed in the operating system images supplied by most clouds. On systemd-based Linux, cloud-init.service usually runs at boot, as a oneshot service. • When started with the init sub-command, cloud-init runs the commands defined in a sequence of modules to specialise the operating system installation for the intended purpose. • Configuration comes from two sources: • Cloud provider-supplied metadata • User-supplied configuration

Slide 8

Slide 8 text

Configuring Cloud-Init

Slide 9

Slide 9 text

Example A Shell Script in User Data

Slide 10

Slide 10 text

A Shell Script in User Data

Slide 11

Slide 11 text

A Shell Script in User Data

Slide 12

Slide 12 text

A Shell Script in User Data

Slide 13

Slide 13 text

Example #cloud-config

Slide 14

Slide 14 text


Slide 15

Slide 15 text


Slide 16

Slide 16 text


Slide 17

Slide 17 text

#cloud-config Schema • #cloud-config is a complex YAML schema, whose valid components are affected by which modules are installed. • Documentation is somewhat hit-and-miss. Most of the information is in the docs, somewhere. All of the information is in the (python) source code of cloud-init. • Unless referring constantly to the code, writing the configuration files can be an iterative process of trial and error until you have a sufficiently large collection of reusable sections which you can cargo-cult into doing what you want. • cloud-init(1) has limited built-in schema validation functionality, but most modern editors will do just as well here with a YAML plugin.

Slide 18

Slide 18 text

#cloud-config Schema

Slide 19

Slide 19 text

#cloud-config Schema

Slide 20

Slide 20 text

#cloud-config Schema

Slide 21

Slide 21 text

Example Host SSH Keys

Slide 22

Slide 22 text

Host SSH Keys

Slide 23

Slide 23 text

Host SSH Keys

Slide 24

Slide 24 text

Host SSH Keys • There is no built-in module for specifying host keys at first boot, so we’ll need to build this ourselves. • Breaking this task down, we’ll need to do a few different things: • Generate some known host keys, and get them to the virtual machine • Move the keys into /etc/ssh before the first time the ssh.service unit starts • To do this correctly, it’s necessary for us to dig into the cloud-init default configuration to see what order modules run in, and choose the correct places to insert our logic.

Slide 25

Slide 25 text

Host SSH Keys • Cloud-init runs in three phases: • Init - essential configuration that must be done early on • Config - configuration that doesn’t affect other stages of boot • Final - configuration that must be run as late as possible • The configuration (by default) lives in /etc/cloud/cloud.cfg, and is YAML. • One of the pieces of configuration sets in cloud.cfg is which modules run in which phase.

Slide 26

Slide 26 text

Host SSH Keys

Slide 27

Slide 27 text

Host SSH Keys

Slide 28

Slide 28 text

Host SSH Keys

Slide 29

Slide 29 text

Host SSH Keys

Slide 30

Slide 30 text

Host SSH Keys

Slide 31

Slide 31 text

More about write-files • File content needs to be provided embedded in the YAML configuration file. • If we wanted to provide them from a remote source, we could use a script to download and verify checksums of whatever files we downloaded. • Properties we can set for each file are: • Content (in one of a variety of encodings) • Path • Owner • Mode

Slide 32

Slide 32 text

Host SSH Keys

Slide 33

Slide 33 text

Generating Host Keys

Slide 34

Slide 34 text

Generating Host Keys

Slide 35

Slide 35 text

Generating Host Keys

Slide 36

Slide 36 text

Generating Host Keys

Slide 37

Slide 37 text

Write-Files Configuration

Slide 38

Slide 38 text

Write-Files Configuration

Slide 39

Slide 39 text

Additional Configuration

Slide 40

Slide 40 text

Multi-part Configuration for User-Data

Slide 41

Slide 41 text

Generating Multi-part Cloud Config

Slide 42

Slide 42 text

Booting a Virtual Machine

Slide 43

Slide 43 text

Booting a Virtual Machine

Slide 44

Slide 44 text

About those docs…

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

ssh_keys Configuration

Slide 48

Slide 48 text

ssh_keys Configuration

Slide 49

Slide 49 text


Slide 50

Slide 50 text

ssh_keys Configuration

Slide 51

Slide 51 text

Other Use Cases • Some of the use cases we didn’t look at today, but are easy enough to accomplish: • Change the filesystem types for attached volumes (e.g. to XFS) • Configure a package repository (yum or apt) and install packages at boot • Install Docker, pull and run an image from Docker Hub • Run Chef or Puppet in standalone mode on boot, after downloading or installing the configuration from a package • Write out an /etc/machine-role to use as a base for accessing SSM Parameter Store trees • Join a node to a Serf or Consul cluster • Post a notification to Slack using the “phone home” module

Slide 52

Slide 52 text

Summary • Cloud-init packs in a huge amount of functionality, however it’s not necessarily very discoverable. • It is worth learning at least the basics if: • You want a runtime-specialisation-based workflow • You work across a diverse range of clouds or operating systems and want a common configuration tool

Slide 53

Slide 53 text