[deSymfony] Docker on every environment

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Containers. How do we transport containers from development to production?

Slide 5

Slide 5 text

You take the red pill, you stay in Wonderland, and I show you how deep the rabbit hole goes.

Slide 6

Slide 6 text

Docker provides a beautiful API that hides the real complexity.

Slide 7

Slide 7 text

Why do we call them containers?

Slide 8

Slide 8 text

ISO standards for containers were published between 1968 and 1970.

Slide 9

Slide 9 text

Software containers try to be the standard for Applications distribution Applications runtime

Slide 10

Slide 10 text

Development Workflow Dev Prod CI / CD

Slide 11

Slide 11 text

CI / CD Development Workflow Dev Prod

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Distribution: Container Images

Slide 14

Slide 14 text

Packaging applications for distribution is not a new problem.

Slide 15

Slide 15 text

❏ Disk images .iso ❏ VMware .vdmk ❏ Vagrant .box ❏ Amazon Machine Images AMI Systems Packaging

Slide 16

Slide 16 text

❏ zip/tgz ❏ Java jar/war ❏ Debian deb ❏ RedHat rpm Application Packaging

Slide 17

Slide 17 text

Container images combine both system and application packaging. Old best practices still apply.

Slide 18

Slide 18 text

Container images can be easily built using manifests.

Slide 19

Slide 19 text

Building the same manifest twice could produce different images.

Slide 20

Slide 20 text

The difference between how you think something works and how it actually works risks hard-to-debug production issues. Gareth Rushgrove @garethr

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

❏ Which OS is it based on? ❏ Which packages are installed? ❏ What application is running inside? Giving a running container

Slide 23

Slide 23 text

❏ Which OS is it based on? ❏ Which packages are installed? ❏ What application is running inside? Giving a running container

Slide 24

Slide 24 text

Operating System Which Alpine? FROM alpine CMD [“echo”, “Knock”, ”Knock”, “Neo”]

Slide 25

Slide 25 text

Operating System Is this better? FROM alpine:3.4 CMD [“echo”, “Knock”, ”Knock”, “Neo”]

Slide 26

Slide 26 text

Operating System Tags can be overwritten! 3.4 won’t be the same in two weeks, probably FROM alpine:3.4 CMD [“echo”, “Knock”, ”Knock”, “Neo”]

Slide 27

Slide 27 text

❏ Which OS is it based on? ❏ Which packages are installed? ❏ What application is running inside? Giving a running container

Slide 28

Slide 28 text

Packages Which pip? FROM alpine:3.4 RUN apk add -‐-‐update py-‐pip CMD [“echo”, “Knock”, ”Knock”, “Neo”]

Slide 29

Slide 29 text

Versions Specify the version FROM alpine:3.4 RUN apk add -‐-‐update py-‐pip=8.1.2-‐r0 CMD [“echo”, “Knock”, ”Knock”, “Neo”]

Slide 30

Slide 30 text

❏ Which OS is it based on? ❏ Which packages are installed? ❏ What application is running inside? Giving a running container

Slide 31

Slide 31 text

Application Which version of our application? FROM alpine:3.4 RUN apk add -‐-‐update py-‐pip=8.1.2-‐r0 COPY app.py /app.py CMD [“python”, “/app.py”]

Slide 32

Slide 32 text

Metadata Use Docker Labels for application metadata FROM alpine:3.4 ARG vcs_ref="Unknown" ARG build_date="Unknown" RUN apk add -‐-‐update py-‐pip=8.1.2-‐r0 LABEL org.label-‐schema.vcs-‐ref=$vcs_ref \ org.label-‐schema.build-‐date=$build_date COPY app.py /app.py CMD [“python”, “/app.py”]

Slide 33

Slide 33 text

Slide 34

Slide 34 text

Standard for Docker labels

Slide 35

Slide 35 text

Use labels to extract info

Slide 36

Slide 36 text

Slide 37

Slide 37 text

Metadata Calculate the values for the labels $ docker build \ -‐-‐build-‐arg vcs_ref=`git rev-‐parse HEAD` \ -‐-‐build-‐arg date=`date -‐u + "%Y-‐%m-‐%dT%H:%MZ"` \ -‐t your_image_name .

Slide 38

Slide 38 text

Open Source Docker Registries

Slide 39

Slide 39 text

Docker Hub

Slide 40

Slide 40 text

Official Docker Registry

Slide 41

Slide 41 text

Harbor (VMware)

Slide 42

Slide 42 text

Port.us (Suse)

Slide 43

Slide 43 text

Paid Docker Registries

Slide 44

Slide 44 text

Docker DataCenter

Slide 45

Slide 45 text

AWS ECR

Slide 46

Slide 46 text

JFrog Artifactory

Slide 47

Slide 47 text

Development Workflow Dev CI / CD Prod

Slide 48

Slide 48 text

Building our images

Slide 49

Slide 49 text

Building the same manifest twice could produce different images.

Slide 50

Slide 50 text

Build once. Promote images to different environments.

Slide 51

Slide 51 text

Jenkins Workflow 1. Detect merge to repository

Slide 52

Slide 52 text

Jenkins Workflow 1. Detect merge to repository 2. If tests pass, build image and push it to pre production registry

Slide 53

Slide 53 text

Jenkins Workflow 1. Detect merge to repository 2. If tests pass, build image and push it to pre production registry 3. Deploy to pre environment

Slide 54

Slide 54 text

Jenkins Workflow 1. Detect merge to repository 2. If tests pass, build image and push it to pre production registry 3. Deploy to pre environment 4. If tests pass, push image to pro registry

Slide 55

Slide 55 text

Jenkins Workflow 1. Detect merge to repository 2. If tests pass, build image and push it to pre production registry 3. Deploy to pre environment 4. If tests pass, push image to pro registry 5. Deploy to production

Slide 56

Slide 56 text

Keep In Mind ❏ Be clear on which versions of docker/docker-compose you allow ❏ Use Jenkins build number or timestamp as image tag ❏ Seek a Generic Build process ❏ Clean old images/containers

Slide 57

Slide 57 text

Clean old images/containers

Slide 58

Slide 58 text

CI / CD Development Workflow Dev Prod

Slide 59

Slide 59 text

Container Runtime

Slide 60

Slide 60 text

Container vs VM comparison

Slide 61

Slide 61 text

It’s better to think about containers as regular unix processes

Slide 62

Slide 62 text

Run only one process

Slide 63

Slide 63 text

❏ Easier debugging ❏ Simplified setup of networks, ports... ❏ Logs have only one format ❏ Resources sharing: CPU vs memory Running one process

Slide 64

Slide 64 text

Processes inside containers are started and stopped the same way a unix system would do it.

Slide 65

Slide 65 text

❏ How they start? ❏ How they stop? Containers and processes share a similar lifecycle

Slide 66

Slide 66 text

Unix Processes: the chosen PID1 (the One, got it? :D)

Slide 67

Slide 67 text

When you run the container, the init (PID1) process is started, which is responsible for starting other processes, creating a tree-like hierarchy.

Slide 68

Slide 68 text

$ ps -‐e -‐o user,pid,ppid,args -‐-‐forest UID PID PPID CMD root 1 0 init root 205 1 /sbin/udevd -‐-‐daemon root 1113 205 \_ /sbin/udevd -‐-‐daemon root 1114 205 \_ /sbin/udevd -‐-‐daemon root 1199 1 crond -‐f -‐d 8 root 1216 1 VBoxService root 1241 1 /usr/local/sbin/acpid root 1249 1 /sbin/udhcpc -‐b -‐i eth1...

Slide 69

Slide 69 text

Every process has a parent process, except for the init process.

Slide 70

Slide 70 text

When a process ends, its entry in the process table remains, keeping the process in a defunct or zombie status.

Slide 71

Slide 71 text

Only after the parent read the child’s exit status to know what happened, the zombie is removed. This is called reaping.

Slide 72

Slide 72 text

The init process also adopts orphans. An orphan is a process that is still executing but whose parent has died.

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

If your process is running as PID1, it’s probably expecting a init process to properly adopting and reaping processes.

Slide 75

Slide 75 text

Running our process as a subcommand of bash would solve this problem. But bash doesn’t handle unix signals.

Slide 76

Slide 76 text

Unix Signals

Slide 77

Slide 77 text

A signal is an asynchronous notification sent to a process in order to notify it of an event that occurred.

Slide 78

Slide 78 text

SIGINT Interrupt from keyboard SIGKILL Kill process immediately SIGTERM Request nice process termination

Slide 79

Slide 79 text

Signals sent by the Docker engine to the containers are handled by the init process.

Slide 80

Slide 80 text

Unless a process has registered a custom signal handler for SIGTERM, the kernel will fall back sending a SIGKILL signal.

Slide 81

Slide 81 text

For PID1, though, the kernel won’t even fall back to SIGKILL. If your process hasn’t registered a handler, SIGTERM will have no effect on the process.

Slide 82

Slide 82 text

No content

Slide 83

Slide 83 text

By default, docker stop sends a SIGTERM and waits 10 seconds for the container to stop. After that, it sends a SIGKILL.

Slide 84

Slide 84 text

nginx expects a SIGQUIT to do a graceful shutdown. Apache expects a SIGWINCH.

Slide 85

Slide 85 text

Options for your container entrypoint ❏ shell form

Slide 86

Slide 86 text

Options for your container entrypoint ❏ shell form ❏ exec form

Slide 87

Slide 87 text

Options for your container entrypoint ❏ shell form ❏ exec form ❏ Using a proper init process

Slide 88

Slide 88 text

shell form ❏ ENTRYPOINT command param1 param2 ❏ The process will be started as a subcommand of /bin/sh -c ❏ Remember, bash does not pass signals

Slide 89

Slide 89 text

exec form ❏ ENTRYPOINT ["executable", "param1"] ❏ The process will be PID1 init process inside the container ❏ Your process should adopt and take care of reaping

Slide 90

Slide 90 text

Using a proper init process ❏ A init process that pass signals, adopt orphans and takes care of reaping. ❏ There are several init processes for containers ❏ tini ❏ dumb-init

Slide 91

Slide 91 text

Demo

Slide 92

Slide 92 text

Wrapping Up

Slide 93

Slide 93 text

We need formal pipelines to promote images from development to production. Build once and promote.

Slide 94

Slide 94 text

Without a proper init system ❏ Zombie processes could become a problem in the system. ❏ Signals are not passed to your process as you may expect, not being able to gracefully stop a process.