Introduction to Docker and Docker Compose

Slide 1

Slide 1 text

101 Engineering Team Minh Nguyen & Luong Vo

Slide 2

Slide 2 text

- Let’s look into Eh Avatar, an application to generate a beautiful avatar for you <3 - https://github.com/Thinkei/eh-avatar - Let’s setup and run the project Before we start

Slide 3

Slide 3 text

Application Flow Web server 1) POST /avatars --data name=”Minh” Postgres Sidekiq server Redis 2) Save into DB 3) Send to sidekiq via Redis 4) Background job to generate avatar RMagick 5) Get /avatars/1

Slide 4

Slide 4 text

Section 1: What is docker?

Slide 5

Slide 5 text

What is Docker First, let’s talk about Linux containers….

Slide 6

Slide 6 text

Linux container technology - Linux containers contain applications in a way that keep them isolated from the host system that they run on. - Allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies - Make it easier to provide a consistent experience between development and production environment - Easy to deploy and replicate deployments

Slide 7

Slide 7 text

But why should we care ?

Slide 8

Slide 8 text

The challenge

Slide 9

Slide 9 text

The challenge…. but simpler

Slide 10

Slide 10 text

The real-life solution

Slide 11

Slide 11 text

The solution that we won’t talk about

Slide 12

Slide 12 text

But...

Slide 13

Slide 13 text

Why Containers but not VM?

Slide 14

Slide 14 text

Why Containers but not VM? - We don’t need an entire operating system, just some components to make everything up and run. - Container use host kernel to ensure the isolation and resource control, instead of full layer of visualization - Therefore, containers are - Fast in both starting up and operation - Lower memory footprint - Lightweight

Slide 15

Slide 15 text

Linux container technology is not new - First form of container was Chroot, 1982 - FreeBSD Jails, 2000 - Solaris Zones, 2004 - Linux OpenVZ, 2005 - LXC, 2008 - Docker, 2013 - Built on LXC - Moved to libcontainer (March 2014) - Moved to runC (July 2015)

Slide 16

Slide 16 text

What is Docker? - Container solution that provides full packages: - Image management - Resource Isolation - File System Isolation - Network Isolation - Change Management - Sharing - Process Management - Service Discovery (DNS since 1.10)

Slide 17

Slide 17 text

Section 2: Working with Docker

Slide 18

Slide 18 text

Docker concepts Source code Dockerfile Build Docker image Push Image registry Host Pull Docker container 1 Docker container 2 Start

Slide 19

Slide 19 text

Image vs Container? - Images are read-only. - An instance of an image is called a container. - You can have many running containers of the same image. - Images are created with the build command, and they'll produce a container when started with run.

Slide 20

Slide 20 text

Docker concepts - Each image has a name and usually attached tags - All docker images with the same name are grouped - The most popular docker image registry is Docker Hub - https://hub.docker.com/r/library/postgres/tags/ - There are other public / private image registries from AWS, Google, etc.

Slide 21

Slide 21 text

Practice 1: Bring up dependencies - Checkout branch docker_1 of the repository - Or view directly on this link: - https://github.com/Thinkei/eh-avatar/blob/docker_1/DOC KER_1.md

Slide 22

Slide 22 text

Practice 1: Bring up dependencies - Install Docker on your local machine. https://docs.docker.com/install/ - Pull image of redis docker pull redis:latest - Pull image of postgres docker pull postgres:latest

Slide 23

Slide 23 text

Practice 1: Bring up dependencies - Start redis docker run --name my-redis redis - Test the redis server by command redis-cli -h localhost - And it fails :troll:

Slide 24

Slide 24 text

Practice 1: Bring up dependencies - Why couldn’t we connect to the redis server in postgres? + All containers are network isolated + It means that containers could not access others’ network and the host could not access containers’ network

Slide 25

Slide 25 text

Docker network diagram (simpliﬁed) Host’s network interface (no port) Network interface A (open 6379) Container A: Redis (port 6379) Network interface B Container B Process Sidekiq Process redis-cli

Slide 26

Slide 26 text

Practice 1: Bring up dependencies - Forward port 6379 of container to outside with port 6378 docker run --name my-redis -p 6378:6379 redis - Test the redis server by command redis-cli -h localhost -p 6378 - And it works

Slide 27

Slide 27 text

Docker network diagram (simpliﬁed) Host’s network interface (open 6378) Network interface A (open 6379) Container A: Redis (port 6379) Network interface B Container B Process Sidekiq Process redis-cli

Slide 28

Slide 28 text

Practice 1: Bring up dependencies - Bring up postgres docker run \ --name my-postgres \ -p 5433:5432 \ -e POSTGRES_PASSWORD=password \ -e POSTGRES_USER=username \ -e POSTGRES_DB=ehavatar \ postgres

Slide 29

Slide 29 text

Practice 1: Bring up dependencies - Setup schema again DATABASE_URL=postgres://username:password@localhost:5433/ ehavatar \ REDIS_URL=redis://localhost:6378 \ bundle exec ruby setup_schema.rb

Slide 30

Slide 30 text

Practice 1: Bring up dependencies - Start web server on host machine DATABASE_URL=postgres://username:password@localhost:5433/ ehavatar \ REDIS_URL=redis://localhost:6378 \ bundle exec puma

Slide 31

Slide 31 text

Practice 1: Bring up dependencies - Start sidekiq on host machine DATABASE_URL=postgres://username:password@localhost:5433/ ehavatar \ REDIS_URL=redis://localhost:6378 \ bundle exec bundle exec sidekiq -r ./config/environment.rb

Slide 32

Slide 32 text

Practice 1: Bring up dependencies - Let’s turn off the postgres docker container and restart - Oops, all data is gone :’(

Slide 33

Slide 33 text

Practice 1: Bring up dependencies - Let’s turn off the postgres docker container and restart - Oops, all data is gone :’( - What did happen? + All containers are ﬁle system isolated + All containers data are not mounted (linked) to the host. When it starts again, no data is retained

Slide 34

Slide 34 text

Docker File system diagram (simpliﬁed) Container A’s Virtual FIle system Container A: Postgres Host’s real file system

Slide 35

Slide 35 text

Practice 1: Bring up dependencies - Try again with this command docker run \ --name my-postgres \ -p 5433:5432 \ -e POSTGRES_PASSWORD=password \ -e POSTGRES_USER=username \ -e POSTGRES_DB=ehavatar \ -e PGDATA=/var/data \ -v $(pwd)/tmp/data:/var/data \ postgres

Slide 36

Slide 36 text

Docker File system diagram (simpliﬁed) Container A’s Virtual FIle system Container A: Postgres Host’s real file system /var/data /Users/ahihi/tmp/data

Slide 37

Slide 37 text

Docker concepts - Dockerﬁle is a text document that contains all the commands a user could call on the command line to assemble an image. - It deﬁnes: - Is current image based on other image? If yes, what is it? - Dependencies installation commands - How to start the container of this image? - What environments are allowed to be passed in? - Which ports the container will expose? - etc.

Slide 38

Slide 38 text

Practice 2: Build your own docker image - Checkout branch docker_2 - Or view online at https://github.com/Thinkei/eh-avatar/blob/docker_1/DOC KER_1.md

Slide 39

Slide 39 text

Practice 2: Build your own docker image - Start Postgres and Redis container again we don't need to expose ports - Create a new ﬁle Dockerﬁle with the following content FROM ruby:2.4.0 RUN mkdir -p /app WORKDIR /app COPY Gemfile Gemfile.lock ./ RUN gem install bundler && bundle install --jobs 20 --retry 5 --without test COPY . ./ EXPOSE 9292 CMD ["bundle", "exec", "puma"]

Slide 40

Slide 40 text

Notable concept that you won’t know - Union File Systems - because sharing is caring

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

Practice 2: Build your own docker image - Build the docker image with command docker build -t ehavatar . - The building process fails. We fail to build RMgick gem. Need to add dependencies. Add this line to Dockeriﬂe RUN apt-get update -qq --fix-missing && apt-get install -y libmagickwand-dev

Slide 43

Slide 43 text

Practice 2: Build your own docker image - Build again. And it is sure to be successful. - Start web server and sidekiq docker run -p 9292:9292 eh-avatar docker run -p 9292:9292 eh-avatar bundle exec sidekiq -r ./config/environment.rb - And it fails again. We fail to connect to redis and postgres

Slide 44

Slide 44 text

Practice 2: Build your own docker image - Using `link` to link the application containers to dependencies docker run \ --link my-redis \ --link my-postgres \ -e DATABASE_URL=postgres://username:password@my-postgres:5432/ehavatar \ -e REDIS_URL=redis://my-redis:6379 \ -v $(pwd)/tmp:/app/tmp \ -p 9292:9292 \ eh-avatar

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Section 3: Docker Compose

Slide 47

Slide 47 text

Docker Compose

Slide 48

Slide 48 text

Docker Compose - A tool for deﬁning and running multi-container Docker applications. - Run multiple isolated environments on a single host with a single command. - Better Development environments. - Easier build / run / scale.

Slide 49

Slide 49 text

Practice 3: Use docker-compose

Slide 50

Slide 50 text

Best practice and pitfalls 1. Containers should be ephemeral 2. Specify a build context 3. Use a .dockerignore ﬁle 4. Use multi-stage builds 5. Avoid installing unnecessary packages 6. Minimize the number of layers 7. ….

Slide 51

Slide 51 text

After this, do you become a master of Docker? - This is just a beginning :troll: - There are a lot of things haven’t been mentioned yet