Linux Containers for Linux Beginners

Slide 1

Slide 1 text

LINUX CONTAINERS FOR LINUX BEGINNERS Ann Guilinger athenahealth Twitter: @aguilinger

Slide 2

Slide 2 text

WHO AM I? ➤Ann Guilinger ➤Software Developer at athenahealth ➤That’s my dog! ➤Twitter: @AGuilinger

Slide 3

Slide 3 text

LEVEL SETTING - THIS TALK ➤ Deep dive into how containers work and the diﬀerent processes/technologies ➤ No assumption that you know those ➤ Not speciﬁcally about LXC (LinuX Containers) ➤ Key takeaways: ➤ The problem containers solve ➤ How containers work ➤ A little on how linux works!

Slide 4

Slide 4 text

A BRIEF HISTORY OF HOW WE GOT TO CONTAINERS

Slide 5

Slide 5 text

HISTORY ➤ Start simple: web software needs to run on a machine ➤ Used to run on single, large, physical servers ➤ There were problems: ➤ Diﬀerent environments/set up ➤ Not enough/too much resources ➤ Fragile/catastrophic if that machine goes down

Slide 6

Slide 6 text

HISTORY ➤ Move towards isolated and replicable infrastructure - usually through virtual machines (VM) ➤ VM is operating system (OS) that can run on another OS ➤ Enabled cloud providers ➤ Problems now: ➤ Large and diﬃcult to move around ➤ Movement necessary to share between developers and deploy to production machinery

Slide 7

Slide 7 text

HISTORY ➤ Solution lighter than a VM: Containers! ➤ Lightweight way to isolate applications ➤ LXC (Literally LinuX Containers) offered full fledged container creation software ➤ Docker eventually came along and offered a robust container management ecosystem ➤ Gained more confidence as a secure solution

Slide 8

Slide 8 text

CONTAINERS ➤ Are a way of isolating processes (applications) ➤ More light weight than a Virtual Machine ➤ Are not a ﬁrst-class citizen of Linux (or any operating system) ➤ Instead, are made up of diﬀerent components such as namespaces and cgroups ➤ To understand these, we need to dig into how Linux works

Slide 9

Slide 9 text

LINUX BASICS *that are important in containers

Slide 10

Slide 10 text

FILES AND PROCESSES AND NETWORKING OH MY!

Slide 11

Slide 11 text

FILES ➤ “Everything is a ﬁle” in Linux

Slide 12

Slide 12 text

FILES ➤ ls -l <file> ➤ Types of file: ➤ - = regular file ➤ d = directory ➤ l = link ➤ c = special file ➤ s = socket ➤ p = pipe ➤ b = block device Type of file Permissions

Slide 13

Slide 13 text

FILES ➤ ls -l <file> ➤ Permissions ➤ r = read ➤ w = write ➤ x = execute ➤ - = no permission ➤ 3 groups of 3 permissions ➤ 1st group = owner of the file ➤ 2nd - members of the group owning the file ➤ 3rd - everyone else

Slide 14

Slide 14 text

PERMISSIONS (USERS AND GROUPS) ➤ Permission fall into three categories: ➤ user ➤ group ➤ everyone else ➤ Users = uniquely identiﬁed users on the system ➤ Groups grant permissions to sets of users based on who is “in” the group ➤ I.E. can have group 2 which contains users 1, 2, 3 ➤ Files owned by user 1 - users 2 and 3 get permission of the group

Slide 15

Slide 15 text

FILESYSTEMS - (1) ➤ First definition of filesystem: ➤ The tree of files in a Linux system starting at / ➤ / is the root directory ➤ Just a way of organizing stored data

Slide 16

Slide 16 text

FILESYSTEMS - (2) ➤ File systems are different ways to handle storing info about files ➤ Example types of filesystems: ➤ Journaling ➤ Log-structured ➤ Copy-on-Write ➤ Difference in how metadata, recovery, storage, access is handled

Slide 17

Slide 17 text

MOUNT ➤ A mount allows attaching another ﬁlesystem to the hierarchical ﬁlesystem of the current linux machine ➤ Example: Floppy, CD-ROM, USB

Slide 18

Slide 18 text

PROCESS ➤ Running instance of a script/program ➤ Uniquely identiﬁed with PID ➤ Files about process are stored in /proc/ ➤ Can get most info from other commands ➤ Example: lsof -p gives ﬁles open by process ➤ Can get same info from /proc/

Slide 19

Slide 19 text

PROCESS - EXAMPLE The script:

Slide 20

Slide 20 text

PROCESS

Slide 21

Slide 21 text

INTER-PROCESS COMMUNICATION ➤ Processes communicate ➤ Shared memory ➤ Read/write to “same location” ➤ Message passing ➤ Send and receive basic messages in a understandable format ➤ OS provides the channel to send the messages ➤ Example: pipes! |

Slide 22

Slide 22 text

INTER-PROCESS COMMUNICATION: SYSTEM V ➤ System V IPC mechanisms ➤ Message queues ➤ Send and receive messages in order ➤ Semaphores ➤ Non-negative integer that is incremented/decremented ➤ Shared memory ➤ Area of memory that appears to be the same between processes

Slide 23

Slide 23 text

NETWORK ➤ Lots could be covered here - but we’re keeping this simple ➤ Network is just a way of connecting many machines and devices ➤ Network stack has routes, ﬁrewalls, devices, etc…

Slide 24

Slide 24 text

USER SPACE VS KERNEL SPACE

Slide 25

Slide 25 text

LINUX BASICS - OS ARCHITECTURE ➤ Kernel space and user space ➤ Kernel: ➤ Has unrestricted access to hardware ➤ Can reference any memory address ➤ User space: ➤ Where applications run ➤ Talks to kernel to get access to hardware/memory

Slide 26

Slide 26 text

LINUX BASICS - OS ARCHITECTURE Kernel CPU Memory Devices Other hardware System calls User space Kernel space Applications (processes)

Slide 27

Slide 27 text

SYSTEM CALLS ➤ Programs eventually make system calls ➤ Example system calls: ➤ Files ➤ open ➤ read ➤ write ➤ close  ➤ Processes ➤ wait ➤ exec ➤ fork ➤ exit ➤ kill

Slide 28

Slide 28 text

LINUX SUMMARY ➤ Everything is a file ➤ Users have different permissions to files ➤ Can mount filesystems (arrangements of files) ➤ Processes are running instances of a program ➤ Processes communicate through interprocess communication (IPC) ➤ Machines/devices communicate over networks ➤ Kernel space and user space

Slide 29

Slide 29 text

VIRTUAL MACHINES

Slide 30

Slide 30 text

LINUX BASICS - OS ARCHITECTURE Kernel CPU Memory Devices Other hardware System calls User space Kernel space Applications (processes)

Slide 31

Slide 31 text

VIRTUAL MACHINES Host Kernel CPU Memory Devices Other hardware Hypervisor Guest OS Virtual Hardware App App Virtual Machine Guest OS Virtual Hardware App App Virtual Machine Non-VM Apps

Slide 32

Slide 32 text

VIRTUAL MACHINES ➤ Hypervisor lives on Host machine ➤ Responsible for sharing resources ➤ Responsible for isolating the VMS ➤ Virtual view of the hardware on guest OS (inside the VM) ➤ Guest OS still acts like it is talking directly to the hardware ➤ Diﬃcult to share virtual images - they are very large since they contain the entirety of the OS Guest OS Virtual Hardware App App Virtual Machine

Slide 33

Slide 33 text

VIRTUAL MACHINES Host Kernel CPU Memory Devices Other hardware Hypervisor Guest OS Virtual Hardware App App Virtual Machine Guest OS Virtual Hardware App App Virtual Machine Non-VM Apps

Slide 34

Slide 34 text

NAMESPACES AND CGROUPS the magic ingredients that make up containers

Slide 35

Slide 35 text

NAMESPACES

Slide 36

Slide 36 text

NAMESPACES ➤ A way to isolate certain properties so they do not interact with other namespaces ➤ Why? ➤ Shared resources can lead to problems ➤ Security ➤ Lot of other cool stuﬀ

Slide 37

Slide 37 text

NAMESPACES ➤ The namespaces for a process can be found in /proc ➤ ls -l /proc//ns

Slide 38

Slide 38 text

NAMESPACES ➤ Namespaces can be created by using one of three system calls: ➤ clone ➤ Creates a new process ➤ unshare ➤ Moves current process to a new namespace ➤ setns ➤ Join process to existing namespace ➤ Pass diﬀerent constants to specify which kind(s) of namespace

Slide 39

Slide 39 text

TYPES OF NAMESPACES ➤ Mount (MNT) ➤ Process ID (PID) ➤ Interprocess communication (IPC) ➤ Unix Timesharing System (UTS) ➤ User ID (USER) ➤ Network (NET) ➤ Control group (cgroup)

Slide 40

Slide 40 text

NAMESPACE - MNT ➤ Mount (MNT) ➤ CLONE_NEWNS ➤ Literally “new namespace” ➤ Allow diﬀerent views of the host ﬁlesystem

Slide 41

Slide 41 text

NAMESPACE - MNT

Slide 42

Slide 42 text

NAMESPACE - MNT Mount namespace 1 Mount namespace 2 usb mounted to /media/myusb no mounts

Slide 43

Slide 43 text

NAMESPACE - PID ➤ Process ID (PID) ➤ CLONE_NEWPID ➤ All PIDs must be unique within a namespace ➤ I.E. Every PID namespace can have a process with PID 1 (generally the init process)

Slide 44

Slide 44 text

NAMESPACE - PID Host Namespace

Slide 45

Slide 45 text

NAMESPACE - IPC ➤ Interprocess communication (IPC) ➤ CLONE_NEWIPC ➤ Isolate IPC resources provided by the system ➤ Namely System V IPC objects

Slide 46

Slide 46 text

NAMESPACE - IPC

Slide 47

Slide 47 text

NAMESPACE - UTS ➤ Unix Timesharing System (UTS) ➤ “Timesharing” = multiple users at once ➤ CLONE_NEWUTS ➤ Allow changing hostname and domain name within the namespace

Slide 48

Slide 48 text

NAMESPACE - UTS

Slide 49

Slide 49 text

NAMESPACE - USER ➤ User ID (USER) ➤ CLONE_NEWUSER ➤ Allows users to have diﬀering privileges inside and outside the namespace

Slide 50

Slide 50 text

NAMESPACE - USER

Slide 51

Slide 51 text

NAMESPACE - NET ➤ Network (NET) ➤ CLONE_NEWNET ➤ Control network capabilities within the namespace ➤ Physical network devices can only connect to one namespace ➤ All other namespaces that need to talk to that network must create a virtual network ➤ Virtual network communicates through a veth pair where there is interfaces on the namespace with the device and the namespace without

Slide 52

Slide 52 text

NAMESPACE - NET

Slide 53

Slide 53 text

CONTROL GROUPS (CGROUPS)

Slide 54

Slide 54 text

CGROUPS ➤ Way to limit and monitor resources ➤ I.E. CPU time, memory available, etc… ➤ Can see what controllers are available in /proc ➤ cat /proc/cgroups ➤ Can see cgroups that a process belongs to in /proc//cgroup

Slide 55

Slide 55 text

CGROUPS ➤ Two versions: v1, v2 ➤ v1 was a free-for-all of people adding controllers ➤ Caused inconsistencies and management became too complex ➤ Rewrote to v2 to make more sane ➤ Did not completely replace v1 ➤ Many controllers still only implemented in v1 ➤ Both still exist so all controllers work - v1 speciﬁc ones drop to the v1 cgroup implementation

Slide 56

Slide 56 text

CGROUPS

Slide 57

Slide 57 text

PUTTING IT TOGETHER

Slide 58

Slide 58 text

CONTAINERS ➤ Combination of the various namespaces and cgroups to create a way to limit a process ➤ Helps running multiple processes on same machine safely

Slide 59

Slide 59 text

LINUX BASICS - OS ARCHITECTURE Kernel CPU Memory Devices Other hardware System calls User space Kernel space Applications (processes)

Slide 60

Slide 60 text

CONTAINERS Kernel CPU Memory Devices Other hardware System calls User space Kernel space Process Process Process Container

Slide 61

Slide 61 text

DOCKER The rise of the usable container

Slide 62

Slide 62 text

DOCKER BASICS ➤ Container creation system ➤ Originally built on LXC ➤ One of the most popular container orchestration systems ➤ Uses containers to isolate the running processes - also wraps extra security, etc… ➤ Environment to run, maintain, share containers ➤ Uses images to cache info on building containers ➤ Stores images in a repository