Linux Containers for Linux Beginners

Deep dive into the tools an technologies underlying containers on the linux operating system

Transcript

1. LINUX CONTAINERS FOR LINUX BEGINNERS Ann Guilinger athenahealth Twitter: @aguilinger
2. WHO AM I? ➤Ann Guilinger ➤Software Developer at athenahealth ➤That’s my dog! ➤Twitter: @AGuilinger
3. LEVEL SETTING - THIS TALK ➤ Deep dive into how containers work and the different processes/technologies ➤ No assumption that you know those ➤ Not specifically about LXC (LinuX Containers) ➤ Key takeaways: ➤ The problem containers solve ➤ How containers work ➤ A little on how linux works!
4. A BRIEF HISTORY OF HOW WE GOT TO CONTAINERS
5. HISTORY ➤ Start simple: web software needs to run on a machine ➤ Used to run on single, large, physical servers ➤ There were problems: ➤ Different environments/set up ➤ Not enough/too much resources ➤ Fragile/catastrophic if that machine goes down
6. HISTORY ➤ Move towards isolated and replicable infrastructure - usually through virtual machines (VM) ➤ VM is operating system (OS) that can run on another OS ➤ Enabled cloud providers ➤ Problems now: ➤ Large and difficult to move around ➤ Movement necessary to share between developers and deploy to production machinery
7. HISTORY ➤ Solution lighter than a VM: Containers! ➤ Lightweight way to isolate applications ➤ LXC (Literally LinuX Containers) offered full fledged container creation software ➤ Docker eventually came along and offered a robust container management ecosystem ➤ Gained more confidence as a secure solution
8. CONTAINERS ➤ Are a way of isolating processes (applications) ➤ More light weight than a Virtual Machine ➤ Are not a first-class citizen of Linux (or any operating system) ➤ Instead, are made up of different components such as namespaces and cgroups ➤ To understand these, we need to dig into how Linux works
9. LINUX BASICS *that are important in containers
10. FILES AND PROCESSES AND NETWORKING OH MY!
11. FILES ➤ “Everything is a file” in Linux
12. FILES ➤ ls -l <file> ➤ Types of file: ➤ - = regular file ➤ d = directory ➤ l = link ➤ c = special file ➤ s = socket ➤ p = pipe ➤ b = block device Type of file Permissions
13. FILES ➤ ls -l <file> ➤ Permissions ➤ r = read ➤ w = write ➤ x = execute ➤ - = no permission ➤ 3 groups of 3 permissions ➤ 1st group = owner of the file ➤ 2nd - members of the group owning the file ➤ 3rd - everyone else
14. PERMISSIONS (USERS AND GROUPS) ➤ Permission fall into three categories: ➤ user ➤ group ➤ everyone else ➤ Users = uniquely identified users on the system ➤ Groups grant permissions to sets of users based on who is “in” the group ➤ I.E. can have group 2 which contains users 1, 2, 3 ➤ Files owned by user 1 - users 2 and 3 get permission of the group
15. FILESYSTEMS - (1) ➤ First definition of filesystem: ➤ The tree of files in a Linux system starting at / ➤ / is the root directory ➤ Just a way of organizing stored data
16. FILESYSTEMS - (2) ➤ File systems are different ways to handle storing info about files ➤ Example types of filesystems: ➤ Journaling ➤ Log-structured ➤ Copy-on-Write ➤ Difference in how metadata, recovery, storage, access is handled
17. MOUNT ➤ A mount allows attaching another filesystem to the hierarchical filesystem of the current linux machine ➤ Example: Floppy, CD-ROM, USB
18. PROCESS ➤ Running instance of a script/program ➤ Uniquely identified with PID ➤ Files about process are stored in /proc/<PID> ➤ Can get most info from other commands ➤ Example: lsof -p <PID> gives files open by process ➤ Can get same info from /proc/<PID>
19. PROCESS - EXAMPLE The script:
20. PROCESS
21. INTER-PROCESS COMMUNICATION ➤ Processes communicate ➤ Shared memory ➤ Read/write to “same location” ➤ Message passing ➤ Send and receive basic messages in a understandable format ➤ OS provides the channel to send the messages ➤ Example: pipes! |
22. INTER-PROCESS COMMUNICATION: SYSTEM V ➤ System V IPC mechanisms ➤ Message queues ➤ Send and receive messages in order ➤ Semaphores ➤ Non-negative integer that is incremented/decremented ➤ Shared memory ➤ Area of memory that appears to be the same between processes
23. NETWORK ➤ Lots could be covered here - but we’re keeping this simple ➤ Network is just a way of connecting many machines and devices ➤ Network stack has routes, firewalls, devices, etc…
24. USER SPACE VS KERNEL SPACE
25. LINUX BASICS - OS ARCHITECTURE ➤ Kernel space and user space ➤ Kernel: ➤ Has unrestricted access to hardware ➤ Can reference any memory address ➤ User space: ➤ Where applications run ➤ Talks to kernel to get access to hardware/memory
26. LINUX BASICS - OS ARCHITECTURE Kernel CPU Memory Devices Other hardware System calls User space Kernel space Applications (processes)
27. SYSTEM CALLS ➤ Programs eventually make system calls ➤ Example system calls: ➤ Files ➤ open ➤ read ➤ write ➤ close
 ➤ Processes ➤ wait ➤ exec ➤ fork ➤ exit ➤ kill
28. LINUX SUMMARY ➤ Everything is a file ➤ Users have different permissions to files ➤ Can mount filesystems (arrangements of files) ➤ Processes are running instances of a program ➤ Processes communicate through interprocess communication (IPC) ➤ Machines/devices communicate over networks ➤ Kernel space and user space
29. VIRTUAL MACHINES
30. LINUX BASICS - OS ARCHITECTURE Kernel CPU Memory Devices Other hardware System calls User space Kernel space Applications (processes)
31. VIRTUAL MACHINES Host Kernel CPU Memory Devices Other hardware Hypervisor Guest OS Virtual Hardware App App Virtual Machine Guest OS Virtual Hardware App App Virtual Machine Non-VM Apps
32. VIRTUAL MACHINES ➤ Hypervisor lives on Host machine ➤ Responsible for sharing resources ➤ Responsible for isolating the VMS ➤ Virtual view of the hardware on guest OS (inside the VM) ➤ Guest OS still acts like it is talking directly to the hardware ➤ Difficult to share virtual images - they are very large since they contain the entirety of the OS Guest OS Virtual Hardware App App Virtual Machine
33. VIRTUAL MACHINES Host Kernel CPU Memory Devices Other hardware Hypervisor Guest OS Virtual Hardware App App Virtual Machine Guest OS Virtual Hardware App App Virtual Machine Non-VM Apps
34. NAMESPACES AND CGROUPS the magic ingredients that make up containers
35. NAMESPACES
36. NAMESPACES ➤ A way to isolate certain properties so they do not interact with other namespaces ➤ Why? ➤ Shared resources can lead to problems ➤ Security ➤ Lot of other cool stuff
37. NAMESPACES ➤ The namespaces for a process can be found in /proc ➤ ls -l /proc/<PID>/ns
38. NAMESPACES ➤ Namespaces can be created by using one of three system calls: ➤ clone ➤ Creates a new process ➤ unshare ➤ Moves current process to a new namespace ➤ setns ➤ Join process to existing namespace ➤ Pass different constants to specify which kind(s) of namespace
39. TYPES OF NAMESPACES ➤ Mount (MNT) ➤ Process ID (PID) ➤ Interprocess communication (IPC) ➤ Unix Timesharing System (UTS) ➤ User ID (USER) ➤ Network (NET) ➤ Control group (cgroup)
40. NAMESPACE - MNT ➤ Mount (MNT) ➤ CLONE_NEWNS ➤ Literally “new namespace” ➤ Allow different views of the host filesystem
41. NAMESPACE - MNT
42. NAMESPACE - MNT Mount namespace 1 Mount namespace 2 usb mounted to /media/myusb no mounts
43. NAMESPACE - PID ➤ Process ID (PID) ➤ CLONE_NEWPID ➤ All PIDs must be unique within a namespace ➤ I.E. Every PID namespace can have a process with PID 1 (generally the init process)
44. NAMESPACE - PID Host Namespace
45. NAMESPACE - IPC ➤ Interprocess communication (IPC) ➤ CLONE_NEWIPC ➤ Isolate IPC resources provided by the system ➤ Namely System V IPC objects
46. NAMESPACE - IPC
47. NAMESPACE - UTS ➤ Unix Timesharing System (UTS) ➤ “Timesharing” = multiple users at once ➤ CLONE_NEWUTS ➤ Allow changing hostname and domain name within the namespace
48. NAMESPACE - UTS
49. NAMESPACE - USER ➤ User ID (USER) ➤ CLONE_NEWUSER ➤ Allows users to have differing privileges inside and outside the namespace
50. NAMESPACE - USER
51. NAMESPACE - NET ➤ Network (NET) ➤ CLONE_NEWNET ➤ Control network capabilities within the namespace ➤ Physical network devices can only connect to one namespace ➤ All other namespaces that need to talk to that network must create a virtual network ➤ Virtual network communicates through a veth pair where there is interfaces on the namespace with the device and the namespace without
52. NAMESPACE - NET
53. CONTROL GROUPS (CGROUPS)
54. CGROUPS ➤ Way to limit and monitor resources ➤ I.E. CPU time, memory available, etc… ➤ Can see what controllers are available in /proc ➤ cat /proc/cgroups ➤ Can see cgroups that a process belongs to in /proc/<PID>/cgroup
55. CGROUPS ➤ Two versions: v1, v2 ➤ v1 was a free-for-all of people adding controllers ➤ Caused inconsistencies and management became too complex ➤ Rewrote to v2 to make more sane ➤ Did not completely replace v1 ➤ Many controllers still only implemented in v1 ➤ Both still exist so all controllers work - v1 specific ones drop to the v1 cgroup implementation
56. CGROUPS
57. PUTTING IT TOGETHER
58. CONTAINERS ➤ Combination of the various namespaces and cgroups to create a way to limit a process ➤ Helps running multiple processes on same machine safely
59. LINUX BASICS - OS ARCHITECTURE Kernel CPU Memory Devices Other hardware System calls User space Kernel space Applications (processes)
60. CONTAINERS Kernel CPU Memory Devices Other hardware System calls User space Kernel space Process Process Process Container
61. DOCKER The rise of the usable container
62. DOCKER BASICS ➤ Container creation system ➤ Originally built on LXC ➤ One of the most popular container orchestration systems ➤ Uses containers to isolate the running processes - also wraps extra security, etc… ➤ Environment to run, maintain, share containers ➤ Uses images to cache info on building containers ➤ Stores images in a repository
63. DEMO TIME Showing the namespaces in action in a Docker container
64. THANK YOU! questions? twitter: @aguilinger