Container Migration and CRIU Details

Join Adrian Reber, Red Hat, to get a detailed technical background on how Checkpoint/Restore In Userspace (CRIU) works and how CRIU enables container migration.

Transcript

1. Container Live Migration Adrian Reber 2020, June 30
2. Red Hat Blog: Container migration with Podman on RHEL https://www.redhat.com/en/blog/container-migration-podman-rhel 2
3. Agenda Use cases Details Demos Future 3
4. Definition: Container Live Migration 4
5. Transfer Running Container 5
6. Serialize on Source System 6
7. Transfer to Destination System 7
8. Checkpoint/Restore in Userspace CRIU 8
9. Multiple Integrations Exist 9
10. Use Cases 10
11. Reboot and Save State 11
12. Host Container 12
13. Host Container 13
14. Host Container 14
15. Host Container 15
16. Quick Startup 16
17. Host Container 17
18. Host Container Container 18
19. Host Container Container Container Container 19
20. Container Live Migration 20
21. Source Container Destination 21
22. Source Container Container Destination 22
23. Source Container Container Destination Container Container 23
24. CRIU 24
25. First Step: Checkpointing 25
26. Seize Process Using ptrace() 26
27. Collect Details From /proc/<PID>/* 27
28. Parasite Code 28
29. Parasite Code Most favorite part 29
30. Parasite Code And the craziest 30
31. Parasite Code Injected into the process 31
32. Parasite Code Daemon waiting for commands 32
33. Parasite Code Removed after usage 33
34. Checkpointed To Be Code Process Original 34
35. Checkpointed To Be Code Process Original 35
36. Checkpointed To Be Parasite Code Process Original 36
37. Checkpointed To Be Code Process Original 37
38. Checkpointing Finished 38
39. Checkpointing Finished All relevant information written 39
40. Checkpointing Finished Target process is killed 40
41. Checkpointing Finished Or continues to run 41
42. Container Live Migration SELinux Linux Security Summit EU 2019 https://sched.co/Tymj 42
43. 2015: CRIU LSM support 43
44. During checkpointing Read /proc/PID/attr/current 44
45. During restore Write /proc/PID/attr/current 45
46. 1 if (!strstartswith(last, "unconfined_")) { 2 pr_err("Non unconfined selinux contexts not supported %s\n", last); 3 freecon(ctx); 4 return -1; 5 } 46
47. • setsockcreatecon(3) for parasite daemon • Write to /proc/PID/attr/current • Allow dyntransition • Do not set context of threads • Allow writing to /proc/sys/kernel/ns_last_pid • Fix socket labels • Pre-create CRIU log files with appropriate labels • Fix file descriptor leaks 47
48. Second/Last Step: Restoring 48
49. Read Checkpoint Images 49
50. clone() For Each PID/TID LPC: CRIU and the PID dance clone3() with Linux 5.5 https://linuxplumbersconf.org/event/4/contributions/472/ 50
51. PID dance open() /proc/sys/kernel/ns_last_pid write() (PID - 1) to ns_last_pid close() ns_last_pid clone() getpid() 51
52. Avoiding the PID dance (2010): eclone() https://lore.kernel.org/patchwork/patch/198220/ 52
53. Avoiding the PID dance (2019): clone3() 53
54. ”In general, clone3() is extensible and allows for the implementation of new features.” https://git.kernel.org/pub/scm/linux/kernel/git/ torvalds/linux.git/commit/?id=7f192e3cd316ba58c 54
55. clone3() with set_tid 55
56. 1 struct clone_args args = {0}; 2 pid_t *set_tid; 3 set_tid[0] = 2020; 4 args.set_tid = set_tid; 5 args.set_tid_size = 1; 6 syscall(__NR_clone3 , args, sizeof(struct clone_args)); 56
57. CRIU Morphs Itself Open and position file descriptors 57
58. CRIU Morphs Itself Map memory pages 58
59. CRIU Morphs Itself Load security settings 59
60. CRIU Morphs Itself Jump into restored process 60
61. Container Live Migration 61
62. Container Live Migration OpenVZ 62
63. Container Live Migration Borg 63
64. Container Live Migration LXC/LXD 64
65. Container Live Migration Docker 65
66. Container Live Migration Podman 66
67. Podman: daemonless 67
68. Podman: rootless 68
69. Podman: Checkpoint/Restore October 2018 69
70. Podman: Checkpoint/Restore Required runc and CRIU changes 70
71. Podman: Container Live Migration June 2019 71
72. Podman: Container Live Migration Required runc, CRIU, SELinux changes 72
73. Checkpoint includes File System Changes 73
74. 1 # podman run --rm -d adrianreber/wildfly -hello 2 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a 3 # podman inspect -l --format "{{.NetworkSettings.IPAddress}}" 4 10.88.0.247 5 # curl 10.88.0.247:8080/helloworld/ 6 0 7 # curl 10.88.0.247:8080/helloworld/ 8 1 9 # podman container checkpoint -l --export=/tmp/chkpt.tar.gz 10 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a 11 # scp /tmp/chkpt.tar.gz rhel08:/tmp 74
75. 1 # podman container restore --import=/tmp/chkpt.tar.gz 2 699f33eb7fecbc5bbb00400be0aa79c888dbc63a54cac7bd2eed836a57d8a68a 3 # podman inspect -l --format "{{.NetworkSettings.IPAddress}}" 4 10.88.0.247 5 # curl 10.88.0.247:8080/helloworld/ 6 2 7 # curl 10.88.0.247:8080/helloworld/ 8 3 75
76. 1 # podman container restore --import=/tmp/chkpt.tar.gz -n hello1 2 d02feeec894d77f66cc82484fe77ae369396a85f6d05594dc156c21e685942dd 3 # podman container restore --import=/tmp/chkpt.tar.gz -n hello2 4 735efb4fee6961d3eee069beb28dde5cbc6fc46c1a32a43ecc993d04c02015b2 5 # podman inspect --format "{{.NetworkSettings.IPAddress}}" hello1 6 10.88.0.248 7 # podman inspect --format "{{.NetworkSettings.IPAddress}}" hello2 8 10.88.0.249 9 # curl 10.88.0.248:8080/helloworld/ 10 2 11 # curl 10.88.0.249:8080/helloworld/ 12 2 76
77. Future: kubectl migrate 77
78. Future: Non-root checkpoint/restore 78
79. Summary • CRIU can checkpoint and restore containers • Integrated in different containers engines • Used in production • Reboot into new kernel without losing container state • Start multiple copies • Migrate running containers 79
80. https://lisas.de/~adrian/container-live-migration-article.pdf https://asciinema.org/a/249922 https://asciinema.org/a/249918 https://lisas.de/~adrian/posts/2019-Apr-10-criu-and-selinux.html https://criu.org/Podman https://twitter.com/adrian__reber https://www.redhat.com/en/blog/container-migration-podman-rhel https://cfp.all-systems-go.io/ASG2019/talk/E88Z7V/ https://sched.co/Tymj https://linuxplumbersconf.org/event/4/contributions/472/ 80
81. Thank you