Introduction to CRIU

- Let’s take a glance at the future of containers!

@ JapanContainerDays 2018/12/05

Transcript

1. Let’s take a glance at the future of containers! Uchio Kondo / GMO Pepabo, Inc. 2018.12.05 JapanContainerDays v18.12 Introduction to CRIU
2. Señor-Principal Engineer @ GMO Pepabo, Inc. Uchio Kondo https://blog.udzura.jp/ @udzura Technical department, Dev Productivity/R&D Team RubyKaigi 2019 at Fukuoka Local Organizer Chair on CNDJ at Fukuoka, 2019.04
3. http://www.fuk-ab.co.jp/network_int.html Hi from Fukuoka
4. ˏ
   1FQBCP
   'VLVPLB
   CSBODI
5. Scope of Today’s Talk •What is CRIU? •Dive into the inside of CRIU •How can we use CRIU? • Migration • Reduction of bootstrap cost •How to combine CRIU into a runtime?
6. #containerdaysjp #TerraceRoom
7. Scope of Today’s Talk •What is CRIU? •Dive into the inside of CRIU •How can we use CRIU? • Migration • Reduction of bootstrap time •How to combine CRIU into a runtime? For Developers/Operators Using Containers For RUNTIME Developers
8. What is CRIU?
9. CRIU is: C/R In Userspace • a project to implement checkpoint/restore(C/R) functionality for Linux • Generally, VMs are able to be dumped and restored. • CRIU is this functionality for processes/containers • https://www.criu.org/Main_Page • ex. crtools
10. Whet CRIU is for • CRIU is developed as a project of Virtuozzo • https://www.virtuozzo.com/ • CRIU is currently used by OpenVZ (https://openvz.org/), 
 LXC/LXD and Docker. You can use criu command alone.
11. CRIU can create checkpoint • ... for PROCESSES. • Dumping memories, fds, socket state...
12. Hey, containers are PROCESSES!!
13. Containers are PROCESSES • So CRIU can create checkpoints for containers! • CRIU has many of functionalities to make container’s checkpoint. e.g. Network, Namespace, cgroup...
14. Docker + CRIU demo
15. Note... https://github.com/moby/moby/issues/35691 • Checkpoint won’t work with 18.03~... I used 17.06 for now • moby@master fixed this issue
16. Enable docker checkpoint • Following instruction in https://github.com/docker/cli/blob/ master/experimental/checkpoint-restore.md • Preparation: Install CRIU by yourself (and Docker v17.06 :) • (Ubuntu Bionic has criu package v3.6) • Add --experimental flag to dockerd startup command, then restart
17. Enable docker checkpoint
18. Checkpoint/Restore demo • Run simple container that count number on memory • e.g. • Then create checkpoint: • And restart with --checkpoint option
 • Thus, the count is rollbacked to checkpoint! • (If no --checkpoint, count is restarted by 0)
19. Checkpoint/Restore demo • Using criu command internally to checkpoint/restore
20. Resources about CRIU • Slide from OpenVZ team: • https://www.slideshare.net/openvz/criu-13dusseldorf • One of most reliable articles written in Japanese: • https://gihyo.jp/admin/serial/01/linux_containers/0032
21. Dive into the inside of CRIU
22. How can we invoke CRIU • There are 2 modes: • Via cli: criu command. Normally we use this • Via API: server/client model
23. cli model Shell CRIU command Kernel Target
 process Syscalls, /proc files ...
24. Server/client model • CRIU can be a service: criu service • Client can access this service via socket, using protobuf • CRIU provides some of protobuf wrapper: • C wrapper (called <libcriu>) • Python wrapper • Go wrapper(experimental)
25. Server/client model Program CRIU service UNIX domain socket Kernel libcriu Target
 process Syscalls, /proc files ... protobuf
26. Detail of container C/R process • docker checkpoint/restore uses CRIU • Let’s look into how docker use CRIU!!!
27. Processes that docker hosts
28. Detailed processes overview dockerd docker-containerd containerd-shim Container’s process \_ \_ \_
29. Linux Namespace dockerd docker-containerd containerd-shim Container’s process \_ \_ \_ Host’s Linux Namespace Container’s Linux Namespace
30. Assigned cgroup dockerd docker-containerd containerd-shim Container’s process \_ \_ \_ Systemd-managed cgroup (docker.service) Container’s Each cgroup
31. How CRIU make images CRIU Target
 process Syscalls, /proc files ... Kernel • CRIU gets the information of process via syscall, /proc file, iproute2 utilities...
32. How CRIU make images • Then dump them into images - normally processes will be killed at this time. Memory dump Network conf File descriptors cgroup params Process attrs ...... CRIU Target
 process Syscalls, /proc files ... Kernel
33. How CRIU restore images CRIU Restored
 process • CRIU will use these images on restore Memory dump Network conf File descriptors cgroup params Process attrs ...... Kernel
34. The raw images
35. crit: image utility • CRIU is bundled with crit command, which can decode images in CRIU format.
36. How can we use CRIU?
37. Case 1: Migration
38. P.Haul Project https://criu.org/P.Haul • Extension to make live migration with CRIU possible. • Super experimental • Not so active
39. P.Haul works? • Example of node-to-node migration using sample process • https://github.com/checkpoint-restore/p.haul/blob/master/test/ mtouch/HOWTO • There is also a example for docker 1.9.0... and cannot reproduce now • https://github.com/checkpoint-restore/p.haul/blob/master/test/ docker/HOWTO
40. Migration demo P.Haul looks too inactive! So I implemented it Using my container... I’ll show later!
41. Case 2: Reduction of
 Bootstrap Cost
42. Containers with slow bootstrap • Especially big applications: Legacy Rails, JVM, ... • These applications cannot enjoy enough the merits of lightweight aspect of containers. • e.g. A small Rails project takes 2,500ms~ to become ready. • Jenkins project takes 5,000ms~ to listen 8080...
43. FYI: “FastContainer” • An architecture to handle containers • A container will be bootstrapped on first request, and automatically shut down after some minutes. • This means containers are restarted repeatedly, and this force containers to be refreshed and clean. • cf. “Phoenix Server” in the book “Infrastructure as Code” • Used in our PaaS service: https://mc.lolipop.jp • See @matsumotory’s paper/slide https://speakerdeck.com/matsumoto_r/fastcontainer-shi-xing-huan-jing-falsebian-hua- nisu-zao-kushi-ying-dekiruheng-chang-xing-wochi-tusisutemuakitekutiya
44. FYI: “FastContainer” Web Proxy Web Request Dispatcher FastContainer Runtime CMDB ❌ FastContainer Killed 1. Check 2. Boot 3. Forward 4. Terminate
45. Experiment overview Environment Containers Bench Host https://github.com/FastContainer/nginx-haconiwa 192.168.199.10 192.168.199.20 Service Meshing: Runtime:
46. Experiment codes ab -g bench-rails.tsv \ -s 120 -c 1 -t 90 -n 1000000 -k -l http://192.168.199.10/ import numpy as np import matplotlib.pyplot as plt data = np.loadtxt("/path/to/bench-rails.tsv", delimiter="\t", skiprows=1, usecols=(1,4), dtype=int) data = np.rot90(sorted(data, key=lambda x:x[0]), k=-1) plt.plot(data[0], data[1], linewidth=1, color="orange") plt.ylim(0, 2700) plt.show() Benchmarker Script For Visualize
47. Needs fast boot up • One of bottleneck of this architecture is “slow boot” apps • Comparison of Apache HTTPD vs Rails application: ms/r unixtime Apache(phpinfo) RoR(no bootsnap)
48. Normal FastCon lifecycle ngx_mruby Haconiwa Containers Restart on next request Stop after “Lifetime” Haconiwa
49. Lifecycle with CRIU ngx_mruby Haconiwa Containers ReSTORE on next request Make image just before stop, In async process haconiwa restore Image
50. Using CRIU to make boot fast • Comparison of hot-start Rails application and cold-start (from criu image) Rails: RoR(no bootsnap/From CRIU image) RoR(no bootsnap)
51. Misc.
52. No-downtime kernel upgrade? • Is it possible?: Yes(logically).
53. Kubernetes integration? • There seems to be no plan yet...(I want more info) • A project in UBC class refers this: • https://www.cs.ubc.ca/~bestchai/teaching/cs416_2017w2/ project2/project_m6r8_s8u8_v5v8_y6x8_proposal.pdf
54. Checkpoint
55. Restore :)
56. How to combine CRIU into a runtime?
57. I’ll introduce My container runtime...
58. https://haconiwa.mruby.org/
59. Haconiwa • Highly Configurable container runtime written in mruby • Non OCI-compatible for now (I am planning...) • Implemented basic container features: • Linux namespace, cgroup, chroot/pivot_root, capability/uid/ gid, rlimit, seccomp, apparmor... • Implemented some “hooks”: • Lifetime hooks, async timeout/interval hooks, sighandlers
60. Haconiwa accepts DSLs
61. What I’m working on now • Bundling CRIU features into Haconiwa • haconiwa checkpoint: • To create checkpoint from a running container • haconiwa restore: • To make a restored container, with some spec changes
62. CRIU deep features: • These are what I used in haconiwa development: • Restoration process hooks(action script) • Change cgroup name on restore • Replace supervisor program by --exec-cmd
63. Restoration process hooks • CRIU has a hooks which are invoked as the checkpointing or restoration is processed: Action Script. • e.g. post-dump, post-restore, setup-namespaces... • Haconiwa use this action script to change container’s IP from dumped one as written in a new DSL.
64. Change cgroup name on restore • Haconiwa’s container has name option, which decides its cgroup name. • When you want to change name between dumped and restored containers, you must also change new one’s cgroup name. • Criu’s --cgroup-root option to solve this
65. Replace supervisor program • Haconiwa has its own hooks, and restore process should also restore these hooks by DSL. • This is out of CRIU’s feature • Hooks are implemented in “container supervisor”, rather than container itself • So I implemented to set “supervisor for restored containers” upon a restored container. And hooks are invoked in SV
66. Replacement process Haconiwa sv \- criu restore \- Container Haconiwa sv \- haconiwa _restored \- Container exec() wait() in new program Restore done!
67. See official document • https://criu.org/Tree_after_restore
68. Haconiwa x CRIU Demo
69. Live Rails Migration Using haconiwa C/R
70. Demo Overview Load Balancer Victim container Restored container Image On shared storage Victim Host Dest Host http://Mac:10080 http://Mac:11080 Nonstop! https://github.com/udzura/nginx-haconiwa/tree/haconiwa-migration
71. Conclusion
72. Conclusion • CRIU can create checkpoints for containers, and restore. • I introduced 2 use cases: • Migration • Reduction of bootstrap cost • There is no Kubernetes integration yet, but may be soon? • I have been developing CRIU integration with my container runtime :)
73. Join Us To Be Cloud Native! Follow us: @pb_recruit