Slide 1

Slide 1 text

RHEL好きの集い v2.1 kpatchの深掘り OSSセキュリティ技術の会 面 和毅

Slide 2

Slide 2 text

RHEL 8.1からkpatchが標準サポート

Slide 3

Slide 3 text

そもそもkpatchってどうやって動くの? 『kernelに動的にパッチを当てる?』 セールストークはわかった。 じゃあ、実際には何をやってるの?

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

取り敢えず読んでみるべ https://github.com/dynup/kpatch とそのリンクに全て書いてありそう -> https://github.com/dynup/kpatch/blob/master/doc/patch-author-guide.md が参考になる

Slide 6

Slide 6 text

そもそも論 登場人物 ・kpatch ・kGraft ・live-patch

Slide 7

Slide 7 text

経緯 kpatch kGraft livepatch (Kernel 4.0から) kpatch(User-Land) kGraft(User-Land) + + 良い所取り

Slide 8

Slide 8 text

ftrace kpatch kGraft livepatch (Kernel 4.0から) kpatch(User-Land) kGraft(User-Land) + + 良い所取り

Slide 9

Slide 9 text

ftraceの動きを見てみる [root@rhel81 ~]# cat /sys/kernel/debug/tracing/available_tracers hwlat blk function_graph wakeup_dl wakeup_rt wakeup function nop /sys/kernel/debug/tracing/current_tracer [root@rhel81 ~]# echo 1 > /sys/kernel/debug/tracing/tracing_on [root@rhel81 ~]# ls anaconda-ks.cfg initial-setup-ks.cfg kpatch.log [root@rhel81 ~]# pwd /root [root@rhel81 ~]# echo 0 > /sys/kernel/debug/tracing/tracing_on

Slide 10

Slide 10 text

ftraceの動きを見てみる [root@rhel81 ~]# cat /sys/kernel/debug/tracing/trace_pipe > trace.log [root@rhel81 ~]# cat trace.log ------------------------------ CPU:0 [LOST 2183845 EVENTS] pool-2670 [000] .... 348.077183: selinux_file_permission <-security_file_permission pool-2670 [000] .... 348.077184: __inode_security_revalidate <-selinux_file_permission pool-2670 [000] .... 348.077184: _cond_resched <-__inode_security_revalidate pool-2670 [000] .... 348.077184: rcu_all_qs <-_cond_resched pool-2670 [000] .... 348.077184: file_has_perm <-security_file_permission pool-2670 [000] .... 348.077184: bpf_fd_pass <-file_has_perm pool-2670 [000] .... 348.077184: inode_has_perm.isra.43 <-file_has_perm 成る程。 関数のTraceとか出来るのね。

Slide 11

Slide 11 text

livepatchを見てみる linux-5.3.1/samples/livepatch 以下のサンプルを見ながら linux-5.3.1/include/linux/livepatch.h を参考に見てみましょう

Slide 12

Slide 12 text

/samples/livepatch/livepatch-sample.c static struct klp_func funcs[] = { { .old_name = "cmdline_proc_show", .new_func = livepatch_cmdline_proc_show, }, { } }; static struct klp_object objs[] = { { /* name being NULL means vmlinux */ .funcs = funcs, }, { } }; static struct klp_patch patch = { .mod = THIS_MODULE, .objs = objs, };

Slide 13

Slide 13 text

include/linux/livepatch.h : klp_func構造体) include/linux/livepatch.h : klp_func構造体) /** * struct klp_func - function structure for live patching * @old_name: name of the function to be patched * @new_func: pointer to the patched function code * @old_sympos: a hint indicating which symbol position the old function * can be found (optional) * @old_func: pointer to the function being patched * @kobj: kobject for sysfs resources * @node: list node for klp_object func_list * @stack_node: list node for klp_ops func_stack list * @old_size: size of the old function * @new_size: size of the new function ...

Slide 14

Slide 14 text

include/linux/livepatch.h : klp_object構造体 /** * struct klp_object - kernel object structure for live patching * @name: module name (or NULL for vmlinux) * @funcs: function entries for functions to be patched in the object * @callbacks: functions to be executed pre/post (un)patching * @kobj: kobject for sysfs resources * @func_list: dynamic list of the function entries * @node: list node for klp_patch obj_list * @mod: kernel module associated with the patched object * (NULL for vmlinux) * @dynamic: temporary object for nop functions; dynamically allocated * @patched: the object's funcs have been added to the klp_ops list */ struct klp_object { /* external */ const char *name;

Slide 15

Slide 15 text

include/linux/livepatch.h : klp_patch構造体 /** * struct klp_patch - patch structure for live patching * @mod: reference to the live patch module * @objs: object entries for kernel objects to be patched * @replace: replace all actively used patches * @list: list node for global list of actively used patches * @kobj: kobject for sysfs resources * @obj_list: dynamic list of the object entries * @enabled: the patch is enabled (but operation may be incomplete) * @forced: was involved in a forced transition * @free_work: patch cleanup from workqueue-context * @finish: for waiting till it is safe to remove the patch module */ struct klp_patch {

Slide 16

Slide 16 text

じゃあ、moduleのinitは?

Slide 17

Slide 17 text

kernel/livepatch/core.c static int __init klp_init(void) { klp_root_kobj = kobject_create_and_add("livepatch", kernel_kobj); if (!klp_root_kobj) return -ENOMEM; return 0; } module_init(klp_init); モジュールの初期化

Slide 18

Slide 18 text

klp_enable_patch static int __klp_enable_patch(struct klp_patch *patch) { ---snip--- /* * Enforce the order of the func->transition writes in * klp_init_transition() and the ops->func_stack writes in * klp_patch_object(), so that klp_ftrace_handler() will see the * func->transition updates before the handler is registered and the * new funcs become visible to the handler. */ ---snip--- ret = klp_patch_object(obj); if (ret) {

Slide 19

Slide 19 text

klp_patch_func static int klp_patch_func(struct klp_func *func) { ---snip--- ops = klp_find_ops(func->old_func); if (!ops) { unsigned long ftrace_loc; ftrace_loc = klp_get_ftrace_location((unsigned long)func->old_func); #ifndef klp_get_ftrace_location static unsigned long klp_get_ftrace_location(unsigned long faddr) { return faddr; } #endif

Slide 20

Slide 20 text

klp_ftrace_handler static void notrace klp_ftrace_handler(unsigned long ip, (略) { --snip-- /* * NOPs are used to replace existing patches with original code. * Do nothing! Setting pc would cause an infinite loop. */ if (func->nop) goto unlock; klp_arch_set_pc(regs, (unsigned long)func->new_func); --------------------- static inline void klp_arch_set_pc(struct pt_regs *regs, unsigned long ip) { regs->ip = ip; }

Slide 21

Slide 21 text

平松さんの資料(YLUG)

Slide 22

Slide 22 text

kpatchとlivepatch sios@localhost:~/SIOS/RHEL/kpatch/tmp/patch$ ls Makefile kpatch-patch-hook.c kpatch.lds.S livepatch-test.mod.o patch-hook.c Module.symvers kpatch-patch.h livepatch-patch-hook.c livepatch-test.o patch-hook.o built-in.o kpatch.h livepatch-test.ko modules.order tmp.ko kpatch-macros.h kpatch.lds livepatch-test.mod.c output.o tmp_output.o

Slide 23

Slide 23 text

結局こういう事 livepatch (Kernel 4.0から) kpatch(User-Land) ftrace

Slide 24

Slide 24 text

kpatchの制限

Slide 25

Slide 25 text

kpatchでpatchが作れないケース 1. データ構造の変更とか (kpatchはあくまでも”関数”を対象) https://github.com/dynup/kpatch/blob/master/doc/patch-author-guide.md “kpatch patches functions, not data. If the original patch involves a change to a data structure, the patch will require some rework, as changes to data structures are not allowed by default.”

Slide 26

Slide 26 text

kpatchでpatchが作れないケース 2. データセマンティックの変更 (shadow variable(kpatchの変数)の追加とか) https://github.com/dynup/kpatch/blob/master/doc/patch-author-guide.md diff --git a/fs/aio.c b/fs/aio.c index ebd06fd0de89..6a33b73c9107 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -280,6 +280,8 @@ static void free_ioctx_rcu(struct rcu_head *head) * and ctx->users has dropped to 0, so we know no more kiocbs can be submitted - * now it's safe to cancel any that need to be. */ +#include +#define KPATCH_SHADOW_REQS_ACTIVE_V2 1

Slide 27

Slide 27 text

kpatchでpatchが作れないケース 3. __init()関数の変更 4. ヘッダーファイルの変更 5. 関数の削除(これは直感的に無理さがわかりやすい ) その他諸々・・・・ でもkpatch-buildで大抵見つけてくれるのでまあ(完全ではないと思う)。

Slide 28

Slide 28 text

kpatchでちょっと作ってみよう

Slide 29

Slide 29 text

kpatchでpatchを作ってみよう https://github.com/dynup/kpatch に方法が載っています。 Fedora/CentOS/RHEL/Debianで出来るらしい。 Fedora31 -> ダメでした(2019/12月現在) なので CentOS7で試す -> うまく行った

Slide 30

Slide 30 text

patchの作成 1. /usr/src/debug/kernel-XX内で linux-XX.new(変更後のソース) linux-XX (オリジナル) でpatchを作成。 diff -Nru linux-3.10.0-1062.4.3.el7.x86_64 linux-3.10.0-1062.4.3.el7.x86_64.new > /root/test.patch

Slide 31

Slide 31 text

kpatchモジュールの作成 [root@cent7 work]# kpatch-build -t vmlinux /root/test.patch Using cache at /root/.kpatch/src Testing patch file(s) Reading special section data Building original source Building patched source Extracting new and modified ELF sections read_write.o: changed function: vfs_write read_write.o: changed function: vfs_read namespace.o: changed function: do_mount read_write.o: changed function: vfs_write read_write.o: changed function: vfs_read namespace.o: changed function: do_mount Patched objects: vmlinux Building patch module: livepatch-test.ko SUCCESS

Slide 32

Slide 32 text

kpatchモジュールの作成とロード [root@cent7 work]# ls livepatch-test.ko [root@cent7 work]# kpatch load livepatch-test.ko loading patch module: livepatch-test.ko waiting (up to 15 seconds) for patch transition to complete... patch transition has stalled! signaling stalled process(es): waiting (up to 60 seconds) for patch transition to complete... transition complete (1 seconds) kpatchモジュール

Slide 33

Slide 33 text

kpatchモジュールのロード dmesg ) [ 1176.794796] livepatch: enabling patch 'livepatch_test' [ 1176.881381] livepatch: 'livepatch_test': starting patching transition [ 1192.031557] livepatch: signaling remaining tasks [ 1192.381182] livepatch: 'livepatch_test': patching complete コマンドだと(kpatch list) [root@cent7 work]# kpatch list Loaded patch modules: livepatch_test [enabled] Installed patch modules:

Slide 34

Slide 34 text

kpatchモジュールのアンロード [root@cent7 work]# kpatch unload livepatch_test disabling patch module: livepatch_test waiting (up to 15 seconds) for patch transition to complete... patch transition has stalled! signaling stalled process(es): waiting (up to 60 seconds) for patch transition to complete... transition complete (1 seconds) unloading patch module: livepatch_test [root@cent7 work]# kpatch list Loaded patch modules: Installed patch modules: [root@cent7 work]#

Slide 35

Slide 35 text

RHELだと

Slide 36

Slide 36 text

RHNからRPMがダウンロードできる

Slide 37

Slide 37 text

RHのkpatchモジュールの適用 普通にrpmコマンドでOK [root@rhel77 ~]# rpm -ivh /home/sios/rhel77/kpatch-patch-3_10_0-1062-1-1.el7.x86_64.rpm 警告: /home/sios/rhel77/kpatch-patch-3_10_0-1062-1-1.el7.x86_64.rpm: ヘッダー V3 RSA/SHA256 Signature、鍵 ID fd431d51: NOKEY 準備しています... ################################# [100%] 更新中 / インストール中... 1:kpatch-patch-3_10_0-1062-1-1.el7 ################################# [100%] loading patch module: /usr/lib/kpatch/3.10.0-1062.el7.x86_64/kpatch-3_10_0-1062-1-1.ko installing /usr/lib/kpatch/3.10.0-1062.el7.x86_64/kpatch-3_10_0-1062-1-1.ko (3.10.0-1062.el7.x86_64) [root@rhel77 ~]# ls /usr/lib/kpatch/3.10.0-1062.el7.x86_64/ kpatch-3_10_0-1062-1-1.ko

Slide 38

Slide 38 text

RHのkpatchモジュールの適用 kpatchがロードされています。 [root@rhel77 ~]# kpatch list Loaded patch modules: kpatch_3_10_0_1062_1_1 [enabled] Installed patch modules: kpatch_3_10_0_1062_1_1 (3.10.0-1062.el7.x86_64) (ちなみに再起動してもロードされます。 「再起動は許すがkernelのバージョンはミドルウェアのサポートの関係で変更できない」 という環境には有用。)

Slide 39

Slide 39 text

その他試してみましょう

Slide 40

Slide 40 text

懸念点1. こんなケースだと大丈夫? No Problem! 論理的に問題なし 更にテスト済み(Sophos-AV で10回以上組み合わせを変 えてテスト。問題なし )

Slide 41

Slide 41 text

懸念点2. パフォーマンスの影響は? 当然、kpatchばっかりで根本的にKernelを入れ替えないのな ら、パフォーマンスに影響が出るんじゃない?

Slide 42

Slide 42 text

vmstatで見てみた(RHEL7.7) パッチによっては メモリが減る

Slide 43

Slide 43 text

Unixbench(RHEL7.7) こっちはほぼ変化なし

Slide 44

Slide 44 text

結果 メモリの方は少しずつ食っていく感じ(パッチによる) -> メモリに”厳しい”環境だと何か有るかも(10Mbytes単位だけど) 結局「恒久的に当てない」ではなく「緊急時の対応」と考えるのが妥当

Slide 45

Slide 45 text

まとめ

Slide 46

Slide 46 text

kpatchの中身を知ると意外に面白い ・kpatchの仕組みは以外にベタだけれど深い ・kpatchを作ることも出来るけど、対象が限られる ・kpatchはRHELで提供されているものを使うのが正義  (検証をきっちりとやっているからね) ・kpatchは、あくまでも一時しのぎ

Slide 47

Slide 47 text

Q&A