Upgrade to Pro — share decks privately, control downloads, hide ads and more …

RHEL好きの集い vol2.1 kpatchの深堀り

OMO
December 20, 2019

RHEL好きの集い vol2.1 kpatchの深堀り

RHEL好きの集い vol2.1でのkpatchの動作を解説した資料です。

OMO

December 20, 2019
Tweet

More Decks by OMO

Other Decks in Technology

Transcript

  1. RHEL好きの集い v2.1 kpatchの深掘り OSSセキュリティ技術の会 面 和毅

  2. RHEL 8.1からkpatchが標準サポート

  3. そもそもkpatchってどうやって動くの? 『kernelに動的にパッチを当てる?』 セールストークはわかった。 じゃあ、実際には何をやってるの?

  4. None
  5. 取り敢えず読んでみるべ https://github.com/dynup/kpatch とそのリンクに全て書いてありそう -> https://github.com/dynup/kpatch/blob/master/doc/patch-author-guide.md が参考になる

  6. そもそも論 登場人物 ・kpatch ・kGraft ・live-patch

  7. 経緯 kpatch kGraft livepatch (Kernel 4.0から) kpatch(User-Land) kGraft(User-Land) + +

    良い所取り
  8. ftrace kpatch kGraft livepatch (Kernel 4.0から) kpatch(User-Land) kGraft(User-Land) + +

    良い所取り
  9. ftraceの動きを見てみる [[email protected] ~]# cat /sys/kernel/debug/tracing/available_tracers hwlat blk function_graph wakeup_dl wakeup_rt

    wakeup function nop /sys/kernel/debug/tracing/current_tracer [[email protected] ~]# echo 1 > /sys/kernel/debug/tracing/tracing_on [[email protected] ~]# ls anaconda-ks.cfg initial-setup-ks.cfg kpatch.log [[email protected] ~]# pwd /root [[email protected] ~]# echo 0 > /sys/kernel/debug/tracing/tracing_on
  10. ftraceの動きを見てみる [[email protected] ~]# cat /sys/kernel/debug/tracing/trace_pipe > trace.log [[email protected] ~]# cat

    trace.log ------------------------------ CPU:0 [LOST 2183845 EVENTS] pool-2670 [000] .... 348.077183: selinux_file_permission <-security_file_permission pool-2670 [000] .... 348.077184: __inode_security_revalidate <-selinux_file_permission pool-2670 [000] .... 348.077184: _cond_resched <-__inode_security_revalidate pool-2670 [000] .... 348.077184: rcu_all_qs <-_cond_resched pool-2670 [000] .... 348.077184: file_has_perm <-security_file_permission pool-2670 [000] .... 348.077184: bpf_fd_pass <-file_has_perm pool-2670 [000] .... 348.077184: inode_has_perm.isra.43 <-file_has_perm 成る程。 関数のTraceとか出来るのね。
  11. livepatchを見てみる linux-5.3.1/samples/livepatch 以下のサンプルを見ながら linux-5.3.1/include/linux/livepatch.h を参考に見てみましょう

  12. /samples/livepatch/livepatch-sample.c static struct klp_func funcs[] = { { .old_name =

    "cmdline_proc_show", .new_func = livepatch_cmdline_proc_show, }, { } }; static struct klp_object objs[] = { { /* name being NULL means vmlinux */ .funcs = funcs, }, { } }; static struct klp_patch patch = { .mod = THIS_MODULE, .objs = objs, };
  13. include/linux/livepatch.h : klp_func構造体) include/linux/livepatch.h : klp_func構造体) /** * struct klp_func

    - function structure for live patching * @old_name: name of the function to be patched * @new_func: pointer to the patched function code * @old_sympos: a hint indicating which symbol position the old function * can be found (optional) * @old_func: pointer to the function being patched * @kobj: kobject for sysfs resources * @node: list node for klp_object func_list * @stack_node: list node for klp_ops func_stack list * @old_size: size of the old function * @new_size: size of the new function ...
  14. include/linux/livepatch.h : klp_object構造体 /** * struct klp_object - kernel object

    structure for live patching * @name: module name (or NULL for vmlinux) * @funcs: function entries for functions to be patched in the object * @callbacks: functions to be executed pre/post (un)patching * @kobj: kobject for sysfs resources * @func_list: dynamic list of the function entries * @node: list node for klp_patch obj_list * @mod: kernel module associated with the patched object * (NULL for vmlinux) * @dynamic: temporary object for nop functions; dynamically allocated * @patched: the object's funcs have been added to the klp_ops list */ struct klp_object { /* external */ const char *name;
  15. include/linux/livepatch.h : klp_patch構造体 /** * struct klp_patch - patch structure

    for live patching * @mod: reference to the live patch module * @objs: object entries for kernel objects to be patched * @replace: replace all actively used patches * @list: list node for global list of actively used patches * @kobj: kobject for sysfs resources * @obj_list: dynamic list of the object entries * @enabled: the patch is enabled (but operation may be incomplete) * @forced: was involved in a forced transition * @free_work: patch cleanup from workqueue-context * @finish: for waiting till it is safe to remove the patch module */ struct klp_patch {
  16. じゃあ、moduleのinitは?

  17. kernel/livepatch/core.c static int __init klp_init(void) { klp_root_kobj = kobject_create_and_add("livepatch", kernel_kobj);

    if (!klp_root_kobj) return -ENOMEM; return 0; } module_init(klp_init); モジュールの初期化
  18. klp_enable_patch static int __klp_enable_patch(struct klp_patch *patch) { ---snip--- /* *

    Enforce the order of the func->transition writes in * klp_init_transition() and the ops->func_stack writes in * klp_patch_object(), so that klp_ftrace_handler() will see the * func->transition updates before the handler is registered and the * new funcs become visible to the handler. */ ---snip--- ret = klp_patch_object(obj); if (ret) {
  19. klp_patch_func static int klp_patch_func(struct klp_func *func) { ---snip--- ops =

    klp_find_ops(func->old_func); if (!ops) { unsigned long ftrace_loc; ftrace_loc = klp_get_ftrace_location((unsigned long)func->old_func); #ifndef klp_get_ftrace_location static unsigned long klp_get_ftrace_location(unsigned long faddr) { return faddr; } #endif
  20. klp_ftrace_handler static void notrace klp_ftrace_handler(unsigned long ip, (略) { --snip--

    /* * NOPs are used to replace existing patches with original code. * Do nothing! Setting pc would cause an infinite loop. */ if (func->nop) goto unlock; klp_arch_set_pc(regs, (unsigned long)func->new_func); --------------------- static inline void klp_arch_set_pc(struct pt_regs *regs, unsigned long ip) { regs->ip = ip; }
  21. 平松さんの資料(YLUG)

  22. kpatchとlivepatch [email protected]:~/SIOS/RHEL/kpatch/tmp/patch$ ls Makefile kpatch-patch-hook.c kpatch.lds.S livepatch-test.mod.o patch-hook.c Module.symvers kpatch-patch.h

    livepatch-patch-hook.c livepatch-test.o patch-hook.o built-in.o kpatch.h livepatch-test.ko modules.order tmp.ko kpatch-macros.h kpatch.lds livepatch-test.mod.c output.o tmp_output.o
  23. 結局こういう事 livepatch (Kernel 4.0から) kpatch(User-Land) ftrace

  24. kpatchの制限

  25. kpatchでpatchが作れないケース 1. データ構造の変更とか (kpatchはあくまでも”関数”を対象) https://github.com/dynup/kpatch/blob/master/doc/patch-author-guide.md “kpatch patches functions, not data.

    If the original patch involves a change to a data structure, the patch will require some rework, as changes to data structures are not allowed by default.”
  26. kpatchでpatchが作れないケース 2. データセマンティックの変更 (shadow variable(kpatchの変数)の追加とか) https://github.com/dynup/kpatch/blob/master/doc/patch-author-guide.md diff --git a/fs/aio.c b/fs/aio.c

    index ebd06fd0de89..6a33b73c9107 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -280,6 +280,8 @@ static void free_ioctx_rcu(struct rcu_head *head) * and ctx->users has dropped to 0, so we know no more kiocbs can be submitted - * now it's safe to cancel any that need to be. */ +#include <linux/livepatch.h> +#define KPATCH_SHADOW_REQS_ACTIVE_V2 1
  27. kpatchでpatchが作れないケース 3. __init()関数の変更 4. ヘッダーファイルの変更 5. 関数の削除(これは直感的に無理さがわかりやすい ) その他諸々・・・・ でもkpatch-buildで大抵見つけてくれるのでまあ(完全ではないと思う)。

  28. kpatchでちょっと作ってみよう

  29. kpatchでpatchを作ってみよう https://github.com/dynup/kpatch に方法が載っています。 Fedora/CentOS/RHEL/Debianで出来るらしい。 Fedora31 -> ダメでした(2019/12月現在) なので CentOS7で試す ->

    うまく行った
  30. patchの作成 1. /usr/src/debug/kernel-XX内で linux-XX.new(変更後のソース) linux-XX (オリジナル) でpatchを作成。 diff -Nru linux-3.10.0-1062.4.3.el7.x86_64

    linux-3.10.0-1062.4.3.el7.x86_64.new > /root/test.patch
  31. kpatchモジュールの作成 [[email protected] work]# kpatch-build -t vmlinux /root/test.patch Using cache at

    /root/.kpatch/src Testing patch file(s) Reading special section data Building original source Building patched source Extracting new and modified ELF sections read_write.o: changed function: vfs_write read_write.o: changed function: vfs_read namespace.o: changed function: do_mount read_write.o: changed function: vfs_write read_write.o: changed function: vfs_read namespace.o: changed function: do_mount Patched objects: vmlinux Building patch module: livepatch-test.ko SUCCESS
  32. kpatchモジュールの作成とロード [[email protected] work]# ls livepatch-test.ko [[email protected] work]# kpatch load livepatch-test.ko

    loading patch module: livepatch-test.ko waiting (up to 15 seconds) for patch transition to complete... patch transition has stalled! signaling stalled process(es): waiting (up to 60 seconds) for patch transition to complete... transition complete (1 seconds) kpatchモジュール
  33. kpatchモジュールのロード dmesg ) [ 1176.794796] livepatch: enabling patch 'livepatch_test' [

    1176.881381] livepatch: 'livepatch_test': starting patching transition [ 1192.031557] livepatch: signaling remaining tasks [ 1192.381182] livepatch: 'livepatch_test': patching complete コマンドだと(kpatch list) [[email protected] work]# kpatch list Loaded patch modules: livepatch_test [enabled] Installed patch modules:
  34. kpatchモジュールのアンロード [[email protected] work]# kpatch unload livepatch_test disabling patch module: livepatch_test

    waiting (up to 15 seconds) for patch transition to complete... patch transition has stalled! signaling stalled process(es): waiting (up to 60 seconds) for patch transition to complete... transition complete (1 seconds) unloading patch module: livepatch_test [[email protected] work]# kpatch list Loaded patch modules: Installed patch modules: [[email protected] work]#
  35. RHELだと

  36. RHNからRPMがダウンロードできる

  37. RHのkpatchモジュールの適用 普通にrpmコマンドでOK [[email protected] ~]# rpm -ivh /home/sios/rhel77/kpatch-patch-3_10_0-1062-1-1.el7.x86_64.rpm 警告: /home/sios/rhel77/kpatch-patch-3_10_0-1062-1-1.el7.x86_64.rpm: ヘッダー

    V3 RSA/SHA256 Signature、鍵 ID fd431d51: NOKEY 準備しています... ################################# [100%] 更新中 / インストール中... 1:kpatch-patch-3_10_0-1062-1-1.el7 ################################# [100%] loading patch module: /usr/lib/kpatch/3.10.0-1062.el7.x86_64/kpatch-3_10_0-1062-1-1.ko installing /usr/lib/kpatch/3.10.0-1062.el7.x86_64/kpatch-3_10_0-1062-1-1.ko (3.10.0-1062.el7.x86_64) [[email protected] ~]# ls /usr/lib/kpatch/3.10.0-1062.el7.x86_64/ kpatch-3_10_0-1062-1-1.ko
  38. RHのkpatchモジュールの適用 kpatchがロードされています。 [[email protected] ~]# kpatch list Loaded patch modules: kpatch_3_10_0_1062_1_1

    [enabled] Installed patch modules: kpatch_3_10_0_1062_1_1 (3.10.0-1062.el7.x86_64) (ちなみに再起動してもロードされます。 「再起動は許すがkernelのバージョンはミドルウェアのサポートの関係で変更できない」 という環境には有用。)
  39. その他試してみましょう

  40. 懸念点1. こんなケースだと大丈夫? No Problem! 論理的に問題なし 更にテスト済み(Sophos-AV で10回以上組み合わせを変 えてテスト。問題なし )

  41. 懸念点2. パフォーマンスの影響は? 当然、kpatchばっかりで根本的にKernelを入れ替えないのな ら、パフォーマンスに影響が出るんじゃない?

  42. vmstatで見てみた(RHEL7.7) パッチによっては メモリが減る

  43. Unixbench(RHEL7.7) こっちはほぼ変化なし

  44. 結果 メモリの方は少しずつ食っていく感じ(パッチによる) -> メモリに”厳しい”環境だと何か有るかも(10Mbytes単位だけど) 結局「恒久的に当てない」ではなく「緊急時の対応」と考えるのが妥当

  45. まとめ

  46. kpatchの中身を知ると意外に面白い ・kpatchの仕組みは以外にベタだけれど深い ・kpatchを作ることも出来るけど、対象が限られる ・kpatchはRHELで提供されているものを使うのが正義  (検証をきっちりとやっているからね) ・kpatchは、あくまでも一時しのぎ

  47. Q&A