Share
Embed
linux kernel auto test by using ktest
introduce a linux kernel auto test tool named ktest. From testing one patch to test set of patches and bisect a problematic patch.
Satoru Takeuchi PRO
October 12, 2017
Transcript
26 slides
-
2 Introduction • Purpose – Learn how to test linux
kernel automatically by using ktest – It's not a ktest reference, but a quick ktest learning guide • Target readers – Linux kernel developers who have x86_64 linux system and the KVM guest system on it (*1) *1) Although ktest itself can be used for many environments, this slide focuses on this specific combination for simplicity. -
3 Table of contents 1. What is ktest? 2. System
requirement of ktest 3. System environment of this document 4. Initial configuration 5. How to test 6. Conclusion 7. References -
4 1. What is ktest? • A kernel automated test
tool which is distributed as part of linux kernel – Author: Steven Rostedt • It keeps working even if test kernel has panic/hangup problems. • This document covers the following features – Basic tests for a kernel • build test→ install test→ boot test→ user defined test – Test each patch of a patchset – Bisect to find the problematic commit from a series of commits -
5 2. System requirement of ktest • The following two
system(VM is OK) – Host: To build test kernel and to watch the progress of test – Target: Run test with test kernel • Can be poweron/off from Host • Can be connected via console from Host • Can be connected via ssh as root from Host • Kernel source – It should be git repository to use some advanced features -
6 3. System environment of this document • I don't
explain how to setup this environment here. • If you try to do the subsequent steps, please replace each expression with appropriate one depend on your environment. Host Target • System: debian/jessy x86_64 • hostname/user: host/sat • Linux kernel's git repository under /home/sat/src/linux • “make mrproper” clean • kernel config file for test kernel can be boot on Target. • /home/sat/kernel.config • System: debian/jessy or CentOS7 x86_64 guest in Host • Hostname/domain name: target • Bootloader: grub2 • “grub-reboot” works fine • Can be connected via “virsh console” from Host • Can be connected via “ssh root@target” without passphrase -
7 4. Initial configuration (1/2): Make workspace $ mkdir -p
/home/sat/ktest $ cd /home/sat/ktest $ mkdir output $ cp /home/sat/src/linux/tools/testing/ktest/ktest.pl . • Run the following commands on Host. -
8 4. Initial Configuration (2/2): Prepare the grub2 entry •
Add the following entry in /boot/grub/grub.conf, the grub2's configuration file, on Target. menuentry 'ktest' { ... linux /boot/ktest-vmlinuz … console=ttyS0,115200n8 ... initrd /boot/ktest-initramfs.img } -
9 5. How to test • Learn how to run
the following tests one by one. 1. Build test. 2. Install test. 3. Boot test. 4. User defined test. 5. Test patchset. 6. Bisect -
10 5.1. Build test (1/2): Configuration • Create /home/sat/ktest/ktest.conf as
follows. All pathnames should be full path here. MACHINE = target # hostname of Target LOG_FILE = /home/sat/ktest/ktest.log # The logfile of ktest BUILD_DIR = /home/sat/src/linux # your kernel source path OUTPUT_DIR = /home/sat/ktest/output # Output dir of kernel build LOCALVERSION = -ktest # local version of the test kernel BUILD_TARGET = arch/x86/boot/bzImage # The name of bzImage TARGET_IMAGE = /boot/ktest-vmlinuz # Full path of vmlinuz image on Target MIN_CONFIG = /home/sat/kernel.config # The base of .config BUILD_TYPE= olddefconfig # configuration target of kernel build BUILD_OPTIONS = -j8 # (optional) kernel build option TEST_START TEST_TYPE = build -
11 5.1. Build test (2/2): Run&Verify • Run ktest as
follows. $ ./ktest.pl • Then ktest builds your test kernel automatically. • If succeeds, test finished with the following log. ... ****************************************** ******************************************* KTEST RESULT: TEST 1 SUCCESS!!!! ** ******************************************* ******************************************* 1 of 1 tests were successful • If fails, diagnose the problem with the logfile. • Failed during build → Your kernel would have some failed-to-build bug. • Failed in other place → ktest.conf would be wrong. -
12 5.2. Install test • Modify ktest.conf + SSH_USER =
root + POST_INSTALL = ssh ${SSH_USER}@${MACHINE} /usr/sbin/mkinitramfs -o /boot/ktest-initramfs.img $KERNEL_VERSION # (optional) The code run after installation on Host. - TEST_TYPE = build + TEST_TYPE = install • If your system is CentOS7, replace POST_INSTALL line as follows. • Run ktest • Then ktest builds and installs your test kernel automatically. • If fails, ktest.conf would be wrong. $ ./ktest.pl + POST_INSTALL = ssh {SSH_USER}@${MACHINE} /usr/sbin/dracut -f /boot/ktest-initramfs.img $KERNEL_VERSION -
13 5.3. Boot test (1/2): Configuration • Modify ktest +
REBOOT_TYPE = grub2 + GRUB_FILE = /boot/grub/grub.cfg + GRUB_MENU = ktest + GRUB_REBOOT = grub-reboot # The command to boot your test kernel only once on Target + POWER_CYCLE = virsh destroy ${MACHINE}; sleep 5; virsh start ${MACHINE} # The commands from Host to force reboot Target. + REBOOT = ssh ${SSH_USER}@${MACHINE} reboot; : # reboot command which ktest uses + REBOOT_ON_ERROR = 1 # (optional) Set if you want to reboot Target on failed-to-boot. + CONSOLE = virsh console ${MACHINE} # Command to connect from Host to Target + CLOSE_CONSOLE_SIGNAL = KILL # Signal to kill the console process. ... - TEST_TYPE = install + TEST_TYPE = boot • Modify ktest.conf • If your guest is CentOS7, replace GRUB_FILE and GRUB_REBOOT lines as follows + GRUB_FILE = /boot/grub2/grub.cfg + GRUB_REBOOT = grub2-reboot -
14 5.3. Boot test (2/2): Run&Verify • Run ktest •
Then ktest builds, installs, boot your test kernel , and reboot with the original kernel automatically. • If fails, your kernel has failed-to-boot bugs or ktest.conf would be wrong. $ ./ktest.pl -
15 5.4. User defined test 1. Make test. Please note
that the test is run under Host. 2. Modify ktest.conf 3. Run ktest. • Then ktest builds, installs, boot your test kernel, run the user defined test with this kernel, reboot with the original kernel automatically. • If failed… • on user defined test → Your kernel or your test would have some bugs. • on other place → ktest.conf would be wrong. $ ./ktest.pl - TEST_TYPE = boot + TEST_TYPE = test + TEST = /home/sat/ktest/hello.test $ cat hello.test ssh 'root@target echo hello ktest!' -
16 5.5. Test patchset (1/4): Assumption • You made the
patchset, consists of the following ten patches, for 3.14-rc6. – 7 th patch has panic-on-boot bug (of course you don't know it beforehand). • You'd like to run boot test for each patch before submitting it to LKML. • $ git log --oneline -11 • 3d02e41 test commit 10 • f3c04a3 test commit 9 • 3fec210 test commit 8 • 0641787 test commit 7 # It has a critical bug! • a172eb6 test commit 6 • 63d9f91 test commit 5 • d6edbe0 test commit 4 • a350e10 test commit 3 • 52a2a97 test commit 2 • 8644b5e test commit 1 • fa389e2 Linux 3.14-rc6 -
17 5.5. Test patchset (2/4): Configuration&Run 1. Modify ktest.conf 2.
Run ktest • Then ktest runs boot test for each patch with the folloing log. - TEST_TYPE = test - TEST = /home/sat/ktest/hello.test + TEST_TYPE = patchcheck + PATCHCHECK_TYPE = boot # Run boot test for each patch. + PATCHCHECK_START = 8644b5e # The first patch of the patchset + PATCHCHECK_END = 3d02e41 # The last patch of the patchset $ ./ktest.pl Processing commit 8644b5e0896b075c19f9d52aea1e9a22d7fdc89d test commit 1 … Processing commit 52a2a973c29dce4d85e3c0d570f4351b248ab311 test commit 2 ... -
18 5.5. Test patchset (3/4): Verify • Ktest detects panic
during the test of 7 th patch with the following log, and finishes. Processing commit 0641787fc60d4926b6d286a84417e4b6bb5eca3b test commit 7 ... [ 0.024472] Kernel panic - not syncing: Attempted to kill the idle task! bug timed out after 1 seconds Test forced to stop after 60 seconds after failure CRITICAL FAILURE... failed - got a bug report REBOOTING ssh root@debian sync ... FAILED! ssh root@debian reboot ... FAILED! virsh destroy debian; sleep 5; virsh start debian ... SUCCESS See /home/sat/ktest/ktest.log for more info. failed - got a bug report ... -
19 5.5. Test patchset (4/4): Further work • Modify PATCHCHECK_TYPE
option if you want to change the test for each patch. • The same value as TEST_TYPE option can be set. • build, install, boot, test (You should also set TEST option here) • ex) Run your own test, hello.test, for each patches. - PATCHCHECK_TYPE = boot + PATCHCHECK_TYPE = test + TEST = /home/sat/ktest/hello.test -
20 5.6 Bisect (1/5): Assumption • 3.14-rc6 boots without any
problem. • 3.14-rc6 with the following ten commits get panic on boot. • 7 th commit has panic-on-boot bug (of course you don't know it beforehand). • You want to find the problematic commit and report it to LKML. • $ git log --oneline -11 • 3d02e41 test commit 10 • f3c04a3 test commit 9 • 3fec210 test commit 8 • 0641787 test commit 7 # It has a critical bug! • a172eb6 test commit 6 • 63d9f91 test commit 5 • d6edbe0 test commit 4 • a350e10 test commit 3 • 52a2a97 test commit 2 • 8644b5e test commit 1 • fa389e2 Linux 3.14-rc6 -
21 5.6. bisect (2/5): Configuration • Modify ktest.conf - TEST_TYPE
= patchcheck - PATCHCHECK_TYPE = boot - PATCHCHECK_START = 8644b5e - PATCHCHECK_END = 3d02e41 + TEST_TYPE = bisect + BISECT_TYPE = boot + BISECT_GOOD = b28a960 # One before the first of suspect commits + BISECT_BAD = 3d02e41 # The last of suspect commits -
22 5.6. bisect (3/5): Run • Run ktest • Then
ktest bisects the suspect commits automatically. git bisect start ... SUCCESS git bisect good fa389e220254c69ffae0d403eac4146171062d08 ... SUCCESS git bisect bad 3d02e41fd18ddda74352beb35aa952266b34b92f ... Bisecting: 4 revisions left to test after this (roughly 2 steps) … git bisect good ... Bisecting: 2 revisions left to test after this (roughly 1 step) … git bisect bad ... Bisecting: 0 revisions left to test after this (roughly 0 steps) ... • $ ./ktest.pl -
23 5.6. Bisect (4/5): Verify • Finally ktest detects 7th
commit from 3.14-rc6 is the bad one git bisect good ... 0641787fc60d4926b6d286a84417e4b6bb5eca3b is the first bad commit commit 0641787fc60d4926b6d286a84417e4b6bb5eca3b Author: Satoru Takeuchi <[email protected]> Date: Sun Feb 16 22:48:02 2014 +0900 test commit 7 ... ******************************************* ******************************************* KTEST RESULT: TEST 1 SUCCESS!!!! ** ******************************************* ******************************************* ... -
24 5.6. Bisect (5/5): Further work • Modify BISECT_TYPE option
if you want to change the test for each bisect. • The same value as TEST_TYPE option can be set. • build, install, boot, test (you should also set TEST option here) • ex) Run your own test, hello.test, for each bisect. - PATCHCHECK_TYPE = boot + PATCHCHECK_TYPE = test + TEST = /home/sat/ktest/hello.test -
25 6. Conclusion • Now you can do the automated
kernel test. Congratulations! • It can reduce plenty of your precious time. • For more information, please read the references attached at the next page. • Feel free to ask me if you have any questions/comments. -
26 7. References • The files under <kernel source>/testing/tools/ktest: •
ktest.pl: The ktest program • sample.conf: The document for all options. • examples/*: Sample configuration files for some Targets • examples/kvm.conf is for the KVM Target • ktest tutorial session by Steven Rostedt: – http://video.linux.com/videos/automated-testing-with-ktestpl • Elkdat: Prepare ktest environment by one command • https://github.com/satoru-takeuchi/elkdat