1 Linux kernel auto test by using ktest Version1.1.6 Satoru Takeuchi [email protected] 

2 Introduction ● Purpose – Learn how to test linux kernel automatically by using ktest – It's not a ktest reference, but a quick ktest learning guide ● Target readers – Linux kernel developers who have x86_64 linux system and the KVM guest system on it (*1) *1) Although ktest itself can be used for many environments, this slide focuses on this specific combination for simplicity. 

3 Table of contents 1. What is ktest? 2. System requirement of ktest 3. System environment of this document 4. Initial configuration 5. How to test 6. Conclusion 7. References 

4 1. What is ktest? ● A kernel automated test tool which is distributed as part of linux kernel – Author: Steven Rostedt ● It keeps working even if test kernel has panic/hangup problems. ● This document covers the following features – Basic tests for a kernel ● build test→ install test→ boot test→ user defined test – Test each patch of a patchset – Bisect to find the problematic commit from a series of commits 

5 2. System requirement of ktest ● The following two system(VM is OK) – Host: To build test kernel and to watch the progress of test – Target: Run test with test kernel ● Can be poweron/off from Host ● Can be connected via console from Host ● Can be connected via ssh as root from Host ● Kernel source – It should be git repository to use some advanced features 

6 3. System environment of this document ● I don't explain how to setup this environment here. ● If you try to do the subsequent steps, please replace each expression with appropriate one depend on your environment. Host Target ● System: debian/jessy x86_64 ● hostname/user: host/sat ● Linux kernel's git repository under /home/sat/src/linux ● “make mrproper” clean ● kernel config file for test kernel can be boot on Target. ● /home/sat/kernel.config ● System: debian/jessy or CentOS7 x86_64 guest in Host ● Hostname/domain name: target ● Bootloader: grub2 ● “grub-reboot” works fine ● Can be connected via “virsh console” from Host ● Can be connected via “ssh root@target” without passphrase 

7 4. Initial configuration (1/2): Make workspace $ mkdir -p /home/sat/ktest $ cd /home/sat/ktest $ mkdir output $ cp /home/sat/src/linux/tools/testing/ktest/ktest.pl . ● Run the following commands on Host. 

8 4. Initial Configuration (2/2): Prepare the grub2 entry ● Add the following entry in /boot/grub/grub.conf, the grub2's configuration file, on Target. menuentry 'ktest' { ... linux /boot/ktest-vmlinuz … console=ttyS0,115200n8 ... initrd /boot/ktest-initramfs.img } 

9 5. How to test ● Learn how to run the following tests one by one. 1. Build test. 2. Install test. 3. Boot test. 4. User defined test. 5. Test patchset. 6. Bisect 

10 5.1. Build test (1/2): Configuration ● Create /home/sat/ktest/ktest.conf as follows. All pathnames should be full path here. MACHINE = target # hostname of Target LOG_FILE = /home/sat/ktest/ktest.log # The logfile of ktest BUILD_DIR = /home/sat/src/linux # your kernel source path OUTPUT_DIR = /home/sat/ktest/output # Output dir of kernel build LOCALVERSION = -ktest # local version of the test kernel BUILD_TARGET = arch/x86/boot/bzImage # The name of bzImage TARGET_IMAGE = /boot/ktest-vmlinuz # Full path of vmlinuz image on Target MIN_CONFIG = /home/sat/kernel.config # The base of .config BUILD_TYPE= olddefconfig # configuration target of kernel build BUILD_OPTIONS = -j8 # (optional) kernel build option TEST_START TEST_TYPE = build 

11 5.1. Build test (2/2): Run&Verify ● Run ktest as follows. $ ./ktest.pl ● Then ktest builds your test kernel automatically. ● If succeeds, test finished with the following log. ... ****************************************** ******************************************* KTEST RESULT: TEST 1 SUCCESS!!!! ** ******************************************* ******************************************* 1 of 1 tests were successful ● If fails, diagnose the problem with the logfile. ● Failed during build → Your kernel would have some failed-to-build bug. ● Failed in other place → ktest.conf would be wrong. 

12 5.2. Install test ● Modify ktest.conf + SSH_USER = root + POST_INSTALL = ssh ${SSH_USER}@${MACHINE} /usr/sbin/mkinitramfs -o /boot/ktest-initramfs.img $KERNEL_VERSION # (optional) The code run after installation on Host. - TEST_TYPE = build + TEST_TYPE = install ● If your system is CentOS7, replace POST_INSTALL line as follows. ● Run ktest ● Then ktest builds and installs your test kernel automatically. ● If fails, ktest.conf would be wrong. $ ./ktest.pl + POST_INSTALL = ssh {SSH_USER}@${MACHINE} /usr/sbin/dracut -f /boot/ktest-initramfs.img $KERNEL_VERSION 

13 5.3. Boot test (1/2): Configuration ● Modify ktest + REBOOT_TYPE = grub2 + GRUB_FILE = /boot/grub/grub.cfg + GRUB_MENU = ktest + GRUB_REBOOT = grub-reboot # The command to boot your test kernel only once on Target + POWER_CYCLE = virsh destroy ${MACHINE}; sleep 5; virsh start ${MACHINE} # The commands from Host to force reboot Target. + REBOOT = ssh ${SSH_USER}@${MACHINE} reboot; : # reboot command which ktest uses + REBOOT_ON_ERROR = 1 # (optional) Set if you want to reboot Target on failed-to-boot. + CONSOLE = virsh console ${MACHINE} # Command to connect from Host to Target + CLOSE_CONSOLE_SIGNAL = KILL # Signal to kill the console process. ... - TEST_TYPE = install + TEST_TYPE = boot ● Modify ktest.conf ● If your guest is CentOS7, replace GRUB_FILE and GRUB_REBOOT lines as follows + GRUB_FILE = /boot/grub2/grub.cfg + GRUB_REBOOT = grub2-reboot 

14 5.3. Boot test (2/2): Run&Verify ● Run ktest ● Then ktest builds, installs, boot your test kernel , and reboot with the original kernel automatically. ● If fails, your kernel has failed-to-boot bugs or ktest.conf would be wrong. $ ./ktest.pl 

15 5.4. User defined test 1. Make test. Please note that the test is run under Host. 2. Modify ktest.conf 3. Run ktest. ● Then ktest builds, installs, boot your test kernel, run the user defined test with this kernel, reboot with the original kernel automatically. ● If failed… ● on user defined test → Your kernel or your test would have some bugs. ● on other place → ktest.conf would be wrong. $ ./ktest.pl - TEST_TYPE = boot + TEST_TYPE = test + TEST = /home/sat/ktest/hello.test $ cat hello.test ssh 'root@target echo hello ktest!' 

16 5.5. Test patchset (1/4): Assumption ● You made the patchset, consists of the following ten patches, for 3.14-rc6. – 7 th patch has panic-on-boot bug (of course you don't know it beforehand). ● You'd like to run boot test for each patch before submitting it to LKML. ● $ git log --oneline -11 ● 3d02e41 test commit 10 ● f3c04a3 test commit 9 ● 3fec210 test commit 8 ● 0641787 test commit 7 # It has a critical bug! ● a172eb6 test commit 6 ● 63d9f91 test commit 5 ● d6edbe0 test commit 4 ● a350e10 test commit 3 ● 52a2a97 test commit 2 ● 8644b5e test commit 1 ● fa389e2 Linux 3.14-rc6 

17 5.5. Test patchset (2/4): Configuration&Run 1. Modify ktest.conf 2. Run ktest ● Then ktest runs boot test for each patch with the folloing log. - TEST_TYPE = test - TEST = /home/sat/ktest/hello.test + TEST_TYPE = patchcheck + PATCHCHECK_TYPE = boot # Run boot test for each patch. + PATCHCHECK_START = 8644b5e # The first patch of the patchset + PATCHCHECK_END = 3d02e41 # The last patch of the patchset $ ./ktest.pl Processing commit 8644b5e0896b075c19f9d52aea1e9a22d7fdc89d test commit 1 … Processing commit 52a2a973c29dce4d85e3c0d570f4351b248ab311 test commit 2 ... 

18 5.5. Test patchset (3/4): Verify ● Ktest detects panic during the test of 7 th patch with the following log, and finishes. Processing commit 0641787fc60d4926b6d286a84417e4b6bb5eca3b test commit 7 ... [ 0.024472] Kernel panic - not syncing: Attempted to kill the idle task! bug timed out after 1 seconds Test forced to stop after 60 seconds after failure CRITICAL FAILURE... failed - got a bug report REBOOTING ssh root@debian sync ... FAILED! ssh root@debian reboot ... FAILED! virsh destroy debian; sleep 5; virsh start debian ... SUCCESS See /home/sat/ktest/ktest.log for more info. failed - got a bug report ... 

19 5.5. Test patchset (4/4): Further work ● Modify PATCHCHECK_TYPE option if you want to change the test for each patch. ● The same value as TEST_TYPE option can be set. ● build, install, boot, test (You should also set TEST option here) ● ex) Run your own test, hello.test, for each patches. - PATCHCHECK_TYPE = boot + PATCHCHECK_TYPE = test + TEST = /home/sat/ktest/hello.test 

20 5.6 Bisect (1/5): Assumption ● 3.14-rc6 boots without any problem. ● 3.14-rc6 with the following ten commits get panic on boot. ● 7 th commit has panic-on-boot bug (of course you don't know it beforehand). ● You want to find the problematic commit and report it to LKML. ● $ git log --oneline -11 ● 3d02e41 test commit 10 ● f3c04a3 test commit 9 ● 3fec210 test commit 8 ● 0641787 test commit 7 # It has a critical bug! ● a172eb6 test commit 6 ● 63d9f91 test commit 5 ● d6edbe0 test commit 4 ● a350e10 test commit 3 ● 52a2a97 test commit 2 ● 8644b5e test commit 1 ● fa389e2 Linux 3.14-rc6 

21 5.6. bisect (2/5): Configuration ● Modify ktest.conf - TEST_TYPE = patchcheck - PATCHCHECK_TYPE = boot - PATCHCHECK_START = 8644b5e - PATCHCHECK_END = 3d02e41 + TEST_TYPE = bisect + BISECT_TYPE = boot + BISECT_GOOD = b28a960 # One before the first of suspect commits + BISECT_BAD = 3d02e41 # The last of suspect commits 

22 5.6. bisect (3/5): Run ● Run ktest ● Then ktest bisects the suspect commits automatically. git bisect start ... SUCCESS git bisect good fa389e220254c69ffae0d403eac4146171062d08 ... SUCCESS git bisect bad 3d02e41fd18ddda74352beb35aa952266b34b92f ... Bisecting: 4 revisions left to test after this (roughly 2 steps) … git bisect good ... Bisecting: 2 revisions left to test after this (roughly 1 step) … git bisect bad ... Bisecting: 0 revisions left to test after this (roughly 0 steps) ... ● $ ./ktest.pl 

23 5.6. Bisect (4/5): Verify ● Finally ktest detects 7th commit from 3.14-rc6 is the bad one git bisect good ... 0641787fc60d4926b6d286a84417e4b6bb5eca3b is the first bad commit commit 0641787fc60d4926b6d286a84417e4b6bb5eca3b Author: Satoru Takeuchi Date: Sun Feb 16 22:48:02 2014 +0900 test commit 7 ... ******************************************* ******************************************* KTEST RESULT: TEST 1 SUCCESS!!!! ** ******************************************* ******************************************* ... 

24 5.6. Bisect (5/5): Further work ● Modify BISECT_TYPE option if you want to change the test for each bisect. ● The same value as TEST_TYPE option can be set. ● build, install, boot, test (you should also set TEST option here) ● ex) Run your own test, hello.test, for each bisect. - PATCHCHECK_TYPE = boot + PATCHCHECK_TYPE = test + TEST = /home/sat/ktest/hello.test 

25 6. Conclusion ● Now you can do the automated kernel test. Congratulations! ● It can reduce plenty of your precious time. ● For more information, please read the references attached at the next page. ● Feel free to ask me if you have any questions/comments. 

26 7. References ● The files under /testing/tools/ktest: ● ktest.pl: The ktest program ● sample.conf: The document for all options. ● examples/*: Sample configuration files for some Targets ● examples/kvm.conf is for the KVM Target ● ktest tutorial session by Steven Rostedt: – http://video.linux.com/videos/automated-testing-with-ktestpl ● Elkdat: Prepare ktest environment by one command ● https://github.com/satoru-takeuchi/elkdat