Making Linux sucks less

From a performance perspective, there’s a lot in the kernel that focuses on optimizations. But not everything is on by default. There’s options in the kernel config that can both improve performance and also slow it down. Then there’s tweaks you can do on a running system. This talk will talk about ways to make your system work a bit faster. Different workloads call for different approaches as there is no “”one size fits all””. Are you focused on performance, reaction time, or power savings. What you care about is the deciding factor on how you set up your environment.

Steven ROSTEDT

Transcript

1. Making the Linux Kernel
2. Making the Linux Kernel SUCK LESS
3. September 19, 2024
4. Making the Linux Kernel SUCK LESS
5. Making the Linux Kernel SUCK LESS PREEMPT_RT
6. In memory of Daniel Bristot de Oliviera
7. Real-Time Going Upstream 6.12!!! • In honor of RT going upstream, need to list what it did for the kernel ◦ mutex_lock ◦ NO HZ ◦ High Resolution Timers ◦ lockdep ◦ Priority Inheritance (futex) ◦ Threaded interrupts ◦ ftrace ◦ printk
8. mutex_lock/unlock • All sleeping locks use to be just a semaphore
9. mutex_lock/unlock • All sleeping locks use to be just a semaphore ◦ semaphores allow a set of tasks to enter a critical section
10. mutex_lock/unlock • All sleeping locks use to be just a semaphore ◦ semaphores allow a set of tasks to enter a critical section ◦ Most semaphores only allow just one task (mutex!)
11. mutex_lock/unlock • All sleeping locks use to be just a semaphore ◦ semaphores allow a set of tasks to enter a critical section ◦ Most semaphores only allow just one task (mutex!) ◦ Semaphores allowed passing of ownership
12. mutex_lock/unlock • All sleeping locks use to be just a semaphore ◦ semaphores allow a set of tasks to enter a critical section ◦ Most semaphores only allow just one task (mutex!) ◦ Semaphores allowed passing of ownership • RT requires an “owner” for all critical sections ◦ This is a requirement for priority inheritance
13. mutex_lock/unlock • All sleeping locks use to be just a semaphore ◦ semaphores allow a set of tasks to enter a critical section ◦ Most semaphores only allow just one task (mutex!) ◦ Semaphores allowed passing of ownership • RT requires an “owner” for all critical sections ◦ This is a requirement for priority inheritance • The RT folks added the mutex_lock/unlock interface
14. NO_HZ and High Resolution Timers • The tick use to always run ◦ Even when the system was idle
15. NO_HZ and High Resolution Timers • The tick use to always run ◦ Even when the system was idle ◦ Prevents the CPU from entering a deep sleep ▪ Expensive for large data centers
16. NO_HZ and High Resolution Timers • The tick use to always run ◦ Even when the system was idle ◦ Prevents the CPU from entering a deep sleep ▪ Expensive for large data centers • High Resolution Timers was being blocked by the timer maintainer
17. NO_HZ and High Resolution Timers • The tick use to always run ◦ Even when the system was idle ◦ Prevents the CPU from entering a deep sleep ▪ Expensive for large data centers • High Resolution Timers was being blocked by the timer maintainer ◦ NO_HZ was made dependent on HR Timers ◦ Couldn’t have NO_HZ without HR Timers
18. NO_HZ and High Resolution Timers • The tick use to always run ◦ Even when the system was idle ◦ Prevents the CPU from entering a deep sleep ▪ Expensive for large data centers • High Resolution Timers was being blocked by the timer maintainer ◦ NO_HZ was made dependent on HR Timers ◦ Couldn’t have NO_HZ without HR Timers ◦ Linus pulled in both
19. NO_HZ and High Resolution Timers • The tick use to always run ◦ Even when the system was idle ◦ Prevents the CPU from entering a deep sleep ▪ Expensive for large data centers • High Resolution Timers was being blocked by the timer maintainer ◦ NO_HZ was made dependent on HR Timers ◦ Couldn’t have NO_HZ without HR Timers ◦ Linus pulled in both ◦ Thomas Gleixner became the new timer maintainer
20. Lockdep Lock A Lock B TASK 1 Lock B Lock A TASK 2
21. Lockdep Lock A Lock B TASK 1 Lock B Lock C TASK 2 Lock C TASK 3 Lock A
22. Lockdep Lock A
23. Lockdep Lock A Interrupt
24. Lockdep Lock A Interrupt Lock A
25. Lockdep Lock A Interrupt Lock A DEADLOCK
26. Lockdep Lock A
27. Lockdep Lock A Interrupt
28. Lockdep Lock A Lock B TASK 1
29. Lockdep Lock A Lock B TASK 1 Lock B TASK 2
30. Lockdep Lock A Lock B TASK 1 Lock B Lock C TASK 2 Lock C TASK 3
31. Lockdep Lock A Interrupt Lock B TASK 1 Lock B Lock C TASK 2 Lock C TASK 3
32. Lockdep Lock A Interrupt Lock B TASK 1 Lock B Lock C TASK 2 Lock C TASK 3 Lock A
33. Lockdep Lock A Interrupt Lock B TASK 1 Lock B Lock C TASK 2 Lock C TASK 3 Lock A DEADLOCK
34. Lockdep Lock A Interrupt Lock B TASK 1 Lock B Lock C TASK 2 Lock C TASK 3
35. Lockdep Lock A kmalloc(size, GFP_KERNEL)
36. Lockdep Lock A kmalloc(size, GFP_KERNEL) MEMORY RECLAIM!
37. Lockdep Lock A kmalloc(size, GFP_KERNEL) Lock A MEMORY RECLAIM!
38. Lockdep Lock A kmalloc(size, GFP_KERNEL) Lock A MEMORY RECLAIM! DEADLOCK
39. Lockdep Lock A MEMORY RECLAIM Lock B TASK 1 Lock B Lock C TASK 2 Lock C TASK 3 Lock A
40. Lockdep Lock A MEMORY RECLAIM Lock B TASK 1 Lock B Lock C TASK 2 Lock C TASK 3 Lock A DEADLOCK
41. Lockdep down_read A TASK 1 down_read B down_read B TASK 2 down_read A
42. Lockdep TASK 1 TASK 2 DEADLOCK?? down_read A down_read B down_read B down_read A
43. Lockdep TASK 1 TASK 2 DEADLOCK?? YES down_read A down_read B down_read B down_read A
44. Lockdep TASK 1 TASK 2 down_write C TASK 3 down_read A down_read B
45. Lockdep TASK 1 TASK 2 TASK 3 down_read A down_read B down_read B down_read A down_write C
46. Lockdep TASK 1 TASK 2 TASK 3 DEADLOCK down_read A down_read B down_read B down_read A down_write C
47. User Space Priority Inheritance
48. User Space Priority Inheritance pthread_mutexattr_setprotocol(&attr, PTHREAD_PRIO_INHERIT)
49. Interrupts
50. Interrupt threads task: irq-123/harddrive
51. Tracing / ftrace • Three month project ◦ Started in January 2008 ◦ Was to port the tracing infrastructure from PREEMPT_RT
52. Tracing / ftrace • Three month project ◦ Started in January 2008 ◦ Was to port the tracing infrastructure from PREEMPT_RT • Still haven’t finished it
53. printk # cyclictest -q -D 90s -i 500 -d 500 T: 0 ( 3634) P: 0 I:500 C: 180000 Min: 10 Act: 54 Avg: 54 Max: 68
54. printk # cyclictest -q -D 90s -i 500 -d 500 T: 0 ( 3634) P: 0 I:500 C: 180000 Min: 10 Act: 54 Avg: 54 Max: 68 # cyclictest -q -D 90s -i 500 -d 500
55. printk # cyclictest -q -D 90s -i 500 -d 500 T: 0 ( 3634) P: 0 I:500 C: 180000 Min: 10 Act: 54 Avg: 54 Max: 68 # cyclictest -q -D 90s -i 500 -d 500 # echo ? > /proc/sysrq-trigger
56. printk # cyclictest -q -D 90s -i 500 -d 500 T: 0 ( 3634) P: 0 I:500 C: 180000 Min: 10 Act: 54 Avg: 54 Max: 68 # cyclictest -q -D 90s -i 500 -d 500 T: 0 ( 3653) P: 0 I:500 C: 179931 Min: 6 Act: 55 Avg: 55 Max: 34658 # echo ? > /proc/sysrq-trigger !!!!!!!!!!!!!!!!!
57. printk • The last blocker of PREEMPT_RT!
58. printk • The last blocker of PREEMPT_RT! • Old printk serializes the output ◦ Does each console one by one!
59. printk • The last blocker of PREEMPT_RT! • Old printk serializes the output ◦ Does each console one by one! • Threaded printk allows all consoles to be printed at once!
60. printk • The last blocker of PREEMPT_RT! • Old printk serializes the output ◦ Does each console one by one! • Threaded printk allows all consoles to be printed at once! • Can now be called in any context! ◦ NMI ◦ Scheduler
61. Build speed • In 2008, ftrace was added
62. Build speed • In 2008, ftrace was added • It originally required a daemon to find the mcount locations ◦ This is needed to convert them to nops
63. Build speed • In 2008, ftrace was added • It originally required a daemon to find the mcount locations ◦ This is needed to convert them to nops • To get rid of the daemon, Perl scripts were added to the build
64. Build speed • In 2008, ftrace was added • It originally required a daemon to find the mcount locations ◦ This is needed to convert them to nops • To get rid of the daemon, Perl scripts were added to the build ◦ It did objdump to find the locations of mcount calls
65. Build speed • In 2008, ftrace was added • It originally required a daemon to find the mcount locations ◦ This is needed to convert them to nops • To get rid of the daemon, Perl scripts were added to the build ◦ It did objdump to find the locations of mcount calls ◦ It then created an assembly file holding these locations
66. Build speed • In 2008, ftrace was added • It originally required a daemon to find the mcount locations ◦ This is needed to convert them to nops • To get rid of the daemon, Perl scripts were added to the build ◦ It did objdump to find the locations of mcount calls ◦ It then created an assembly file holding these locations ◦ It then compiled and relinked the file into the original object file
67. Build speed • In 2008, ftrace was added • It originally required a daemon to find the mcount locations ◦ This is needed to convert them to nops • To get rid of the daemon, Perl scripts were added to the build ◦ It did objdump to find the locations of mcount calls ◦ It then created an assembly file holding these locations ◦ It then compiled and relinked the file into the original object file • Needless to say, this slowed down the build
68. Build speed • In 2008, ftrace was added • It originally required a daemon to find the mcount locations ◦ This is needed to convert them to nops • To get rid of the daemon, Perl scripts were added to the build ◦ It did objdump to find the locations of mcount calls ◦ It then created an assembly file holding these locations ◦ It then compiled and relinked the file into the original object file • Needless to say, this slowed down the build • In 2010, recordmcount.c was written to do this in C ◦ Written by John Reiser ◦ sped up the build again!
69. Build speed • In 2008, Linus complained about build speed
70. Build speed • In 2008, Linus complained about build speed ◦ We ask users to bisect the bugs they report
71. Build speed • In 2008, Linus complained about build speed ◦ We ask users to bisect the bugs they report ◦ They only have the distro configs
72. Build speed • In 2008, Linus complained about build speed ◦ We ask users to bisect the bugs they report ◦ They only have the distro configs ◦ It can take them 13 hours to build on their machines
73. Build speed • In 2008, Linus complained about build speed ◦ We ask users to bisect the bugs they report ◦ They only have the distro configs ◦ It can take them 13 hours to build on their machines • Linus told Kernel Summit attendees to fix it
74. Build speed • In 2008, Linus complained about build speed ◦ We ask users to bisect the bugs they report ◦ They only have the distro configs ◦ It can take them 13 hours to build on their machines • Linus told Kernel Summit attendees to fix it (I wasn’t invited)
75. Build speed • In 2008, Linus complained about build speed ◦ We ask users to bisect the bugs they report ◦ They only have the distro configs ◦ It can take them 13 hours to build on their machines • Linus told Kernel Summit attendees to fix it ◦ Thomas Gleixner said we already have a script
76. Build speed • In 2008, Linus complained about build speed ◦ We ask users to bisect the bugs they report ◦ They only have the distro configs ◦ It can take them 13 hours to build on their machines • Linus told Kernel Summit attendees to fix it ◦ Thomas Gleixner said we already have a script ◦ “Steven Rostedt has this streamline-config.pl”
77. Build speed • In 2008, Linus complained about build speed ◦ We ask users to bisect the bugs they report ◦ They only have the distro configs ◦ It can take them 13 hours to build on their machines • Linus told Kernel Summit attendees to fix it ◦ Thomas Gleixner said we already have a script ◦ “Steven Rostedt has this streamline-config.pl” ◦ Linus asked him why it’s not already in the kernel?
78. Build speed • In 2008, Linus complained about build speed ◦ We ask users to bisect the bugs they report ◦ They only have the distro configs ◦ It can take them 13 hours to build on their machines • Linus told Kernel Summit attendees to fix it ◦ Thomas Gleixner said we already have a script ◦ “Steven Rostedt has this streamline-config.pl” ◦ Linus asked him why it’s not already in the kernel? • Later Steven was asked to add it
79. Build speed • In 2008, Linus complained about build speed ◦ We ask users to bisect the bugs they report ◦ They only have the distro configs ◦ It can take them 13 hours to build on their machines • Linus told Kernel Summit attendees to fix it ◦ Thomas Gleixner said we already have a script ◦ “Steven Rostedt has this streamline-config.pl” ◦ Linus asked him why it’s not already in the kernel? • Later Steven was asked to add it ◦ This became make localmodconfig ◦ It sped up the build tremendously
80. Build speed • In 2008, Linus complained about build speed ◦ We ask users to bisect the bugs they report ◦ They only have the distro configs ◦ It can take them 13 hours to build on their machines • Linus told Kernel Summit attendees to fix it ◦ Thomas Gleixner said we already have a script ◦ “Steven Rostedt has this streamline-config.pl” ◦ Linus asked him why it’s not already in the kernel? • Later Steven was asked to add it ◦ This became make localmodconfig ◦ It sped up the build tremendously ◦ It also hid the slow down caused by ftrace perl scripts!
81. text_poke() Ftrace was the first to add runtime modification of code <schedule>: 0f 1f 44 00 00 nop 53 push %rbx 65 48 8b 1c 25 00 61 mov %gs:0x16100,%rbx 01 00 ffffffff81a1491b: R_X86_64_32S current_task 48 8b 43 10 mov 0x10(%rbx),%rax 48 85 c0 test %rax,%rax 74 10 je ffffffff81a14938 <schedule+0x28> f6 43 24 20 testb $0x20,0x24(%rbx) 75 49 jne ffffffff81a14977 <schedule+0x67> 48 83 bb 20 0c 00 00 cmpq $0x0,0xc20(%rbx) 00 74 1f je ffffffff81a14957 <schedule+0x47> 31 ff xor %edi,%edi e8 a1 f8 ff ff callq ffffffff81a141e0 <__schedule>
82. text_poke() Ftrace was the first to add runtime modification of code <schedule>: <cc> 1f 44 00 00 <int3>nop 53 push %rbx 65 48 8b 1c 25 00 61 mov %gs:0x16100,%rbx 01 00 ffffffff81a1491b: R_X86_64_32S current_task 48 8b 43 10 mov 0x10(%rbx),%rax 48 85 c0 test %rax,%rax 74 10 je ffffffff81a14938 <schedule+0x28> f6 43 24 20 testb $0x20,0x24(%rbx) 75 49 jne ffffffff81a14977 <schedule+0x67> 48 83 bb 20 0c 00 00 cmpq $0x0,0xc20(%rbx) 00 74 1f je ffffffff81a14957 <schedule+0x47> 31 ff xor %edi,%edi e8 a1 f8 ff ff callq ffffffff81a141e0 <__schedule>
83. text_poke() Ftrace was the first to add runtime modification of code <schedule>: <int3>nop push %rbx mov %gs:0x16100,%rbx mov 0x10(%rbx),%rax test %rax,%rax
84. text_poke() Ftrace was the first to add runtime modification of code <schedule>: <int3>nop push %rbx mov %gs:0x16100,%rbx mov 0x10(%rbx),%rax test %rax,%rax do_int3(struct pt_regs *regs) { regs->ip += 5; return }
85. text_poke() Ftrace was the first to add runtime modification of code <schedule>: <int3>nop push %rbx mov %gs:0x16100,%rbx mov 0x10(%rbx),%rax test %rax,%rax do_int3(struct pt_regs *regs) { regs->ip += 5; return }
86. text_poke() Ftrace was the first to add runtime modification of code <schedule>: <int3>nop push %rbx mov %gs:0x16100,%rbx mov 0x10(%rbx),%rax test %rax,%rax do_int3(struct pt_regs *regs) { regs->ip += 5; return }
87. text_poke() Ftrace was the first to add runtime modification of code <schedule>: <int3>nop push %rbx mov %gs:0x16100,%rbx mov 0x10(%rbx),%rax test %rax,%rax do_int3(struct pt_regs *regs) { regs->ip += 5; return }
88. text_poke() Ftrace was the first to add runtime modification of code <schedule>: <cc> 1f 44 00 00 <int3>nop 53 push %rbx 65 48 8b 1c 25 00 61 mov %gs:0x16100,%rbx 01 00 ffffffff81a1491b: R_X86_64_32S current_task 48 8b 43 10 mov 0x10(%rbx),%rax 48 85 c0 test %rax,%rax 74 10 je ffffffff81a14938 <schedule+0x28> f6 43 24 20 testb $0x20,0x24(%rbx) 75 49 jne ffffffff81a14977 <schedule+0x67> 48 83 bb 20 0c 00 00 cmpq $0x0,0xc20(%rbx) 00 74 1f je ffffffff81a14957 <schedule+0x47> 31 ff xor %edi,%edi e8 a1 f8 ff ff callq ffffffff81a141e0 <__schedule>
89. text_poke() Ftrace was the first to add runtime modification of code <schedule>: <cc> 1b d0 1e 00 <int3>callq ffffffff81c01930 <__fentry__> 53 push %rbx 65 48 8b 1c 25 00 61 mov %gs:0x16100,%rbx 01 00 ffffffff81a1491b: R_X86_64_32S current_task 48 8b 43 10 mov 0x10(%rbx),%rax 48 85 c0 test %rax,%rax 74 10 je ffffffff81a14938 <schedule+0x28> f6 43 24 20 testb $0x20,0x24(%rbx) 75 49 jne ffffffff81a14977 <schedule+0x67> 48 83 bb 20 0c 00 00 cmpq $0x0,0xc20(%rbx) 00 74 1f je ffffffff81a14957 <schedule+0x47> 31 ff xor %edi,%edi e8 a1 f8 ff ff callq ffffffff81a141e0 <__schedule>
90. text_poke() Ftrace was the first to add runtime modification of code <schedule>: e8 1b d0 1e 00 callq ffffffff81c01930 <__fentry__> 53 push %rbx 65 48 8b 1c 25 00 61 mov %gs:0x16100,%rbx 01 00 ffffffff81a1491b: R_X86_64_32S current_task 48 8b 43 10 mov 0x10(%rbx),%rax 48 85 c0 test %rax,%rax 74 10 je ffffffff81a14938 <schedule+0x28> f6 43 24 20 testb $0x20,0x24(%rbx) 75 49 jne ffffffff81a14977 <schedule+0x67> 48 83 bb 20 0c 00 00 cmpq $0x0,0xc20(%rbx) 00 74 1f je ffffffff81a14957 <schedule+0x47> 31 ff xor %edi,%edi e8 a1 f8 ff ff callq ffffffff81a141e0 <__schedule>
91. text_poke() Static branches can use the same method! if (static_key_false(&somekey)) { /* Do something unlikely */ }
92. text_poke() Static branches can use the same method! static int static_key_false(struct static_key *key) { asm goto("1:" ".byte 0xe9 \n\t .long 0\n\t /* nop */" ".pushsection __jump_table, \"a\" \n\t" _ASM_PTR "1b, %l[" #label "], %c0 \n\t" ".popsection \n\t" : : "i" (key) : : l_yes); return false; l_yes: return true; }
93. text_poke() Static branches can use the same method! static int static_key_false(struct static_key *key) { asm goto("1:" ".byte 0xe9 \n\t .long 0\n\t /* nop */" ".pushsection __jump_table, \"a\" \n\t" _ASM_PTR "1b, %l[" #label "], %c0 \n\t" ".popsection \n\t" : : "i" (key) : : l_yes); return false; l_yes: return true; } NOP
94. text_poke() Static branches can use the same method! static int static_key_false(struct static_key *key) { asm goto("1:" "jmp l_yes\n\t" ".pushsection __jump_table, \"a\" \n\t" _ASM_PTR "1b, %l[" #label "], %c0 \n\t" ".popsection \n\t" : : "i" (key) : : l_yes); return false; l_yes: return true; }
95. text_poke() Static branches can use the same method! static int static_key_false(struct static_key *key) { asm goto("1:" "jmp l_yes\n\t" ".pushsection __jump_table, \"a\" \n\t" _ASM_PTR "1b, %l[" #label "], %c0 \n\t" ".popsection \n\t" : : "i" (key) : : l_yes); return false; l_yes: return true; }
96. text_poke() Static branches can use the same method! if (static_key_false(&somekey)) { /* Do something unlikely */ }
97. text_poke() Static branches can use the same method! if (0) { /* Do something unlikely */ }
98. text_poke() Static branches can use the same method! if (1) { /* Do something unlikely */ }
99. text_poke() • But you must enable CONFIG_JUMP_LABEL ◦ General architecture-dependent options → Optimize very unlikely/likely branches
100. text_poke() • But you must enable CONFIG_JUMP_LABEL ◦ General architecture-dependent options → Optimize very unlikely/likely branches • Without this, it’s just a conditional branch if (somekey->enabled) { /* Do something unlikely */ }
101. cond_resched() • In CONFIG_PREEMPT_NONE the kernel does not preempt!
102. cond_resched() • In CONFIG_PREEMPT_NONE the kernel does not preempt! • If there is a long loop, the watchdog timer will trigger ◦ Need to notify areas that can schedule
103. cond_resched() • In CONFIG_PREEMPT_NONE the kernel does not preempt! • If there is a long loop, the watchdog timer will trigger ◦ Need to notify areas that can schedule • Approximately 1455 cond_resch() in v6.11
104. cond_resched() kernel/bpf/verifier.c: for (i = 0; i < env->subprog_cnt; i++) { old_bpf_func = func[i]->bpf_func; tmp = bpf_int_jit_compile(func[i]); if (tmp != func[i] || func[i]->bpf_func != old_bpf_func) { verbose(env, "JIT doesn't support bpf-to-bpf calls\n"); err = -ENOTSUPP; goto out_free; } }
105. cond_resched() kernel/bpf/verifier.c: for (i = 0; i < env->subprog_cnt; i++) { old_bpf_func = func[i]->bpf_func; tmp = bpf_int_jit_compile(func[i]); if (tmp != func[i] || func[i]->bpf_func != old_bpf_func) { verbose(env, "JIT doesn't support bpf-to-bpf calls\n"); err = -ENOTSUPP; goto out_free; } + cond_resched(); }
106. What’s wrong with CONFIG_PREEMPT? • It can preempt when locks are held • Some workloads can take a big hit from it
107. CONFIG_PREEMPT_RT
108. CONFIG_PREEMPT_RT
109. CONFIG_PREEMPT_RT
110. NEED_RESCHED CONFIG_PREEMPT_RT
111. NEED_RESCHED CONFIG_PREEMPT_RT
112. NEED_RESCHED CONFIG_PREEMPT_RT
113. NEED_RESCHED CONFIG_PREEMPT_RT
114. NEED_RESCHED CONFIG_PREEMPT_RT
115. NEED_RESCHED CONFIG_PREEMPT_RT
116. NEED_RESCHED CONFIG_PREEMPT_RT
117. NEED_RESCHED CONFIG_PREEMPT_RT
118. LAZY_NEED_RESCHED
119. LAZY_NEED_RESCHED
120. LAZY_NEED_RESCHED
121. LAZY_NEED_RESCHED
122. LAZY_NEED_RESCHED
123. LAZY_NEED_RESCHED
124. LAZY_NEED_RESCHED
125. LAZY_NEED_RESCHED NEED_RESCHED
126. CONFIG_PREEMPT_AUTO • Takes the lessons learned from LAZY_NEED_RESCHED of RT Preempt Type Schedule Tick might_sleep() PREEMPT_NONE x x PREEMPT_VOLUNTARY x ✅ PREEMPT ✅ ✅
127. CONFIG_PREEMPT_AUTO LAZY_NEED_RESCHED
128. LAZY_NEED_RESCHED Go to user space CONFIG_PREEMPT_AUTO
129. LAZY_NEED_RESCHED Go to user space schedule CONFIG_PREEMPT_AUTO
130. LAZY_NEED_RESCHED CONFIG_PREEMPT_AUTO
131. LAZY_NEED_RESCHED CONFIG_PREEMPT_AUTO
132. LAZY_NEED_RESCHED NEED_RESCHED CONFIG_PREEMPT_AUTO
133. LAZY_NEED_RESCHED NEED_RESCHED schedule CONFIG_PREEMPT_AUTO
134. Questions?
135. Questions?