Schedule Recipes

In this talk, we’ll explore a collection of minimal schedulers built using the Linux kernel’s sched_ext framework.

Some are practical, others (intentionally) less so, but all highlight the expressive power and creativity that sched_ext can bring. Like a good set of recipes, these examples are meant to nspire experimentation, lower the barrier of kernel scheduler development, and help you cook up your own ideas, whether it’s for performance tuning, research, or just for fun.

Andrea RIGHI

Transcript

1. Scheduling recipes Andrea Righi <arighi@nvidia.com>
2. What is a scheduler? • Kernel component that determines ◦ Where each task needs to run ◦ When each task needs to run ◦ For how long each task needs to run • Scheduler ◦ CPU allocator (space) ◦ Task allocator (time)
3. sched_ext: extensible scheduler class • Technology in the Linux kernel that allows to implement scheduling policies as BPF programs (GPLv2) • Available since Linux v6.12 • Key features ◦ Bespoke scheduling policies ◦ Rapid experimentation ◦ Safety (can’t crash the kernel)
4. BPF + user-space schedulers Kernel BPF User space sched_ext core BPF scheduler libbpf libbpf-rs User-space scheduler sched_ext callbacks
5. Gaming: fps consistency EEVDF scx_flash
6. Server workload: Llama benchmark ~2.6x throughput
7. Ingredients
8. Shared runqueue
9. Per-CPU runqueues
10. Runtime allocation (fairness)
11. Topology complexity NUMA node 0 L3 cache Core 0 (big) CPU 0 CPU 1 Core 1 (big) CPU 0 CPU 1 L3 cache Core 2 (LITTLE) CPU 0 CPU 1 Core 3 (LITTLE) CPU 0 CPU 1 NUMA node 1 L3 cache Core 0 (big) CPU 0 CPU 1 Core 1 (big) CPU 0 CPU 1 L3 cache Core 2 (LITTLE) CPU 0 CPU 1 Core 3 (LITTLE) CPU 0 CPU 1
12. Task wake-up CPU 0 CPU 1 Waker Wakee Cache Cache Wake up
13. Recipes
14. Recipe #1: The empty scheduler • Empty scheduler (use sched_ext default) ◦ Global run-queue ◦ Round-robin scheduler (20ms time slice) ◦ Built-in idle CPU selection policy
15. Recipe #2: The global vs local runqueue scheduler • Round robin scheduler that can use either a single global queue or multiple per-CPU runqueues ◦ Time slice scaled proportionally to the task’s priority (weight) and inversely proportional to the amount of contending tasks (fairness) ◦ Perfect load balancing (global) vs bad load balancing (local) ◦ Not really good if you overload the system (round-robin) ◦ Bad for cache locality and scalability (global) vs good for cache locality and scalability (local)
16. Recipe #3: The “yell at your PC” scheduler • Adjust the number of CPUs used based on the noise level around your PC ◦ Show potential BPF / user space interactions ◦ Good for energy consumption (if the environment is quiet) ◦ Not usable in public places (i.e., library, open space office, etc.)
17. Conclusion
18. Key takeaways • The one-size-fits-all scheduler approach is no longer sufficient • CPU allocation can be more effective than time allocation • Shared queues vs local queues can be relevant • Prioritizing waker –> wakee pipelines can help improve responsiveness • Hybrid schedulers (BPF + user space) have great potential
19. References • Demo schedulers: https://github.com/sched-ext/scx/tree/kr-demo
20. Questions