Sound of Scheduling: Writing Linux Schedulers in Java with eBPF

Slide 1

Slide 1 text

Sound of Scheduling timetableworld.com Writing Linux Schedulers in Java Johannes Bechberger (SAP SE)

Slide 2

Slide 2 text

You all know the problem:

Slide 3

Slide 3 text

You all know the problem:

Slide 4

Slide 4 text

You want to produce:

Slide 5

Slide 5 text

But that's not your kitchen

Slide 6

Slide 6 text

It's this

Slide 7

Slide 7 text

Focaccia Cantucini ... Oven

Slide 8

Slide 8 text

Focaccia Cantucini Oven Time

Slide 9

Slide 9 text

Focaccia Cantucini Oven Time Cantucini Focaccia Cantucini

Slide 10

Slide 10 text

Focaccia Cantucini Oven Time Cantucini Focaccia Cantucini

Slide 11

Slide 11 text

Focaccia Cantucini Oven Time Cantucini Focaccia Cantucini

Slide 12

Slide 12 text

Focaccia Cantucini Oven Time Cantucini Focaccia

Slide 13

Slide 13 text

Focaccia Cantucini Oven Time Cantucini Focaccia

Slide 14

Slide 14 text

Scheduling

Slide 15

Slide 15 text

What is scheduling?

Slide 16

Slide 16 text

Process A Process B CPU 1 ... CPU 2 ...

Slide 17

Slide 17 text

Process A Process B CPU Time

Slide 18

Slide 18 text

Process A Process B CPU A Time

Slide 19

Slide 19 text

Process A Process B CPU A Time

Slide 20

Slide 20 text

Process A Process B CPU A Time B

Slide 21

Slide 21 text

Process A Process B CPU A Time B A

Slide 22

Slide 22 text

Process A Process B CPU A Time B A A

Slide 23

Slide 23 text

Process A Process B CPU A Time B A B A

Slide 24

Slide 24 text

Process A Process B CPU A Time B A B A A

Slide 25

Slide 25 text

Hear this sound?

Slide 26

Slide 26 text

It's my scheduler

Slide 27

Slide 27 text

Written in Java

Slide 28

Slide 28 text

Why?

Slide 29

Slide 29 text

The only way of disco- vering the limits of the possible is to venture a little way past them into the impossible. Clarke’s second law https://www.flickr.com/photos/itupictures/16636142906 “

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

How?

Slide 33

Slide 33 text

How to modify the kernel?

Slide 34

Slide 34 text

Traditional ways 1.Change the Kernel 2.Kernel module

Slide 35

Slide 35 text

Traditional ways 1.Change the Kernel 2.Kernel module Interfaces are complicated

Slide 36

Slide 36 text

Traditional ways 1.Change the Kernel 2.Kernel module Not possible with schedulers

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

eBPF is a crazy technology, it’s like putting JavaScript into the Linux kernel Brendan Gregg “ https://www.youtube.com/watch?v=tDacjrSCeq4

Slide 39

Slide 39 text

eBPF is a crazy technology, it’s like putting JavaScript into the Linux kernel Brendan Gregg “ https://www.facesofopensource.com/brendan-gregg/

Slide 40

Slide 40 text

https://ebpf.io/what-is-ebpf/

Slide 41

Slide 41 text

Courtesy of Mohammed Aboullaite eBPF runtime

Slide 42

Slide 42 text

Courtesy of Mohammed Aboullaite eBPF runtime

Slide 43

Slide 43 text

Courtesy of Mohammed Aboullaite

Slide 44

Slide 44 text

How to share data? https://mostlynerdless.de/blog/2024/01/12/hello-ebpf-recording-data-in-basic-ebpf-maps-2/

Slide 45

Slide 45 text

How to share data? https://mostlynerdless.de/blog/2024/01/12/hello-ebpf-recording-data-in-basic-ebpf-maps-2/ Any Problems?

Slide 46

Slide 46 text

How to share data? https://mostlynerdless.de/blog/2024/01/12/hello-ebpf-recording-data-in-basic-ebpf-maps-2/

Slide 47

Slide 47 text

eBPF Maps Courtesy of Mohammed Aboullaite

Slide 48

Slide 48 text

https://ebpf.io

Slide 49

Slide 49 text

https://www.youtube.com/watch?v=X3AWV5lJ6RY

Slide 50

Slide 50 text

user land https://www.youtube.com/watch?v=X3AWV5lJ6RY

Slide 51

Slide 51 text

https://ebpf.io

Slide 52

Slide 52 text

Back to scheduling

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

1.Ease of experimentation and exploration 2.Customization 3.Rapid scheduler deployments https://lwn.net/Articles/978911/ “

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

Let's create a scheduler

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

What is the performance? Good* * For a typical Java benchmark

Slide 59

Slide 59 text

How does it work?

Slide 60

Slide 60 text

CPU sched-ext User land Kernel land Hardware Java Code C Code Byte Code Loaded attach

Slide 61

Slide 61 text

Task A CPU 1 Local Queue CPU 2 Local Queue Global Queue Scheduler ... ...

Slide 62

Slide 62 text

Schedulers can be simple

Slide 63

Slide 63 text

@BPF(license = "GPL") public abstract class SampleScheduler extends BPFProgram implements Scheduler, Runnable { static final long SHARED_DSQ_ID = 0; @Type static class Stats { long global; long local; } @Override public int init() { return scx_bpf_create_dsq(SHARED_DSQ_ID, -1); } }

Slide 64

Slide 64 text

@BPF(license = "GPL") public abstract class SampleScheduler { @BPFMapDefinition(maxEntries = 100) BPFHashMap statsPerCPU; @BPFMapDefinition(maxEntries = 100000) BPFLRUHashMap<@Unsigned Integer, @Unsigned Long> enqueuesPerProcess; }

Slide 65

Slide 65 text

@BPF(license = "GPL") public abstract class SampleScheduler { @Override public int selectCPU(Ptr p, int prev_cpu, long wake_flags) { boolean is_idle = false; int cpu = scx_bpf_select_cpu_dfl(p, prev_cpu, wake_flags, Ptr.of(is_idle)); if (is_idle) { !/* !!... / } return cpu; } } *

Slide 66

Slide 66 text

@BPF(license = "GPL") public abstract class SampleScheduler { @Override public void enqueue(Ptr p, long enq_flags) { incrementStats(false); recordEnqueue(p); scx_bpf_dispatch(p, SHARED_DSQ_ID, SCX_SLICE_DFL.value(), enq_flags); } }

Slide 67

Slide 67 text

@BPF(license = "GPL") public abstract class SampleScheduler { @Override public void dispatch(int cpu, Ptr prev) { scx_bpf_consume(SHARED_DSQ_ID); } }

Slide 68

Slide 68 text

But they behave differently

Slide 69

Slide 69 text

Demo

Slide 70

Slide 70 text

Lottery Scheduler

Slide 71

Slide 71 text

Task 1 CPU 1 CPU 2 Scheduling Queue treated as a lottery bowl ... ... Draw randomly from queue Ask for new task Return finished task Enqueue task for the first time Task 2 2 1 3 4 7 6 1

Slide 72

Slide 72 text

Demo

Slide 73

Slide 73 text

What else can we do with eBPF

Slide 74

Slide 74 text

Fin. Thanks to Dylan Reimerink sched-ext Slack Johannes Bechberger mostlynerdless.de OpenJDK Developer, SAP