Memory API : Patterns, Performance et Cas d'Utilisation

Slide 1

Slide 1 text

Memory API: Patterns, Uses Cases, and Performance From the Panama Foreign Functions and Memory API José Paumard Java Developer Advocate Java Platform Group Rémi Forax Maître de conferences Université Gustave Eiffel

Slide 2

Slide 2 text

https://twitter.com/Nope! https://github.com/forax https://speakerdeck.com/forax OpenJDK, ASM, Tatoo, Pro, etc… One of the Father of invokedynamic (Java 7) Lambda (Java 8), Module (Java 9) Constant dynamic (Java 11) Record, text blocks, sealed types (Java 14 / 15) Valhalla (Java 25+)

Slide 3

Slide 3 text

https://dev.java @josepaumard.bsky.social

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text

Slide 11

Slide 11 text

Slide 12

Slide 12 text

Slide 13

Slide 13 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 13 Heal the rift between Java and C Fixing issues in the Java NIO API Namely, fix and update what you can do with ByteBuffer ByteBuffer where released in Java 4, in 2002 What is Panama About?

Slide 14

Slide 14 text

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Slide 17

Slide 17 text

Slide 18

Slide 18 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 18 The World with NIO heap memory native memory write() read() buf.put() buf.get() ByteBuffer position limit capacity ... Servlet doPost() doGet() ...

Slide 19

Slide 19 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 19 Off-heap allocation File mapping Creating a ByteBuffer var buffer = ByteBuffer.allocateDirect(1_024); // int var buffer = FileChannel.map( READ_WRITE, position, size); // longs

Slide 20

Slide 20 text

Slide 21

Slide 21 text

Slide 22

Slide 22 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 22 Too high level for a Memory Access API position, capacity, reset are not needed 32 bits indexing only Allow for unaligned access, but may be very slow Non-deterministic deallocation! closing a mapped file does not close the ByteBuffer Issues with the ByteBuffer API

Slide 23

Slide 23 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 23 The GC - selects a region containing the ByteBuffer - then sees that the ByteBuffer is dead - then a Cleaner code (weakref) is pushed to a Cleaner queue Later, a cleaner thread dequeues the Cleaner code and calls free on the off-heap memory (or not…) How is Deallocation Working?

Slide 24

Slide 24 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 24 New API: the MemorySegment API - lower level than the ByteBuffer API - ByteBuffer are now built on top of MemorySegment Goals: - fix ByteBuffer issues - better interaction with C code Welcome to Panama

Slide 25

Slide 25 text

Slide 26

Slide 26 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 26 A MemorySegment: - is safe (cannot be used once freed) - gives you control over the allocation / deallocation - brings close to C performance (and Unsafe) - offers direct access, indexed 64 bits access, structured access - opt-in unsafe access (for C interop, may crash later) - retrofit ByteBuffer on top MemorySegment

Slide 27

Slide 27 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 27 It is unsafe! - Close to C performance - No use after free protection (security) - Can peek/poke everywhere (may crash later) - No null check for on heap array access (may crash) Memory access methods are - deprecated for removal (JEP 471, del. JDK 23) - warnings since 2006 What about sun.misc.Unsafe?

Slide 28

Slide 28 text

Slide 29

Slide 29 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 29 Most CPU require your data to be aligned in memory These are properly aligned Alignement 0x00FFA000 0x00FFA004 0x00FFA008 0x00FFA00C 0x00FFA010 byte short int short byte

Slide 30

Slide 30 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 30 Most CPU require your data to be aligned in memory These are misaligned Alignement 0x00FFA000 0x00FFA004 0x00FFA008 0x00FFA00C 0x00FFA010 short int

Slide 31

Slide 31 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 31 Off-heap allocation, direct access The MemorySegment API var arena = Arena.global(); var segment = arena.allocation(1_024L); // long, off-heap segment.set(ValueLayout.JAVA_INT, 4L, 42); var value = segment.get(ValueLayout.JAVA_INT, 4L);

Slide 32

Slide 32 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 32 Heap allocation, indexed access The MemorySegment API var ints = new int[] {1, 2, 3, 4}; var segment = MemorySegment.ofArray(ints); // on-heap segment.setAtIndex(ValueLayout.JAVA_INT, 2L, 65); var cell = segment.getAtIndex(ValueLayout.JAVA_INT, 2L);

Slide 33

Slide 33 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 34 Copy from on-heap to off-heap The MemorySegment API: File Mapping var ints = new int[] {1, 2, 3, 4}; var arraySegment = MemorySegment.ofArray(ints); // on-heap var offHeapSegment = Arena.global().allocate(64L); // off-heap offHeapSegment.copyFrom(arraySegment);

Slide 34

Slide 34 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 35 Writing data to a mapped file: you need a ByteBuffer The MemorySegment API: File Mapping var ints = new int[] {1, 2, 3, 4}; var arraySegment = MemorySegment.ofArray(ints); // on-heap var offHeapSegment = Arena.global().allocate(64L); // off-heap offHeapSegment.copyFrom(arraySegment); var byteBuffer = offHeapSegment.asByteBuffer(); // this is a view! byteBuffer.limit( ints.length * (int) ValueLayout.JAVA_INT.byteSize()); try (var file = FileChannel.open(path, CREATE, WRITE)) { file.write(byteBuffer); }

Slide 35

Slide 35 text

Slide 36

Slide 36 text

Slide 37

Slide 37 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 38 An Arena can create off-heap memory segments It initializes memory segment with zeroes It is AutoCloseable (more on this in a mn) It deallocates the memory segments it created on close() Introducing Arena

Slide 38

Slide 38 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 39 There are four of them: What is this Arena object? var global = Arena.global(); // singleton var confined = Arena.ofConfined(); var shared = Arena.ofShared(); var auto = Arena.ofAuto(); // supports legacy // ByteBuffer semantics

Slide 39

Slide 39 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 40 There are four of them: What is this Arena object? Bounded Lifetime Closed by the User Shared among threads Global No No Yes Auto Confined Shared

Slide 40

Slide 40 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 41 There are four of them: What is this Arena object? Bounded Lifetime Closed by the User Shared among threads Global No No Yes Auto Yes No (GC) Yes Confined Shared

Slide 41

Slide 41 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 42 There are four of them: What is this Arena object? Bounded Lifetime Closed by the User Shared among threads Global No No Yes Auto Yes No (GC) Yes Confined Yes Yes No Shared

Slide 42

Slide 42 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 43 There are four of them: What is this Arena object? Bounded Lifetime Closed by the User Shared among threads Global No No Yes Auto Yes No (GC) Yes Confined Yes Yes No Shared Yes Yes Yes

Slide 43

Slide 43 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 44 int[] vs memorySegment.get(JAVA_INT, ...) Benchmarks Array 0.728 ± 0.009 ns/op OfArray 1.358 ± 0.003 ns/op Confined 1.255 ± 0.002 ns/op Auto 1.254 ± 0.002 ns/op Shared 1.254 ± 0.013 ns/op Global 1.258 ± 0.026 ns/op Unsafe 0.627 ± 0.001 ns/op

Slide 44

Slide 44 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 45 Looping and summing 512 ints Benchmarks Array 128.338 ± 0.084 ns/op OfArray 131.927 ± 0.761 ns/op Confined 131.829 ± 0.077 ns/op Auto 131.832 ± 0.491 ns/op Shared 131.760 ± 0.068 ns/op Global 131.727 ± 0.137 ns/op Unsafe 128.083 ± 0.131 ns/op

Slide 45

Slide 45 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 46 Access time is independent of the type of arena For random direct access - Overhead is important (2x) - 3 checks 1. Is it the right thread? 2. Has the Arena been Closed? 3. Is access in bounds? Random Access Performance

Slide 46

Slide 46 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 47 Access time is independent of the type of arena For loop + indexed access - Fixed cost at the beginning of the loop - 3 Checks are hoisted out of the loop 1. Is it the right thread? Is done once 2. Has the Arena been Closed? Is done once 3. Is access in bounds? Is elided Loop Performance

Slide 47

Slide 47 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 48 Memory fragmentation Application integrity Jextract Memory Layout Structured memory access with offsets and VarHandle Lazy allocation using Stable Value (prev. 25) After the Break

Slide 48

Slide 48 text

Coffee Break!

Slide 49

Slide 49 text

Slide 50

Slide 50 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 51 Allocation/Deallocation of an Arena + MemorySegment Benchmarks Array 2.522 ± 0.015 ns/op OfArray 6.494 ± 0.093 ns/op Confined 82.287 ± 1.530 ns/op Unsafe (malloc) 22.834 ± 0.097 ns/op Unsafe with init 72.338 ± 0.390 ns/op

Slide 51

Slide 51 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 52 With two arenas How Do Arenas Allocate Segments? var arenaA = Arena.ofConfined(); var a1 = arenaA.allocate(...); heap memory native memory Arena JVM a1

Slide 52

Slide 52 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 53 With two arenas How Do Arenas Allocate Segments? var arenaA = Arena.ofConfined(); var a1 = arenaA.allocate(...); var a2 = arenaA.allocate(...); heap memory native memory Arena JVM a1 a2

Slide 53

Slide 53 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 54 With two arenas How Do Arenas Allocate Segments? var arenaA = Arena.ofConfined(); var arenaB = Arena.ofConfined(); var b1 = arenaB.allocate(...); var a1 = arenaA.allocate(...); var b2 = arenaB.allocate(...); var a2 = arenaA.allocate(...); var b3 = arenaB.allocate(...); var b4 = arenaB.allocate(...); memory b2 a1 b3 a2 b4 b1

Slide 54

Slide 54 text

Slide 55

Slide 55 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 56 Then arenaC is created How Do Arenas Allocate Segments? memory var arenaC = Arena.ofConfined(); var c1 = arenaC.allocate(...); b2 b3 b4 c1 b1

Slide 56

Slide 56 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 57 Then arenaC is created How Do Arenas Allocate Segments? memory var arenaC = Arena.ofConfined(); var c1 = arenaC.allocate(...); var c2 = arenaC.allocate(...); c2 X X b2 b3 b4 b1 c1

Slide 57

Slide 57 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 58 Then arenaC is created And you end up with fragmentation! How Do Arenas Allocate Segments? memory var arenaC = Arena.ofConfined(); var c1 = arenaC.allocate(...); var c2 = arenaC.allocate(...); c2 b2 b3 b4 b1 X X c1

Slide 58

Slide 58 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 59 Holes in native memory It leads to two problems: - Allocation is slow, you need to find a large enough space for your memory segment - Not enough contiguous free memory may prevent the creation of a large memory segment Fragmentation

Slide 59

Slide 59 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 60 Default arenas create fragmentation But Arena is an interface! So you can implement your own allocation strategy https://docs.oracle.com/en/java/javase/24/docs/api/java.base/j ava/lang/foreign/Arena.html#custom-arenas Custom Arenas

Slide 60

Slide 60 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 61 Allocation/Deallocation of an Arena + MemorySegment Benchmarks Array 2.522 ± 0.015 ns/op OfArray 6.494 ± 0.093 ns/op Confined 82.287 ± 1.530 ns/op Auto 434.694 ± 247.868 ns/op Shared 6696.144 ± 37.833 ns/op Unsafe (malloc) 22.834 ± 0.097 ns/op Unsafe with init 72.338 ± 0.390 ns/op

Slide 61

Slide 61 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 62 ofAuto(): close() has the same semantics as ByteBuffer Slooooow! In the worst case scenario it calls System.gc() (even slooooooower!) Deallocation / close()

Slide 62

Slide 62 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 63 ofShared(): you want to avoid having a volatile access in get() (to know if the arena has been closed) close() performs a VM Handcheck with all other threads checks method on top of the stack is annotated as performing an access checks if the locals contains the closing arena ⇒ Linear with the number of platform threads Deallocation / close()

Slide 63

Slide 63 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 64 Confined: default choice, manual deallocation Shared = confined + multi thread access Not the malloc API, you should try to group allocations Global: permanent memory Auto: legacy, multi thread access, GC triggered deallocation Arena Uses Cases

Slide 64

Slide 64 text

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Slide 67

Slide 67 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 68 Memory segments are safe by default - even creating a memory segment from a long is safe (byteSize is 0) MemorySegment.reinterpret(newSize) - opt-in to unsafe, only for native memory - requires --enable-native-access on the command line - emits a warning in Java 23, will be an error in the future Unsafe MemorySegment

Slide 68

Slide 68 text

Slide 69

Slide 69 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 70 JEP 261: Module System JEP 260: Encapsulate Most Internal APIs JEP 396: Strongly Encapsulate JDK Internals by Default JEP 403: Strongly Encapsulate JDK Internals JEP 451: Prepare to Disallow the Dynamic Loading of Agents JEP 471: Deprecate the Memory-Access Methods in sun.misc.Unsafe for Removal JEP 472: Prepare to Restrict the Use of JNI JEP 498: Warn upon Use of Memory-Access Methods in sun.misc.Unsafe Draft JEP: Integrity by Default

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Slide 72

Slide 72 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 73 Simplify C interoperability - Jextract takes a .h file and creates java classes from it - It creates one class for the .h with the function definitions - Then one per struct What is Jextract?

Slide 73

Slide 73 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 74 Uses LLVM internally To correctly parse C declarations And to extract platform/OS definitions (eg: what is the size of an int) It is an external tool, that needs to be downloaded separately https://jdk.java.net/jextract What is Jextract?

Slide 74

Slide 74 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 75 It is an interface that describe a piece of memory What is MemoryLayout? MemoryLayout SequencedLayout GroupLayout PaddingLayout ValueLayout StructLayout UnionLayout

Slide 75

Slide 75 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 76 Defining a struct Point A MemoryLayout can be Named var pointLayout = MemoryLayout.structLayout( ValueLayout.JAVA_INT, ValueLayout.JAVA_INT );

Slide 76

Slide 76 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 77 Size of the struct Point, offset of x and y in Point MemoryLayout Size and Offset var pointLayout = MemoryLayout.structLayout( ValueLayout.JAVA_INT, ValueLayout.JAVA_INT ); long pointLayoutSize = pointLayout.byteSize(); long xOffset = pointLayout.byteOffset(0); // by index

Slide 77

Slide 77 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 78 Size of the struct Point, offset of x and y in Point MemoryLayout Size and Offset var pointLayout = MemoryLayout.structLayout( ValueLayout.JAVA_INT.withName("x"), ValueLayout.JAVA_INT.withName("y") ).withName("point"); long pointLayoutSize = pointLayout.byteSize(); long xOffset = pointLayout.byteOffset(0); // by index long yOffset = pointLayout.byteOffset("y"); // by name

Slide 78

Slide 78 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 79 Memory layouts need padding Alignment and Padding struct { char kind; int payload; char extra; } 0 32 ki 64 16 48 payload padding static final MemoryLayout LAYOUT = MemoryLayout.structLayout( ValueLayout.JAVA_BYTE.withName("kind"), MemoryLayout.paddingLayout(3), ValueLayout.JAVA_INT.withName("payload"), ValueLayout.JAVA_BYTE.withName("extra"), MemoryLayout.paddingLayout(3) ); 80 96 ex padding

Slide 79

Slide 79 text

Slide 80

Slide 80 text

Slide 81

Slide 81 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 82 An object that gives access to fields with different semantics: - a get / set access (plain, opaque, volatile) - and concurrent access: compareAndSet, getAndAdd, … It hides the offset and size computations to access the elements of your memory layout What is a VarHandle?

Slide 82

Slide 82 text

Slide 83

Slide 83 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 84 1) The compiler doesn’t do any type checking: it trusts you! (and the different IDE are not there to hep you…) 2) It allows conversion at runtime It can be convenient, but can lead to autoboxing You can use withInvokeExactBehavior() VarHandle Caveats

Slide 84

Slide 84 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 85 Compute the sum of 512 point.x + point.y OfArray with offset 214.479 ± 1.712 ns/op OfArray with VarHandle 141.404 ± 0.081 ns/op Unsafe with offset 137.518 ± 0.476 ns/op Arena with offset 212.881 ± 3.436 ns/op Arena with VarHandle 141.190 ± 0.107 ns/op Benchmarks

Slide 85

Slide 85 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 86 Offset computation by the user is slow VarHandle offers an access pattern to the JVM, which gives you better performance Jextract does not create VarHandles, only offsets Using Offsets or VarHandle?

Slide 86

Slide 86 text

Slide 87

Slide 87 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 88 A stable value holds an eventually non-modifiable data Three guarantees: 1) Initialized when the value is first requested Not initialized at application startup nor at class initialization 2) Initialization code is run once 3) Once initialized, treated as (a real) constant by the JVM Stable Value

Slide 88

Slide 88 text

Slide 89

Slide 89 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 90 Current state of the high-level API (prev. in 25) Stable Value class StableValue { static Supplier supplier(Supplier supplier) { ... } static List list(int size, IntFunction mapper) { ... } static Map map(Set keys, Function mapper) { ... } }

Slide 90

Slide 90 text

4/15/2025 Copyright © 2025, Oracle and/or its affiliates 91 Java 24 confinedStableMapLoop avgt 5 140.513 ± 0.132 ns/op confinedStableValueLoop avgt 5 140.973 ± 0.468 ns/op confinedVarHandleLoop avgt 5 140.862 ± 0.583 ns/op Java 25 (EA) confinedStableMapLoop avgt 5 22294.893 ± 99.458 ns/op confinedStableValueLoop avgt 5 140.634 ± 0.319 ns/op confinedVarHandleLoop avgt 5 140.533 ± 0.082 ns/op Stable Value – Performance