SIMD Parallel Programming with the Vector API

SIMD Parallel Programming with the Vector API Advanced parallel computing
José Paumard Java Developer Advocate Java Platform Group

https://twitter.com/JosePaumard https://github.com/JosePaumard https://www.youtube.com/c/JosePaumard01 https://www.youtube.com/user/java https://www.youtube.com/hashtag/jepcafe https://fr.slideshare.net/jpaumard https://www.pluralsight.com/authors/josepaumard https://dev.java

Tune in! Copyright © 2021, Oracle and/or its affiliates |
4 Inside Java Newscast JEP Café Road To 21 series Inside.java Inside Java Podcast Sip of Java Cracking the Java coding interview

4/24/2024 Copyright © 2021, Oracle and/or its affiliates | 5
JEP 469, 7th Incubator in 22, and…

4/24/2024 Copyright © 2021, Oracle and/or its affiliates | Confidential:
Internal/Restricted/Highly Restricted 6 What is SIMD computing? Single Instruction Multiple Data

Parallel streams Computing Things in Parallel int[] aLotOfData = ...; Arrays.stream(aLotOfData) .parallel() .map(Some::mapping) .filter(Some::filtering) .reduce(identityElement, Some::reduction);

Parallel arrays Computing Things in Parallel int[] aLotOfData = ...; Arrays.parallelSort(aLotOfData);

It relies on concurrency and the splitting of your data Computing Things in Parallel

At some point your data elements are small enough to be processed Computing Things in Parallel

Then these partial results are merged using a merge function Computing Things in Parallel R1 R2 R3 R4 R5 R6 R7 R8

Until the final result can be computed Computing Things in Parallel M(R1 , R2 ) M(R3 , R4 ) M(R5 , R6 ) M(R7 , R8 )

Until the final result can be computed Computing Things in Parallel M(M1 , M2 ) M(M3 , M4 )

Until the final result can be computed The merge function can be the same as the reduction operation (SUM, MIN, MAX) But it can be different (merge sort vs sort) Computing Things in Parallel Final Result

All this is computed in different threads, from the Common Fork Join Pool So you may come across visibility, race condition, or blocking issues Don’t do I/O in a parallel stream! Don’t use non-splitable data sources! Don’t use some operations (skip, limit) Computing Things in Parallel

What kind of gain can you expect? It depends on your merge operation! If it is the same: the number of cores, minus the overhead If it is not: well, it depends! (measure, don’t guess!) Computing Things in Parallel

SIMD computing is completely different

What is SIMD? Short answer: an old concept Parallel Computing of Things

Less short answer: it has to do with how CPU are working Parallel Computing of Things Reg A PU Reg B Result

The PU loads instruction, that can load data Parallel Computing of Things 10 PU 20 LOAD

The PU loads instruction, that can load data, and do something with it Parallel Computing of Things 10 PU 20 30 LOAD ADD

SIMD works with several PU and registers Parallel Computing of Things memory PGM

SIMD works with several PU and registers, can load all of them on the same CPU cycle Parallel Computing of Things 10 10 LOAD 21 30 5 12 8 3 memory

SIMD works with several PU and registers, can load all of them on the same CPU cycle, and add them Parallel Computing of Things 10 10 20 LOAD ADD 21 30 51 5 12 17 8 3 11 memory

No threads, no splitting of your data, no partial reductions Overhead? Parallel Computing of Things

Two questions: 1) What is this data memory? How can you access it? 2) What are the available operations? Parallel Computing of Things

SIMD relies on: 1) specific hardware elements on each CPU core 2) specific assembly instructions AVX-256, AVX-512 No concurrency! You need to check what is your CPU capable of! The SIMD Way of Computing

This memory is a specific array in your CPU From 512 bits to 128, with different instructions sets Parallel Computing of Things 256 bits 4 longs or doubles 8 ints or floats 16 shorts 32 bytes

Pros: 1) Does not rely on concurrent programming 2) No overhead for splitting and merging data Cons: 1) Still an overhead to load and unload your vectors 2) Algorithms can be more complex 3) Efficient code depends on what your CPU can do The SIMD Way of Computing

Let us add two vectors How is the Vector API Working? int[] v1 = ...; int[] v2 = ...; int[] result = new int[v1.length]; var V1 = IntVector.fromArray(species, v1, 0); var V2 = IntVector.fromArray(species, v2, 0);

Let us add two vectors The species object carries info on the CPU and the vector type How is the Vector API Working? int[] v1 = ...; int[] v2 = ...; int[] result = new int[v1.length]; var species = IntVector.SPECIES_PREFERRED; var V1 = IntVector.fromArray(species, v1, 0); var V2 = IntVector.fromArray(species, v2, 0);

Let us add two vectors The species object carries info on the CPU and the vector type How is the Vector API Working? private static final Species species = IntVector.SPECIES_PREFERRED; int[] v1 = ...; int[] v2 = ...; int[] result = new int[v1.length]; var V1 = IntVector.fromArray(species, v1, 0); var V2 = IntVector.fromArray(species, v2, 0);

Let us add two vectors How is the Vector API Working? int[] v1 = ...; int[] v2 = ...; int[] result = new int[v1.length]; var V1 = IntVector.fromArray(species, v1, 0); var V2 = IntVector.fromArray(species, v2, 0); var RESULT = V1.add(V2); RESULT.intoArray(result, 0);

Let us add two vectors This add operation maps to an assembly instruction, that adds all the components of V1 and V2 in 1 CPU cycle The overhead is the copying of the arrays into the vectors, back and forth. How is the Vector API Working? var RESULT = V1.add(V2);

ADD, SUB, MULT, DIV ABS, NEG, MIN, MAX EQ, GT, LT Bit-wise operations Parallel Operations

The size of the array has to match the size of the vector your SIMD kernel can handle. For 256 bits: 8 ints. How can you handle larger arrays? For loop FTW! Caveat!

Large Arrays Species species = IntVector.SPECIES_PREFERRED; for (int index = 0; index < v1.length; index += 8) { var V1 = IntVector.fromArray(species, v1, index); var V2 = IntVector.fromArray(species, v2, index); var RESULT = V1.add(V2); RESULT.intoArray(result, index); }

Large Arrays Species species = IntVector.SPECIES_PREFERRED; for (int index = 0; index < v1.length; index += species.length()) { var V1 = IntVector.fromArray(species, v1, index); var V2 = IntVector.fromArray(species, v2, index); var RESULT = V1.add(V2); RESULT.intoArray(result, index); }

Hmmm… not quite! What if your array has 2222 elements? 2222 = 277x8 + 6 There will be 6 leftover elements… Two solutions, depending on what your CPU can do Are we Done?

1st Solution Species species = IntVector.SPECIES_PREFERRED; for (int index = 0; index < v1.length; index += species.length()) { var mask = species.indexInRange(index, v1.length); var V1 = IntVector.fromArray(species, v1, index, mask); var V2 = IntVector.fromArray(species, v2, index, mask); var RESULT = V1.add(V2, mask); RESULT.intoArray(result, index, mask); }

It relies on the use of masks … this is not supported by all SIMD kernels / CPU So there is still another pattern 1st Solution

2nd Solution Species species = IntVector.SPECIES_PREFERRED; int index = 0; for (; index < species.loopBound(v1.length); index += species.length()) { var V1 = IntVector.fromArray(species, v1, index); var V2 = IntVector.fromArray(species, v2, index); var RESULT = V1.add(V2); RESULT.intoArray(result, index); } for (int index2 = index; index2 < v1.length; index2++) { result[index2] = v1[index2] + v2[index2]; }

Measure, don’t guess! If your CPU does not support masking, the 2nd pattern may be faster Which One is Better?

In the context of the Vector API Vectors have lanes and not components So lanes and components are actually the same thing Lane = the way your data flows through the SIMD machine of your computer Some Vocabulary

Two kinds of operations 1) Lane-wise operations: operate on a given lane for two vectors ADD, SUB, etc… are lane-wise operations 2) Cross-lane operations: operate on the different lanes of a vector MAX, MIN, SORT are cross-lanes operations Operations on Lanes

Computing the Norm of a Vector – V1 Species species = FloatVector.SPECIES_PREFERRED; float sum = 0f; for ( ) { var V = FloatVector.fromArray(...); var V2 = V.mult(V); sum += V2.reduceLanes(VectorOperators.ADD); } float norm = Math.sqrt(norm);

Computing the Norm of a Vector – V2 Species species = FloatVector.SPECIES_PREFERRED; var SUM = FloatVector.zero(species); // can call broadcast with a value for ( ) { var V = FloatVector.fromArray(...); var V2 = V.mult(V); SUM = SUM.add(V2); } float norm = Math.sqrt(SUM.reduceLanes(VectorOperators.ADD));

Vector Operators Java Day Copyright © 2024, Oracle and/or its
affiliates 53 VectorOperator Unary Binary Ternary Test Conversion Comparison

affiliates 54 Unary NOT ZOMO ABS NEG BIT_COUNT TRAILING_ZERO_COUNT LEADING_ZERO_COUNT REVERSE REVERSE_BYTES SIN COS TAN SINH COSH TANH ASIN ACOS ATAN EXP LOG LOG10 EXPM1 LOG1P SQRT CBRT

affiliates 55 Binary SUB DIV AND_NOT LSHL ASHR LSHR ROL ROR COMPRESS_BITS EXPAND_BITS ATAN2 POW HYPOT Associative ADD MULT MIN MAX FIRST_NON_ZERO AND OR XOR OR_UNCHECKED

affiliates 56 Ternary BITWISE_BLEND FMA

affiliates 57 Test IS_DEFAULT IS_NEGATIVE IS_FINITE IS_NAN IS_INFINITE

affiliates 58 Comparison EQ NE LT LE GT GE UNSIGNED_LT UNSIGNED_LE UNSIGNED_GT UNSIGNED_GE

affiliates 59 Conversion Byte Short Integer Long Float Double Byte B2S B2I B2L B2F B2D Short S2B S2I S2L S2F S2D Integer I2B I2S I2L I2F I2D Long L2B L2S L2I L2F L2D Float F2B F2S F2I F2L F2D Double D2B D2S D2I D2L D2F

affiliates 60 Conversion Double -> Long REINTERPRET_D2L Long -> Double REINTERPRET_L2D Float -> Integer REINTERPRET_F2I Integer -> Float REINTERPRET_I2F

affiliates 61 Conversion Byte -> Short ZERO_EXTEND_B2S Byte -> Integer ZERO_EXTEND_B2I Byte -> Long ZERO_EXTEND_B2L Short -> Integer ZERO_EXTEND_S2I Short -> Long ZERO_EXTEND_S2L Integer -> Long ZERO_EXTEND_I2L

Computing an Average Species species = FloatVector.SPECIES_PREFERRED; for ( ) { var V = FloatVector.fromArray(...); var sum = SUM.reduceLanes(VectorOperators.ADD) } float average = sum / v1.length;

Because the VectorOperators objects map specific assembly instructions of your SIMD kernel Why Cant you Pass a Lambda?

Filtering is about selecting the lanes that match a criteria You cant avoid masking here! Filtering Lanes 3 8 5 1 6 9 7 5 > 5

Filtering is about selecting the lanes that match a criteria You cant avoid masking here! Filtering Lanes 3 8 5 1 6 9 7 5 > 5 F T F F T T T F = mask

Filtering is about selecting the lanes that match a criteria You cant avoid masking here! Filtering Lanes 3 8 5 1 6 9 7 5 > 5 F T F F T T T F = mask 8 6 9 7 - - - - compress:

Compression + copy into the result array Filtering Lanes V1 Species species = IntVector.SPECIES_PREFERRED; int maxIndex = 0; for ( ) { var mask = V.compare(VectorOperators.GT, 5); V.compress(mask).intoArray(result, maxIndex); maxIndex += mask.trueCount(); // the number of compressed elements }

You can also copy with a mask into the result array Filtering Lanes V2 Species species = IntVector.SPECIES_PREFERRED; int maxIndex = 0; for ( ) { var mask = V.compare(VectorOperators.GT, 5); V.intoArray(result, maxIndex, mask); maxIndex += mask.trueCount(); // the number of compressed elements }

No need to compress Filtering Lanes + Reduction Species species = DoubleSpecies.SPECIES_PREFERRED; double sum = 0d; double count = 0d; for ( ) { var mask = V.compare(VectorOperators.GT, 5); sum += V.reduceLanes(VectorOperators.ADD, mask); count += mask.trueCount(); // the number of added elements } double average = sum / count;

https://github.com/openjdk/jdk/tree/master/test/micro/org/openjdk/bench/jdk/incubator/vector https://bit.ly/vector-api Performances

String comparison! String size JDK Scalar JDK Vector API 16 18ns ±2ns 17ns ±1ns 32 40ns ±4ns 8ns ±1ns 64 61ns ±1ns 10ns ±1ns 128 123ns ±3ns 16ns ±1ns 1024 940ns ±9ns 90ns ±1ns Performances

The Vector API can be used in many places in the JDK to improve the performances of common operations Array and String comparison String compare ignore case String charset conversion Hash code of String and arrays computation Array sorting (no merge sort!) Arc Tangent computation Applications

The Vector API can be used in many places in the JDK to improve the performances of common operations Linear Algebra! Neural Networks Machine Learning Artificial Intelligence Applications

The Vector API rocks!

The Vector API rocks! Questions?

SIMD Parallel Programming with the Vector API

SIMD Parallel Programming with the Vector API

More Decks by José

Other Decks in Programming

Featured

Transcript