Let's Build a Cross-platform 3D Engine with Kotlin/Multiplatform

Slide 1

Slide 1 text

LET'S BUILD A CROSS PLATFORM 3D ENGINE WITH KOTLIN/MULTIPLATFORM

Slide 2

Slide 2 text

David Wursteisen

Slide 3

Slide 3 text

"#$%&'($#&

Slide 4

Slide 4 text

John Carmack

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

Game   development Kotlin Multiplatform

Slide 9

Slide 9 text

You, at the end of   this conference Game   development Kotlin Multiplatform

Slide 10

Slide 10 text

GAME DEVELOPMENT

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

public void render(ﬂoat delta) {  movePlayer(delta);  moveEnemies(delta);  moveWorld(delta);    renderWorld();  renderEnemies();  renderPlayer();  }

Slide 17

Slide 17 text

public void render(ﬂoat delta) {  movePlayer(delta);  moveEnemies(delta);  moveWorld(delta);    renderWorld();  renderEnemies();  renderPlayer();  }

Slide 18

Slide 18 text

public void render(ﬂoat delta) {  movePlayer(delta);  moveEnemies(delta);  moveWorld(delta);    renderWorld();  renderEnemies();  renderPlayer();  }

Slide 19

Slide 19 text

public void render(ﬂoat delta) {  movePlayer(delta);  moveEnemies(delta);  moveWorld(delta);    renderWorld();  renderEnemies();  renderPlayer();  } 60x  seconds

Slide 20

Slide 20 text

(0,0) x y

Slide 21

Slide 21 text

PICKING A GAME ENGINE

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

❤

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

Hair Dash - http://cleancutgames.com/

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

Core iOS GWT Android Desktop

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Hi all. My studio, Sickhead, did the port of Slay the Spire to Switch. We struggled for a bit trying to convert the Java to C++ code directly using existing third party tools. The resulting C++ code pretty much didn't work at all and took tons of manual changes to make work. After several months of failure taking that path we changed gears. We instead converted Java to C# which required much less manual ﬁxes to make work and solved all the GC memory management issues. This means we had a C# version of LibGDX in there as well as all the game code. We then just added a new LibGDC backend with new OpenGL bindings and new sound, input implementations. At that point we had a PC port of Spire using C#. We then used our own mature C# IL to C++ cross compiler and runtime we developed to port XNA and MonoGame titles to generate binaries which run on Switch, PS4, XB1, and other platforms. While Java -> C# -> C++ was a crazy idea when we started... it actually works beautifully and makes perfect sense now. HTTPS://WWW.BADLOGICGAMES.COM/FORUM/VIEWTOPIC.PHP?F=11&T=29625

Slide 32

Slide 32 text

Slide 33

Slide 33 text

KOTLIN MULTIPLATFORM FEEDBACK LOOP

Slide 34

Slide 34 text

LET'S DIVE IN BUILDING A 3D ENGINE

Slide 35

Slide 35 text

void render(float delta) { }

Slide 36

Slide 36 text

fun render(delta: float) { }

Slide 37

Slide 37 text

KOTLIN/MULTIPLATFOM KOTLIN/JVM KOTLIN/NATIVE KOTLIN/JS KOTLIN/ANDROID

Slide 38

Slide 38 text

Common Js Native Android Jvm

Slide 39

Slide 39 text

KOTLIN/JVM KOTLIN/JS KOTLIN/ANDROID KOTLIN/NATIVE COMMON LIBGDX LIBGDX PLAYCANVAS RAYLIB

Slide 40

Slide 40 text

JS NATIVE JVM

Slide 41

Slide 41 text

KOTLIN/NATIVE KOTLIN/JS *.H CINTEROP *.KT *.D.TS DUKAT *.KT

Slide 42

Slide 42 text

external open class ScrollbarComponentSystem(app: pc.Application) : ComponentSystem { override fun on(name: String, callback: HandleEvent, scope: Any): EventHandler override fun off(name: String, callback: HandleEvent, scope: Any): EventHandler override fun fire(name: Any, arg1: Any, arg2: Any, arg3: Any, arg4: Any): EventHandler override fun once(name: String, callback: HandleEvent, scope: Any): EventHandler override fun hasEvent(name: String): Boolean }

Slide 43

Slide 43 text

external interface `T$8` { var volume: Number? get() = definedExternally set(value) = definedExternally var pitch: Number? get() = definedExternally set(value) = definedExternally var loop: Boolean? get() = definedExternally set(value) = definedExternally )

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

OpenGL

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

OpenGL ES was deprecated in iOS 12.

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

VERTEX VERTEX VERTEX

Slide 53

Slide 53 text

HTTPS://WWW.LABRI.FR/PERSO/NROUGIER/PYTHON-OPENGL/#MODERN-OPENGL VERTICES VERTEX  SHADER FRAGMENT  SHADER OpenGL (GPU) Kotlin (CPU)

Slide 54

Slide 54 text

void main() {  vec4 sum = vec4(0);    float step = iResolution.y;  // y  for(float i = -area ; i <= area ; i += 1.) {  // x  for(float j = -area ; j <= area ; j += 1.) {  float x = v_texCoords.x + i / iResolution.x;  float y = v_texCoords.y + j / iResolution.y;  sum += texture2D(u_texture, vec2(x, y)) * 0.005;  }  }  gl_FragColor = texture2D(u_texture, v_texCoords) + sum;  } OpenGL Shading Language (GLSL)

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

BUILDING A 3D ENGINE BASED ON OpenGL

Slide 60

Slide 60 text

KOTLIN/JVM KOTLIN/JS KOTLIN/ANDROID KOTLIN/NATIVE COMMON LWJGL OpenGL ES WebGL OpenGL

Slide 61

Slide 61 text

WHAT YOU EXPECT

Slide 62

Slide 62 text

WHAT YOU'LL GET

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

A SHORT HIKE

Slide 65

Slide 65 text

class GameExample : Game { override fun render(delta: Float) { } } Unit ?

Slide 66

Slide 66 text

typealias Seconds = Float class GameExample : Game { override fun render(delta: Seconds) { } }

Slide 67

Slide 67 text

interface CanMove { fun rotateX(angle: Float): CanMove } Radian or Degree ?

Slide 68

Slide 68 text

typealias Degree = Float interface CanMove { fun rotateX(angle: Degree): CanMove }

Slide 69

Slide 69 text

Y X Z

Slide 70

Slide 70 text

1. Translation 2. Rotation 3. Scale

Slide 71

Slide 71 text

MATRIX

Slide 72

Slide 72 text

MATRIX [ 1 | 0 | 0 | 0 ]  [ 0 | 1 | 0 | 0 ]  [ 0 | 0 | 1 | 0 ]  [ 0 | 0 | 0 | 1 ]

Slide 73

Slide 73 text

Scale MATRIX [ 1 | 0 | 0 | 0 ]  [ 0 | 1 | 0 | 0 ]  [ 0 | 0 | 1 | 0 ]  [ 0 | 0 | 0 | 1 ] X Z Y Translation Rotation

Slide 74

Slide 74 text

TRANSFORMATION = IDENTITY

Slide 75

Slide 75 text

TRANSFORMATION = IDENTITY * TRANSLATION

Slide 76

Slide 76 text

TRANSFORMATION = IDENTITY * TRANSLATION * TRANSLATION

Slide 77

Slide 77 text

TRANSFORMATION = IDENTITY * TRANSLATION * TRANSLATION * ROTATION

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

HTTPS://WWW.YOUTUBE.COM/WATCH?V=LJYDCYP0WG4

Slide 80

Slide 80 text

data class Mat4(…) { // ... operator fun times(m: Mat4): Mat4 { val t = transpose(this) return Mat4( Float4(dot(t.x, m.x), dot(t.y, m.x), dot(t.z, m.x), dot(t.w, m.x)), Float4(dot(t.x, m.y), dot(t.y, m.y), dot(t.z, m.y), dot(t.w, m.y)), Float4(dot(t.x, m.z), dot(t.y, m.z), dot(t.z, m.z), dot(t.w, m.z)), Float4(dot(t.x, m.w), dot(t.y, m.w), dot(t.z, m.w), dot(t.w, m.w)) ) } }

Slide 81

Slide 81 text

val result: Mat4 = Mat4(…) * Mat4(…)

Slide 82

Slide 82 text

MATRIX [ 1 | 0 | 0 | 0 ]  [ 0 | 1 | 0 | 0 ]  [ 0 | 0 | 1 | 0 ]  [ 0 | 0 | 0 | 1 ] How to access it?

Slide 83

Slide 83 text

val mat: Mat4 = … val x = mat[0][4] val y = mat[1][4] val z = mat[2][4]    

Slide 84

Slide 84 text

data class Mat4(…) { // ... inline val scale: Float3 get() = … inline val translation: Float3 get() = … val rotation: Float3 get() { … } }    

Slide 85

Slide 85 text

data class Float3(var x: Float = 0.0f, var y: Float = 0.0f, var z: Float = 0.0f) { inline var r: Float get() = x set(value) { x = value } inline var g: Float get() = y set(value) { y = value } inline var b: Float get() = z set(value) { z = value } }

Slide 86

Slide 86 text

data class Mat4(…) { // ... inline val scale: Float3 get() = … inline val translation: Float3 get() = … val rotation: Float3 get() { … } }

Slide 87

Slide 87 text

data class Float3( var x: Float = 0.0f, var y: Float = 0.0f, var z: Float = 0.0f) val translation: Float3 = mat4.translation val (x, y, z) = translation

Slide 88

Slide 88 text

CONVERTING INTO MULTIPLATFORM

Slide 89

Slide 89 text

plugins { kotlin("multiplatform") version "1.3.60" } kotlin { js { } jvm { } sourceSets { val commonMain by getting { dependencies { implementation(kotlin("stdlib-common")) } } js().compilations["main"].defaultSourceSet { dependencies { implementation(kotlin("stdlib-js")) } } jvm().compilations["main"].defaultSourceSet { dependencies { implementation(kotlin("stdlib-jdk8")) } } } }

Slide 90

Slide 90 text

inline fun pow(x: Float, y: Float) = StrictMath.pow( x.toDouble(), y.toDouble() ).toFloat() inline fun pow(x: Float, y: Float) = x.pow(y) java.lang.StrictMath

Slide 91

Slide 91 text

KOTLIN-MATH.JAR (JVM) KOTLIN-MATH.JS (JS) KOTLIN-MATH.KLIB (NATIVE)

Slide 92

Slide 92 text

BUILDING FOR MULTIPLATFORM

Slide 93

Slide 93 text

interface GL { fun uniform1i(uniform: Uniform, data: Int) fun uniform2f(uniform: Uniform, first: Float, second: Float) fun uniform3f(uniform: Uniform, first: Float, second: Float, third: Float) // … } class WebGL(private val gl: WebGLRenderingContext) : GL { override fun uniform1i(uniform: Uniform, data: Int) { gl.uniform1i(uniform.uniformLocation, data) } override fun uniform2f(uniform: Uniform, first: Float, second: Float) { gl.uniform2f(uniform.uniformLocation, first, second) } override fun uniform3f(uniform: Uniform, first: Float, second: Float, third: Float) { gl.uniform3f(uniform.uniformLocation, first, second, third) } } COMMON

Slide 94

Slide 94 text

Slide 95

Slide 95 text

expect class GLContext { internal fun createContext(): GL } actual class GLContext { internal actual fun createContext(): GL { return LwjglGL(…) } } COMMON

Slide 96

Slide 96 text

expect class GLContext { internal fun createContext(): GL } actual class GLContext { internal actual fun createContext(): GL { return LwjglGL(…) } } COMMON KOTLIN/JVM

Slide 97

Slide 97 text

IN OTHER LANGUAGES, THIS CAN OFTEN BE ACCOMPLISHED BY BUILDING A SET OF INTERFACES IN THE COMMON CODE AND IMPLEMENTING THESE INTERFACES IN PLATFORM-SPECIFIC MODULES. HOWEVER, THIS APPROACH IS NOT IDEAL IN CASES WHEN YOU HAVE A LIBRARY ON ONE OF THE PLATFORMS THAT IMPLEMENTS THE FUNCTIONALITY YOU NEED, AND YOU'D LIKE TO USE THE API OF THIS LIBRARY DIRECTLY WITHOUT EXTRA WRAPPERS.

Slide 98

Slide 98 text

Slide 99

Slide 99 text

// Common expect class Uniform // JVM actual class Uniform(val address: Int) // JVM override fun uniform2f(uniform: Uniform, first: Float, second: Float) { glUniform2f(uniform.address, first, second) } COMMON KOTLIN/JVM 1. Producing 2. Consuming

Slide 100

Slide 100 text

Slide 101

Slide 101 text

DRAWING FIRST TRIANGLE

Slide 102

Slide 102 text

Slide 103

Slide 103 text

Slide 104

Slide 104 text

val colors = gl.createBuffer() gl.bindBuffer(GL.ARRAY_BUFFER, colors) gl.bufferData(GL.ARRAY_BUFFER, mesh.vertices.convertColors(), GL.STATIC_DRAW) gl.bindBuffer(GL.ARRAY_BUFFER, colors) gl.vertexAttribPointer( index = shader.getAttrib("aVertexColor"), size = 4, type = GL.FLOAT, normalized = false, stride = 0, offset = 0 ) gl.enableVertexAttribArray(shader.getAttrib("aVertexColor"))

Slide 105

Slide 105 text

Slide 106

Slide 106 text

val verticesOrder = gl.createBuffer() gl.bindBuffer(GL.ELEMENT_ARRAY_BUFFER, verticesOrder) gl.bufferData(GL.ELEMENT_ARRAY_BUFFER, mesh.verticesOrder.convertOrder(), GL.STATIC_DRAW) gl.drawElements(mesh.drawType.glType, mesh.verticesOrder.size, GL.UNSIGNED_SHORT, 0)

Slide 107

Slide 107 text

A B C D E

Slide 108

Slide 108 text

A B C D E

Slide 109

Slide 109 text

val vertices = gl.createBuffer() gl.bindBuffer(GL.ARRAY_BUFFER, vertices) gl.bufferData(GL.ARRAY_BUFFER, mesh.vertices.convertPositions(), GL.STATIC_DRAW) gl.vertexAttribPointer( index = shader.getAttrib("aVertexPosition"), size = 3, type = GL.FLOAT, normalized = false, stride = 0, offset = 0 ) gl.enableVertexAttribArray(shader.getAttrib("aVertexPosition")) val colors = gl.createBuffer() gl.bindBuffer(GL.ARRAY_BUFFER, colors) gl.bufferData(GL.ARRAY_BUFFER, mesh.vertices.convertColors(), GL.STATIC_DRAW) gl.bindBuffer(GL.ARRAY_BUFFER, colors) gl.vertexAttribPointer( index = shader.getAttrib("aVertexColor"), size = 4, type = GL.FLOAT, normalized = false, stride = 0, offset = 0 ) gl.enableVertexAttribArray(shader.getAttrib("aVertexColor")) val verticesOrder = gl.createBuffer() gl.bindBuffer(GL.ELEMENT_ARRAY_BUFFER, verticesOrder) gl.bufferData(GL.ELEMENT_ARRAY_BUFFER, mesh.verticesOrder.convertOrder(), GL.STATIC_DRAW) gl.drawElements(mesh.drawType.glType, mesh.verticesOrder.size, GL.UNSIGNED_SHORT, 0) Vertices Colors Order

Slide 110

Slide 110 text

attribute vec3 aVertexPosition; void main() { gl_Position = aVertexPosition; }

Slide 111

Slide 111 text

attribute vec4 aVertexColor; void main() { gl_FragColor = aVertexColor; }

Slide 112

Slide 112 text

Color interpolation

Slide 113

Slide 113 text

Perspective Camera

Slide 114

Slide 114 text

Perspective Camera  (wide angle)

Slide 115

Slide 115 text

Orthographic

Slide 116

Slide 116 text

Perspective Camera  Frustum

Slide 117

Slide 117 text

Perspective Camera  Frustum

Slide 118

Slide 118 text

https://webglfundamentals.org/webgl/lessons/webgl-3d-camera.html

Slide 119

Slide 119 text

https://webglfundamentals.org/webgl/lessons/webgl-3d-camera.html

Slide 120

Slide 120 text

No content

Slide 121

Slide 121 text

0.1253255324554 145343224242.12

Slide 122

Slide 122 text

No content

Slide 123

Slide 123 text

attribute vec3 aVertexPosition; void main() { gl_Position = aVertexPosition; }

Slide 124

Slide 124 text

attribute vec4 aVertexPosition; uniform mat4 uModelViewMatrix; uniform mat4 uProjectionMatrix; void main() { gl_Position = uProjectionMatrix * uModelViewMatrix * aVertexPosition; }

Slide 125

Slide 125 text

attribute vec4 aVertexPosition; uniform mat4 uModelViewMatrix; uniform mat4 uProjectionMatrix; void main() { gl_Position = uProjectionMatrix * uModelViewMatrix * aVertexPosition; } Change for every vertex Common for all vertices

Slide 126

Slide 126 text

Try me

Slide 127

Slide 127 text

PLATFORM DIFFERENCES

Slide 128

Slide 128 text

interface Game { fun render(delta: Seconds) }

Slide 129

Slide 129 text

override fun render(delta: Seconds) { // --- act --- model.rotateZ(delta * 20) }

Slide 130

Slide 130 text

MOBILE APP  VERSUS   DESKTOP APP

Slide 131

Slide 131 text

TOUCH  VERSUS   CLICK

Slide 132

Slide 132 text

LEFT CONTROL  VERSUS   CONTROL

Slide 133

Slide 133 text

WEB INPUTS DESKTOP INPUTS MOBILE INPUTS TARGET API

Slide 134

Slide 134 text

IMPLEMENTATION DETAILS

Slide 135

Slide 135 text

WEB JVM ANDROID

Slide 136

Slide 136 text

FIXING PLATFORMS IMPLEMENTATION DETAILS IS HARD AND CAN GIVE YOU HEADACHE

Slide 137

Slide 137 text

BUT ONE PLATFORM CAN HELP YOU TO WORK ON ANOTHER ONE. AND THAT'S COOL

Slide 138

Slide 138 text

GRADLEW -T GRADLE BROWSER WEBPACK Reload on change Reload on change

Slide 139

Slide 139 text

SPECTORJS

Slide 140

Slide 140 text

SKELETON BASED ANIMATIONS

Slide 141

Slide 141 text

No content

Slide 142

Slide 142 text

No content

Slide 143

Slide 143 text

No content

Slide 144

Slide 144 text

No content

Slide 145

Slide 145 text

class DemoSuzanne : Game { private val model: Drawable = fileHandler.get("suzanne.protobuf") fun render(delta: Seconds) { model.draw() } } Usage Synchronous loading

Slide 146

Slide 146 text

class DemoSuzanne : Game { private val model: Drawable by fileHandler.get("suzanne.protobuf") fun render(delta: Seconds) { model.draw() } }

Slide 147

Slide 147 text

open class Content(val filename: String) { private var isLoaded: Boolean = false private var content: R? = null operator fun getValue(thisRef: Any?, property: KProperty<*>): R { if (isLoaded) { return content!! } else { throw EarlyAccessException(filename, property.name) } } }

Slide 148

Slide 148 text

private fun asyncContent(filename: String, enc: (ByteArray) -> T): Content { val url = computeUrl(filename) val jsonFile = XMLHttpRequest() jsonFile.responseType = XMLHttpRequestResponseType.Companion.ARRAYBUFFER jsonFile.open("GET", url, true) val content = Content(filename) jsonFile.onload = { _ -> if (jsonFile.readyState == 4.toShort() && jsonFile.status == 200.toShort()) { val element = enc((jsonFile.response as ArrayBuffer).toByteArray()) content.load(element) } } jsonFile.send() return content }

Slide 149

Slide 149 text

class DemoSuzanne : Game { private val model: Drawable by fileHandler.get("suzanne.protobuf") fun render(delta: Seconds) { model.draw() } } Content of the wrapper  Not the wrapper itself

Slide 150

Slide 150 text

actual fun run(gameFactory: () -> Game) { this.game = gameFactory() window.requestAnimationFrame(::loading) } private fun loading(now: Double) { if (!fileHandler.isFullyLoaded()) { window.requestAnimationFrame(::loading) } else { game.create() game.resume() window.requestAnimationFrame(::render) } } Get all resources to load Start the game Wait for resources to be loaded

Slide 151

Slide 151 text

{ "asset": { "generator": "Khronos glTF Blender I/O v1.1.46", "version": "2.0" }, "nodes": [ { "name": "Light", "rotation": [ 0.16907575726509094, 0.7558803558349609, -0.27217137813568115, 0.570947527885437 ] } ], "buffers": [ { "byteLength": 26400, "uri": "data:application/octet-stream;base64,…" } ] } GLTF

Slide 152

Slide 152 text

JVM JS ANDROID GLTF Export PROTOBUF Conversion kotlinx.serialization

Slide 153

Slide 153 text

@Serializable class Mesh( @ProtoId(1) val vertices: List = emptyList(), @ProtoId(2) val verticesOrder: IntArray = intArrayOf() ) @Serializable class Vertex( @ProtoId(1) val position: Position, @ProtoId(2) val color: Color, @ProtoId(3) val normal: Normal, @ProtoId(4) val influence: Influence )

Slide 154

Slide 154 text

fun readProtobuf(data: ByteArray): Model { val deserializer = ProtoBuf(context = serialModule()) return deserializer.load(serializer(), data) } fun writeProtobuf(model: Model): ByteArray { val serializer = ProtoBuf(context = serialModule()) return serializer.dump(Model.serializer(), model) }

Slide 155

Slide 155 text

Joint Relative to

Slide 156

Slide 156 text

No content

Slide 157

Slide 157 text

No content

Slide 158

Slide 158 text

Reference position Target position Move transformation

Slide 159

Slide 159 text

HOW TO GET   THE MOVE TRANSFORMATION ?

Slide 160

Slide 160 text

Origin A B OA + AB = OB

Slide 161

Slide 161 text

Origin A B AB = OB - OA Inverse

Slide 162

Slide 162 text

val animationTransformation = B.globalTransformation * inverse(A.globalTransformation)

Slide 163

Slide 163 text

uniform mat4 uJointTransformationMatrix[MAX_JOINTS]; attribute vec3 aVertexPosition; attribute int aJointId; void main() { mat4 uJointMatrix = uJointTransformationMatrix[aJointId]; gl_Position = uJointMatrix * vec4(aVertexPosition, 1.0); } All animation moves transformation Get transformation for this vertex Apply transformation