Let's Build a Cross-platform 3D Engine with Kotlin/Multiplatform

When creating a game, you want your game to be available on all platforms to target as many players as possible. Developing a game engine is a challenge: you have to deal with mathematical concepts, animations, player interactions... Creating an engine that runs on different platforms is even more challenging. Is Kotlin and Kotlin/Multiplatform could be the right tool to build this kind of engine? Let's check this out together.

Message for you, so you can already be aware what you can expect from this session:
Using existing Java game engines, you can create a game that runs on the JVM, in a browser or on Android and iOS. But it requires some complex tooling, when tools are available for the platform you're targeting.

This session is about the creation of a game engine that runs the same way in a browser, in a JVM, on iOS or Android.

(Conference For Kotliners - 05/06/2020 - Online)

Transcript

1. LET'S BUILD A CROSS PLATFORM 3D ENGINE WITH KOTLIN/MULTIPLATFORM
2. David Wursteisen
3. "#$%&'($#&
4. John Carmack
8. Game 
 development Kotlin Multiplatform
9. You, at the end of 
 this conference Game 
 development Kotlin Multiplatform
10. GAME DEVELOPMENT
16. public void render(float delta) {
 movePlayer(delta);
 moveEnemies(delta);
 moveWorld(delta);
 
 renderWorld();
 renderEnemies();
 renderPlayer();
 }
17. public void render(float delta) {
 movePlayer(delta);
 moveEnemies(delta);
 moveWorld(delta);
 
 renderWorld();
 renderEnemies();
 renderPlayer();
 }
18. public void render(float delta) {
 movePlayer(delta);
 moveEnemies(delta);
 moveWorld(delta);
 
 renderWorld();
 renderEnemies();
 renderPlayer();
 }
19. public void render(float delta) {
 movePlayer(delta);
 moveEnemies(delta);
 moveWorld(delta);
 
 renderWorld();
 renderEnemies();
 renderPlayer();
 } 60x
 seconds
20. (0,0) x y
21. PICKING A GAME ENGINE
23. ❤
25. Hair Dash - http://cleancutgames.com/
29. Core iOS GWT Android Desktop
31. Hi all. My studio, Sickhead, did the port of Slay the Spire to Switch. We struggled for a bit trying to convert the Java to C++ code directly using existing third party tools. The resulting C++ code pretty much didn't work at all and took tons of manual changes to make work. After several months of failure taking that path we changed gears. We instead converted Java to C# which required much less manual fixes to make work and solved all the GC memory management issues. This means we had a C# version of LibGDX in there as well as all the game code. We then just added a new LibGDC backend with new OpenGL bindings and new sound, input implementations. At that point we had a PC port of Spire using C#. We then used our own mature C# IL to C++ cross compiler and runtime we developed to port XNA and MonoGame titles to generate binaries which run on Switch, PS4, XB1, and other platforms. While Java -> C# -> C++ was a crazy idea when we started... it actually works beautifully and makes perfect sense now. HTTPS://WWW.BADLOGICGAMES.COM/FORUM/VIEWTOPIC.PHP?F=11&T=29625
32. Hi all. My studio, Sickhead, did the port of Slay the Spire to Switch. We struggled for a bit trying to convert the Java to C++ code directly using existing third party tools. The resulting C++ code pretty much didn't work at all and took tons of manual changes to make work. After several months of failure taking that path we changed gears. We instead converted Java to C# which required much less manual fixes to make work and solved all the GC memory management issues. This means we had a C# version of LibGDX in there as well as all the game code. We then just added a new LibGDC backend with new OpenGL bindings and new sound, input implementations. At that point we had a PC port of Spire using C#. We then used our own mature C# IL to C++ cross compiler and runtime we developed to port XNA and MonoGame titles to generate binaries which run on Switch, PS4, XB1, and other platforms. While Java -> C# -> C++ was a crazy idea when we started... it actually works beautifully and makes perfect sense now. HTTPS://WWW.BADLOGICGAMES.COM/FORUM/VIEWTOPIC.PHP?F=11&T=29625 MY STUDIO, SICKHEAD, 
 DID THE PORT OF SLAY THE SPIRE TO SWITCH.
 WHILE JAVA -> C# -> C++ WAS A CRAZY IDEA WHEN WE STARTED... IT ACTUALLY WORKS BEAUTIFULLY AND MAKES PERFECT SENSE NOW.
33. KOTLIN MULTIPLATFORM FEEDBACK LOOP
34. LET'S DIVE IN BUILDING A 3D ENGINE
35. void render(float delta) { }
36. fun render(delta: float) { }
37. KOTLIN/MULTIPLATFOM KOTLIN/JVM KOTLIN/NATIVE KOTLIN/JS KOTLIN/ANDROID
38. Common Js Native Android Jvm
39. KOTLIN/JVM KOTLIN/JS KOTLIN/ANDROID KOTLIN/NATIVE COMMON LIBGDX LIBGDX PLAYCANVAS RAYLIB
40. JS NATIVE JVM
41. KOTLIN/NATIVE KOTLIN/JS *.H CINTEROP *.KT *.D.TS DUKAT *.KT
42. external open class ScrollbarComponentSystem(app: pc.Application) : ComponentSystem { override fun on(name: String, callback: HandleEvent, scope: Any): EventHandler override fun off(name: String, callback: HandleEvent, scope: Any): EventHandler override fun fire(name: Any, arg1: Any, arg2: Any, arg3: Any, arg4: Any): EventHandler override fun once(name: String, callback: HandleEvent, scope: Any): EventHandler override fun hasEvent(name: String): Boolean }
43. external interface `T$8` { var volume: Number? get() = definedExternally set(value) = definedExternally var pitch: Number? get() = definedExternally set(value) = definedExternally var loop: Boolean? get() = definedExternally set(value) = definedExternally )
46. OpenGL
50. OpenGL ES was deprecated in iOS 12.
52. VERTEX VERTEX VERTEX
53. HTTPS://WWW.LABRI.FR/PERSO/NROUGIER/PYTHON-OPENGL/#MODERN-OPENGL VERTICES VERTEX
 SHADER FRAGMENT
 SHADER OpenGL (GPU) Kotlin (CPU)
54. void main() {
 vec4 sum = vec4(0);
 
 float step = iResolution.y;
 // y
 for(float i = -area ; i <= area ; i += 1.) {
 // x
 for(float j = -area ; j <= area ; j += 1.) {
 float x = v_texCoords.x + i / iResolution.x;
 float y = v_texCoords.y + j / iResolution.y;
 sum += texture2D(u_texture, vec2(x, y)) * 0.005;
 }
 }
 gl_FragColor = texture2D(u_texture, v_texCoords) + sum;
 } OpenGL Shading Language (GLSL)
59. BUILDING A 3D ENGINE BASED ON OpenGL
60. KOTLIN/JVM KOTLIN/JS KOTLIN/ANDROID KOTLIN/NATIVE COMMON LWJGL OpenGL ES WebGL OpenGL
61. WHAT YOU EXPECT
62. WHAT YOU'LL GET
64. A SHORT HIKE
65. class GameExample : Game { override fun render(delta: Float) { } } Unit ?
66. typealias Seconds = Float class GameExample : Game { override fun render(delta: Seconds) { } }
67. interface CanMove { fun rotateX(angle: Float): CanMove } Radian or Degree ?
68. typealias Degree = Float interface CanMove { fun rotateX(angle: Degree): CanMove }
69. Y X Z
70. 1. Translation 2. Rotation 3. Scale
71. MATRIX
72. MATRIX [ 1 | 0 | 0 | 0 ]
 [ 0 | 1 | 0 | 0 ]
 [ 0 | 0 | 1 | 0 ]
 [ 0 | 0 | 0 | 1 ]
73. Scale MATRIX [ 1 | 0 | 0 | 0 ]
 [ 0 | 1 | 0 | 0 ]
 [ 0 | 0 | 1 | 0 ]
 [ 0 | 0 | 0 | 1 ] X Z Y Translation Rotation
74. TRANSFORMATION = IDENTITY
75. TRANSFORMATION = IDENTITY * TRANSLATION
76. TRANSFORMATION = IDENTITY * TRANSLATION * TRANSLATION
77. TRANSFORMATION = IDENTITY * TRANSLATION * TRANSLATION * ROTATION
79. HTTPS://WWW.YOUTUBE.COM/WATCH?V=LJYDCYP0WG4
80. data class Mat4(…) { // ... operator fun times(m: Mat4): Mat4 { val t = transpose(this) return Mat4( Float4(dot(t.x, m.x), dot(t.y, m.x), dot(t.z, m.x), dot(t.w, m.x)), Float4(dot(t.x, m.y), dot(t.y, m.y), dot(t.z, m.y), dot(t.w, m.y)), Float4(dot(t.x, m.z), dot(t.y, m.z), dot(t.z, m.z), dot(t.w, m.z)), Float4(dot(t.x, m.w), dot(t.y, m.w), dot(t.z, m.w), dot(t.w, m.w)) ) } }
81. val result: Mat4 = Mat4(…) * Mat4(…)
82. MATRIX [ 1 | 0 | 0 | 0 ]
 [ 0 | 1 | 0 | 0 ]
 [ 0 | 0 | 1 | 0 ]
 [ 0 | 0 | 0 | 1 ] How to access it?
83. val mat: Mat4 = … val x = mat[0][4] val y = mat[1][4] val z = mat[2][4] 
 

84. data class Mat4(…) { // ... inline val scale: Float3 get() = … inline val translation: Float3 get() = … val rotation: Float3 get() { … } } 
 

85. data class Float3(var x: Float = 0.0f, var y: Float = 0.0f, var z: Float = 0.0f) { inline var r: Float get() = x set(value) { x = value } inline var g: Float get() = y set(value) { y = value } inline var b: Float get() = z set(value) { z = value } }
86. data class Mat4(…) { // ... inline val scale: Float3 get() = … inline val translation: Float3 get() = … val rotation: Float3 get() { … } }
87. data class Float3( var x: Float = 0.0f, var y: Float = 0.0f, var z: Float = 0.0f) val translation: Float3 = mat4.translation val (x, y, z) = translation
88. CONVERTING INTO MULTIPLATFORM
89. plugins { kotlin("multiplatform") version "1.3.60" } kotlin { js { } jvm { } sourceSets { val commonMain by getting { dependencies { implementation(kotlin("stdlib-common")) } } js().compilations["main"].defaultSourceSet { dependencies { implementation(kotlin("stdlib-js")) } } jvm().compilations["main"].defaultSourceSet { dependencies { implementation(kotlin("stdlib-jdk8")) } } } }
90. inline fun pow(x: Float, y: Float) = StrictMath.pow( x.toDouble(), y.toDouble() ).toFloat() inline fun pow(x: Float, y: Float) = x.pow(y) java.lang.StrictMath
91. KOTLIN-MATH.JAR (JVM) KOTLIN-MATH.JS (JS) KOTLIN-MATH.KLIB (NATIVE)
92. BUILDING FOR MULTIPLATFORM
93. interface GL { fun uniform1i(uniform: Uniform, data: Int) fun uniform2f(uniform: Uniform, first: Float, second: Float) fun uniform3f(uniform: Uniform, first: Float, second: Float, third: Float) // … } class WebGL(private val gl: WebGLRenderingContext) : GL { override fun uniform1i(uniform: Uniform, data: Int) { gl.uniform1i(uniform.uniformLocation, data) } override fun uniform2f(uniform: Uniform, first: Float, second: Float) { gl.uniform2f(uniform.uniformLocation, first, second) } override fun uniform3f(uniform: Uniform, first: Float, second: Float, third: Float) { gl.uniform3f(uniform.uniformLocation, first, second, third) } } COMMON
94. interface GL { fun uniform1i(uniform: Uniform, data: Int) fun uniform2f(uniform: Uniform, first: Float, second: Float) fun uniform3f(uniform: Uniform, first: Float, second: Float, third: Float) // … } class WebGL(private val gl: WebGLRenderingContext) : GL { override fun uniform1i(uniform: Uniform, data: Int) { gl.uniform1i(uniform.uniformLocation, data) } override fun uniform2f(uniform: Uniform, first: Float, second: Float) { gl.uniform2f(uniform.uniformLocation, first, second) } override fun uniform3f(uniform: Uniform, first: Float, second: Float, third: Float) { gl.uniform3f(uniform.uniformLocation, first, second, third) } } COMMON KOTLIN/JS
95. expect class GLContext { internal fun createContext(): GL } actual class GLContext { internal actual fun createContext(): GL { return LwjglGL(…) } } COMMON
96. expect class GLContext { internal fun createContext(): GL } actual class GLContext { internal actual fun createContext(): GL { return LwjglGL(…) } } COMMON KOTLIN/JVM
97. IN OTHER LANGUAGES, THIS CAN OFTEN BE ACCOMPLISHED BY BUILDING A SET OF INTERFACES IN THE COMMON CODE AND IMPLEMENTING THESE INTERFACES IN PLATFORM-SPECIFIC MODULES. HOWEVER, THIS APPROACH IS NOT IDEAL IN CASES WHEN YOU HAVE A LIBRARY ON ONE OF THE PLATFORMS THAT IMPLEMENTS THE FUNCTIONALITY YOU NEED, AND YOU'D LIKE TO USE THE API OF THIS LIBRARY DIRECTLY WITHOUT EXTRA WRAPPERS.
98. IN OTHER LANGUAGES, THIS CAN OFTEN BE ACCOMPLISHED BY BUILDING A SET OF INTERFACES IN THE COMMON CODE AND IMPLEMENTING THESE INTERFACES IN PLATFORM-SPECIFIC MODULES. HOWEVER, THIS APPROACH IS NOT IDEAL IN CASES WHEN YOU HAVE A LIBRARY ON ONE OF THE PLATFORMS THAT IMPLEMENTS THE FUNCTIONALITY YOU NEED, AND YOU'D LIKE TO USE THE API OF THIS LIBRARY DIRECTLY WITHOUT EXTRA WRAPPERS.
99. // Common expect class Uniform // JVM actual class Uniform(val address: Int) // JVM override fun uniform2f(uniform: Uniform, first: Float, second: Float) { glUniform2f(uniform.address, first, second) } COMMON KOTLIN/JVM 1. Producing 2. Consuming
100. // Common expect class Uniform // JVM actual class Uniform(val address: Int) // JVM override fun uniform2f(uniform: Uniform, first: Float, second: Float) { glUniform2f(uniform.address, first, second) } COMMON KOTLIN/JVM 1. Producing 2. Consuming
101. DRAWING FIRST TRIANGLE
102. val vertices = gl.createBuffer() gl.bindBuffer(GL.ARRAY_BUFFER, vertices) gl.bufferData(GL.ARRAY_BUFFER, mesh.vertices.convertPositions(), GL.STATIC_DRAW) gl.vertexAttribPointer( index = shader.getAttrib("aVertexPosition"), size = 3, type = GL.FLOAT, normalized = false, stride = 0, offset = 0 ) gl.enableVertexAttribArray(shader.getAttrib("aVertexPosition"))
103. val vertices = gl.createBuffer() gl.bindBuffer(GL.ARRAY_BUFFER, vertices) gl.bufferData(GL.ARRAY_BUFFER, mesh.vertices.convertPositions(), GL.STATIC_DRAW) gl.vertexAttribPointer( index = shader.getAttrib("aVertexPosition"), size = 3, type = GL.FLOAT, normalized = false, stride = 0, offset = 0 ) gl.enableVertexAttribArray(shader.getAttrib("aVertexPosition")) X Y Z
104. val colors = gl.createBuffer() gl.bindBuffer(GL.ARRAY_BUFFER, colors) gl.bufferData(GL.ARRAY_BUFFER, mesh.vertices.convertColors(), GL.STATIC_DRAW) gl.bindBuffer(GL.ARRAY_BUFFER, colors) gl.vertexAttribPointer( index = shader.getAttrib("aVertexColor"), size = 4, type = GL.FLOAT, normalized = false, stride = 0, offset = 0 ) gl.enableVertexAttribArray(shader.getAttrib("aVertexColor"))
105. val colors = gl.createBuffer() gl.bindBuffer(GL.ARRAY_BUFFER, colors) gl.bufferData(GL.ARRAY_BUFFER, mesh.vertices.convertColors(), GL.STATIC_DRAW) gl.bindBuffer(GL.ARRAY_BUFFER, colors) gl.vertexAttribPointer( index = shader.getAttrib("aVertexColor"), size = 4, type = GL.FLOAT, normalized = false, stride = 0, offset = 0 ) gl.enableVertexAttribArray(shader.getAttrib("aVertexColor")) Red Green Blue Alpha
106. val verticesOrder = gl.createBuffer() gl.bindBuffer(GL.ELEMENT_ARRAY_BUFFER, verticesOrder) gl.bufferData(GL.ELEMENT_ARRAY_BUFFER, mesh.verticesOrder.convertOrder(), GL.STATIC_DRAW) gl.drawElements(mesh.drawType.glType, mesh.verticesOrder.size, GL.UNSIGNED_SHORT, 0)
107. A B C D E
108. A B C D E
109. val vertices = gl.createBuffer() gl.bindBuffer(GL.ARRAY_BUFFER, vertices) gl.bufferData(GL.ARRAY_BUFFER, mesh.vertices.convertPositions(), GL.STATIC_DRAW) gl.vertexAttribPointer( index = shader.getAttrib("aVertexPosition"), size = 3, type = GL.FLOAT, normalized = false, stride = 0, offset = 0 ) gl.enableVertexAttribArray(shader.getAttrib("aVertexPosition")) val colors = gl.createBuffer() gl.bindBuffer(GL.ARRAY_BUFFER, colors) gl.bufferData(GL.ARRAY_BUFFER, mesh.vertices.convertColors(), GL.STATIC_DRAW) gl.bindBuffer(GL.ARRAY_BUFFER, colors) gl.vertexAttribPointer( index = shader.getAttrib("aVertexColor"), size = 4, type = GL.FLOAT, normalized = false, stride = 0, offset = 0 ) gl.enableVertexAttribArray(shader.getAttrib("aVertexColor")) val verticesOrder = gl.createBuffer() gl.bindBuffer(GL.ELEMENT_ARRAY_BUFFER, verticesOrder) gl.bufferData(GL.ELEMENT_ARRAY_BUFFER, mesh.verticesOrder.convertOrder(), GL.STATIC_DRAW) gl.drawElements(mesh.drawType.glType, mesh.verticesOrder.size, GL.UNSIGNED_SHORT, 0) Vertices Colors Order
110. attribute vec3 aVertexPosition; void main() { gl_Position = aVertexPosition; }
111. attribute vec4 aVertexColor; void main() { gl_FragColor = aVertexColor; }
112. Color interpolation
113. Perspective Camera
114. Perspective Camera
 (wide angle)
115. Orthographic
116. Perspective Camera
 Frustum
117. Perspective Camera
 Frustum
118. https://webglfundamentals.org/webgl/lessons/webgl-3d-camera.html
119. https://webglfundamentals.org/webgl/lessons/webgl-3d-camera.html
121. 0.1253255324554 145343224242.12
123. attribute vec3 aVertexPosition; void main() { gl_Position = aVertexPosition; }
124. attribute vec4 aVertexPosition; uniform mat4 uModelViewMatrix; uniform mat4 uProjectionMatrix; void main() { gl_Position = uProjectionMatrix * uModelViewMatrix * aVertexPosition; }
125. attribute vec4 aVertexPosition; uniform mat4 uModelViewMatrix; uniform mat4 uProjectionMatrix; void main() { gl_Position = uProjectionMatrix * uModelViewMatrix * aVertexPosition; } Change for every vertex Common for all vertices
126. Try me
127. PLATFORM DIFFERENCES
128. interface Game { fun render(delta: Seconds) }
129. override fun render(delta: Seconds) { // --- act --- model.rotateZ(delta * 20) }
130. MOBILE APP
 VERSUS 
 DESKTOP APP
131. TOUCH
 VERSUS 
 CLICK
132. LEFT CONTROL
 VERSUS 
 CONTROL
133. WEB INPUTS DESKTOP INPUTS MOBILE INPUTS TARGET API
134. IMPLEMENTATION DETAILS
135. WEB JVM ANDROID
136. FIXING PLATFORMS IMPLEMENTATION DETAILS IS HARD AND CAN GIVE YOU HEADACHE
137. BUT ONE PLATFORM CAN HELP YOU TO WORK ON ANOTHER ONE. AND THAT'S COOL
138. GRADLEW <TARGET> -T GRADLE BROWSER WEBPACK Reload on change Reload on change
139. SPECTORJS
140. SKELETON BASED ANIMATIONS
145. class DemoSuzanne : Game { private val model: Drawable = fileHandler.get("suzanne.protobuf") fun render(delta: Seconds) { model.draw() } } Usage Synchronous loading
146. class DemoSuzanne : Game { private val model: Drawable by fileHandler.get("suzanne.protobuf") fun render(delta: Seconds) { model.draw() } }
147. open class Content<R>(val filename: String) { private var isLoaded: Boolean = false private var content: R? = null operator fun getValue(thisRef: Any?, property: KProperty<*>): R { if (isLoaded) { return content!! } else { throw EarlyAccessException(filename, property.name) } } }
148. private fun <T> asyncContent(filename: String, enc: (ByteArray) -> T): Content<T> { val url = computeUrl(filename) val jsonFile = XMLHttpRequest() jsonFile.responseType = XMLHttpRequestResponseType.Companion.ARRAYBUFFER jsonFile.open("GET", url, true) val content = Content<T>(filename) jsonFile.onload = { _ -> if (jsonFile.readyState == 4.toShort() && jsonFile.status == 200.toShort()) { val element = enc((jsonFile.response as ArrayBuffer).toByteArray()) content.load(element) } } jsonFile.send() return content }
149. class DemoSuzanne : Game { private val model: Drawable by fileHandler.get("suzanne.protobuf") fun render(delta: Seconds) { model.draw() } } Content of the wrapper
 Not the wrapper itself
150. actual fun run(gameFactory: () -> Game) { this.game = gameFactory() window.requestAnimationFrame(::loading) } private fun loading(now: Double) { if (!fileHandler.isFullyLoaded()) { window.requestAnimationFrame(::loading) } else { game.create() game.resume() window.requestAnimationFrame(::render) } } Get all resources to load Start the game Wait for resources to be loaded
151. { "asset": { "generator": "Khronos glTF Blender I/O v1.1.46", "version": "2.0" }, "nodes": [ { "name": "Light", "rotation": [ 0.16907575726509094, 0.7558803558349609, -0.27217137813568115, 0.570947527885437 ] } ], "buffers": [ { "byteLength": 26400, "uri": "data:application/octet-stream;base64,…" } ] } GLTF
152. JVM JS ANDROID GLTF Export PROTOBUF Conversion kotlinx.serialization
153. @Serializable class Mesh( @ProtoId(1) val vertices: List<Vertex> = emptyList(), @ProtoId(2) val verticesOrder: IntArray = intArrayOf() ) @Serializable class Vertex( @ProtoId(1) val position: Position, @ProtoId(2) val color: Color, @ProtoId(3) val normal: Normal, @ProtoId(4) val influence: Influence )
154. fun readProtobuf(data: ByteArray): Model { val deserializer = ProtoBuf(context = serialModule()) return deserializer.load(serializer(), data) } fun writeProtobuf(model: Model): ByteArray { val serializer = ProtoBuf(context = serialModule()) return serializer.dump(Model.serializer(), model) }
155. Joint Relative to
158. Reference position Target position Move transformation
159. HOW TO GET 
 THE MOVE TRANSFORMATION ?
160. Origin A B OA + AB = OB
161. Origin A B AB = OB - OA Inverse
162. val animationTransformation = B.globalTransformation * inverse(A.globalTransformation)
163. uniform mat4 uJointTransformationMatrix[MAX_JOINTS]; attribute vec3 aVertexPosition; attribute int aJointId; void main() { mat4 uJointMatrix = uJointTransformationMatrix[aJointId]; gl_Position = uJointMatrix * vec4(aVertexPosition, 1.0); } All animation moves transformation Get transformation for this vertex Apply transformation
164. Try me
168. KOOL
169. SO KOTLIN MULTIPLATFORM? Try me