The Day I Reverse Engineered a Gameboy Advance Game

Gameboy Advance was one of the most popular video game platforms of its time and, because of it, many people worked together as a community to study and to document its architecture and develop ROM hacking and other interesting tools.

It turns out that this video game is a fantastic way to start studying reverse engineering: its architecture is very well documented and simpler if compared to the current game console generation – and, of course, it is very fun to work in a game-related project.

So... what do you think of learning reverse engineering through this challenge: developing a level editor for a GBA game called "Klonoa: Empire of Dreams"?
We need to understand the architecture behind ARM hardware, apply reverse engineering in order to discover how the logic of the game works, and then use our front-end knowledge (JS + React.js) to build a level editor.

Transcript

1. The day I reverse engineered a Gameboy Advance game With @ythecombinator AND @macabeus
2. WTF?! THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
3. """
5. soooo…
7. This is macabeus
8. This is MATHEUS
9. This is US PLAYING KLONOA
10. AND THIS IS US BORED
11. soooo…
12. WANTED MORE LEVELS TO PLAY WITH
13. WANTED DIFFERENT LEVELS
14. soooo…
15. soooo…
16. soooo…
17. soooo…AND THIS IS KLONNOA
18. soooo…AND THIS IS KLONNOA
22. http://bit.ly/gba-the-conf-project
23. Stretching the bridge THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
24. https://www.nogba.com
25. https://www.hex-rays.com/products/ida
28. This is the tile ID. In this example, all tiles with the ID 9B are exactly the bridge part, which is the left corner. 9A is the ID of the continuous part of the bridge.
29. This tells us where the tile is stored in the memory. In this example, it is at 0600F1AD.
30. https://www.coranac.com/tonc/text/hardware.htm
31. https://www.coranac.com/tonc/text/hardware.htm The address found is expected 0600F1AD given that everything between 0600 and 0601 is part of the section known as VRAM.
32. Our bridge starts here!
33. Our bridge starts here!
34. And now we grow it bigger!
35. And now we grow it bigger!
36. And now we grow it bigger! WTF??!
37. Understanding the DMA THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
40. VRAM pulls the tilemap
 containing exactly what the player has to see from somewhere else that has the entire thing.
41. https://www.coranac.com/tonc/text/regbg.htm
42. https://www.coranac.com/tonc/text/regbg.htm Direct Memory Access
43. https://www.coranac.com/tonc/text/regbg.htm It is a feature of computer systems that allows certain hardware subsystems to access main system memory (random- access memory), independent of the CPU
44. https://www.coranac.com/tonc/text/regbg.htm tldr; a faster copy and paste
45. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0 0600F000 80000200 DMA3: 03004800 07000000 84000048
46. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0 0600F000 80000200 DMA3: 03004800 07000000 84000048 The closest to the bytes of our bridge (0600F1AD).
47. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0 0600F000 80000200 DMA3: 03004800 07000000 84000048 This address is the VRAM data source. It’s located in a region called Fast WRAM.
49. Our bridge starts here!
50. Stretching it…
51. Stretching it… …It works! And…
52. Stretching it… …It works! And… WTF??!
53. Stretching it… …It works! And… WTF??!
56. Our extra tiles disappear when out of the player vision range.
57. LET’S TALK ABOUT PHYSICS THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
58. Object Attribute Memory
59. OAM
61. This is the address in which Klonoa is stored on the memory
62. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0 0600F000 80000200 DMA3: 03004800 07000000 8400003E
63. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0 0600F000 80000200 DMA3: 03004800 07000000 8400003E We can see exactly the byte that writes on Klonoa’s OAM (03004800).
64. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0 0600F000 80000200 DMA3: 03004800 07000000 8400003E We can see exactly the byte that writes on Klonoa’s OAM (03004800).
70. 1 2
82. Here is where Y fixes Klonoa’s position
83. Here is where Y fixes Klonoa’s position Here is where which decides if should fix or not Klonoa’s position
85. https://www.coranac.com/tonc/text/hardware.htm
86. https://www.coranac.com/tonc/text/hardware.htm
90. Let’s find the tilemap on rom THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
91. Cartridge > > > > The data flow
92. Cartridge Slow WRAM > > > > The data flow
93. Cartridge Fast WRAM Slow WRAM > > > > The data flow
94. Cartridge Fast WRAM Slow WRAM VRAM > > > > The data flow
95. Cartridge Fast WRAM Display Slow WRAM VRAM > > > > The data flow
96. Cartridge Fast WRAM Display VRAM Slow WRAM > > > > Our research flow
98. swi 11h what?
99. Swi
100. SoftWare Interrupt
101. SoftWare Interrupt = Bios calls
103. BIOS is a firmware that is intended to initialise the physical components of the system
104. BIOS is a firmware that is intended to initialise the physical components of the system In GBA, its BIOS exposes many functions often used in games, including data compression and decompression
105. BIOS is a firmware that is intended to initialise the physical components of the system In GBA, its BIOS exposes many functions often used in games, including data compression and decompression Each function has an associated numeric code, which must be used as a parameter in the SWI statement
106. Input: How division works on ARM Assembly swi 0x06 Output:
107. Input: How division works on ARM Assembly swi 0x06 R0
 numerator R1
 denominator Output:
108. Input: How division works on ARM Assembly swi 0x06 R0
 numerator R1
 denominator R0
 numerator / denominator R1
 numerator % denominator R3
 abs(numerator / denominator) Output:
110. Huffman + lz77 = deflate
116. 6E 6E 6E B0 B1 BD 77 AB B3 AB B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E
117. 6E 6E 6E B0 B1 BD 77 AB B3 AB B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E
118. 6E 6E 6E B0 B1 BD 77 AB B3 AB B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E 6E 6E 6E B0 B1 BD 77 AB B3 AB B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E
119. 6E 6E 6E B0 B1 BD 77 AB B3 AB B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E 6E 6E 6E B0 B1 BD 77 AB B3 AB B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E
120. 6E 6E 6E B0 B1 BD 77 AB B3 AB B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E 6E 6E 6E B0 B1 BD 77 AB B3 AB B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E =
121. Let’s paint the tilemap THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
122. 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F
123. 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F
124. 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F
125. 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F
126. H · W = 25023
 H,W ∈ ℕ
128. x x x x a x x x x x x x x b x x x x I’m on tile a
129. x x x x a x x x x x x x x b x x x x I’m on tile a
130. x x x x a x x x x x x x x b x x x x I’m on tile a x x x x a x x x x x x x x b x x x x From the tile a to b there are 9 bytes (= 9 tiles), so the length of the tilemap is exactly 9
132. Breakpoint here
133. …is located here This byte…
135. As I change this byte…
136. …it reflects right here
138. Changing these bytes…
139. …we drop Klonoa one level below
141. Changing these bytes… …we found the B tile!
142. 020085DB The length of the level is 420! - 02008437 = 1A4 (420)
143. https://github.com/oliver-moran/jimp
144. const Jimp = require('jimp') const { drop } = require('ramda') const fs = require('fs') const data = drop(3, fs.readFileSync('dump/level-2/tilemap'))
145. const getPixelColor = hexTile "# { if (hexTile === 0) { return 0x00000000 } return 0xFFFFFFFF }
146. new Jimp(300, 600, (err, image) "# { let x = 0 let y = 0 for (let i = 0; i < data.length; i += 1) { const value = data[i] x += 1 if (x === 300) { y += 1 x = 0 } let color = getPixelColor(value) image.setPixelColor(color, x, y) } image.write('image.png') })
153. let’s see the project THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
154. Coding THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
155. soooo…
156. soooo…
157. Stack
158. React.js ✈
159. Webpack ✈
160. Ramda.js ✈
161. FormerKit ✈
162. GH Pages ✈
163. architecture
164. ✂ Brush Scissors
165. klo-gba.js/scissors/src/visions/
166. export default { rom: { tilemap: [0x1B27FC, 0x1B36F3], oam: [0xE2B90, 0xE2F59], portals: [0xD48C8, 0xD48EF], }, tilemap: { totalStages: 3, height: 60, width: 420, scheme: [ { name: 'grass', ids: [0x7D, 0x80, 0x81, 0x82, 0x8E, 0x8F, 0x90], },
167. export default { rom: { tilemap: [0x1B27FC, 0x1B36F3], oam: [0xE2B90, 0xE2F59], portals: [0xD48C8, 0xD48EF], }, tilemap: { totalStages: 3, height: 60, width: 420, scheme: [ { name: 'grass', ids: [0x7D, 0x80, 0x81, 0x82, 0x8E, 0x8F, 0x90], },
168. export default { rom: { tilemap: [0x1B27FC, 0x1B36F3], oam: [0xE2B90, 0xE2F59], portals: [0xD48C8, 0xD48EF], }, tilemap: { totalStages: 3, height: 60, width: 420, scheme: [ { name: 'grass', ids: [0x7D, 0x80, 0x81, 0x82, 0x8E, 0x8F, 0x90], },
169. { name: 'rock', ids: [0x54, 0x53, 0x55, 0x56, 0x57, 0x58, 0x59, }, { name: 'darkRock', ids: [0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, }, { name: 'wood', ids: [0x52, 0x94, 0x98, 0x99, 0x9A, 0x9B, 0x9D, }, { name: 'bridgeRope', ids: [0x04, 0x05, 0x0B, 0x0C, 0x11, 0x13, 0x14,
170. { name: 'spike', ids: [0x3D], }, ], },
171. Provider Pattern
173. ✘ Nah$$.Not necessary at all.
174. ROMProvider VisionProvider
175. import React from 'react' const ROMContext = React.createContext() export default ROMContext import React from 'react' const VisionContext = React.createContext() export default VisionContext klo-gba.js/brush/src/context/ROMContext.js klo-gba.js/brush/src/context/VisionContext.js
176. import React from 'react' const ROMContext = React.createContext() export default ROMContext import React from 'react' const VisionContext = React.createContext() export default VisionContext klo-gba.js/brush/src/context/ROMContext.js klo-gba.js/brush/src/context/VisionContext.js
177. const ROMProvider = (props) "# { const [romBuffer, setROMBuffer] = useROMBuffer() return ( <ROMContext.Provider value={{ romBufferMemory: romBuffer.memory, romBufferStatus: romBuffer.status, setROMBuffer, }} > {props.children} </ROMContext.Provider> ) }
178. const SelectVision = () "# { const { romBufferMemory } = useContext(ROMContext) const { updateVision, vision: { state: visionState }, visionIndex, visionWorld, } = useContext(VisionContext) const setNewVision = (newVision) "# { const [newVisionWorld, newVisionIndex] = newVision.split('-') updateVision(romBufferMemory, newVisionWorld, newVisionIndex) }
179. const SelectVision = () "# { const { romBufferMemory } = useContext(ROMContext) const { updateVision, vision: { state: visionState }, visionIndex, visionWorld, } = useContext(VisionContext) const setNewVision = (newVision) "# { const [newVisionWorld, newVisionIndex] = newVision.split('-') updateVision(romBufferMemory, newVisionWorld, newVisionIndex) }
180. Custom hooks
181. import { useContext, useEffect } from 'react' import VisionContext from '../context/VisionContext' const useWhenVisionChanges = (callback) "# { const { visionIndex, visionWorld } = useContext(VisionContext) useEffect(callback, [visionIndex, visionWorld]) } export default useWhenVisionChanges
182. const LoadedRom = () "# { const { vision } = useContext(VisionContext) const [highlightCoordinates, setHighlightCoordinates] = useState([-1, -1]) const [optShowGrid, setOptShowGrid] = useState(false) const [optShowOAM, setOptShowOAM] = useState(true) const [optShowPortals, setOptShowPortals] = useState(true) const [toolState, setToolState] = useTool() const [resolution, setResolution] = useState(1) useWhenVisionChanges(() "# { setHighlightCoordinates([-1, -1]) }) $( $$.
183. const LoadedRom = () "# { const { vision } = useContext(VisionContext) const [highlightCoordinates, setHighlightCoordinates] = useState([-1, -1]) const [optShowGrid, setOptShowGrid] = useState(false) const [optShowOAM, setOptShowOAM] = useState(true) const [optShowPortals, setOptShowPortals] = useState(true) const [toolState, setToolState] = useTool() const [resolution, setResolution] = useState(1) useWhenVisionChanges(() "# { setHighlightCoordinates([-1, -1]) }) $( $$.
184. import { useState } from 'react' const useTool = () "# { const [{ currentTool, toolsValue }, setToolsStates] = useState({ currentTool: 'magnifyingGlass', toolsValue: { brush: null, magnifyingGlass: null, }, })
185. const updateToolState = (tool, newToolState) "# { if (newToolState ))* undefined) { setToolsStates({ currentTool: tool, toolsValue: { $$.toolsValue, [tool]: newToolState, }, }) return } setToolsStates({ currentTool: tool, toolsValue, }) }
186. return [{ name: currentTool, value: toolsValue[currentTool], }, updateToolState] } export default useTool
187. Web assembly
188. Tilemap compressed Old C code to uncompress it Decompressed Tilemap > >
189. Tilemap compressed Old C code to uncompress it Decompressed Tilemap > > Can’t run C code on browser!
190. Tilemap compressed Old C code to uncompress it Decompressed Tilemap > >
191. Tilemap compressed Old C code to uncompress it Decompressed Tilemap > > >Emscripten >WebAssembly
192. huffman.c lzss.c > > Emscripten huffman.js lzss.js huffman.wasm lzss.wasm +
194. klo-gba.js/scissors/webpack.config.js const rules = [ { loader: 'file-loader', test: /\.wasm$/, type: 'javascript/auto', }, … ]
195. const huffmanWasm = require('./wasm/huffman.wasm') const huffmanModule = require('./wasm/huffman.js')({ locateFile (path) { if (path.endsWith('.wasm')) { return `static/wasm/${huffmanWasm}` } return path }, })
196. class HuffmanDecodeError extends Error { constructor (e) { super(`Error when tried to use huffman decoder: ${e}`) this.name = 'HuffmanDecodeError' } } huffmanModule.onRuntimeInitialized = () "# {} const huffmanDecode = (buffer) "# { FS.writeFile('file', buffer)
197. class HuffmanDecodeError extends Error { constructor (e) { super(`Error when tried to use huffman decoder: ${e}`) this.name = 'HuffmanDecodeError' } } huffmanModule.onRuntimeInitialized = () "# {} const huffmanDecode = (buffer) "# { FS.writeFile('file', buffer)
198. const huffmanDecode = (buffer) "# { FS.writeFile('file', buffer) huffmanModule._HUF_Decode() try { return huffmanModule.FS.readFile('file', { encoding: 'binary' }) } catch (e) { throw new HuffmanDecodeError(e) } }
199. function docker_run_emscripten { local filename="$1" echo "Compiling $filename$$." docker run \ --rm -it \ -v $(pwd)/scissors/src/wasm:/src \ trzeci/emscripten \ emcc -s WASM=1 -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s EXTRA_EXPORTED_RUNTIME_METHODS=[\"FS\"] -s EXPORT_NAME=\"$filename\" -o ./$filename.js $filename.c } docker_run_emscripten huffman docker_run_emscripten lzss
200. function docker_run_emscripten { local filename="$1" echo "Compiling $filename$$." docker run \ --rm -it \ -v $(pwd)/scissors/src/wasm:/src \ trzeci/emscripten \ emcc -s WASM=1 -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s EXTRA_EXPORTED_RUNTIME_METHODS=[\"FS\"] -s EXPORT_NAME=\"$filename\" -o ./$filename.js $filename.c } docker_run_emscripten huffman docker_run_emscripten lzss
202. const extractFullTilemap = (romBuffer, [addressStart, addressEnd]) "# romBuffer.slice(addressStart, addressEnd) |> huffmanDecode |> lzssDecode
203. Tilemap
205. ✔ Display tiles and OAM ✔ Allow the user to edit it
206. https://github.com/konvajs/konva
208. 1st load: ˜3 seconds! 2nd load: ˜13 seconds!
209. After some React.js performance work…
210. 1st load
211. 1st load ˜1.8 seconds!
212. 1st load 2nd load
213. 1st load 2nd load ˜3.6 seconds!
214. 1st load 2nd load 1st load: ˜1.8 seconds! 2nd load: ˜3.6 seconds!
215. After switching from Canvas to WebGL…
216. 1st load
217. 1st load ˜1.6 seconds!
218. 1st load ˜1.6 seconds! 2nd load
219. 1st load ˜1.6 seconds! 2nd load ˜2.8 seconds!
220. After doing more optimizations on WebGL…
221. 1st load
222. 1st load ˜625 ms!
223. 1st load 2nd load ˜625 ms!
224. 1st load 2nd load ˜536 ms! ˜625 ms!
225. 1st load: ˜3 seconds! 2nd load: ˜13 seconds! Perf recap: No one is gonna play with this
226. 1st load: ˜3 1.8 seconds! 2nd load: ˜13 3.6 seconds! Perf recap: Removing wasted renders and other React optimisation stuff.
227. 1st load: ˜3 1.8 1.6 seconds! 2nd load: ˜13 3.6 2.8 seconds! Perf recap: Canvas → WebGL
228. 1st load: ˜3 1.8 1.6 0.625 seconds! 2nd load: ˜13 3.6 2.8 0.536 seconds! Perf recap: React → WebGL
229. Result THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
236. THE Future THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
240. WE ARE... THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
241. MACABEUS macabeus ____ www.macalogs.com.br _ _ Software Engineer, full-stack @LOGGI Always creating crazy projects
242. MATHEUS ALBUQUERQUE YTHECOMBINATOR www.ythecombinator.space LAND@YTHECOMBINATOR.SPACE Senior Software Engineer, Front-End @STRV addicted to emojis, memes and beer
243. Raul peres behance.net/Kniksis illustrator
244. This talk THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME
245. http://bit.ly/gba-reactsp-slides
246. http://bit.ly/gba-the-conf-project
247. http://bit.ly/medium-gba
249. We got stickers!
250. OBRIGADO THANKS @ythecombinator AND @macabeus