The Day I Reverse Engineered a Gameboy Advance Game

97d21da8e0ffa8f81218a293482c253a?s=47 Matheus
August 17, 2019

The Day I Reverse Engineered a Gameboy Advance Game

Gameboy Advance was one of the most popular video game platforms of its time and, because of it, many people worked together as a community to study and to document its architecture and develop ROM hacking and other interesting tools.

It turns out that this video game is a fantastic way to start studying reverse engineering: its architecture is very well documented and simpler if compared to the current game console generation – and, of course, it is very fun to work in a game-related project.

So... what do you think of learning reverse engineering through this challenge: developing a level editor for a GBA game called "Klonoa: Empire of Dreams"?
We need to understand the architecture behind ARM hardware, apply reverse engineering in order to discover how the logic of the game works, and then use our front-end knowledge (JS + React.js) to build a level editor.

97d21da8e0ffa8f81218a293482c253a?s=128

Matheus

August 17, 2019
Tweet

Transcript

  1. The day I reverse engineered a Gameboy Advance game With

    @ythecombinator AND @macabeus
  2. WTF?! THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME

  3. """

  4. None
  5. soooo…

  6. None
  7. This is macabeus

  8. This is MATHEUS

  9. This is US PLAYING KLONOA

  10. AND THIS IS US BORED

  11. soooo…

  12. WANTED MORE LEVELS TO PLAY WITH

  13. WANTED DIFFERENT LEVELS

  14. soooo…

  15. soooo…

  16. soooo…

  17. soooo…AND THIS IS KLONNOA

  18. soooo…AND THIS IS KLONNOA

  19. None
  20. None
  21. None
  22. http://bit.ly/gba-the-conf-project

  23. Stretching the bridge THE DAY I REVERSE ENGINEERED A GAMEBOY

    ADVANCE GAME
  24. https://www.nogba.com

  25. https://www.hex-rays.com/products/ida

  26. None
  27. None
  28. This is the tile ID. In this example, all tiles

    with the ID 9B are exactly the bridge part, which is the left corner. 9A is the ID of the continuous part of the bridge.
  29. This tells us where the tile is stored in the

    memory. In this example, it is at 0600F1AD.
  30. https://www.coranac.com/tonc/text/hardware.htm

  31. https://www.coranac.com/tonc/text/hardware.htm The address found is expected 0600F1AD given that everything

    between 0600 and 0601 is part of the section known as VRAM.
  32. Our bridge starts here!

  33. Our bridge starts here!

  34. And now we grow it bigger!

  35. And now we grow it bigger!

  36. And now we grow it bigger! WTF??!

  37. Understanding the DMA THE DAY I REVERSE ENGINEERED A GAMEBOY

    ADVANCE GAME
  38. None
  39. None
  40. VRAM pulls the tilemap
 containing exactly what the player has

    to see from somewhere else that has the entire thing.
  41. https://www.coranac.com/tonc/text/regbg.htm

  42. https://www.coranac.com/tonc/text/regbg.htm Direct Memory Access

  43. https://www.coranac.com/tonc/text/regbg.htm It is a feature of computer systems that allows

    certain hardware subsystems to access main system memory (random- access memory), independent of the CPU
  44. https://www.coranac.com/tonc/text/regbg.htm tldr; a faster copy and paste

  45. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0

    0600F000 80000200 DMA3: 03004800 07000000 84000048
  46. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0

    0600F000 80000200 DMA3: 03004800 07000000 84000048 The closest to the bytes of our bridge (0600F1AD).
  47. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0

    0600F000 80000200 DMA3: 03004800 07000000 84000048 This address is the VRAM data source. It’s located in a region called Fast WRAM.
  48. None
  49. Our bridge starts here!

  50. Stretching it…

  51. Stretching it… …It works! And…

  52. Stretching it… …It works! And… WTF??!

  53. Stretching it… …It works! And… WTF??!

  54. None
  55. None
  56. Our extra tiles disappear when out of the player vision

    range.
  57. LET’S TALK ABOUT PHYSICS THE DAY I REVERSE ENGINEERED A

    GAMEBOY ADVANCE GAME
  58. Object Attribute Memory

  59. OAM

  60. None
  61. This is the address in which Klonoa is stored on

    the memory
  62. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0

    0600F000 80000200 DMA3: 03004800 07000000 8400003E
  63. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0

    0600F000 80000200 DMA3: 03004800 07000000 8400003E We can see exactly the byte that writes on Klonoa’s OAM (03004800).
  64. DMA3: 03000900 0600E000 80000400 DMA3: 03001100 0600E800 80000400 DMA3: 03004DB0

    0600F000 80000200 DMA3: 03004800 07000000 8400003E We can see exactly the byte that writes on Klonoa’s OAM (03004800).
  65. None
  66. None
  67. None
  68. None
  69. None
  70. 1 2

  71. None
  72. None
  73. None
  74. None
  75. None
  76. None
  77. None
  78. None
  79. None
  80. None
  81. None
  82. Here is where Y fixes Klonoa’s position

  83. Here is where Y fixes Klonoa’s position Here is where

    which decides if should fix or not Klonoa’s position
  84. None
  85. https://www.coranac.com/tonc/text/hardware.htm

  86. https://www.coranac.com/tonc/text/hardware.htm

  87. None
  88. None
  89. None
  90. Let’s find the tilemap on rom THE DAY I REVERSE

    ENGINEERED A GAMEBOY ADVANCE GAME
  91. Cartridge > > > > The data flow

  92. Cartridge Slow WRAM > > > > The data flow

  93. Cartridge Fast WRAM Slow WRAM > > > > The

    data flow
  94. Cartridge Fast WRAM Slow WRAM VRAM > > > >

    The data flow
  95. Cartridge Fast WRAM Display Slow WRAM VRAM > > >

    > The data flow
  96. Cartridge Fast WRAM Display VRAM Slow WRAM > > >

    > Our research flow
  97. None
  98. swi 11h what?

  99. Swi

  100. SoftWare Interrupt

  101. SoftWare Interrupt = Bios calls

  102. None
  103. BIOS is a firmware that is intended to initialise the

    physical components of the system
  104. BIOS is a firmware that is intended to initialise the

    physical components of the system In GBA, its BIOS exposes many functions often used in games, including data compression and decompression
  105. BIOS is a firmware that is intended to initialise the

    physical components of the system In GBA, its BIOS exposes many functions often used in games, including data compression and decompression Each function has an associated numeric code, which must be used as a parameter in the SWI statement
  106. Input: How division works on ARM Assembly swi 0x06 Output:

  107. Input: How division works on ARM Assembly swi 0x06 R0


    numerator R1
 denominator Output:
  108. Input: How division works on ARM Assembly swi 0x06 R0


    numerator R1
 denominator R0
 numerator / denominator R1
 numerator % denominator R3
 abs(numerator / denominator) Output:
  109. None
  110. Huffman + lz77 = deflate

  111. None
  112. None
  113. None
  114. None
  115. None
  116. 6E 6E 6E B0 B1 BD 77 AB B3 AB

    B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E
  117. 6E 6E 6E B0 B1 BD 77 AB B3 AB

    B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E
  118. 6E 6E 6E B0 B1 BD 77 AB B3 AB

    B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E 6E 6E 6E B0 B1 BD 77 AB B3 AB B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E
  119. 6E 6E 6E B0 B1 BD 77 AB B3 AB

    B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E 6E 6E 6E B0 B1 BD 77 AB B3 AB B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E
  120. 6E 6E 6E B0 B1 BD 77 AB B3 AB

    B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E 6E 6E 6E B0 B1 BD 77 AB B3 AB B3 7C 6E 6E 6E 6E 6E 6E 6E 6E 7C 6E 6E 6E 6E 7C 6E 6E =
  121. Let’s paint the tilemap THE DAY I REVERSE ENGINEERED A

    GAMEBOY ADVANCE GAME
  122. 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F

  123. 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F 0x1A 0x1B 0x1C 0x1D

    0x1E 0x1F 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F
  124. 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F 0x1A 0x1B 0x1C 0x1D

    0x1E 0x1F 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F
  125. 0x1A 0x1B 0x1C 0x1D 0x1E 0x1F

  126. H · W = 25023
 H,W ∈ ℕ

  127. None
  128. x x x x a x x x x x

    x x x b x x x x I’m on tile a
  129. x x x x a x x x x x

    x x x b x x x x I’m on tile a
  130. x x x x a x x x x x

    x x x b x x x x I’m on tile a x x x x a x x x x x x x x b x x x x From the tile a to b there are 9 bytes (= 9 tiles), so the length of the tilemap is exactly 9
  131. None
  132. Breakpoint here

  133. …is located here This byte…

  134. None
  135. As I change this byte…

  136. …it reflects right here

  137. None
  138. Changing these bytes…

  139. …we drop Klonoa one level below

  140. None
  141. Changing these bytes… …we found the B tile!

  142. 020085DB The length of the level is 420! - 02008437

    = 1A4 (420)
  143. https://github.com/oliver-moran/jimp

  144. const Jimp = require('jimp') const { drop } = require('ramda')

    const fs = require('fs') const data = drop(3, fs.readFileSync('dump/level-2/tilemap'))
  145. const getPixelColor = hexTile "# { if (hexTile === 0)

    { return 0x00000000 } return 0xFFFFFFFF }
  146. new Jimp(300, 600, (err, image) "# { let x =

    0 let y = 0 for (let i = 0; i < data.length; i += 1) { const value = data[i] x += 1 if (x === 300) { y += 1 x = 0 } let color = getPixelColor(value) image.setPixelColor(color, x, y) } image.write('image.png') })
  147. None
  148. None
  149. None
  150. None
  151. None
  152. None
  153. let’s see the project THE DAY I REVERSE ENGINEERED A

    GAMEBOY ADVANCE GAME
  154. Coding THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME

  155. soooo…

  156. soooo…

  157. Stack

  158. React.js ✈

  159. Webpack ✈

  160. Ramda.js ✈

  161. FormerKit ✈

  162. GH Pages ✈

  163. architecture

  164. ✂ Brush Scissors

  165. klo-gba.js/scissors/src/visions/

  166. export default { rom: { tilemap: [0x1B27FC, 0x1B36F3], oam: [0xE2B90,

    0xE2F59], portals: [0xD48C8, 0xD48EF], }, tilemap: { totalStages: 3, height: 60, width: 420, scheme: [ { name: 'grass', ids: [0x7D, 0x80, 0x81, 0x82, 0x8E, 0x8F, 0x90], },
  167. export default { rom: { tilemap: [0x1B27FC, 0x1B36F3], oam: [0xE2B90,

    0xE2F59], portals: [0xD48C8, 0xD48EF], }, tilemap: { totalStages: 3, height: 60, width: 420, scheme: [ { name: 'grass', ids: [0x7D, 0x80, 0x81, 0x82, 0x8E, 0x8F, 0x90], },
  168. export default { rom: { tilemap: [0x1B27FC, 0x1B36F3], oam: [0xE2B90,

    0xE2F59], portals: [0xD48C8, 0xD48EF], }, tilemap: { totalStages: 3, height: 60, width: 420, scheme: [ { name: 'grass', ids: [0x7D, 0x80, 0x81, 0x82, 0x8E, 0x8F, 0x90], },
  169. { name: 'rock', ids: [0x54, 0x53, 0x55, 0x56, 0x57, 0x58,

    0x59, }, { name: 'darkRock', ids: [0x40, 0x41, 0x42, 0x43, 0x44, 0x45, 0x46, }, { name: 'wood', ids: [0x52, 0x94, 0x98, 0x99, 0x9A, 0x9B, 0x9D, }, { name: 'bridgeRope', ids: [0x04, 0x05, 0x0B, 0x0C, 0x11, 0x13, 0x14,
  170. { name: 'spike', ids: [0x3D], }, ], },

  171. Provider Pattern

  172. None
  173. ✘ Nah$$.Not necessary at all.

  174. ROMProvider VisionProvider

  175. import React from 'react' const ROMContext = React.createContext() export default

    ROMContext import React from 'react' const VisionContext = React.createContext() export default VisionContext klo-gba.js/brush/src/context/ROMContext.js klo-gba.js/brush/src/context/VisionContext.js
  176. import React from 'react' const ROMContext = React.createContext() export default

    ROMContext import React from 'react' const VisionContext = React.createContext() export default VisionContext klo-gba.js/brush/src/context/ROMContext.js klo-gba.js/brush/src/context/VisionContext.js
  177. const ROMProvider = (props) "# { const [romBuffer, setROMBuffer] =

    useROMBuffer() return ( <ROMContext.Provider value={{ romBufferMemory: romBuffer.memory, romBufferStatus: romBuffer.status, setROMBuffer, }} > {props.children} </ROMContext.Provider> ) }
  178. const SelectVision = () "# { const { romBufferMemory }

    = useContext(ROMContext) const { updateVision, vision: { state: visionState }, visionIndex, visionWorld, } = useContext(VisionContext) const setNewVision = (newVision) "# { const [newVisionWorld, newVisionIndex] = newVision.split('-') updateVision(romBufferMemory, newVisionWorld, newVisionIndex) }
  179. const SelectVision = () "# { const { romBufferMemory }

    = useContext(ROMContext) const { updateVision, vision: { state: visionState }, visionIndex, visionWorld, } = useContext(VisionContext) const setNewVision = (newVision) "# { const [newVisionWorld, newVisionIndex] = newVision.split('-') updateVision(romBufferMemory, newVisionWorld, newVisionIndex) }
  180. Custom hooks

  181. import { useContext, useEffect } from 'react' import VisionContext from

    '../context/VisionContext' const useWhenVisionChanges = (callback) "# { const { visionIndex, visionWorld } = useContext(VisionContext) useEffect(callback, [visionIndex, visionWorld]) } export default useWhenVisionChanges
  182. const LoadedRom = () "# { const { vision }

    = useContext(VisionContext) const [highlightCoordinates, setHighlightCoordinates] = useState([-1, -1]) const [optShowGrid, setOptShowGrid] = useState(false) const [optShowOAM, setOptShowOAM] = useState(true) const [optShowPortals, setOptShowPortals] = useState(true) const [toolState, setToolState] = useTool() const [resolution, setResolution] = useState(1) useWhenVisionChanges(() "# { setHighlightCoordinates([-1, -1]) }) $( $$.
  183. const LoadedRom = () "# { const { vision }

    = useContext(VisionContext) const [highlightCoordinates, setHighlightCoordinates] = useState([-1, -1]) const [optShowGrid, setOptShowGrid] = useState(false) const [optShowOAM, setOptShowOAM] = useState(true) const [optShowPortals, setOptShowPortals] = useState(true) const [toolState, setToolState] = useTool() const [resolution, setResolution] = useState(1) useWhenVisionChanges(() "# { setHighlightCoordinates([-1, -1]) }) $( $$.
  184. import { useState } from 'react' const useTool = ()

    "# { const [{ currentTool, toolsValue }, setToolsStates] = useState({ currentTool: 'magnifyingGlass', toolsValue: { brush: null, magnifyingGlass: null, }, })
  185. const updateToolState = (tool, newToolState) "# { if (newToolState ))*

    undefined) { setToolsStates({ currentTool: tool, toolsValue: { $$.toolsValue, [tool]: newToolState, }, }) return } setToolsStates({ currentTool: tool, toolsValue, }) }
  186. return [{ name: currentTool, value: toolsValue[currentTool], }, updateToolState] } export

    default useTool
  187. Web assembly

  188. Tilemap compressed Old C code to uncompress it Decompressed Tilemap

    > >
  189. Tilemap compressed Old C code to uncompress it Decompressed Tilemap

    > > Can’t run C code on browser!
  190. Tilemap compressed Old C code to uncompress it Decompressed Tilemap

    > >
  191. Tilemap compressed Old C code to uncompress it Decompressed Tilemap

    > > >Emscripten >WebAssembly
  192. huffman.c lzss.c > > Emscripten huffman.js lzss.js huffman.wasm lzss.wasm +

  193. None
  194. klo-gba.js/scissors/webpack.config.js const rules = [ { loader: 'file-loader', test: /\.wasm$/,

    type: 'javascript/auto', }, … ]
  195. const huffmanWasm = require('./wasm/huffman.wasm') const huffmanModule = require('./wasm/huffman.js')({ locateFile (path)

    { if (path.endsWith('.wasm')) { return `static/wasm/${huffmanWasm}` } return path }, })
  196. class HuffmanDecodeError extends Error { constructor (e) { super(`Error when

    tried to use huffman decoder: ${e}`) this.name = 'HuffmanDecodeError' } } huffmanModule.onRuntimeInitialized = () "# {} const huffmanDecode = (buffer) "# { FS.writeFile('file', buffer)
  197. class HuffmanDecodeError extends Error { constructor (e) { super(`Error when

    tried to use huffman decoder: ${e}`) this.name = 'HuffmanDecodeError' } } huffmanModule.onRuntimeInitialized = () "# {} const huffmanDecode = (buffer) "# { FS.writeFile('file', buffer)
  198. const huffmanDecode = (buffer) "# { FS.writeFile('file', buffer) huffmanModule._HUF_Decode() try

    { return huffmanModule.FS.readFile('file', { encoding: 'binary' }) } catch (e) { throw new HuffmanDecodeError(e) } }
  199. function docker_run_emscripten { local filename="$1" echo "Compiling $filename$$." docker run

    \ --rm -it \ -v $(pwd)/scissors/src/wasm:/src \ trzeci/emscripten \ emcc -s WASM=1 -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s EXTRA_EXPORTED_RUNTIME_METHODS=[\"FS\"] -s EXPORT_NAME=\"$filename\" -o ./$filename.js $filename.c } docker_run_emscripten huffman docker_run_emscripten lzss
  200. function docker_run_emscripten { local filename="$1" echo "Compiling $filename$$." docker run

    \ --rm -it \ -v $(pwd)/scissors/src/wasm:/src \ trzeci/emscripten \ emcc -s WASM=1 -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s EXTRA_EXPORTED_RUNTIME_METHODS=[\"FS\"] -s EXPORT_NAME=\"$filename\" -o ./$filename.js $filename.c } docker_run_emscripten huffman docker_run_emscripten lzss
  201. None
  202. const extractFullTilemap = (romBuffer, [addressStart, addressEnd]) "# romBuffer.slice(addressStart, addressEnd) |>

    huffmanDecode |> lzssDecode
  203. Tilemap

  204. None
  205. ✔ Display tiles and OAM ✔ Allow the user to

    edit it
  206. https://github.com/konvajs/konva

  207. None
  208. 1st load: ˜3 seconds! 2nd load: ˜13 seconds!

  209. After some React.js performance work…

  210. 1st load

  211. 1st load ˜1.8 seconds!

  212. 1st load 2nd load

  213. 1st load 2nd load ˜3.6 seconds!

  214. 1st load 2nd load 1st load: ˜1.8 seconds! 2nd load:

    ˜3.6 seconds!
  215. After switching from Canvas to WebGL…

  216. 1st load

  217. 1st load ˜1.6 seconds!

  218. 1st load ˜1.6 seconds! 2nd load

  219. 1st load ˜1.6 seconds! 2nd load ˜2.8 seconds!

  220. After doing more optimizations on WebGL…

  221. 1st load

  222. 1st load ˜625 ms!

  223. 1st load 2nd load ˜625 ms!

  224. 1st load 2nd load ˜536 ms! ˜625 ms!

  225. 1st load: ˜3 seconds! 2nd load: ˜13 seconds! Perf recap:

    No one is gonna play with this
  226. 1st load: ˜3 1.8 seconds! 2nd load: ˜13 3.6 seconds!

    Perf recap: Removing wasted renders and other React optimisation stuff.
  227. 1st load: ˜3 1.8 1.6 seconds! 2nd load: ˜13 3.6

    2.8 seconds! Perf recap: Canvas → WebGL
  228. 1st load: ˜3 1.8 1.6 0.625 seconds! 2nd load: ˜13

    3.6 2.8 0.536 seconds! Perf recap: React → WebGL
  229. Result THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE GAME

  230. None
  231. None
  232. None
  233. None
  234. None
  235. None
  236. THE Future THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE

    GAME
  237. None
  238. None
  239. None
  240. WE ARE... THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE

    GAME
  241. MACABEUS macabeus ____ www.macalogs.com.br _ _ Software Engineer, full-stack @LOGGI

    Always creating crazy projects
  242. MATHEUS ALBUQUERQUE YTHECOMBINATOR www.ythecombinator.space LAND@YTHECOMBINATOR.SPACE Senior Software Engineer, Front-End @STRV

    addicted to emojis, memes and beer
  243. Raul peres behance.net/Kniksis illustrator

  244. This talk THE DAY I REVERSE ENGINEERED A GAMEBOY ADVANCE

    GAME
  245. http://bit.ly/gba-reactsp-slides

  246. http://bit.ly/gba-the-conf-project

  247. http://bit.ly/medium-gba

  248. None
  249. We got stickers!

  250. OBRIGADO THANKS @ythecombinator AND @macabeus