Playing Video Games One Frame at a Time

Video games are hard and as I age my reaction time has gone down. But I'm also a programmer and I can write tools that solve this for me. This talk will go over what it means to play video games with the assistance of tools and show a few examples of how this can be done using C++. From games that support creating tools to injecting your own code into a game. This is an introductory talk on this topic but it will cover a few lower level systems interactions. No video game knowledge is needed.

Transcript

1. 1 TEST SLIDE Text is usually about this size and color It might go this far down. ^ code will be this color on this background and this example is roughly the smallest it will get. It will usually be bigger. If you can’t read it, move closer please.
2. Playing Video Games One Frame at a Time Ólafur Waage
3. Ólafur Waage Senior Software Developer - Factlines @olafurw on most social platforms 1 3
7. “ People say nothing is impossible, but I do nothing every day - Winnie the Pooh 7
8. “ People say nothing is impossible, but I do nothing every day - Winnie the Pooh 8
9. WHO AM I?
10. Worked on ATC systems for a couple of years. In the gaming industry for 7 years. As a consultant in Norway for a couple of years.
11. Worked on ATC systems for a couple of years. In the gaming industry for 7 years. As a consultant in Norway for a couple of years.
12. Worked on ATC systems for a couple of years. In the gaming industry for 7 years. As a consultant in Norway for a couple of years.
13. WHAT IS THIS TALK ABOUT?
14. What’s going on here? A good way to describe this talk is: 14
15. What’s going on here? A good way to describe this talk is: - A bit of silly fun 15
16. What’s going on here? A good way to describe this talk is: - A bit of silly fun - A bit of curiosity 16
17. What’s going on here? A good way to describe this talk is: - A bit of silly fun - A bit of curiosity - A bit of real life use cases 17
18. What’s going on here? A good way to describe this talk is: - A bit of silly fun - A bit of curiosity - A bit of real life use cases This is probably not an area a lot of people have thought about, so let’s explore together. 18
19. Last year? Also… 19
20. Last year? Also… I did a talk last year, 20
21. Last year? Also… I did a talk last year, that was way more serious than this one. 21
22. Last year? Also… I did a talk last year, that was way more serious than this one. (about how humans learn information) 22
23. Last year? Also… I did a talk last year, that was way more serious than this one. (about how humans learn information) And in that talk I said this… 23
24. "Unfortunately, in this case there is considerable evidence that in many situations this strategy…
25. "Unfortunately, in this case there is considerable evidence that in many situations this strategy (1 hour lectures)
26. "Unfortunately, in this case there is considerable evidence that in many situations this strategy (1 hour lectures) is rather poor (Bligh, 1971).
27. "Unfortunately, in this case there is considerable evidence that in many situations this strategy (1 hour lectures) is rather poor (Bligh, 1971). A lecture may inspire, motivate and inform, but rarely fulfils the principles for effective learning; it is certainly inadequate for developing high-level skills"
28. Last year? So, I can’t in good conscience make a talk this year that tries to effectively teach you something. 28
29. Last year? So, I can’t in good conscience make a talk this year that tries to effectively teach you something. Then instead I decided to make a fun talk that does have some real life applications that you might not have thought about. 29
30. WARNING
31. Before I start 31
32. Before I start I will do stuff to video games in this talk. 32
33. Before I start I will do stuff to video games in this talk. If you are going to do similar stuff, please, and I’m asking nicely, only do this to single player non-online games. 33
34. Before I start I will do stuff to video games in this talk. If you are going to do similar stuff, please, and I’m asking nicely, only do this to single player non-online games. I’m not going to do crazy stuff here, but I feel the need to add this warning. 34
35. BORING BACKSTORY
36. Playing video games is great fun, but I’m also a programmer.
37. Playing video games is great fun, but I’m also a programmer. How can I marry these two things together?
38. Playing video games is great fun, but I’m also a programmer. How can I marry these two things together? Well, you can make games.
39. Playing video games is great fun, but I’m also a programmer. How can I marry these two things together? Well, you can make games. But what if there was another way?
40. Games are just applications, so what can we do to control those applications remotely?
41. Games are just applications, so what can we do to control those applications remotely? What are the different challenges you will face when trying to interact with complex software like this?
42. Games are just applications, so what can we do to control those applications remotely? What are the different challenges you will face when trying to interact with complex software like this? And can some of this apply to interacting with other software?
43. Level 1
44. Level 1 What if the game you want to play supports playing it programmatically, right out of the box?
46. Doom is a 1993 first person shooter.
47. Doom is a 1993 first person shooter. And it kind of redefined a few aspects of video games.
48. Doom is a 1993 first person shooter. And it kind of redefined a few aspects of video games. - The look and feel of first person shooters
49. Doom is a 1993 first person shooter. And it kind of redefined a few aspects of video games. - The look and feel of first person shooters - Online multiplayer
50. Doom is a 1993 first person shooter. And it kind of redefined a few aspects of video games. - The look and feel of first person shooters - Online multiplayer - Game modification
51. Doom is a 1993 first person shooter. And it kind of redefined a few aspects of video games. - The look and feel of first person shooters - Online multiplayer - Game modification - Video Game Speedrunning
52. Doom is a 1993 first person shooter. And it kind of redefined a few aspects of video games. - The look and feel of first person shooters - Online multiplayer - Game modification - Video Game Speedrunning I can’t go over everything but let’s look at the last one.
53. Whenever there is a timer, people will try to be the best, be the fastest.
54. Whenever there is a timer, people will try to be the best, be the fastest. Doom is no exception.
55. Whenever there is a timer, people will try to be the best, be the fastest. Doom is no exception. Because at the end of every stage, there was a timer.
56. Whenever there is a timer, people will try to be the best, be the fastest. Doom is no exception. Because at the end of every stage, there was a timer. And as you can see, people are still playing, still trying to get better.
57. And if you download these files, you’ll notice that they’re actually pretty small.
58. And if you download these files, you’ll notice that they’re actually pretty small. The reason is that these files basically only contain what inputs the user pressed and during what frame.
65. This is possible because of one thing.
66. This is possible because of one thing. Doom is deterministic
67. This is possible because of one thing. Doom is deterministic Doom is a single threaded application where every single entity in the map patiently waits their turn, always in the same order.
68. This is possible because of one thing. Doom is deterministic Doom is a single threaded application where every single entity in the map patiently waits their turn, always in the same order.
69. This is possible because of one thing. Doom is deterministic Doom is a single threaded application where every single entity in the map patiently waits their turn, always in the same order. And when they want to do something random, they extract a number from this table
70. This is possible because of one thing. Doom is deterministic Doom is a single threaded application where every single entity in the map patiently waits their turn, always in the same order. And when they want to do something random, they extract a number from this table Then an index is incremented and the next entity will get the next number.
71. So what if you wanted to take Doom and convert it to C++ (or some language)
72. So what if you wanted to take Doom and convert it to C++ (or some language) How can you move forwards with that project knowing that your code changes and refactorings haven’t broken the game?
73. So what if you wanted to take Doom and convert it to C++ (or some language) How can you move forwards with that project knowing that your code changes and refactorings haven’t broken the game? We could try playing the game, or writing unit tests.
74. So what if you wanted to take Doom and convert it to C++ (or some language) How can you move forwards with that project knowing that your code changes and refactorings haven’t broken the game? We could try playing the game, or writing unit tests. But we have the tool in our hands right now.
75. Approval testing is a methodology where you focus on comparing the output of the old known bit of code with the output of your new code.
76. Approval testing is a methodology where you focus on comparing the output of the old known bit of code with the output of your new code. Think of it like a git diff. If there is a difference, then something broke.
77. Approval testing is a methodology where you focus on comparing the output of the old known bit of code with the output of your new code. Think of it like a git diff. If there is a difference, then something broke. This is useful when you want to do a complete rewrite or when the original source might not be available in full.
78. Doom can run the old demo files and then give you a nice statistics screen at the end.
79. Doom can run the old demo files and then give you a nice statistics screen at the end. How long you played for, how many monsters you killed, how many items you got and how many secrets you found.
80. Doom can run the old demo files and then give you a nice statistics screen at the end. How long you played for, how many monsters you killed, how many items you got and how many secrets you found. Well, if I play this demo on the new version, I should get the exact same results.
81. Another tidbit. The game doesn’t need to render the game to the screen for the simulation to run.
82. Another tidbit. The game doesn’t need to render the game to the screen for the simulation to run. The graphics are just a nice visual representation of the game state.
83. Another tidbit. The game doesn’t need to render the game to the screen for the simulation to run. The graphics are just a nice visual representation of the game state. So we can add (and modify) some game flags where the game can run in the terminal.
84. Another tidbit. The game doesn’t need to render the game to the screen for the simulation to run. The graphics are just a nice visual representation of the game state. So we can add (and modify) some game flags where the game can run in the terminal. This also means we don’t need to run the game at a nice frame rate, we can just run it as fast as possible.
85. Another tidbit. The game doesn’t need to render the game to the screen for the simulation to run. The graphics are just a nice visual representation of the game state. So we can add (and modify) some game flags where the game can run in the terminal. This also means we don’t need to run the game at a nice frame rate, we can just run it as fast as possible. The simulation is the same.
86. Putting 1 and 1 together. We can play the game using a known good version of the game.
87. Putting 1 and 1 together. We can play the game using a known good version of the game. Collect the resulting Stats output.
88. Putting 1 and 1 together. We can play the game using a known good version of the game. Collect the resulting Stats output. Save the demo files and the stats text files in a folder.
89. Putting 1 and 1 together. We can play the game using a known good version of the game. Collect the resulting Stats output. Save the demo files and the stats text files in a folder. Run the game, in command line using those demos and check if the resulting stats data match the corresponding text file.
90. Doom running in GitHub Actions.
91. Doom running in GitHub Actions.
92. Doom running in GitHub Actions.
93. Doom running in GitHub Actions.
94. Level 2
95. Level 2 What if the game you want to play doesn't have the support Doom does?
97. Bejeweled 3 is a match three game.
98. Bejeweled 3 is a match three game. Pick a gem, move it left/right or up/down to create three of a kind.
99. Bejeweled 3 is a match three game. Pick a gem, move it left/right or up/down to create three of a kind. You can also join four or more gems for bonus points.
100. Bejeweled 3 is a match three game. Pick a gem, move it left/right or up/down to create three of a kind. You can also join four or more gems for bonus points. After a match has been made, those gems go away and new ones drop down.
101. For Bejeweled 3 the plan of action is pretty simple
102. For Bejeweled 3 the plan of action is pretty simple - What Gem is where
103. For Bejeweled 3 the plan of action is pretty simple - What Gem is where - Pick a good move
104. For Bejeweled 3 the plan of action is pretty simple - What Gem is where - Pick a good move - Move that gem
105. For Bejeweled 3 the plan of action is pretty simple - What Gem is where - Pick a good move - Move that gem - Repeat
106. The Project Project Setup: - C++ - CMake - VCPKG - and OpenCV 106
107. The Project Project Setup: - C++ - CMake - VCPKG - and OpenCV 107
108. Step 1 What Gem is Where?
110. 64px Step 1 What Gem is Where? The game board is fixed resolution, which means each cell is 64 by 64 pixels.
111. w: 261px h: 37px Step 1 What Gem is Where? The game board is fixed resolution, which means each cell is 64 by 64 pixels. That means that the board starts at 261x37 from the top left of the game screen.
112. Step 1 What Gem is Where? The game board is fixed resolution, which means each cell is 64 by 64 pixels. That means that the board starts at 261x37 from the top left of the game screen. From that information, I know exactly where each cell on the board is.
113. Step 1 What Gem is Where? But how do I know that this gem is the green one?
114. Step 1 What Gem is Where? But how do I know that this gem is the green one? I started simple, let’s take the center pixel of that gem and check the RGB color values.
115. Step 1 What Gem is Where? But how do I know that this gem is the green one? I started simple, let’s take the center pixel of that gem and check the RGB color values. But there’s a problem.
116. Step 1 What Gem is Where? But how do I know that this gem is the green one? I started simple, let’s take the center pixel of that gem and check the RGB color values. But there’s a problem.
117. Step 1 What Gem is Where? Let’s use OpenCV to our advantage.
118. Step 1 What Gem is Where? Let’s use OpenCV to our advantage.
119. Step 1 What Gem is Where? Let’s use OpenCV to our advantage. Using Template Matching we can look for our gems on the board.
120. Step 1 What Gem is Where? Let’s use OpenCV to our advantage. Using Template Matching we can look for our gems on the board. I created these nice sample images of each gem.
124. Step 1 What Gem is Where? Let’s use OpenCV to our advantage. Using Template Matching we can look for our gems on the board. I created these nice sample images of each gem. So why doesn’t this work?
125. Step 1 What Gem is Where? Let’s use OpenCV to our advantage. Using Template Matching we can look for our gems on the board. I created these nice sample images of each gem. So why doesn’t this work? Many feature detection algorithms’ first step is to make the source image B&W
126. Step 1 What Gem is Where? Let’s go back to the idea of color.
127. Step 1 What Gem is Where? Let’s go back to the idea of color. What if I don’t check for 1 pixel in the center, but the general … hue … of the center area.
128. Step 1 What Gem is Where? Let’s go back to the idea of color. What if I don’t check for 1 pixel in the center, but the general … hue … of the center area. So I grab a 32x32 area in the center of the gem.
129. Step 1 What Gem is Where? Let’s go back to the idea of color. What if I don’t check for 1 pixel in the center, but the general … hue … of the center area. So I grab a 32x32 area in the center of the gem. And ask OpenCV what the average color of that area.
130. Step 1 What Gem is Where? Let’s go back to the idea of color. What if I don’t check for 1 pixel in the center, but the general … hue … of the center area. So I grab a 32x32 area in the center of the gem. And ask OpenCV what the average color of that area. And check against HSV and not RGB
131. Step 1 What Gem is Where? Let’s go back to the idea of color. What if I don’t check for 1 pixel in the center, but the general … hue … of the center area. So I grab a 32x32 area in the center of the gem. And ask OpenCV what the average color of that area. And check against HSV and not RGB
132. Step 1 What Gem is Where? So I went through and checked the Hue value of every single gem (when stationary and while spinning).
133. Step 1 What Gem is Where? So I went through and checked the Hue value of every single gem (when stationary and while spinning). And made a chart of where they overlap and where they are isolated.
134. Step 1 What Gem is Where? So I went through and checked the Hue value of every single gem (when stationary and while spinning). And made a chart of where they overlap and where they are isolated.
135. Step 1 What Gem is Where? So I went through and checked the Hue value of every single gem (when stationary and while spinning). And made a chart of where they overlap and where they are isolated. And then wrote a function that tries to disambiguate between the colors (and I used RGB to help also)
136. Step 1 What Gem is Where? So I went through and checked the Hue value of every single gem (when stationary and while spinning). And made a chart of where they overlap and where they are isolated. And then wrote a function that tries to disambiguate between the colors (and I used RGB to help also)
137. Step 1 What Gem is Where? So I went through and checked the Hue value of every single gem (when stationary and while spinning). And made a chart of where they overlap and where they are isolated. And then wrote a function that tries to disambiguate between the colors (and I used RGB to help also)
138. Step 1 What Gem is Where? So I went through and checked the Hue value of every single gem (when stationary and while spinning). And made a chart of where they overlap and where they are isolated. And then wrote a function that tries to disambiguate between the colors (and I used RGB to help also)
139. Step 1 What Gem is Where? So I went through and checked the Hue value of every single gem (when stationary and while spinning). And made a chart of where they overlap and where they are isolated. And then wrote a function that tries to disambiguate between the colors (and I used RGB to help also)
141. Step 2 Pick a good move.
142. Step 2 Pick a good move. So what is a good move?
143. Step 2 Pick a good move. So what is a good move? Initially I was thinking of writing some cool algorithm, get some deep learning AI into the mix.
144. Step 2 Pick a good move. So what is a good move? Initially I was thinking of writing some cool algorithm, get some deep learning AI into the mix. But after the color fiasko, I wanted to see if something simple was enough.
145. Step 2 Pick a good move. So what is a good move? Initially I was thinking of writing some cool algorithm, get some deep learning AI into the mix. But after the color fiasko, I wanted to see if something simple was enough. And it was.
146. Step 2 Pick a good move. Let’s start by playing the game a bit and see if anything pops out.
147. Step 2 Pick a good move. Let’s start by playing the game a bit and see if anything pops out. So I played the game a little bit. Just a bit.
148. Step 2 Pick a good move. Let’s start by playing the game a bit and see if anything pops out. So I played the game a little bit. Just a bit. And I had an idea.
149. Step 2 Pick a good move. Let’s start by playing the game a bit and see if anything pops out. So I played the game a little bit. Just a bit. And I had an idea. There aren’t actually that many moves in this game.
150. Step 2 Pick a good move. Let’s start by playing the game a bit and see if anything pops out. So I played the game a little bit. Just a bit. And I had an idea. There aren’t actually that many moves in this game. Can we just map them out?
161. Step 2 Pick a good move. But which move should we pick?
162. Step 2 Pick a good move. But which move should we pick? Well a high value move is always sorted above other moves. But what if all moves have the same value?
163. Step 2 Pick a good move. But which move should we pick? Well a high value move is always sorted above other moves. But what if all moves have the same value? Then we pick a move that is lower on the board. Why?
164. Step 2 Pick a good move. But which move should we pick? Well a high value move is always sorted above other moves. But what if all moves have the same value? Then we pick a move that is lower on the board. Why? Because it creates more matching opportunities when tiles fall down. A move at the top only creates one.
165. Step 3 Move that gem. Now it’s time to interact with the game application directly.
166. Step 3 Move that gem. Now it’s time to interact with the game application directly. The Windows SDK has a lot of nice mouse input functions. So let’s use them.
167. Step 3 Move that gem. Now it’s time to interact with the game application directly. The Windows SDK has a lot of nice mouse input functions. So let’s use them. I basically want a function like this. I know where the gem is and I know what direction I want to move to.
179. Bejeweled 3 DEMO
180. Level 3
181. Level 3 What if a game has none of these features?
184. VVVVVV Does not have built in support for giving the game the inputs.
185. VVVVVV Does not have built in support for giving the game the inputs. Does not have a simple game state.
186. VVVVVV Does not have built in support for giving the game the inputs. Does not have a simple game state. Is fast paced and you need to be pretty accurate with your movement.
189. Before I continue with the madness. There is one concept I need to talk about.
190. Before I continue with the madness. There is one concept I need to talk about. The Game Loop
191. Before I continue with the madness. There is one concept I need to talk about. The Game Loop Games are applications like any other, and they usually follow a certain set of steps.
192. Before I continue with the madness. There is one concept I need to talk about. The Game Loop Games are applications like any other, and they usually follow a certain set of steps. - Get input from user
193. Before I continue with the madness. There is one concept I need to talk about. The Game Loop Games are applications like any other, and they usually follow a certain set of steps. - Get input from user - Update the state
194. Before I continue with the madness. There is one concept I need to talk about. The Game Loop Games are applications like any other, and they usually follow a certain set of steps. - Get input from user - Update the state - Draw the current state
195. There are a few caveats with these steps that makes our work harder.
196. There are a few caveats with these steps that makes our work harder. Get input from user There might be no input or if we handle it badly, we might delay it or miss it.
197. There are a few caveats with these steps that makes our work harder. Get input from user There might be no input or if we handle it badly, we might delay it or miss it. Update the state This can take a long time and we might multi-thread it.
198. There are a few caveats with these steps that makes our work harder. Get input from user There might be no input or if we handle it badly, we might delay it or miss it. Update the state This can take a long time and we might multi-thread it. Draw the current state We might draw old state or we might offload the draw to happen in another thread.
199. 199 Initialize
200. 200 Initialize Process input
201. 201 Initialize Process input Update game state
202. 202 Initialize Process input Update game state Draw game state
203. 203 Initialize Process input Update game state Draw game state
204. 204 Initialize Process input Update game state Draw game state “Frame”
205. 205 Initialize Process input Update game state Draw game state <1ms “Frame”
206. 206 Initialize Process input Update game state Draw game state <1ms “Frame” Depends
207. 207 Initialize Process input Update game state Draw game state <1ms “Frame” Depends Depends
208. 208 Initialize Process input Update game state Draw game state “Frame” 60 fps game 16.666… ms
209. 209 Initialize Process input Update game state Draw game state
210. 210 Initialize Process input Update game state Draw game state At least once per 16.666 ms
211. 211 Initialize Process input Update game state Draw game state *As often as we can At least once per 16.666 ms
224. VVVVVV actually has a few advantages and is why I picked this game for the project.
225. VVVVVV actually has a few advantages and is why I picked this game for the project. It’s open source, so we can just go check.
226. VVVVVV actually has a few advantages and is why I picked this game for the project. It’s open source, so we can just go check. It’s single threaded.
227. VVVVVV actually has a few advantages and is why I picked this game for the project. It’s open source, so we can just go check. It’s single threaded. It has a lot of customization for such a simple game.
228. So here is my plan Loader Target process dll
229. So here is my plan Launch the game as a paused child process of some loader. Loader
230. So here is my plan Launch the game as a paused child process of some loader. Loader Game Process
231. So here is my plan Launch the game as a paused child process of some loader. Loader Game Process dll
232. So here is my plan Launch the game as a paused child process of some loader. Hook the function that creates the DirectX Device. Loader Game Process dll
233. So here is my plan Launch the game as a paused child process of some loader. Hook the function that creates the DirectX Device. Get the address of that device. Loader Game Process dll
234. So here is my plan Launch the game as a paused child process of some loader. Hook the function that creates the DirectX Device. Get the address of that device. Intercept the game’s draw call and call our draw function first. Loader Game Process dll
235. So here is my plan Launch the game as a paused child process of some loader. Hook the function that creates the DirectX Device. Get the address of that device. Intercept the game’s draw call and call our draw function first. Before each frame, send our keyboard inputs. Loader Game Process dll
236. 236 Initialize Process input Update game state Draw game state
237. 237 Initialize Process input Update game state Draw game state
238. 238 Initialize Process input Update game state Draw game state Intercept - Call game’s draw function - Send our inputs
239. 239 Initialize Process input Update game state Draw game state We could hook the input polling function.
240. 240 Initialize Process input Update game state Draw game state We could hook the input polling function. But we actually want to do a bit of drawing ourselves, to make a nice debug UI.
272. A summary of what we’ve done so far. Loader Game Process dll
273. A summary of what we’ve done so far. ✅ Spawn the game process paused. Loader Game Process dll
274. A summary of what we’ve done so far. ✅ Spawn the game process paused. ✅ Attach our DLL to the game process. Loader Game Process dll
275. A summary of what we’ve done so far. ✅ Spawn the game process paused. ✅ Attach our DLL to the game process. ✅ Hook the “CreateDevice” function the game is going to call once it starts. Loader Game Process dll
276. A summary of what we’ve done so far. ✅ Spawn the game process paused. ✅ Attach our DLL to the game process. ✅ Hook the “CreateDevice” function the game is going to call once it starts. Once the game calls the “CreateDevice” function, get the device it just made and hook end “EndScene” function. Loader Game Process dll
291. VVVVVV DEMO
292. Summary: Games are neat and it’s fun to play with them programmatically. Loader Game Process dll
293. Summary: Games are neat and it’s fun to play with them programmatically. When you have an application you don’t have the source code to, there are still ways to automatically interact with it. Loader Game Process dll
294. Summary: Games are neat and it’s fun to play with them programmatically. When you have an application you don’t have the source code to, there are still ways to automatically interact with it. AKA. Yes, you can still test that old legacy app. Yes you can still do safe(er) refactoring. Loader Game Process dll
295. Summary: Games are neat and it’s fun to play with them programmatically. When you have an application you don’t have the source code to, there are still ways to automatically interact with it. AKA. Yes, you can still test that old legacy app. Yes you can still do safe(er) refactoring. Write more tests! Loader Game Process dll
296. TASVideos.org Loader Game Process dll
297. TASVideos.org Loader Game Process dll
298. dsdarchive.com Loader Game Process dll
299. www.classicdoom.com/odddemos.htm Loader Game Process dll
300. VVVVVV https://github.com/olafurw/vvvvvv-hook/ Bejeweled 3 https://github.com/olafurw/bejeweled-player Doom https://github.com/patricia-gallardo/cpp-doom Loader Game Process dll
301. Ólafur Waage Senior Software Developer - Factlines @olafurw on most social platforms 1 301