Playing Video Games One Frame at a Time

Video games are hard and as I age my reaction time has gone down. But I'm also a programmer and I can write tools that solve this for me. This talk will go over what it means to play video games with the assistance of tools and show a few examples of how this can be done using C++. From games that support creating tools to injecting your own code into a game. This is an introductory talk on this topic but it will cover a few lower level systems interactions. No video game knowledge is needed.

Transcript

1. 1
   TEST SLIDE
   Text is usually about this
   size and color
   It might go this far
   down.
   ^ code will be this color on this
   background and this example is
   roughly the smallest it will get.
   It will usually be bigger.
   If you can’t read it,
   move closer please.
   

   View Slide

2. Playing Video Games
   One Frame at a Time
   Ólafur Waage
   

   View Slide

3. Ólafur Waage
   Senior Software Developer - Factlines
   @olafurw on most social platforms
   1
   3
   

   View Slide

4. View Slide

5. View Slide

6. View Slide

7. “
   People say nothing is impossible,
   but I do nothing every day
   - Winnie the Pooh
   7
   

   View Slide

8. “
   People say nothing is impossible,
   but I do nothing every day
   - Winnie the Pooh
   8
   

   View Slide

9. WHO AM I?
   

   View Slide

10. Worked on ATC systems for
    a couple of years.
    In the gaming industry for 7
    years.
    As a consultant in Norway
    for a couple of years.
    

    View Slide

11. Worked on ATC systems for
    a couple of years.
    In the gaming industry for 7
    years.
    As a consultant in Norway
    for a couple of years.
    

    View Slide

12. Worked on ATC systems for
    a couple of years.
    In the gaming industry for 7
    years.
    As a consultant in Norway
    for a couple of years.
    

    View Slide

13. WHAT IS THIS TALK ABOUT?
    

    View Slide

14. What’s going on here?
    A good way to describe this talk is:
    14
    

    View Slide

15. What’s going on here?
    A good way to describe this talk is:
    - A bit of silly fun
    15
    

    View Slide

16. What’s going on here?
    A good way to describe this talk is:
    - A bit of silly fun
    - A bit of curiosity
    16
    

    View Slide

17. What’s going on here?
    A good way to describe this talk is:
    - A bit of silly fun
    - A bit of curiosity
    - A bit of real life use cases
    17
    

    View Slide

18. What’s going on here?
    A good way to describe this talk is:
    - A bit of silly fun
    - A bit of curiosity
    - A bit of real life use cases
    This is probably not an area a lot of people have
    thought about, so let’s explore together.
    18
    

    View Slide

19. Last year?
    Also…
    19
    

    View Slide

20. Last year?
    Also…
    I did a talk last year,
    20
    

    View Slide

21. Last year?
    Also…
    I did a talk last year,
    that was way more serious than this one.
    21
    

    View Slide

22. Last year?
    Also…
    I did a talk last year,
    that was way more serious than this one.
    (about how humans learn information)
    22
    

    View Slide

23. Last year?
    Also…
    I did a talk last year,
    that was way more serious than this one.
    (about how humans learn information)
    And in that talk I said this…
    23
    

    View Slide

24. "Unfortunately, in this case there is considerable evidence that
    in many situations this strategy…
    

    View Slide

25. "Unfortunately, in this case there is considerable evidence that
    in many situations this strategy (1 hour lectures)
    

    View Slide

26. "Unfortunately, in this case there is considerable evidence that
    in many situations this strategy (1 hour lectures) is rather poor
    (Bligh, 1971).
    

    View Slide

27. "Unfortunately, in this case there is considerable evidence that
    in many situations this strategy (1 hour lectures) is rather poor
    (Bligh, 1971).
    A lecture may inspire, motivate and inform, but rarely fulfils the
    principles for effective learning; it is certainly inadequate for
    developing high-level skills"
    

    View Slide

28. Last year?
    So, I can’t in good conscience make a talk this year
    that tries to effectively teach you something.
    28
    

    View Slide

29. Last year?
    So, I can’t in good conscience make a talk this year
    that tries to effectively teach you something.
    Then instead I decided to make a fun talk that does
    have some real life applications that you might not
    have thought about.
    29
    

    View Slide

30. WARNING
    

    View Slide

31. Before I start
    31
    

    View Slide

32. Before I start
    I will do stuff to video games in this talk.
    32
    

    View Slide

33. Before I start
    I will do stuff to video games in this talk.
    If you are going to do similar stuff, please, and I’m
    asking nicely, only do this to single player
    non-online games.
    33
    

    View Slide

34. Before I start
    I will do stuff to video games in this talk.
    If you are going to do similar stuff, please, and I’m
    asking nicely, only do this to single player
    non-online games.
    I’m not going to do crazy stuff here, but I feel the
    need to add this warning.
    34
    

    View Slide

35. BORING BACKSTORY
    

    View Slide

36. Playing video games is great
    fun, but I’m also a
    programmer.
    

    View Slide

37. Playing video games is great
    fun, but I’m also a
    programmer.
    How can I marry these two
    things together?
    

    View Slide

38. Playing video games is great
    fun, but I’m also a
    programmer.
    How can I marry these two
    things together?
    Well, you can make games.
    

    View Slide

39. Playing video games is great
    fun, but I’m also a
    programmer.
    How can I marry these two
    things together?
    Well, you can make games.
    But what if there was another
    way?
    

    View Slide

40. Games are just applications,
    so what can we do to control
    those applications remotely?
    

    View Slide

41. Games are just applications,
    so what can we do to control
    those applications remotely?
    What are the different
    challenges you will face when
    trying to interact with
    complex software like this?
    

    View Slide

42. Games are just applications,
    so what can we do to control
    those applications remotely?
    What are the different
    challenges you will face when
    trying to interact with
    complex software like this?
    And can some of this apply
    to interacting with other
    software?
    

    View Slide

43. Level 1
    

    View Slide

44. Level 1
    What if the game you want to play
    supports playing it programmatically,
    right out of the box?
    

    View Slide

45. View Slide

46. Doom is a 1993 first person
    shooter.
    

    View Slide

47. Doom is a 1993 first person
    shooter.
    And it kind of redefined a few
    aspects of video games.
    

    View Slide

48. Doom is a 1993 first person
    shooter.
    And it kind of redefined a few
    aspects of video games.
    - The look and feel of first
    person shooters
    

    View Slide

49. Doom is a 1993 first person
    shooter.
    And it kind of redefined a few
    aspects of video games.
    - The look and feel of first
    person shooters
    - Online multiplayer
    

    View Slide

50. Doom is a 1993 first person
    shooter.
    And it kind of redefined a few
    aspects of video games.
    - The look and feel of first
    person shooters
    - Online multiplayer
    - Game modification
    

    View Slide

51. Doom is a 1993 first person
    shooter.
    And it kind of redefined a few
    aspects of video games.
    - The look and feel of first
    person shooters
    - Online multiplayer
    - Game modification
    - Video Game
    Speedrunning
    

    View Slide

52. Doom is a 1993 first person
    shooter.
    And it kind of redefined a few
    aspects of video games.
    - The look and feel of first
    person shooters
    - Online multiplayer
    - Game modification
    - Video Game
    Speedrunning
    I can’t go over everything but
    let’s look at the last one.
    

    View Slide

53. Whenever there is a timer,
    people will try to be the best,
    be the fastest.
    

    View Slide

54. Whenever there is a timer,
    people will try to be the best,
    be the fastest.
    Doom is no exception.
    

    View Slide

55. Whenever there is a timer,
    people will try to be the best,
    be the fastest.
    Doom is no exception.
    Because at the end of every
    stage, there was a timer.
    

    View Slide

56. Whenever there is a timer,
    people will try to be the best,
    be the fastest.
    Doom is no exception.
    Because at the end of every
    stage, there was a timer.
    And as you can see, people
    are still playing, still trying to
    get better.
    

    View Slide

57. And if you download these
    files, you’ll notice that they’re
    actually pretty small.
    

    View Slide

58. And if you download these
    files, you’ll notice that they’re
    actually pretty small.
    The reason is that these files
    basically only contain what
    inputs the user pressed and
    during what frame.
    

    View Slide

59. View Slide

60. View Slide

61. View Slide

62. View Slide

63. View Slide

64. View Slide

65. This is possible because of
    one thing.
    

    View Slide

66. This is possible because of
    one thing.
    Doom is deterministic
    

    View Slide

67. This is possible because of
    one thing.
    Doom is deterministic
    Doom is a single threaded
    application where every
    single entity in the map
    patiently waits their turn,
    always in the same order.
    

    View Slide

68. This is possible because of
    one thing.
    Doom is deterministic
    Doom is a single threaded
    application where every
    single entity in the map
    patiently waits their turn,
    always in the same order.
    

    View Slide

69. This is possible because of
    one thing.
    Doom is deterministic
    Doom is a single threaded
    application where every
    single entity in the map
    patiently waits their turn,
    always in the same order.
    And when they want to do
    something random, they
    extract a number from this
    table
    

    View Slide

70. This is possible because of
    one thing.
    Doom is deterministic
    Doom is a single threaded
    application where every
    single entity in the map
    patiently waits their turn,
    always in the same order.
    And when they want to do
    something random, they
    extract a number from this
    table
    Then an index is incremented
    and the next entity will get
    the next number.
    

    View Slide

71. So what if you wanted to
    take Doom and convert it to
    C++ (or some language)
    

    View Slide

72. So what if you wanted to
    take Doom and convert it to
    C++ (or some language)
    How can you move forwards
    with that project knowing
    that your code changes and
    refactorings haven’t broken
    the game?
    

    View Slide

73. So what if you wanted to
    take Doom and convert it to
    C++ (or some language)
    How can you move forwards
    with that project knowing
    that your code changes and
    refactorings haven’t broken
    the game?
    We could try playing the
    game, or writing unit tests.
    

    View Slide

74. So what if you wanted to
    take Doom and convert it to
    C++ (or some language)
    How can you move forwards
    with that project knowing
    that your code changes and
    refactorings haven’t broken
    the game?
    We could try playing the
    game, or writing unit tests.
    But we have the tool in our
    hands right now.
    

    View Slide

75. Approval testing is a
    methodology where you
    focus on comparing the
    output of the old known bit
    of code with the output of
    your new code.
    

    View Slide

76. Approval testing is a
    methodology where you
    focus on comparing the
    output of the old known bit
    of code with the output of
    your new code.
    Think of it like a git diff. If
    there is a difference, then
    something broke.
    

    View Slide

77. Approval testing is a
    methodology where you
    focus on comparing the
    output of the old known bit
    of code with the output of
    your new code.
    Think of it like a git diff. If
    there is a difference, then
    something broke.
    This is useful when you want
    to do a complete rewrite or
    when the original source
    might not be available in full.
    

    View Slide

78. Doom can run the old demo
    files and then give you a nice
    statistics screen at the end.
    

    View Slide

79. Doom can run the old demo
    files and then give you a nice
    statistics screen at the end.
    How long you played for,
    how many monsters you
    killed, how many items you
    got and how many secrets
    you found.
    

    View Slide

80. Doom can run the old demo
    files and then give you a nice
    statistics screen at the end.
    How long you played for,
    how many monsters you
    killed, how many items you
    got and how many secrets
    you found.
    Well, if I play this demo on
    the new version, I should get
    the exact same results.
    

    View Slide

81. Another tidbit. The game
    doesn’t need to render the
    game to the screen for the
    simulation to run.
    

    View Slide

82. Another tidbit. The game
    doesn’t need to render the
    game to the screen for the
    simulation to run.
    The graphics are just a nice
    visual representation of the
    game state.
    

    View Slide

83. Another tidbit. The game
    doesn’t need to render the
    game to the screen for the
    simulation to run.
    The graphics are just a nice
    visual representation of the
    game state.
    So we can add (and modify)
    some game flags where the
    game can run in the
    terminal.
    

    View Slide

84. Another tidbit. The game
    doesn’t need to render the
    game to the screen for the
    simulation to run.
    The graphics are just a nice
    visual representation of the
    game state.
    So we can add (and modify)
    some game flags where the
    game can run in the
    terminal.
    This also means we don’t
    need to run the game at a
    nice frame rate, we can just
    run it as fast as possible.
    

    View Slide

85. Another tidbit. The game
    doesn’t need to render the
    game to the screen for the
    simulation to run.
    The graphics are just a nice
    visual representation of the
    game state.
    So we can add (and modify)
    some game flags where the
    game can run in the
    terminal.
    This also means we don’t
    need to run the game at a
    nice frame rate, we can just
    run it as fast as possible.
    The simulation is the same.
    

    View Slide

86. Putting 1 and 1 together.
    We can play the game using
    a known good version of the
    game.
    

    View Slide

87. Putting 1 and 1 together.
    We can play the game using
    a known good version of the
    game.
    Collect the resulting Stats
    output.
    

    View Slide

88. Putting 1 and 1 together.
    We can play the game using
    a known good version of the
    game.
    Collect the resulting Stats
    output.
    Save the demo files and the
    stats text files in a folder.
    

    View Slide

89. Putting 1 and 1 together.
    We can play the game using
    a known good version of the
    game.
    Collect the resulting Stats
    output.
    Save the demo files and the
    stats text files in a folder.
    Run the game, in command
    line using those demos and
    check if the resulting stats
    data match the
    corresponding text file.
    

    View Slide

90. Doom running in
    GitHub Actions.
    

    View Slide

91. Doom running in
    GitHub Actions.
    

    View Slide

92. Doom running in
    GitHub Actions.
    

    View Slide

93. Doom running in
    GitHub Actions.
    

    View Slide

94. Level 2
    

    View Slide

95. Level 2
    What if the game you want to play
    doesn't have the support Doom does?
    

    View Slide

96. View Slide

97. Bejeweled 3 is a match three
    game.
    

    View Slide

98. Bejeweled 3 is a match three
    game.
    Pick a gem, move it left/right
    or up/down to create three
    of a kind.
    

    View Slide

99. Bejeweled 3 is a match three
    game.
    Pick a gem, move it left/right
    or up/down to create three
    of a kind.
    You can also join four or
    more gems for bonus points.
    

    View Slide

100. Bejeweled 3 is a match three
     game.
     Pick a gem, move it left/right
     or up/down to create three
     of a kind.
     You can also join four or
     more gems for bonus points.
     After a match has been
     made, those gems go away
     and new ones drop down.
     

     View Slide

101. For Bejeweled 3 the plan of
     action is pretty simple
     

     View Slide

102. For Bejeweled 3 the plan of
     action is pretty simple
     - What Gem is where
     

     View Slide

103. For Bejeweled 3 the plan of
     action is pretty simple
     - What Gem is where
     - Pick a good move
     

     View Slide

104. For Bejeweled 3 the plan of
     action is pretty simple
     - What Gem is where
     - Pick a good move
     - Move that gem
     

     View Slide

105. For Bejeweled 3 the plan of
     action is pretty simple
     - What Gem is where
     - Pick a good move
     - Move that gem
     - Repeat
     

     View Slide

106. The Project
     Project Setup:
     - C++
     - CMake
     - VCPKG
     - and OpenCV
     106
     

     View Slide

107. The Project
     Project Setup:
     - C++
     - CMake
     - VCPKG
     - and OpenCV
     107
     

     View Slide

108. Step 1
     What Gem is Where?
     

     View Slide

109. View Slide

110. 64px
     Step 1
     What Gem is Where?
     The game board is fixed
     resolution, which means
     each cell is 64 by 64 pixels.
     

     View Slide

111. w: 261px
     h: 37px
     Step 1
     What Gem is Where?
     The game board is fixed
     resolution, which means
     each cell is 64 by 64 pixels.
     That means that the board
     starts at 261x37 from the top
     left of the game screen.
     

     View Slide

112. Step 1
     What Gem is Where?
     The game board is fixed
     resolution, which means
     each cell is 64 by 64 pixels.
     That means that the board
     starts at 261x37 from the top
     left of the game screen.
     From that information, I
     know exactly where each cell
     on the board is.
     

     View Slide

113. Step 1
     What Gem is Where?
     But how do I know that this
     gem is the green one?
     

     View Slide

114. Step 1
     What Gem is Where?
     But how do I know that this
     gem is the green one?
     I started simple, let’s take the
     center pixel of that gem and
     check the RGB color values.
     

     View Slide

115. Step 1
     What Gem is Where?
     But how do I know that this
     gem is the green one?
     I started simple, let’s take the
     center pixel of that gem and
     check the RGB color values.
     But there’s a problem.
     

     View Slide

116. Step 1
     What Gem is Where?
     But how do I know that this
     gem is the green one?
     I started simple, let’s take the
     center pixel of that gem and
     check the RGB color values.
     But there’s a problem.
     

     View Slide

117. Step 1
     What Gem is Where?
     Let’s use OpenCV to our
     advantage.
     

     View Slide

118. Step 1
     What Gem is Where?
     Let’s use OpenCV to our
     advantage.
     

     View Slide

119. Step 1
     What Gem is Where?
     Let’s use OpenCV to our
     advantage.
     Using Template Matching
     we can look for our gems
     on the board.
     

     View Slide

120. Step 1
     What Gem is Where?
     Let’s use OpenCV to our
     advantage.
     Using Template Matching
     we can look for our gems
     on the board.
     I created these nice sample
     images of each gem.
     

     View Slide

121. View Slide

122. View Slide

123. View Slide

124. Step 1
     What Gem is Where?
     Let’s use OpenCV to our
     advantage.
     Using Template Matching
     we can look for our gems
     on the board.
     I created these nice sample
     images of each gem.
     So why doesn’t this work?
     

     View Slide

125. Step 1
     What Gem is Where?
     Let’s use OpenCV to our
     advantage.
     Using Template Matching
     we can look for our gems
     on the board.
     I created these nice sample
     images of each gem.
     So why doesn’t this work?
     Many feature detection
     algorithms’ first step is to
     make the source image
     B&W
     

     View Slide

126. Step 1
     What Gem is Where?
     Let’s go back to the idea of
     color.
     

     View Slide

127. Step 1
     What Gem is Where?
     Let’s go back to the idea of
     color.
     What if I don’t check for 1
     pixel in the center, but the
     general … hue … of the
     center area.
     

     View Slide

128. Step 1
     What Gem is Where?
     Let’s go back to the idea of
     color.
     What if I don’t check for 1
     pixel in the center, but the
     general … hue … of the
     center area.
     So I grab a 32x32 area in the
     center of the gem.
     

     View Slide

129. Step 1
     What Gem is Where?
     Let’s go back to the idea of
     color.
     What if I don’t check for 1
     pixel in the center, but the
     general … hue … of the
     center area.
     So I grab a 32x32 area in the
     center of the gem.
     And ask OpenCV what the
     average color of that area.
     

     View Slide

130. Step 1
     What Gem is Where?
     Let’s go back to the idea of
     color.
     What if I don’t check for 1
     pixel in the center, but the
     general … hue … of the
     center area.
     So I grab a 32x32 area in the
     center of the gem.
     And ask OpenCV what the
     average color of that area.
     And check against HSV and
     not RGB
     

     View Slide

131. Step 1
     What Gem is Where?
     Let’s go back to the idea of
     color.
     What if I don’t check for 1
     pixel in the center, but the
     general … hue … of the
     center area.
     So I grab a 32x32 area in the
     center of the gem.
     And ask OpenCV what the
     average color of that area.
     And check against HSV and
     not RGB
     

     View Slide

132. Step 1
     What Gem is Where?
     So I went through and
     checked the Hue value of
     every single gem (when
     stationary and while
     spinning).
     

     View Slide

133. Step 1
     What Gem is Where?
     So I went through and
     checked the Hue value of
     every single gem (when
     stationary and while
     spinning).
     And made a chart of where
     they overlap and where
     they are isolated.
     

     View Slide

134. Step 1
     What Gem is Where?
     So I went through and
     checked the Hue value of
     every single gem (when
     stationary and while
     spinning).
     And made a chart of where
     they overlap and where
     they are isolated.
     

     View Slide

135. Step 1
     What Gem is Where?
     So I went through and
     checked the Hue value of
     every single gem (when
     stationary and while
     spinning).
     And made a chart of where
     they overlap and where
     they are isolated.
     And then wrote a function
     that tries to disambiguate
     between the colors (and I
     used RGB to help also)
     

     View Slide

136. Step 1
     What Gem is Where?
     So I went through and
     checked the Hue value of
     every single gem (when
     stationary and while
     spinning).
     And made a chart of where
     they overlap and where
     they are isolated.
     And then wrote a function
     that tries to disambiguate
     between the colors (and I
     used RGB to help also)
     

     View Slide

137. Step 1
     What Gem is Where?
     So I went through and
     checked the Hue value of
     every single gem (when
     stationary and while
     spinning).
     And made a chart of where
     they overlap and where
     they are isolated.
     And then wrote a function
     that tries to disambiguate
     between the colors (and I
     used RGB to help also)
     

     View Slide

138. Step 1
     What Gem is Where?
     So I went through and
     checked the Hue value of
     every single gem (when
     stationary and while
     spinning).
     And made a chart of where
     they overlap and where
     they are isolated.
     And then wrote a function
     that tries to disambiguate
     between the colors (and I
     used RGB to help also)
     

     View Slide

139. Step 1
     What Gem is Where?
     So I went through and
     checked the Hue value of
     every single gem (when
     stationary and while
     spinning).
     And made a chart of where
     they overlap and where
     they are isolated.
     And then wrote a function
     that tries to disambiguate
     between the colors (and I
     used RGB to help also)
     

     View Slide

140. View Slide

141. Step 2
     Pick a good move.
     

     View Slide

142. Step 2
     Pick a good move.
     So what is a good move?
     

     View Slide

143. Step 2
     Pick a good move.
     So what is a good move?
     Initially I was thinking of
     writing some cool algorithm,
     get some deep learning AI
     into the mix.
     

     View Slide

144. Step 2
     Pick a good move.
     So what is a good move?
     Initially I was thinking of
     writing some cool algorithm,
     get some deep learning AI
     into the mix.
     But after the color fiasko, I
     wanted to see if something
     simple was enough.
     

     View Slide

145. Step 2
     Pick a good move.
     So what is a good move?
     Initially I was thinking of
     writing some cool algorithm,
     get some deep learning AI
     into the mix.
     But after the color fiasko, I
     wanted to see if something
     simple was enough.
     And it was.
     

     View Slide

146. Step 2
     Pick a good move.
     Let’s start by playing the
     game a bit and see if
     anything pops out.
     

     View Slide

147. Step 2
     Pick a good move.
     Let’s start by playing the
     game a bit and see if
     anything pops out.
     So I played the game a little
     bit. Just a bit.
     

     View Slide

148. Step 2
     Pick a good move.
     Let’s start by playing the
     game a bit and see if
     anything pops out.
     So I played the game a little
     bit. Just a bit.
     And I had an idea.
     

     View Slide

149. Step 2
     Pick a good move.
     Let’s start by playing the
     game a bit and see if
     anything pops out.
     So I played the game a little
     bit. Just a bit.
     And I had an idea.
     There aren’t actually that
     many moves in this game.
     

     View Slide

150. Step 2
     Pick a good move.
     Let’s start by playing the
     game a bit and see if
     anything pops out.
     So I played the game a little
     bit. Just a bit.
     And I had an idea.
     There aren’t actually that
     many moves in this game.
     Can we just map them out?
     

     View Slide

151. View Slide

152. View Slide

153. View Slide

154. View Slide

155. View Slide

156. View Slide

157. View Slide

158. View Slide

159. View Slide

160. View Slide

161. Step 2
     Pick a good move.
     But which move should we
     pick?
     

     View Slide

162. Step 2
     Pick a good move.
     But which move should we
     pick?
     Well a high value move is
     always sorted above other
     moves. But what if all moves
     have the same value?
     

     View Slide

163. Step 2
     Pick a good move.
     But which move should we
     pick?
     Well a high value move is
     always sorted above other
     moves. But what if all moves
     have the same value?
     Then we pick a move that is
     lower on the board. Why?
     

     View Slide

164. Step 2
     Pick a good move.
     But which move should we
     pick?
     Well a high value move is
     always sorted above other
     moves. But what if all moves
     have the same value?
     Then we pick a move that is
     lower on the board. Why?
     Because it creates more
     matching opportunities
     when tiles fall down. A move
     at the top only creates one.
     

     View Slide

165. Step 3
     Move that gem.
     Now it’s time to interact with
     the game application
     directly.
     

     View Slide

166. Step 3
     Move that gem.
     Now it’s time to interact with
     the game application
     directly.
     The Windows SDK has a lot
     of nice mouse input
     functions. So let’s use them.
     

     View Slide

167. Step 3
     Move that gem.
     Now it’s time to interact with
     the game application
     directly.
     The Windows SDK has a lot
     of nice mouse input
     functions. So let’s use them.
     I basically want a function
     like this. I know where the
     gem is and I know what
     direction I want to move to.
     

     View Slide

168. View Slide

169. View Slide

170. View Slide

171. View Slide

172. View Slide

173. View Slide

174. View Slide

175. View Slide

176. View Slide

177. View Slide

178. View Slide

179. Bejeweled 3 DEMO
     

     View Slide

180. Level 3
     

     View Slide

181. Level 3
     What if a game has none of these
     features?
     

     View Slide

182. View Slide

183. View Slide

184. VVVVVV
     Does not have built in
     support for giving the game
     the inputs.
     

     View Slide

185. VVVVVV
     Does not have built in
     support for giving the game
     the inputs.
     Does not have a simple
     game state.
     

     View Slide

186. VVVVVV
     Does not have built in
     support for giving the game
     the inputs.
     Does not have a simple
     game state.
     Is fast paced and you need
     to be pretty accurate with
     your movement.
     

     View Slide

187. View Slide

188. View Slide

189. Before I continue with the
     madness. There is one
     concept I need to talk about.
     

     View Slide

190. Before I continue with the
     madness. There is one
     concept I need to talk about.
     The Game Loop
     

     View Slide

191. Before I continue with the
     madness. There is one
     concept I need to talk about.
     The Game Loop
     Games are applications like
     any other, and they usually
     follow a certain set of steps.
     

     View Slide

192. Before I continue with the
     madness. There is one
     concept I need to talk about.
     The Game Loop
     Games are applications like
     any other, and they usually
     follow a certain set of steps.
     - Get input from user
     

     View Slide

193. Before I continue with the
     madness. There is one
     concept I need to talk about.
     The Game Loop
     Games are applications like
     any other, and they usually
     follow a certain set of steps.
     - Get input from user
     - Update the state
     

     View Slide

194. Before I continue with the
     madness. There is one
     concept I need to talk about.
     The Game Loop
     Games are applications like
     any other, and they usually
     follow a certain set of steps.
     - Get input from user
     - Update the state
     - Draw the current state
     

     View Slide

195. There are a few caveats with
     these steps that makes our
     work harder.
     

     View Slide

196. There are a few caveats with
     these steps that makes our
     work harder.
     Get input from user
     There might be no input or if
     we handle it badly, we might
     delay it or miss it.
     

     View Slide

197. There are a few caveats with
     these steps that makes our
     work harder.
     Get input from user
     There might be no input or if
     we handle it badly, we might
     delay it or miss it.
     Update the state
     This can take a long time and
     we might multi-thread it.
     

     View Slide

198. There are a few caveats with
     these steps that makes our
     work harder.
     Get input from user
     There might be no input or if
     we handle it badly, we might
     delay it or miss it.
     Update the state
     This can take a long time and
     we might multi-thread it.
     Draw the current state
     We might draw old state or we
     might offload the draw to
     happen in another thread.
     

     View Slide

199. 199
     Initialize
     

     View Slide

200. 200
     Initialize
     Process input
     

     View Slide

201. 201
     Initialize
     Process input
     Update game
     state
     

     View Slide

202. 202
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     

     View Slide

203. 203
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     

     View Slide

204. 204
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     “Frame”
     

     View Slide

205. 205
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     <1ms
     “Frame”
     

     View Slide

206. 206
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     <1ms
     “Frame” Depends
     

     View Slide

207. 207
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     <1ms
     “Frame” Depends
     Depends
     

     View Slide

208. 208
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     “Frame”
     60 fps game
     16.666… ms
     

     View Slide

209. 209
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     

     View Slide

210. 210
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     At least once
     per 16.666 ms
     

     View Slide

211. 211
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     *As often as we can
     At least once
     per 16.666 ms
     

     View Slide

212. View Slide

213. View Slide

214. View Slide

215. View Slide

216. View Slide

217. View Slide

218. View Slide

219. View Slide

220. View Slide

221. View Slide

222. View Slide

223. View Slide

224. VVVVVV actually has a few
     advantages and is why I picked
     this game for the project.
     

     View Slide

225. VVVVVV actually has a few
     advantages and is why I picked
     this game for the project.
     It’s open source, so we can just
     go check.
     

     View Slide

226. VVVVVV actually has a few
     advantages and is why I picked
     this game for the project.
     It’s open source, so we can just
     go check.
     It’s single threaded.
     

     View Slide

227. VVVVVV actually has a few
     advantages and is why I picked
     this game for the project.
     It’s open source, so we can just
     go check.
     It’s single threaded.
     It has a lot of customization for
     such a simple game.
     

     View Slide

228. So here is my plan Loader
     Target process
     dll
     

     View Slide

229. So here is my plan
     Launch the game as a paused
     child process of some loader.
     Loader
     

     View Slide

230. So here is my plan
     Launch the game as a paused
     child process of some loader.
     Loader
     Game Process
     

     View Slide

231. So here is my plan
     Launch the game as a paused
     child process of some loader.
     Loader
     Game Process
     dll
     

     View Slide

232. So here is my plan
     Launch the game as a paused
     child process of some loader.
     Hook the function that creates
     the DirectX Device.
     Loader
     Game Process
     dll
     

     View Slide

233. So here is my plan
     Launch the game as a paused
     child process of some loader.
     Hook the function that creates
     the DirectX Device.
     Get the address of that device.
     Loader
     Game Process
     dll
     

     View Slide

234. So here is my plan
     Launch the game as a paused
     child process of some loader.
     Hook the function that creates
     the DirectX Device.
     Get the address of that device.
     Intercept the game’s draw call
     and call our draw function first.
     Loader
     Game Process
     dll
     

     View Slide

235. So here is my plan
     Launch the game as a paused
     child process of some loader.
     Hook the function that creates
     the DirectX Device.
     Get the address of that device.
     Intercept the game’s draw call
     and call our draw function first.
     Before each frame, send our
     keyboard inputs.
     Loader
     Game Process
     dll
     

     View Slide

236. 236
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     

     View Slide

237. 237
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     

     View Slide

238. 238
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     Intercept
     - Call game’s draw function
     - Send our inputs
     

     View Slide

239. 239
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     We could hook the input polling
     function.
     

     View Slide

240. 240
     Initialize
     Process input
     Update game
     state
     Draw
     game state
     We could hook the input polling
     function.
     But we actually want to do a
     bit of drawing ourselves, to
     make a nice debug UI.
     

     View Slide

241. View Slide

242. View Slide

243. View Slide

244. View Slide

245. View Slide

246. View Slide

247. View Slide

248. View Slide

249. View Slide

250. View Slide

251. View Slide

252. View Slide

253. View Slide

254. View Slide

255. View Slide

256. View Slide

257. View Slide

258. View Slide

259. View Slide

260. View Slide

261. View Slide

262. View Slide

263. View Slide

264. View Slide

265. View Slide

266. View Slide

267. View Slide

268. View Slide

269. View Slide

270. View Slide

271. View Slide

272. A summary of what we’ve
     done so far. Loader
     Game Process
     dll
     

     View Slide

273. A summary of what we’ve
     done so far.
     ✅ Spawn the game process
     paused.
     Loader
     Game Process
     dll
     

     View Slide

274. A summary of what we’ve
     done so far.
     ✅ Spawn the game process
     paused.
     ✅ Attach our DLL to the game
     process.
     Loader
     Game Process
     dll
     

     View Slide

275. A summary of what we’ve
     done so far.
     ✅ Spawn the game process
     paused.
     ✅ Attach our DLL to the game
     process.
     ✅ Hook the “CreateDevice”
     function the game is going to
     call once it starts.
     Loader
     Game Process
     dll
     

     View Slide

276. A summary of what we’ve
     done so far.
     ✅ Spawn the game process
     paused.
     ✅ Attach our DLL to the game
     process.
     ✅ Hook the “CreateDevice”
     function the game is going to
     call once it starts.
     Once the game calls the
     “CreateDevice” function, get the
     device it just made and hook
     end “EndScene” function.
     Loader
     Game Process
     dll
     

     View Slide

277. View Slide

278. View Slide

279. View Slide

280. View Slide

281. View Slide

282. View Slide

283. View Slide

284. View Slide

285. View Slide

286. View Slide

287. View Slide

288. View Slide

289. View Slide

290. View Slide

291. VVVVVV DEMO
     

     View Slide

292. Summary:
     Games are neat and it’s fun to
     play with them
     programmatically.
     Loader
     Game Process
     dll
     

     View Slide

293. Summary:
     Games are neat and it’s fun to
     play with them
     programmatically.
     When you have an application
     you don’t have the source code
     to, there are still ways to
     automatically interact with it.
     Loader
     Game Process
     dll
     

     View Slide

294. Summary:
     Games are neat and it’s fun to
     play with them
     programmatically.
     When you have an application
     you don’t have the source code
     to, there are still ways to
     automatically interact with it.
     AKA. Yes, you can still test that
     old legacy app. Yes you can still
     do safe(er) refactoring.
     Loader
     Game Process
     dll
     

     View Slide

295. Summary:
     Games are neat and it’s fun to
     play with them
     programmatically.
     When you have an application
     you don’t have the source code
     to, there are still ways to
     automatically interact with it.
     AKA. Yes, you can still test that
     old legacy app. Yes you can still
     do safe(er) refactoring.
     Write more tests!
     Loader
     Game Process
     dll
     

     View Slide

296. TASVideos.org
     Loader
     Game Process
     dll
     

     View Slide

297. TASVideos.org
     Loader
     Game Process
     dll
     

     View Slide

298. dsdarchive.com
     Loader
     Game Process
     dll
     

     View Slide

299. www.classicdoom.com/odddemos.htm
     Loader
     Game Process
     dll
     

     View Slide

300. VVVVVV
     https://github.com/olafurw/vvvvvv-hook/
     Bejeweled 3
     https://github.com/olafurw/bejeweled-player
     Doom
     https://github.com/patricia-gallardo/cpp-doom
     Loader
     Game Process
     dll
     

     View Slide

301. Ólafur Waage
     Senior Software Developer - Factlines
     @olafurw on most social platforms
     1
     301
     

     View Slide