Haskell SpriteKit — A Purely Functional API for a Stateful Animation System & Physics Engine

VIDEOS:
* https://www.youtube.com/watch?v=kd8mlbN0Mws (Functional Conf 2017, best choice right now)
* https://www.youtube.com/watch?v=H_z4NKvxf1U (Curry On 2017, more detail than YOW! at the end, but slides out of sync for first 1.5min)
* https://www.youtube.com/watch?v=GaorHAlUkVs (YOW! Lambda Jam 2017, slightly shorter)

Paper with additional details: http://www.cse.unsw.edu.au/%7Echak/papers/CK17.html

Graphics, animation, and games programming in Haskell faces a dilemma. We can either use existing frameworks with their highly imperative APIs (such as OpenGL, Cocos2D, or SpriteKit) or we waste a lot of energy trying to re-engineer those rather complex systems from scratch. Or, maybe, we can escape the dilemma. Instead of a Haskell program directly manipulating the mutable object-graph of existing high-level frameworks, we provide an API for purely functional transformations of a Haskell data structure, together with an adaptation layer that transcribes those transformations into edits of the mutable object-graph.

I this talk, I will explain how I used this approach to architect a Haskell binding to the animation system and physics engine of Apple’s SpriteKit framework. I will discuss both how the binding is structured internally and how it achieves the translation of Haskell side changes to SpriteKit and vice versa, such that it is sufficiently efficient. Moreover, I will explain how to use the Haskell library to implement animations and games.

This talk is aimed at an intermediate level. It will use Haskell and SpriteKit as concrete technology in all examples, but it requires neither previous experience with Haskell nor with SpriteKit. In a similar vain, the talk is based on Apple’s SpriteKit framework, as there is no reasonable (in terms of effort) way around platform-specific technology if you want something that is simultaneously state-of-the-art, practically useful, and convenient to use. Nevertheless, the concepts explained in the talk transcend the specific technology and are generally applicable. Moreover, the entire code base of the Haskell binding is open source and available from https://github.com/mchakravarty/HaskellSpriteKit

Presented at
* YOW! Lambda Jam, Sydney, May 2017: http://lambdajam.yowconference.com.au/speakers/manuel-chakravarty-4/
* IFIP WG2.8, Edinburgh, June 2017
* Curry On, Barcelona, June 2017: http://www.curry-on.org/2017/sessions/haskell-spritekit-a-case-study-in-turning-a-stateful-into-a-purely-functional-api.html
* Functional Conf, Bangalore, November 2017: https://functionalconf.com/proposal.html?id=3939

Transcript

1. Manuel M T Chakravarty & Gabriele Keller Applicative & UNSW Sydney Haskell SpriteKit A Purely Functional API for a Stateful Animation System & Physics Engine mchakravarty TacticalGrace justtesting.org haskellformac.com
2. The Dilemma
3. The Dilemma
4. The Dilemma SKShapeNode *shipOverlayShape = [[SKShapeNode alloc] init]; shipOverlayShape.path = boundingPath; shipOverlayShape.strokeColor = [SKColor clearColor]; shipOverlayShape.fillColor = [SKColor colorWithRed:0.0 green:1.0 blue:0.0 alpha:0.5]; [ship addChild:shipOverlayShape];
5. The Dilemma SKShapeNode *shipOverlayShape = [[SKShapeNode alloc] init]; shipOverlayShape.path = boundingPath; shipOverlayShape.strokeColor = [SKColor clearColor]; shipOverlayShape.fillColor = [SKColor colorWithRed:0.0 green:1.0 blue:0.0 alpha:0.5]; [ship addChild:shipOverlayShape];
6. The Dilemma SKShapeNode *shipOverlayShape = [[SKShapeNode alloc] init]; shipOverlayShape.path = boundingPath; shipOverlayShape.strokeColor = [SKColor clearColor]; shipOverlayShape.fillColor = [SKColor colorWithRed:0.0 green:1.0 blue:0.0 alpha:0.5]; [ship addChild:shipOverlayShape]; data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] }
7. The Compromise Purely Functional API data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] }
8. The Compromise Purely Functional API data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } functional imperative object-oriented
9. The Compromise Purely Functional API represent scenes with algebraic data types data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } functional imperative object-oriented
10. The Compromise Purely Functional API translate transformations into state changes represent scenes with algebraic data types data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } functional imperative object-oriented
11. Purely Functional API data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } functional imperative object-oriented
12. Purely Functional API data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } functional imperative object-oriented Step ❶ — state of the art
13. Purely Functional API data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } functional imperative object-oriented Step ❶ — state of the art Step ❷ — purification
14. Purely Functional API data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } functional imperative object-oriented Step ❶ — state of the art Step ❷ — purification Step ❸ — efficiency
15. SpriteKit Physics, fields & particles Animation & lighting Joints & constraints
16. SpriteKit Physics, fields & particles Animation & lighting Joints & constraints Lazy Lambda Simple side scroller
17. Step ❶ The State of the Art
22. 0 pipes moving nodes pipe pair l ground physics score contact l l
23. — node without visuals 0 pipes moving nodes pipe pair l ground physics score contact l l
24. — node without visuals — attached actions 0 pipes moving nodes pipe pair l ground physics score contact l l
25. — node without visuals — attached actions — attached physics body 0 pipes moving nodes pipe pair l ground physics score contact l l
26. — node without visuals — attached actions — attached physics body 0 pipes moving nodes pipe pair l ground physics score contact l l
27. 0 pipes moving nodes pipe pair l ground physics score contact l l SKScene SKLabelNode SKNode SKNode SKSpriteNode SKNode SKNode SKNode SKSpriteNode SKSpriteNode SKSpriteNode SKSpriteNode
28. SKLabelNode SKScene SKNode SKSpriteNode SKEffectNode OOism #1: Subclassing
29. SKLabelNode SKScene SKNode SKSpriteNode SKEffectNode SKShapeNode SKLightNode ộ OOism #1: Subclassing
30. SKLabelNode SKScene SKNode SKSpriteNode SKEffectNode SKShapeNode SKLightNode ộ OOism #1: Subclassing class SKNode: … { var frame: CGRect func calculatedFrame() -> CGRect var position: CGPoint var zRotation: CGFloat var xScale: CGFloat ⋮ }
31. SKLabelNode SKScene SKNode SKSpriteNode SKEffectNode SKShapeNode SKLightNode ộ OOism #1: Subclassing this one is odd class SKNode: … { var frame: CGRect func calculatedFrame() -> CGRect var position: CGPoint var zRotation: CGFloat var xScale: CGFloat ⋮ }
32. SKLabelNode SKScene SKNode SKSpriteNode SKEffectNode SKShapeNode SKLightNode ộ NSResponder LambdaScene
33. SKLabelNode SKScene SKNode SKSpriteNode SKEffectNode SKShapeNode SKLightNode ộ NSResponder LambdaScene class NSResponder: NSObject { func mouseDown(with: NSEvent) func mouseUp(with: NSEvent) func keyDown(with: NSEvent) func keyUp(with: NSEvent) ⋮ }
34. SKLabelNode SKScene SKNode SKSpriteNode SKEffectNode SKShapeNode SKLightNode ộ NSResponder LambdaScene class NSResponder: NSObject { func mouseDown(with: NSEvent) func mouseUp(with: NSEvent) func keyDown(with: NSEvent) func keyUp(with: NSEvent) ⋮ } user-defined subclass to override event handlers
35. OOism #2: Mutable Properties
36. OOism #2: Mutable Properties Lambda tilts (rotation) depending on vertical velocity
37. OOism #2: Mutable Properties class SKNode: … { var frame: CGRect var position: CGPoint var zRotation: CGFloat var xScale: CGFloat ⋮ } Lambda tilts (rotation) depending on vertical velocity mutation
39. 0 pipes moving nodes pipe pair l ground physics score contact l l
40. 0 pipes moving nodes pipe pair l ground physics score contact l l class SKScene: SKEffectNode { var backgroundColor: NSColor func update(_ currentTime: TimeInterval) ⋮ }
41. 0 pipes moving nodes pipe pair l ground physics score contact l l class SKScene: SKEffectNode { var backgroundColor: NSColor func update(_ currentTime: TimeInterval) ⋮ } called once per frame
42. OOism #3: In-place Graph Edits
43. OOism #3: In-place Graph Edits Pipe pairs appear and disappear at regular intervals
44. OOism #3: In-place Graph Edits Pipe pairs appear and disappear at regular intervals class SKNode: … { var children: [SKNode] { get } func addChild(_ node: SKNode) ⋮ } mutation
45. 0 pipes moving nodes ground physics l l pipe pair score contact
46. 0 pipes moving nodes ground physics l l pipe pair score contact pipe pair score contact
47. 0 pipes moving nodes ground physics l l pipe pair score contact pipe pair score contact pipe pair score contact
48. 0 pipes moving nodes ground physics l l pipe pair score contact pipe pair score contact
49. The Three Issues
50. The Three Issues Subclassing Inherited properties & event handlers
51. The Three Issues Subclassing Inherited properties & event handlers Mutable properties Inplace updates by callbacks
52. The Three Issues Subclassing Inherited properties & event handlers Mutable properties Inplace updates by callbacks Inplace graph edits Add, delete & rearrange nodes
53. Step ❷ Pure Functions & Datatypes
54. Subclassing Inherited properties & event handlers
55. SKLabelNode SKScene SKNode SKSpriteNode SKEffectNode SKShapeNode SKLightNode ộ
56. SKLabelNode SKScene SKNode SKSpriteNode SKEffectNode SKShapeNode SKLightNode ộ
57. SKNode SKLabelNode SKSpriteNode SKShapeNode
58. data Node u = Node { nodeName :: Maybe String , nodePosition :: Point , nodeZRotation :: GFloat , nodeUserData :: u ⋮ } | Label { nodeName :: Maybe String , nodePosition :: Point , nodeZRotation :: GFloat , nodeUserData :: u ⋮ , labelText :: String ⋮ } | Sprite { … } | Shape { … } ⋮
59. data Node u = Node { nodeName :: Maybe String , nodePosition :: Point , nodeZRotation :: GFloat , nodeUserData :: u ⋮ } | Label { nodeName :: Maybe String , nodePosition :: Point , nodeZRotation :: GFloat , nodeUserData :: u ⋮ , labelText :: String ⋮ } | Sprite { … } | Shape { … } ⋮ shared fields of ”superclass” node included in each variant
60. data Node u = Node { nodeName :: Maybe String , nodePosition :: Point , nodeZRotation :: GFloat , nodeUserData :: u ⋮ } | Label { nodeName :: Maybe String , nodePosition :: Point , nodeZRotation :: GFloat , nodeUserData :: u ⋮ , labelText :: String ⋮ } | Sprite { … } | Shape { … } ⋮ shared fields of ”superclass” node included in each variant each variant (except Node) has specific fields
61. data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] , sceneData :: sceneData , sceneBackgroundColor :: Color ⋮ }
62. data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] , sceneData :: sceneData , sceneBackgroundColor :: Color ⋮ } children of a Scene are all Nodes
63. data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] , sceneData :: sceneData , sceneBackgroundColor :: Color ⋮ } children of a Scene are all Nodes the Scene and the Nodes may contain extra user-defined data
65. Mutable properties Inplace updates by callbacks
66. data Node u = Node { … } | Sprite { nodeName :: Maybe String , nodeZRotation :: GFloat ⋮ } ⋮ data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] ⋮ }
67. data Node u = Node { … } | Sprite { nodeName :: Maybe String , nodeZRotation :: GFloat ⋮ } ⋮ data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] ⋮ } immutable, but must change to tilt the bird
68. data Node u = Node { … } | Sprite { nodeName :: Maybe String , nodeZRotation :: GFloat ⋮ } ⋮ data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] , sceneUpdate :: Maybe (SceneUpdate sceneData nodeData) ⋮ } immutable, but must change to tilt the bird
69. data Node u = Node { … } | Sprite { nodeName :: Maybe String , nodeZRotation :: GFloat ⋮ } ⋮ data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] , sceneUpdate :: Maybe (SceneUpdate sceneData nodeData) ⋮ } immutable, but must change to tilt the bird callback called once per frame
70. data Node u = Node { … } | Sprite { nodeName :: Maybe String , nodeZRotation :: GFloat ⋮ } ⋮ data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] , sceneUpdate :: Maybe (SceneUpdate sceneData nodeData) ⋮ } immutable, but must change to tilt the bird type SceneUpdate sd nd = Scene sd nd -> TimeInterval -> Scene sd nd callback called once per frame pure scene transformer
71. data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] , sceneUpdate :: Maybe (SceneUpdate sceneData nodeData) ⋮ } type SceneUpdate sd nd = Scene sd nd -> TimeInterval -> Scene sd nd
72. data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] , sceneUpdate :: Maybe (SceneUpdate sceneData nodeData) , sceneHandleEvent :: Maybe (EventHandler sceneData) ⋮ } type SceneUpdate sd nd = Scene sd nd -> TimeInterval -> Scene sd nd type EventHandler sceneData = Event -> sceneData -> Maybe sceneData
73. data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] , sceneUpdate :: Maybe (SceneUpdate sceneData nodeData) , sceneHandleEvent :: Maybe (EventHandler sceneData) ⋮ } type SceneUpdate sd nd = Scene sd nd -> TimeInterval -> Scene sd nd type EventHandler sceneData = Event -> sceneData -> Maybe sceneData transforms game state in response to input events
75. Inplace graph edits Add, delete & rearrange nodes
76. data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] , sceneData :: sceneData ⋮ } data Node u = Node { … , nodeChildren :: [Node u] ⋮ } | Label { … , nodeChildren :: [Node u] ⋮ } ⋮
77. data Scene sceneData nodeData = Scene { sceneName :: Maybe String , sceneChildren :: [Node nodeData] , sceneData :: sceneData ⋮ } data Node u = Node { … , nodeChildren :: [Node u] ⋮ } | Label { … , nodeChildren :: [Node u] ⋮ } ⋮ these fields describe the scene tree
79. Step ❸ Hidden State & Efficient Marshalling
80. 0 pip movin groun l l pip score
81. 0 pip movin groun l l pip score ObjC -> Haskell
82. 0 pip movin groun l l pip score spawnPipePair ObjC -> Haskell
83. 0 pip movin groun l l pip score 0 pi movin groun l l pip score pip score spawnPipePair new subtree ObjC -> Haskell Haskell -> ObjC
84. 0 pip movin groun l l pip score 0 pi movin groun l l pip score pip score spawnPipePair new subtree old subtree ObjC -> Haskell Haskell -> ObjC
85. Eager Marshalling Is Infeasible
86. Eager Marshalling Is Infeasible Hidden State Nodes have private state that would get lost
87. Eager Marshalling Is Infeasible Hidden State Nodes have private state that would get lost Efficiency Marshalling a whole tree to change a few properties
88. data Scene sc nd = Scene { … } — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame 0 moving nodes ground physics l l l
89. data Scene sc nd = Scene { … } — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling 0 moving nodes ground physics l l l
90. data Scene sc nd = Scene { … } — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling SceneUpdate sc nd Scene sc nd 0 moving nodes ground physics l l l
91. data Scene sc nd = Scene { … } — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling SceneUpdate sc nd Scene sc nd compute diff 0 moving nodes ground physics l l l
92. data Scene sc nd = Scene { … } — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling SceneUpdate sc nd Scene sc nd compute diff apply changes 0 moving nodes ground physics l l l
93. Lazy Marshalling 0 moving nodes ground physics l l l
94. Lazy Marshalling 0 moving nodes ground physics l l l lazy marshalling Scene { sceneName = , sceneChildren = , sceneData = , sceneBackgroundColor = ⋮ }
95. Lazy Marshalling 0 moving nodes ground physics l l l lazy marshalling Scene { sceneName = , sceneChildren = , sceneData = , sceneBackgroundColor = ⋮ } thunks (suspended computations)
96. Lazy Marshalling 0 moving nodes ground physics l l l lazy marshalling Scene { sceneName = , sceneChildren = , sceneData = , sceneBackgroundColor = ⋮ } thunks (suspended computations)
97. Lazy Marshalling marshalSKScene :: SKScene -> Scene sceneData nodeData marshalSKScene skScene = Scene { sceneName = unsafePerformIO $(objc … ) , sceneChildren = unsafePerformIO $ do { nodes <- $(objc … ) ; unsafeInterleaveNSArrayTolistOfNode nodes } ⋮ }
98. Lazy Marshalling marshalSKScene :: SKScene -> Scene sceneData nodeData marshalSKScene skScene = Scene { sceneName = unsafePerformIO $(objc … ) , sceneChildren = unsafePerformIO $ do { nodes <- $(objc … ) ; unsafeInterleaveNSArrayTolistOfNode nodes } ⋮ } produces a thunk (unevaluated expression)
99. Lazy Marshalling marshalSKScene :: SKScene -> Scene sceneData nodeData marshalSKScene skScene = Scene { sceneName = unsafePerformIO $(objc … ) , sceneChildren = unsafePerformIO $ do { nodes <- $(objc … ) ; unsafeInterleaveNSArrayTolistOfNode nodes } ⋮ } produces a thunk (unevaluated expression) inline Objective-C
100. 0 moving nodes ground physics l l l Scene { sceneName = , sceneChildren = , sceneData = , sceneBackgroundColor = ⋮ } Compute Diff & Update
101. 0 moving nodes ground physics l l l Scene { sceneName = , sceneChildren = , sceneData = , sceneBackgroundColor = ⋮ } Compute Diff & Update Scene { sceneName = ”New Name” , sceneChildren = , sceneData = , sceneBackgroundColor = ⋮ } SceneUpdate sc nd
102. 0 moving nodes ground physics l l l Scene { sceneName = , sceneChildren = , sceneData = , sceneBackgroundColor = ⋮ } Compute Diff & Update Scene { sceneName = ”New Name” , sceneChildren = , sceneData = , sceneBackgroundColor = ⋮ } SceneUpdate sc nd compute diff
103. 0 moving nodes ground physics l l l Scene { sceneName = , sceneChildren = , sceneData = , sceneBackgroundColor = ⋮ } Compute Diff & Update Scene { sceneName = ”New Name” , sceneChildren = , sceneData = , sceneBackgroundColor = ⋮ } SceneUpdate sc nd compute diff
104. Compute Diff & Update marshalScene :: Scene sceneData nodeData — original scene -> Scene sceneData nodeData — updated scene -> IO SKScene marshalScene originalScene Scene{..} = do { … ; case reallyUnsafePtrEquality# originalName sceneName of 1# -> return () _ -> $(objc …) ; updateChildren skNode originalChildren sceneChildren ⋮ }
105. Compute Diff & Update marshalScene :: Scene sceneData nodeData — original scene -> Scene sceneData nodeData — updated scene -> IO SKScene marshalScene originalScene Scene{..} = do { … ; case reallyUnsafePtrEquality# originalName sceneName of 1# -> return () _ -> $(objc …) ; updateChildren skNode originalChildren sceneChildren ⋮ }
106. Object Caching & Identity
107. Object Caching & Identity data Node u = Node { nodeName :: Maybe String , nodePosition :: Point ⋮ , nodeForeign :: Maybe SKNode } | Label { nodeName :: Maybe String , nodePosition :: Point ⋮ , nodeForeign :: Maybe SKNode , labelText :: String ⋮ } ⋮ if available, basis for marshalling this node to ObjC land
108. Haskell SpriteKit is open source! https://github.com/mchakravarty/HaskellSpriteKit mchakravarty TacticalGrace justtesting.org haskellformac.com >< state types Want to see the gory details?
109. Thank you!
110. Image Attribution https://pixabay.com/photo-51675/ https://pixabay.com/photo-509871/ https://pixabay.com/photo-1294844/