Playing with Graphics and Animations in Haskell

TALK VIDEO: https://www.youtube.com/watch?v=9dk7_GDNocQ (some code is hard to read in the video, you may like to refer to these slides for that)

NB: The talk includes live coding. In this version of the slides, these sections have been replaced by links to screencasts of the live coding sessions on Vimeo as Speaker Deck does not support embedded video.

Graphics and animations are not only fun, they are also an effective learning tool. For example, fractal tree structures nicely illustrate recursion, and animations are often more motivating and engaging than examples and exercises spewing out text. In this talk, I will demonstrate graphics and animations in Haskell playgrounds — aka the REPL on steroids. The immediate feedback of playgrounds facilitates an explorative style of programming that is especially attractive for graphics programming.

Graphics, animation, and games programming in Haskell faces a dilemma. We can either use existing frameworks with their highly imperative APIs or we waste a lot of energy trying to re-engineer those rather complex systems from scratch. Or, maybe, we can escape the dilemma. Instead of a Haskell program directly manipulating the mutable object-graph of existing frameworks, we provide an API for purely functional transformations of a Haskell data structure, together with an adaptation layer that transcribes those transformations into edits of the mutable object-graph. I will illustrate this approach by describing the architecture of a Haskell binding to the animation system of Apple’s SpriteKit framework.

This talk was the opening keynote of Compose :: Melbourne: http://www.composeconference.org/2016-melbourne/day-one-program/

The code of the Lazy Lambda game is open source at https://github.com/mchakravarty/lazy-lambda and the code for the Haskell SpriteKit binding is at https://github.com/mchakravarty/HaskellSpriteKit

Transcript

1. Manuel M T Chakravarty Applicative & UNSW Australia Playing with Graphics and Animations in Haskell mchakravarty TacticalGrace justtesting.org haskellformac.com 1 Thursday, 1 September 16 45 minute time slot (inclusive Q&A) [5min from REPL to Playgrounds; 10min Pythagorean trees; 7min SpriteKit & Haskell; 15min Lazy Lambda game] » Haskell is a great language for graphics, but …
2. Our Legacy Problem The Status Quo 2 Thursday, 1 September 16 » …we don’t make use of it — largely, because the tool chain makes it hard…
3. 3 Thursday, 1 September 16 * Who has used Haskell at all? Who is confident in using Haskell? * The REPL has served us well since Lisp. * but it also hasn’t changed much in that time. » Which leads to the question…
4. The Venerable REPL Read Eval Print Loop Introduced with Lisp. Remained essentially unchanged. 3 Thursday, 1 September 16 * Who has used Haskell at all? Who is confident in using Haskell? * The REPL has served us well since Lisp. * but it also hasn’t changed much in that time. » Which leads to the question…
5. “Can we do better at facilitating explorative & graphical development?” 4 Thursday, 1 September 16 * Functional programming has advanced a lot since the introduction of Lisp. * Why did the REPL stay still?
6. Classic Haskell Development 5 Thursday, 1 September 16 * Variants include running ghcid and/or using ghc-mod/haskell-ide via editor script/plugin » This presents four main shortcomings…
7. Shortcomings Lack of persistence 6 Thursday, 1 September 16 * REPL statements are not persistent (cumbersome workaround: REPL history) * Limited interaction area, no highlighting etc (line-based) * The user needs to re-issue commands repeatedly & manually * Lack of support for richer media types (HTML, graphics, animations, etc)
8. Shortcomings Lack of persistence Line-based interface 6 Thursday, 1 September 16 * REPL statements are not persistent (cumbersome workaround: REPL history) * Limited interaction area, no highlighting etc (line-based) * The user needs to re-issue commands repeatedly & manually * Lack of support for richer media types (HTML, graphics, animations, etc)
9. Shortcomings Lack of persistence Line-based interface Manual re-execution 6 Thursday, 1 September 16 * REPL statements are not persistent (cumbersome workaround: REPL history) * Limited interaction area, no highlighting etc (line-based) * The user needs to re-issue commands repeatedly & manually * Lack of support for richer media types (HTML, graphics, animations, etc)
10. Shortcomings Lack of persistence Line-based interface Manual re-execution Display limited to text results 6 Thursday, 1 September 16 * REPL statements are not persistent (cumbersome workaround: REPL history) * Limited interaction area, no highlighting etc (line-based) * The user needs to re-issue commands repeatedly & manually * Lack of support for richer media types (HTML, graphics, animations, etc)
11. Tightening the Feedback Loop Haskell Playgrounds 7 Thursday, 1 September 16
12. Haskell Playgrounds Live environment facilitates exploration Encourages writing tests first 8 Thursday, 1 September 16 * Playgrounds encourages exploration and to write the tests first. * While writing/editing the code, you can see how the results change. » Moreover, we can gracefully integrate non-textual results…
13. Visualising Recursion Fractal Trees 9 Thursday, 1 September 16 * Recursion often is the first serious obstacle for people learning Haskell. * We can visualise recursion using fractal tree structures. Instructive and fun! » How do they work?
14. Pythagorean Trees 10 Thursday, 1 September 16 * We start from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
15. Pythagorean Trees 10 Thursday, 1 September 16 * We start from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
16. Pythagorean Trees 10 Thursday, 1 September 16 * We start from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
17. Pythagorean Trees 10 Thursday, 1 September 16 * We start from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
18. Pythagorean Trees 10 Thursday, 1 September 16 * We start from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
19. Pythagorean Trees 10 Thursday, 1 September 16 * We start from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
20. Pythagorean Trees 10 Thursday, 1 September 16 * We start from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
21. Shapes data Point = Point{ pointX, pointY :: Float } 11 Thursday, 1 September 16 * We need to be able to draw polygons. * Simple shapes library based on the vector drawing package Rasterific: https:// hackage.haskell.org/package/Rasterific
22. Shapes data Point = Point{ pointX, pointY :: Float } data Line = Line{ lineStart, lineEnd :: Point } 11 Thursday, 1 September 16 * We need to be able to draw polygons. * Simple shapes library based on the vector drawing package Rasterific: https:// hackage.haskell.org/package/Rasterific
23. Shapes data Point = Point{ pointX, pointY :: Float } data Line = Line{ lineStart, lineEnd :: Point } type Path = [Point] 11 Thursday, 1 September 16 * We need to be able to draw polygons. * Simple shapes library based on the vector drawing package Rasterific: https:// hackage.haskell.org/package/Rasterific
24. Shapes polygon :: Path -> PictureObject type Picture = [PictureObject] data Point = Point{ pointX, pointY :: Float } data Line = Line{ lineStart, lineEnd :: Point } type Path = [Point] 11 Thursday, 1 September 16 * We need to be able to draw polygons. * Simple shapes library based on the vector drawing package Rasterific: https:// hackage.haskell.org/package/Rasterific
25. Shapes scaleLine :: Float -> Line -> Line 12 Thursday, 1 September 16 * Scale line by scale factor, keeping the starting point where it is. * Rotate line around starting point by angle (in radians). » Let’s use this to draw fractal trees…
26. Shapes scaleLine :: Float -> Line -> Line 12 Thursday, 1 September 16 * Scale line by scale factor, keeping the starting point where it is. * Rotate line around starting point by angle (in radians). » Let’s use this to draw fractal trees…
27. Shapes scaleLine :: Float -> Line -> Line rotateLine :: Float -> Line -> Line 12 Thursday, 1 September 16 * Scale line by scale factor, keeping the starting point where it is. * Rotate line around starting point by angle (in radians). » Let’s use this to draw fractal trees…
28. Shapes scaleLine :: Float -> Line -> Line rotateLine :: Float -> Line -> Line 12 Thursday, 1 September 16 * Scale line by scale factor, keeping the starting point where it is. * Rotate line around starting point by angle (in radians). » Let’s use this to draw fractal trees…
29. Watch this screencast at https://vimeo.com/180978157 13 Thursday, 1 September 16 * Basic recursive structure of fractal trees. * We can vary (a) colour, (b) rotation factor, and the width/height ratio — even dynamically in dependence on the recursion depth. » Going through that code is not so interesting, but here are some example results…
30. 14 Thursday, 1 September 16
31. 15 Thursday, 1 September 16
32. 16 Thursday, 1 September 16
33. For more details see our tutorial http://learn.hfm.io/fractals.html 17 Thursday, 1 September 16 * http://learn.hfm.io/fractals.html and subsequent chapters » These are nice pictures, but they are static. How about animation?
34. “How can we code animations in Haskell?” 18 Thursday, 1 September 16
35. Shapes with Actions shapeNodeWithPath :: Path -> Node userData data Node u = … | Shape { nodePosition :: Point , nodeZRotation :: GFloat , nodeActionDirectives :: [Directive u] , nodePhysicsBody :: Maybe PhysicsBody , nodeChildren :: [Node u] …many more… } 19 Thursday, 1 September 16 * Rasterific can only do static images, so we move to Apple’s animation framework SpriteKit. * Shape nodes for polygons and Bezier curves. * Long list of graphical and animation-related attributes.
36. Shapes with Actions shapeNodeWithPath :: Path -> Node userData data Node u = … | Shape { nodePosition :: Point , nodeZRotation :: GFloat , nodeActionDirectives :: [Directive u] , nodePhysicsBody :: Maybe PhysicsBody , nodeChildren :: [Node u] …many more… } specification of animation actions 19 Thursday, 1 September 16 * Rasterific can only do static images, so we move to Apple’s animation framework SpriteKit. * Shape nodes for polygons and Bezier curves. * Long list of graphical and animation-related attributes.
37. Shapes with Actions shapeNodeWithPath :: Path -> Node userData data Node u = … | Shape { nodePosition :: Point , nodeZRotation :: GFloat , nodeActionDirectives :: [Directive u] , nodePhysicsBody :: Maybe PhysicsBody , nodeChildren :: [Node u] …many more… } specification of animation actions physical properties for physics simulation 19 Thursday, 1 September 16 * Rasterific can only do static images, so we move to Apple’s animation framework SpriteKit. * Shape nodes for polygons and Bezier curves. * Long list of graphical and animation-related attributes.
38. nodeActionDirectives = [runAction (moveBy vector)] Movement vector 20 Thursday, 1 September 16
39. nodeActionDirectives = [runAction (moveBy vector)] Movement vector 20 Thursday, 1 September 16
40. nodeActionDirectives = [runAction (rotateByAngle angle)] Rotation angle 21 Thursday, 1 September 16
41. nodeActionDirectives = [runAction (rotateByAngle angle)] Rotation angle 21 Thursday, 1 September 16
42. nodePhysicsBody = Just (bodyWithPolygonFromPath path) Gravity 22 Thursday, 1 September 16
43. nodePhysicsBody = Just (bodyWithPolygonFromPath path) Gravity 22 Thursday, 1 September 16
44. Hierarchical Graphics Elements 23 Thursday, 1 September 16 * Animations benefit from more structure — we want to capture the parent-child relationship.
45. Hierarchical Graphics Elements Absolute positioning relative to origin of coordinate system 23 Thursday, 1 September 16 * Animations benefit from more structure — we want to capture the parent-child relationship.
46. Hierarchical Graphics Elements Relative positioning relative to position of parent node nodeChildren 23 Thursday, 1 September 16 * Animations benefit from more structure — we want to capture the parent-child relationship.
47. Watch this screencast at https://vimeo.com/180981399 24 Thursday, 1 September 16 » We are using a purely functional SpriteKit API. How does this work?
48. Purely Functional SpriteKit 25 Thursday, 1 September 16 * Apple’s macOS/iOS/tvOS/watchOS framework: https://developer.apple.com/spritekit/ * SpriteKit: 2D animation and games framework including physics simulation with advanced features (inverse kinematic etc). » Support for graphics in Haskell is generally pretty poor. Why is that?
49. The Dilemma 26 Thursday, 1 September 16 * Existing toolkits: powerful, but rather imperative APIs (mutable, cyclic objected graphs) * Haskell native: functional, but a lot of work (and hence, very underpowered) » Sacrifice some elegance & gain modern graphics API…
50. The Dilemma SKShapeNode *shipOverlayShape = [[SKShapeNode alloc] init]; shipOverlayShape.path = boundingPath; shipOverlayShape.strokeColor = [SKColor clearColor]; shipOverlayShape.fillColor = [SKColor colorWithRed:0.0 green:1.0 blue:0.0 alpha:0.5]; [ship addChild:shipOverlayShape]; 26 Thursday, 1 September 16 * Existing toolkits: powerful, but rather imperative APIs (mutable, cyclic objected graphs) * Haskell native: functional, but a lot of work (and hence, very underpowered) » Sacrifice some elegance & gain modern graphics API…
51. The Dilemma SKShapeNode *shipOverlayShape = [[SKShapeNode alloc] init]; shipOverlayShape.path = boundingPath; shipOverlayShape.strokeColor = [SKColor clearColor]; shipOverlayShape.fillColor = [SKColor colorWithRed:0.0 green:1.0 blue:0.0 alpha:0.5]; [ship addChild:shipOverlayShape]; 26 Thursday, 1 September 16 * Existing toolkits: powerful, but rather imperative APIs (mutable, cyclic objected graphs) * Haskell native: functional, but a lot of work (and hence, very underpowered) » Sacrifice some elegance & gain modern graphics API…
52. The Dilemma SKShapeNode *shipOverlayShape = [[SKShapeNode alloc] init]; shipOverlayShape.path = boundingPath; shipOverlayShape.strokeColor = [SKColor clearColor]; shipOverlayShape.fillColor = [SKColor colorWithRed:0.0 green:1.0 blue:0.0 alpha:0.5]; [ship addChild:shipOverlayShape]; data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } 26 Thursday, 1 September 16 * Existing toolkits: powerful, but rather imperative APIs (mutable, cyclic objected graphs) * Haskell native: functional, but a lot of work (and hence, very underpowered) » Sacrifice some elegance & gain modern graphics API…
53. The Compromise data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } Purely Functional API 27 Thursday, 1 September 16 * Functional API to wrap imperative toolkit * Structure transformations translated into graph edits * FFI based on ”Inline Objective-C in Haskell” (Haskell Symposium 2014): https:// speakerdeck.com/mchakravarty/foreign-inline-code-in-haskell-haskell-symposium-2014
54. The Compromise data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } Purely Functional API represent scenes with algebraic data types 27 Thursday, 1 September 16 * Functional API to wrap imperative toolkit * Structure transformations translated into graph edits * FFI based on ”Inline Objective-C in Haskell” (Haskell Symposium 2014): https:// speakerdeck.com/mchakravarty/foreign-inline-code-in-haskell-haskell-symposium-2014
55. The Compromise data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } Purely Functional API translate transformations into state changes represent scenes with algebraic data types 27 Thursday, 1 September 16 * Functional API to wrap imperative toolkit * Structure transformations translated into graph edits * FFI based on ”Inline Objective-C in Haskell” (Haskell Symposium 2014): https:// speakerdeck.com/mchakravarty/foreign-inline-code-in-haskell-haskell-symposium-2014
56. SpriteKit 28 Thursday, 1 September 16 * SpriteKit scenes are rooted in an SKScene node (with back edges) * SKScene nodes and other nodes can have child nodes with specific functionality *
57. SpriteKit SKScene 28 Thursday, 1 September 16 * SpriteKit scenes are rooted in an SKScene node (with back edges) * SKScene nodes and other nodes can have child nodes with specific functionality *
58. SpriteKit SKScene SKSprite SKShape SKEmitter 28 Thursday, 1 September 16 * SpriteKit scenes are rooted in an SKScene node (with back edges) * SKScene nodes and other nodes can have child nodes with specific functionality *
59. SpriteKit SKScene SKSprite SKShape SKEmitter SKEmitter 28 Thursday, 1 September 16 * SpriteKit scenes are rooted in an SKScene node (with back edges) * SKScene nodes and other nodes can have child nodes with specific functionality *
60. SpriteKit SKScene SKSprite SKShape SKEmitter SKEmitter Physics, fields & particles Animation & lighting Joints & constraints 28 Thursday, 1 September 16 * SpriteKit scenes are rooted in an SKScene node (with back edges) * SKScene nodes and other nodes can have child nodes with specific functionality *
61. SKScene - (void)update:(NSTimeInterval)time; once per frame method 29 Thursday, 1 September 16 * As animations etc progress, the node graph is being mutated (edges & properties) * There are many more such methods (event handlers, contact handlers, etc) * Properties are also changed by the physics engine and the animation system
62. SKScene - (void)update:(NSTimeInterval)time; once per frame method 29 Thursday, 1 September 16 * As animations etc progress, the node graph is being mutated (edges & properties) * There are many more such methods (event handlers, contact handlers, etc) * Properties are also changed by the physics engine and the animation system
63. SKScene - (void)update:(NSTimeInterval)time; once per frame Object graph edits Property changes method 29 Thursday, 1 September 16 * As animations etc progress, the node graph is being mutated (edges & properties) * There are many more such methods (event handlers, contact handlers, etc) * Properties are also changed by the physics engine and the animation system
64. SKScene data Scene sc nd = Scene { … } — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame 30 Thursday, 1 September 16 * Wrapper translates transformations into state changes. * This is similar to React (Native). » SpriteKit is aimed at games. Let’s make one!
65. SKScene data Scene sc nd = Scene { … } — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling 30 Thursday, 1 September 16 * Wrapper translates transformations into state changes. * This is similar to React (Native). » SpriteKit is aimed at games. Let’s make one!
66. SKScene data Scene sc nd = Scene { … } — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling SceneUpdate sc nd Scene sc nd 30 Thursday, 1 September 16 * Wrapper translates transformations into state changes. * This is similar to React (Native). » SpriteKit is aimed at games. Let’s make one!
67. SKScene data Scene sc nd = Scene { … } — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling SceneUpdate sc nd Scene sc nd compute diff 30 Thursday, 1 September 16 * Wrapper translates transformations into state changes. * This is similar to React (Native). » SpriteKit is aimed at games. Let’s make one!
68. SKScene data Scene sc nd = Scene { … } — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling SceneUpdate sc nd Scene sc nd compute diff apply changes 30 Thursday, 1 September 16 * Wrapper translates transformations into state changes. * This is similar to React (Native). » SpriteKit is aimed at games. Let’s make one!
69. Game On Lazy Lambda 31 Thursday, 1 September 16 » Haskell version of well known game…
70. Watch this screencast at https://vimeo.com/180981926 32 Thursday, 1 September 16 Live Coding
71. 33 Thursday, 1 September 16 * We create the impression of infinite side scrolling by repeated moving and resting of ground tiles. * We’ll do the same with the sky background, but at a slower speed to create depth.
72. 33 Thursday, 1 September 16 * We create the impression of infinite side scrolling by repeated moving and resting of ground tiles. * We’ll do the same with the sky background, but at a slower speed to create depth.
73. 33 Thursday, 1 September 16 * We create the impression of infinite side scrolling by repeated moving and resting of ground tiles. * We’ll do the same with the sky background, but at a slower speed to create depth.
74. 33 Thursday, 1 September 16 * We create the impression of infinite side scrolling by repeated moving and resting of ground tiles. * We’ll do the same with the sky background, but at a slower speed to create depth.
75. 33 Thursday, 1 September 16 * We create the impression of infinite side scrolling by repeated moving and resting of ground tiles. * We’ll do the same with the sky background, but at a slower speed to create depth.
76. Watch this screencast at https://vimeo.com/180981971 34 Thursday, 1 September 16 Live Coding
77. 35 Thursday, 1 September 16 * Volume-based physics body have a mass and are usually affected by gravity and other forces. * Edge-based physics body have no mass and simply serve as a form of positional sentinels.
78. volume- based physics body 35 Thursday, 1 September 16 * Volume-based physics body have a mass and are usually affected by gravity and other forces. * Edge-based physics body have no mass and simply serve as a form of positional sentinels.
79. edge- based physics body volume- based physics body 35 Thursday, 1 September 16 * Volume-based physics body have a mass and are usually affected by gravity and other forces. * Edge-based physics body have no mass and simply serve as a form of positional sentinels.
80. Collision invokes contact handler (also physics) 36 Thursday, 1 September 16 * On collision a contact handler is invoked (and objects behave according to physical properties).
81. Watch this screencast at https://vimeo.com/180982052 37 Thursday, 1 September 16 Live Coding
82. 38 Thursday, 1 September 16
83. edge- based contact sentinel 38 Thursday, 1 September 16
84. 38 Thursday, 1 September 16
85. Watch this screencast at https://vimeo.com/180982081 39 Thursday, 1 September 16 Live Coding
86. mchakravarty TacticalGrace justtesting.org haskellformac.com Haskell SpriteKit open-sourced today! https://github.com/mchakravarty/HaskellSpriteKit 40 Thursday, 1 September 16 * Many of the replaced tests are often omitted as they are hard (UI testing, concurrency, …)
87. Thank you! 41 Thursday, 1 September 16