Playing with Graphics and Animations in Haskell

Playing with Graphics and Animations in Haskell

TALK VIDEO: https://www.youtube.com/watch?v=9dk7_GDNocQ (some code is hard to read in the video, you may like to refer to these slides for that)

NB: The talk includes live coding. In this version of the slides, these sections have been replaced by links to screencasts of the live coding sessions on Vimeo as Speaker Deck does not support embedded video.

Graphics and animations are not only fun, they are also an effective learning tool. For example, fractal tree structures nicely illustrate recursion, and animations are often more motivating and engaging than examples and exercises spewing out text. In this talk, I will demonstrate graphics and animations in Haskell playgrounds — aka the REPL on steroids. The immediate feedback of playgrounds facilitates an explorative style of programming that is especially attractive for graphics programming.

Graphics, animation, and games programming in Haskell faces a dilemma. We can either use existing frameworks with their highly imperative APIs or we waste a lot of energy trying to re-engineer those rather complex systems from scratch. Or, maybe, we can escape the dilemma. Instead of a Haskell program directly manipulating the mutable object-graph of existing frameworks, we provide an API for purely functional transformations of a Haskell data structure, together with an adaptation layer that transcribes those transformations into edits of the mutable object-graph. I will illustrate this approach by describing the architecture of a Haskell binding to the animation system of Apple’s SpriteKit framework.

This talk was the opening keynote of Compose :: Melbourne: http://www.composeconference.org/2016-melbourne/day-one-program/

The code of the Lazy Lambda game is open source at https://github.com/mchakravarty/lazy-lambda and the code for the Haskell SpriteKit binding is at https://github.com/mchakravarty/HaskellSpriteKit

2cc5323ccdfc09b921f1be34b3d78a69?s=128

Manuel Chakravarty

August 29, 2016
Tweet

Transcript

  1. 1.

    Manuel M T Chakravarty Applicative & UNSW Australia Playing with

    Graphics and Animations in Haskell mchakravarty TacticalGrace justtesting.org haskellformac.com 1 Thursday, 1 September 16 45 minute time slot (inclusive Q&A) [5min from REPL to Playgrounds; 10min Pythagorean trees; 7min SpriteKit & Haskell; 15min Lazy Lambda game] » Haskell is a great language for graphics, but …
  2. 2.

    Our Legacy Problem The Status Quo 2 Thursday, 1 September

    16 » …we don’t make use of it — largely, because the tool chain makes it hard…
  3. 3.

    3 Thursday, 1 September 16 * Who has used Haskell

    at all? Who is confident in using Haskell? * The REPL has served us well since Lisp. * but it also hasn’t changed much in that time. » Which leads to the question…
  4. 4.

    The Venerable REPL Read Eval Print Loop Introduced with Lisp.

    Remained essentially unchanged. 3 Thursday, 1 September 16 * Who has used Haskell at all? Who is confident in using Haskell? * The REPL has served us well since Lisp. * but it also hasn’t changed much in that time. » Which leads to the question…
  5. 5.

    “Can we do better at facilitating explorative & graphical development?”

    4 Thursday, 1 September 16 * Functional programming has advanced a lot since the introduction of Lisp. * Why did the REPL stay still?
  6. 6.

    Classic Haskell Development 5 Thursday, 1 September 16 * Variants

    include running ghcid and/or using ghc-mod/haskell-ide via editor script/plugin » This presents four main shortcomings…
  7. 7.

    Shortcomings Lack of persistence 6 Thursday, 1 September 16 *

    REPL statements are not persistent (cumbersome workaround: REPL history) * Limited interaction area, no highlighting etc (line-based) * The user needs to re-issue commands repeatedly & manually * Lack of support for richer media types (HTML, graphics, animations, etc)
  8. 8.

    Shortcomings Lack of persistence Line-based interface 6 Thursday, 1 September

    16 * REPL statements are not persistent (cumbersome workaround: REPL history) * Limited interaction area, no highlighting etc (line-based) * The user needs to re-issue commands repeatedly & manually * Lack of support for richer media types (HTML, graphics, animations, etc)
  9. 9.

    Shortcomings Lack of persistence Line-based interface Manual re-execution 6 Thursday,

    1 September 16 * REPL statements are not persistent (cumbersome workaround: REPL history) * Limited interaction area, no highlighting etc (line-based) * The user needs to re-issue commands repeatedly & manually * Lack of support for richer media types (HTML, graphics, animations, etc)
  10. 10.

    Shortcomings Lack of persistence Line-based interface Manual re-execution Display limited

    to text results 6 Thursday, 1 September 16 * REPL statements are not persistent (cumbersome workaround: REPL history) * Limited interaction area, no highlighting etc (line-based) * The user needs to re-issue commands repeatedly & manually * Lack of support for richer media types (HTML, graphics, animations, etc)
  11. 12.

    Haskell Playgrounds Live environment facilitates exploration Encourages writing tests first

    8 Thursday, 1 September 16 * Playgrounds encourages exploration and to write the tests first. * While writing/editing the code, you can see how the results change. » Moreover, we can gracefully integrate non-textual results…
  12. 13.

    Visualising Recursion Fractal Trees 9 Thursday, 1 September 16 *

    Recursion often is the first serious obstacle for people learning Haskell. * We can visualise recursion using fractal tree structures. Instructive and fun! » How do they work?
  13. 14.

    Pythagorean Trees 10 Thursday, 1 September 16 * We start

    from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
  14. 15.

    Pythagorean Trees 10 Thursday, 1 September 16 * We start

    from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
  15. 16.

    Pythagorean Trees 10 Thursday, 1 September 16 * We start

    from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
  16. 17.

    Pythagorean Trees 10 Thursday, 1 September 16 * We start

    from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
  17. 18.

    Pythagorean Trees 10 Thursday, 1 September 16 * We start

    from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
  18. 19.

    Pythagorean Trees 10 Thursday, 1 September 16 * We start

    from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
  19. 20.

    Pythagorean Trees 10 Thursday, 1 September 16 * We start

    from the square given by a ”base” line. * A triangle on top of the square completes the ”stump”. * The two other sides of the triangle form the base for two recursive applications of the same schema.
  20. 21.

    Shapes data Point = Point{ pointX, pointY :: Float }

    11 Thursday, 1 September 16 * We need to be able to draw polygons. * Simple shapes library based on the vector drawing package Rasterific: https:// hackage.haskell.org/package/Rasterific
  21. 22.

    Shapes data Point = Point{ pointX, pointY :: Float }

    data Line = Line{ lineStart, lineEnd :: Point } 11 Thursday, 1 September 16 * We need to be able to draw polygons. * Simple shapes library based on the vector drawing package Rasterific: https:// hackage.haskell.org/package/Rasterific
  22. 23.

    Shapes data Point = Point{ pointX, pointY :: Float }

    data Line = Line{ lineStart, lineEnd :: Point } type Path = [Point] 11 Thursday, 1 September 16 * We need to be able to draw polygons. * Simple shapes library based on the vector drawing package Rasterific: https:// hackage.haskell.org/package/Rasterific
  23. 24.

    Shapes polygon :: Path -> PictureObject type Picture = [PictureObject]

    data Point = Point{ pointX, pointY :: Float } data Line = Line{ lineStart, lineEnd :: Point } type Path = [Point] 11 Thursday, 1 September 16 * We need to be able to draw polygons. * Simple shapes library based on the vector drawing package Rasterific: https:// hackage.haskell.org/package/Rasterific
  24. 25.

    Shapes scaleLine :: Float -> Line -> Line 12 Thursday,

    1 September 16 * Scale line by scale factor, keeping the starting point where it is. * Rotate line around starting point by angle (in radians). » Let’s use this to draw fractal trees…
  25. 26.

    Shapes scaleLine :: Float -> Line -> Line 12 Thursday,

    1 September 16 * Scale line by scale factor, keeping the starting point where it is. * Rotate line around starting point by angle (in radians). » Let’s use this to draw fractal trees…
  26. 27.

    Shapes scaleLine :: Float -> Line -> Line rotateLine ::

    Float -> Line -> Line 12 Thursday, 1 September 16 * Scale line by scale factor, keeping the starting point where it is. * Rotate line around starting point by angle (in radians). » Let’s use this to draw fractal trees…
  27. 28.

    Shapes scaleLine :: Float -> Line -> Line rotateLine ::

    Float -> Line -> Line 12 Thursday, 1 September 16 * Scale line by scale factor, keeping the starting point where it is. * Rotate line around starting point by angle (in radians). » Let’s use this to draw fractal trees…
  28. 29.

    Watch this screencast at https://vimeo.com/180978157 13 Thursday, 1 September 16

    * Basic recursive structure of fractal trees. * We can vary (a) colour, (b) rotation factor, and the width/height ratio — even dynamically in dependence on the recursion depth. » Going through that code is not so interesting, but here are some example results…
  29. 33.

    For more details see our tutorial http://learn.hfm.io/fractals.html 17 Thursday, 1

    September 16 * http://learn.hfm.io/fractals.html and subsequent chapters » These are nice pictures, but they are static. How about animation?
  30. 35.

    Shapes with Actions shapeNodeWithPath :: Path -> Node userData data

    Node u = … | Shape { nodePosition :: Point , nodeZRotation :: GFloat , nodeActionDirectives :: [Directive u] , nodePhysicsBody :: Maybe PhysicsBody , nodeChildren :: [Node u] …many more… } 19 Thursday, 1 September 16 * Rasterific can only do static images, so we move to Apple’s animation framework SpriteKit. * Shape nodes for polygons and Bezier curves. * Long list of graphical and animation-related attributes.
  31. 36.

    Shapes with Actions shapeNodeWithPath :: Path -> Node userData data

    Node u = … | Shape { nodePosition :: Point , nodeZRotation :: GFloat , nodeActionDirectives :: [Directive u] , nodePhysicsBody :: Maybe PhysicsBody , nodeChildren :: [Node u] …many more… } specification of animation actions 19 Thursday, 1 September 16 * Rasterific can only do static images, so we move to Apple’s animation framework SpriteKit. * Shape nodes for polygons and Bezier curves. * Long list of graphical and animation-related attributes.
  32. 37.

    Shapes with Actions shapeNodeWithPath :: Path -> Node userData data

    Node u = … | Shape { nodePosition :: Point , nodeZRotation :: GFloat , nodeActionDirectives :: [Directive u] , nodePhysicsBody :: Maybe PhysicsBody , nodeChildren :: [Node u] …many more… } specification of animation actions physical properties for physics simulation 19 Thursday, 1 September 16 * Rasterific can only do static images, so we move to Apple’s animation framework SpriteKit. * Shape nodes for polygons and Bezier curves. * Long list of graphical and animation-related attributes.
  33. 44.

    Hierarchical Graphics Elements 23 Thursday, 1 September 16 * Animations

    benefit from more structure — we want to capture the parent-child relationship.
  34. 45.

    Hierarchical Graphics Elements Absolute positioning relative to origin of coordinate

    system 23 Thursday, 1 September 16 * Animations benefit from more structure — we want to capture the parent-child relationship.
  35. 46.

    Hierarchical Graphics Elements Relative positioning relative to position of parent

    node nodeChildren 23 Thursday, 1 September 16 * Animations benefit from more structure — we want to capture the parent-child relationship.
  36. 47.

    Watch this screencast at https://vimeo.com/180981399 24 Thursday, 1 September 16

    » We are using a purely functional SpriteKit API. How does this work?
  37. 48.

    Purely Functional SpriteKit 25 Thursday, 1 September 16 * Apple’s

    macOS/iOS/tvOS/watchOS framework: https://developer.apple.com/spritekit/ * SpriteKit: 2D animation and games framework including physics simulation with advanced features (inverse kinematic etc). » Support for graphics in Haskell is generally pretty poor. Why is that?
  38. 49.

    The Dilemma 26 Thursday, 1 September 16 * Existing toolkits:

    powerful, but rather imperative APIs (mutable, cyclic objected graphs) * Haskell native: functional, but a lot of work (and hence, very underpowered) » Sacrifice some elegance & gain modern graphics API…
  39. 50.

    The Dilemma SKShapeNode *shipOverlayShape = [[SKShapeNode alloc] init]; shipOverlayShape.path =

    boundingPath; shipOverlayShape.strokeColor = [SKColor clearColor]; shipOverlayShape.fillColor = [SKColor colorWithRed:0.0 green:1.0 blue:0.0 alpha:0.5]; [ship addChild:shipOverlayShape]; 26 Thursday, 1 September 16 * Existing toolkits: powerful, but rather imperative APIs (mutable, cyclic objected graphs) * Haskell native: functional, but a lot of work (and hence, very underpowered) » Sacrifice some elegance & gain modern graphics API…
  40. 51.

    The Dilemma SKShapeNode *shipOverlayShape = [[SKShapeNode alloc] init]; shipOverlayShape.path =

    boundingPath; shipOverlayShape.strokeColor = [SKColor clearColor]; shipOverlayShape.fillColor = [SKColor colorWithRed:0.0 green:1.0 blue:0.0 alpha:0.5]; [ship addChild:shipOverlayShape]; 26 Thursday, 1 September 16 * Existing toolkits: powerful, but rather imperative APIs (mutable, cyclic objected graphs) * Haskell native: functional, but a lot of work (and hence, very underpowered) » Sacrifice some elegance & gain modern graphics API…
  41. 52.

    The Dilemma SKShapeNode *shipOverlayShape = [[SKShapeNode alloc] init]; shipOverlayShape.path =

    boundingPath; shipOverlayShape.strokeColor = [SKColor clearColor]; shipOverlayShape.fillColor = [SKColor colorWithRed:0.0 green:1.0 blue:0.0 alpha:0.5]; [ship addChild:shipOverlayShape]; data GameState = GameState { gsSize :: (Double, Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } 26 Thursday, 1 September 16 * Existing toolkits: powerful, but rather imperative APIs (mutable, cyclic objected graphs) * Haskell native: functional, but a lot of work (and hence, very underpowered) » Sacrifice some elegance & gain modern graphics API…
  42. 53.

    The Compromise data GameState = GameState { gsSize :: (Double,

    Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } Purely Functional API 27 Thursday, 1 September 16 * Functional API to wrap imperative toolkit * Structure transformations translated into graph edits * FFI based on ”Inline Objective-C in Haskell” (Haskell Symposium 2014): https:// speakerdeck.com/mchakravarty/foreign-inline-code-in-haskell-haskell-symposium-2014
  43. 54.

    The Compromise data GameState = GameState { gsSize :: (Double,

    Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } Purely Functional API represent scenes with algebraic data types 27 Thursday, 1 September 16 * Functional API to wrap imperative toolkit * Structure transformations translated into graph edits * FFI based on ”Inline Objective-C in Haskell” (Haskell Symposium 2014): https:// speakerdeck.com/mchakravarty/foreign-inline-code-in-haskell-haskell-symposium-2014
  44. 55.

    The Compromise data GameState = GameState { gsSize :: (Double,

    Double) , time :: Millisecond , level :: Int , lifes :: Int , towers :: [Tower] , creeps :: [Creep] , shots :: [Shot] } Purely Functional API translate transformations into state changes represent scenes with algebraic data types 27 Thursday, 1 September 16 * Functional API to wrap imperative toolkit * Structure transformations translated into graph edits * FFI based on ”Inline Objective-C in Haskell” (Haskell Symposium 2014): https:// speakerdeck.com/mchakravarty/foreign-inline-code-in-haskell-haskell-symposium-2014
  45. 56.

    SpriteKit 28 Thursday, 1 September 16 * SpriteKit scenes are

    rooted in an SKScene node (with back edges) * SKScene nodes and other nodes can have child nodes with specific functionality *
  46. 57.

    SpriteKit SKScene 28 Thursday, 1 September 16 * SpriteKit scenes

    are rooted in an SKScene node (with back edges) * SKScene nodes and other nodes can have child nodes with specific functionality *
  47. 58.

    SpriteKit SKScene SKSprite SKShape SKEmitter 28 Thursday, 1 September 16

    * SpriteKit scenes are rooted in an SKScene node (with back edges) * SKScene nodes and other nodes can have child nodes with specific functionality *
  48. 59.

    SpriteKit SKScene SKSprite SKShape SKEmitter SKEmitter 28 Thursday, 1 September

    16 * SpriteKit scenes are rooted in an SKScene node (with back edges) * SKScene nodes and other nodes can have child nodes with specific functionality *
  49. 60.

    SpriteKit SKScene SKSprite SKShape SKEmitter SKEmitter Physics, fields & particles

    Animation & lighting Joints & constraints 28 Thursday, 1 September 16 * SpriteKit scenes are rooted in an SKScene node (with back edges) * SKScene nodes and other nodes can have child nodes with specific functionality *
  50. 61.

    SKScene - (void)update:(NSTimeInterval)time; once per frame method 29 Thursday, 1

    September 16 * As animations etc progress, the node graph is being mutated (edges & properties) * There are many more such methods (event handlers, contact handlers, etc) * Properties are also changed by the physics engine and the animation system
  51. 62.

    SKScene - (void)update:(NSTimeInterval)time; once per frame method 29 Thursday, 1

    September 16 * As animations etc progress, the node graph is being mutated (edges & properties) * There are many more such methods (event handlers, contact handlers, etc) * Properties are also changed by the physics engine and the animation system
  52. 63.

    SKScene - (void)update:(NSTimeInterval)time; once per frame Object graph edits Property

    changes method 29 Thursday, 1 September 16 * As animations etc progress, the node graph is being mutated (edges & properties) * There are many more such methods (event handlers, contact handlers, etc) * Properties are also changed by the physics engine and the animation system
  53. 64.

    SKScene data Scene sc nd = Scene { … }

    — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame 30 Thursday, 1 September 16 * Wrapper translates transformations into state changes. * This is similar to React (Native). » SpriteKit is aimed at games. Let’s make one!
  54. 65.

    SKScene data Scene sc nd = Scene { … }

    — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling 30 Thursday, 1 September 16 * Wrapper translates transformations into state changes. * This is similar to React (Native). » SpriteKit is aimed at games. Let’s make one!
  55. 66.

    SKScene data Scene sc nd = Scene { … }

    — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling SceneUpdate sc nd Scene sc nd 30 Thursday, 1 September 16 * Wrapper translates transformations into state changes. * This is similar to React (Native). » SpriteKit is aimed at games. Let’s make one!
  56. 67.

    SKScene data Scene sc nd = Scene { … }

    — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling SceneUpdate sc nd Scene sc nd compute diff 30 Thursday, 1 September 16 * Wrapper translates transformations into state changes. * This is similar to React (Native). » SpriteKit is aimed at games. Let’s make one!
  57. 68.

    SKScene data Scene sc nd = Scene { … }

    — Haskell record type SceneUpdate sc nd = Scene sc nd -> TimeInterval -> Scene sc nd once per frame Scene sc nd lazy marshalling SceneUpdate sc nd Scene sc nd compute diff apply changes 30 Thursday, 1 September 16 * Wrapper translates transformations into state changes. * This is similar to React (Native). » SpriteKit is aimed at games. Let’s make one!
  58. 69.

    Game On Lazy Lambda 31 Thursday, 1 September 16 »

    Haskell version of well known game…
  59. 71.

    33 Thursday, 1 September 16 * We create the impression

    of infinite side scrolling by repeated moving and resting of ground tiles. * We’ll do the same with the sky background, but at a slower speed to create depth.
  60. 72.

    33 Thursday, 1 September 16 * We create the impression

    of infinite side scrolling by repeated moving and resting of ground tiles. * We’ll do the same with the sky background, but at a slower speed to create depth.
  61. 73.

    33 Thursday, 1 September 16 * We create the impression

    of infinite side scrolling by repeated moving and resting of ground tiles. * We’ll do the same with the sky background, but at a slower speed to create depth.
  62. 74.

    33 Thursday, 1 September 16 * We create the impression

    of infinite side scrolling by repeated moving and resting of ground tiles. * We’ll do the same with the sky background, but at a slower speed to create depth.
  63. 75.

    33 Thursday, 1 September 16 * We create the impression

    of infinite side scrolling by repeated moving and resting of ground tiles. * We’ll do the same with the sky background, but at a slower speed to create depth.
  64. 77.

    35 Thursday, 1 September 16 * Volume-based physics body have

    a mass and are usually affected by gravity and other forces. * Edge-based physics body have no mass and simply serve as a form of positional sentinels.
  65. 78.

    volume- based physics body 35 Thursday, 1 September 16 *

    Volume-based physics body have a mass and are usually affected by gravity and other forces. * Edge-based physics body have no mass and simply serve as a form of positional sentinels.
  66. 79.

    edge- based physics body volume- based physics body 35 Thursday,

    1 September 16 * Volume-based physics body have a mass and are usually affected by gravity and other forces. * Edge-based physics body have no mass and simply serve as a form of positional sentinels.
  67. 80.

    Collision invokes contact handler (also physics) 36 Thursday, 1 September

    16 * On collision a contact handler is invoked (and objects behave according to physical properties).
  68. 86.

    mchakravarty TacticalGrace justtesting.org haskellformac.com Haskell SpriteKit open-sourced today! https://github.com/mchakravarty/HaskellSpriteKit 40

    Thursday, 1 September 16 * Many of the replaced tests are often omitted as they are hard (UI testing, concurrency, …)