Growing a Language for Graphics - DotJS

"Growing a Language for Graphics" is about JavaScript as seen from the eyes of a data visualization / graphics person. By having a look at GLSL - a domain specific language (DSL) for graphics used in WebGL - I talk about what language features would help graphic developers on the Web and I show the long history of trying to standardize operator overloading into JavaScript.
This presentation was given at DotJS in Paris, France in December 2013.

Transcript

1. [Growing] a Language for
   Graphics
   @philogb dotJS Dec. 2013
   

   View Slide

2. View Slide

3. View Slide

4. WebGL
   Canvas
   SVG
   

   View Slide

5. WebGL
   

   View Slide

6. OpenGL OpenGL ES WebGL
   WebGL
   

   View Slide

7. Vertex Shader
   image source: http://duriansoftware.com/joe/An-intro-to-modern-OpenGL.-Chapter-1:-The-Graphics-Pipeline.html
   WebGL - Pipeline
   

   View Slide

8. Vertex Shader
   image source: http://duriansoftware.com/joe/An-intro-to-modern-OpenGL.-Chapter-1:-The-Graphics-Pipeline.html
   Triangle Assembly
   WebGL - Pipeline
   

   View Slide

9. Vertex Shader
   image source: http://duriansoftware.com/joe/An-intro-to-modern-OpenGL.-Chapter-1:-The-Graphics-Pipeline.html
   Triangle Assembly
   Rasterization
   WebGL - Pipeline
   

   View Slide

10. Vertex Shader
    image source: http://duriansoftware.com/joe/An-intro-to-modern-OpenGL.-Chapter-1:-The-Graphics-Pipeline.html
    Triangle Assembly
    Rasterization
    Fragment Shader
    WebGL - Pipeline
    

    View Slide

11. JavaScript
    GPU (Compiled Program)
    WebGL JS API
    WebGL - API
    

    View Slide

12. JavaScript
    Fragment Shader
    WebGL JS API
    Vertex Shader
    GLSL API
    GLSL API
    WebGL - API
    

    View Slide

13. GLSL
    A DSL for Graphics
    •C-like
    •Built-in types, functions for graphics
    •Operator overloading
    

    View Slide

14. 1 vec4 vector = vec4( 0, 1, vec2(0, 0) );
    2 vec3 point = vector.xyz;
    3 vec3 rgb = vector.rgb;
    4 mat4 m = mat4(vector);
    5 vec4 ans = vector * m;
    6
    7 float delta = 0.3;
    8 vec4 from = vec4(0);
    9 vec4 to = vec4(1);
    10 vec4 current = from + (to - from) * delta;
    11 vec4 current2 = mix(from, to, delta);
    12
    GLSL
    Syntax
    

    View Slide

15. GLSL
    Built-in functions
    radians degrees sin
    cos tan asin
    acos atan pow
    exp log exp2
    log2 sqrt inversesqrt
    abs sign floor
    ceil fract mod
    min max clamp
    mix step smoothstep
    length distance dot
    cross normalize faceforward
    reflect refract matrixCompMult
    

    View Slide

16. GLSL & JS
    Simple interpolation
    vec4 current = from + (to - from) * delta;
    var current = (to.add(from.scale(-1))).scale(delta).add(from);
    

    View Slide

17. vec4 current = from + (to - from) * delta;
    var current = (to.add(from.scale(-1))).scale(delta).add(from);
    GLSL & JS
    Simple interpolation
    

    View Slide

18. View Slide

19. View Slide

20. •The Lambda Papers (Scheme)
    

    View Slide

21. •The Lambda Papers (Scheme)
    •Java (1994), Sun fellow
    

    View Slide

22. •The Lambda Papers (Scheme)
    •Java (1994), Sun fellow
    •ECMA TC39 (1st Edition)
    

    View Slide

23. •The Lambda Papers (Scheme)
    •Java (1994), Sun fellow
    •ECMA TC39 (1st Edition)
    •The C Language (X3J11)
    

    View Slide

24. •The Lambda Papers (Scheme)
    •Java (1994), Sun fellow
    •ECMA TC39 (1st Edition)
    •The C Language (X3J11)
    •Fortran (X3J3)
    

    View Slide

25. •The Lambda Papers (Scheme)
    •Java (1994), Sun fellow
    •ECMA TC39 (1st Edition)
    •The C Language (X3J11)
    •Fortran (X3J3)
    •Common Lisp (X3J13)
    

    View Slide

26. Growing a language
    1998 - OOPSLA
    

    View Slide

27. General Rationale
    Some people want
    •Complex Number
    •Rational Number
    •Interval
    •Vector{2,3,4}
    •Matrix
    Should these be added to the language?
    

    View Slide

28. Value Objects
    

    View Slide

29. Value Objects
    •Many use-cases (int64, complex, decimal...)
    

    View Slide

30. Value Objects
    •Many use-cases (int64, complex, decimal...)
    •Overloadable Operators
    

    View Slide

31. Value Objects
    •Many use-cases (int64, complex, decimal...)
    •Overloadable Operators
    •Preserving Boolean Algebra
    

    View Slide

32. Value Objects
    •Many use-cases (int64, complex, decimal...)
    •Overloadable Operators
    •Preserving Boolean Algebra
    •Preserving Relational Relations (trichotomy)
    

    View Slide

33. Value Objects
    •Many use-cases (int64, complex, decimal...)
    •Overloadable Operators
    •Preserving Boolean Algebra
    •Preserving Relational Relations (trichotomy)
    •Strict Equality Operators unchanged
    

    View Slide

34. Value Objects
    •Many use-cases (int64, complex, decimal...)
    •Overloadable Operators
    •Preserving Boolean Algebra
    •Preserving Relational Relations (trichotomy)
    •Strict Equality Operators unchanged
    •Literal Syntax
    

    View Slide

35. Value Objects
    But how did we get here?
    

    View Slide

36. Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic Functions Double Dispatch Revisited Value Types Value Proxies Value Objects
    Operator Overloading
    

    View Slide

37. Generic Functions
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic FunctionsDouble Dispatch Revisited Value Types Value Proxies Value Objects
    ES4 draft
    1 class Complex! { ... }
    2
    3 generic intrinsic function +( a: Complex, b: Complex )
    4 new Complex( a.real + b.real, a.imag + b.imag )
    5
    6 generic intrinsic function +( a: Complex, b: AnyNumber )
    7 a + Complex(b)
    8
    9 generic intrinsic function +( a: AnyNumber, b: Complex )
    10 Complex(a) + b
    

    View Slide

38. Generic Functions
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic FunctionsDouble Dispatch Revisited Value Types Value Proxies Value Objects
    Dynamic Dispatch - How does it work?
    1 class Person {
    2 public void eat (Food f) { println ("eating food"); }
    3 public void eat(Pasta p) { println ("eating pasta"); }
    4 }
    5 /*...*/
    6 Food food = new Pasta();
    7 somePerson.eat(food);
    Java does not have dynamic dispatch in its method
    arguments
    

    View Slide

39. Generic Functions
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic Functions Double Dispatch Revisited Value Types Value Proxies Value Objects
    But what happens with “instances”?
    1 class John extends Person {
    2 public void eat(Food f) { println ("mmm...."); }
    3 public void eat(Pasta p) { println ("Yummy!"); }
    4 }
    5
    6 Person me = new John();
    7 Food pasta = new Pasta();
    8 me.eat(pasta);
    Dynamic Dispatch - How does it work?
    

    View Slide

40. Generic Functions
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic Functions Double Dispatch Revisited Value Types Value Proxies Value Objects
    Dynamic Dispatch
    Java: First arg. is dynamically dispatched
    1 public void function eat(John john, Food f) { ... }
    2 public void function eat(John john, Pasta p) { ... }
    3
    4 public void function eat(Person person, Food f) { ... }
    5 public void function eat(Person person, Pasta p) { ... }
    

    View Slide

41. Generic Functions
    Dynamic Dispatch
    CLOS: All args. are dynamically dispatched
    1 (defmethod eat ((john John) (f Food)) ... )
    2 (defmethod eat ((john John) (p Pasta)) ... )
    3
    4 (defmethod eat ((person Person) (f Food)) ... )
    5 (defmethod eat ((person Person) (p Pasta)) ... )
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic Functions Double Dispatch Revisited Value Types Value Proxies Value Objects
    

    View Slide

42. Double Dispatch
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic Functions Double Dispatch Revisited Value Types Value Proxies Value Objects
    Same principle as Smalltalk
    1 Number.prototype['+'] = function(arg) {
    2 if (isNumber(arg)) {
    3 return primAdd(this, arg);
    4 } else {
    5 return arg['reverse+'](this);
    6 }
    7 };
    8
    9 Number.prototype['reverse+'] = function(receiver) {
    10 if (isNumber(receiver)) {
    11 return primAdd(receiver, this);
    12 } else {
    13 throw ...;
    14 }
    15 };
    16
    

    View Slide

43. Double Dispatch
    Was proposed before based on ExtendScript
    1 Array.prototype['*'] = function(k, rev) {
    2 if( k.constructor != Number ) return;
    3
    4 var i = this.length,
    5 r = this.concat();
    6
    7 while( i-- ) r[i] *= k;
    8 return r;
    9 };
    10
    11 // Sample code:
    12 var a = [1,2,3];
    13 alert( a * 5 ); // [5,10,15]
    14 // (reversed == true)
    15 alert( 5 * a ); // [5,10,15] (works also!)
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic Functions Double Dispatch Revisited Value Types Value Proxies Value Objects
    

    View Slide

44. Double Dispatch
    Problems on extensibility
    1 Point.prototype['+'] = function (that) {
    2 if (that instanceof Point) {
    3 return new Point(this.x + that.x, this.y + that.y);
    4 } else if (that instanceof Number) {
    6 return new Point(this.x + that, this.y + that);
    7 } else if (...) {
    8 //x 10000
    9 } else {
    10 return that['reverse+'](this);
    11 }
    12 };
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic Functions Double Dispatch Revisited Value Types Value Proxies Value Objects
    

    View Slide

45. OO Revisited
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic Functions Double Dispatch Revisited Value Types Value Proxies Value Objects
    1 function pointPlusNumber(a, b) {
    2 return new Point(a.x + b, a.y + b);
    3 }
    4
    5 Function.defineOperator('+', Point, Number, pointPlusNumber);
    6
    7 function numberPlusPoint(a, b) {
    8 return new Point(a + b.x, a + b.y);
    9 }
    10
    11 Function.defineOperator('+', Number, Point, numberPlusPoint);
    12
    13 function addPoints(a, b) {
    14 return new Point(a.x + b.x, a.y + b.y);
    15 }
    16
    17 Function.defineOperator('+', Point, Point, addPoints);
    

    View Slide

46. Value Types
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic Functions Double Dispatch Revisited Value Types Value Proxies Value Objects
    “Value types would be objects that behave like primitives such as number and string:
    they are immutable or appear to be (...) In order to be usable, value types must
    "overload" (...) operator syntax; they also need to extend literal syntax (e.g. 1.1m for
    decimal 1.1).”
    

    View Slide

47. Value Proxies
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic Functions Double Dispatch Revisited Value Types Value Proxies Value Objects
    Proxies
    1 var p = new Proxy(target, handler);
    delete proxy.name deleteProperty
    Object.freeze(proxy) freeze
    name in proxy has
    proxy.name get
    proxy.name = val set
    for (prop in proxy) enumerate
    proxy.apply(thisValue, args) apply
    new proxy(...args) construct
    

    View Slide

48. Value Objects
    Oct. 2007 Jan. 2009 Jun. 2009 Jan. 2010 Mar. 2012
    Dec. 2010
    Generic Functions Double Dispatch Revisited Value Types Value Proxies Value Objects
    ...are not user-defined
    

    View Slide

49. After 7 years still no consensus on how users can
    perform operator overloading
    

    View Slide

50. View Slide

51. Thanks
    @philogb
    http://philogb.github.io/
    

    View Slide

52. # Bibliography
    ## Why operator overloading
    http://www.jroller.com/cpurdy/entry/the_seven_habits_of_highly1
    ## ES4 overview
    http://www.ecmascript.org/es4/spec/overview.pdf
    ## Generic functions
    https://mail.mozilla.org/pipermail/es-discuss/2007-November/005043.html
    https://mail.mozilla.org/pipermail/es-discuss/2007-November/005049.html
    https://mail.mozilla.org/pipermail/es-discuss/2007-November/005052.html
    https://mail.mozilla.org/pipermail/es-discuss/2007-November/005053.html
    ## 2009 Operator Overloading proposal
    https://mail.mozilla.org/pipermail/es-discuss/2009-January/008535.html
    https://mail.mozilla.org/pipermail/es-discuss/2009-January/008541.html
    http://wiki.ecmascript.org/doku.php?id=strawman:operator_overloading_with_double_dispatch
    ## ExtendScript OO
    http://forums.adobe.com/thread/646268
    http://www.indiscripts.com/post/2010/05/operator-overloading-with-extendscript
    ## Operator Overloading revisited
    https://mail.mozilla.org/pipermail/es-discuss/2009-June/009603.html
    https://mail.mozilla.org/pipermail/es-discuss/2009-June/009604.html
    https://mail.mozilla.org/pipermail/es-discuss/2009-June/
    ## Value types
    http://wiki.ecmascript.org/doku.php?id=strawman:value_types
    Decimal type:
    http://intertwingly.net/stories/2008/09/20/estest.html
    https://mail.mozilla.org/pipermail/es-discuss/2009-January/008646.html
    Value types page started:
    https://mail.mozilla.org/pipermail/es-discuss/2010-January/010677.html
    ## Value objects
    http://wiki.ecmascript.org/doku.php?id=strawman:value_objects
    https://mail.mozilla.org/pipermail/es-discuss/2012-March/021411.html
    http://www.slideshare.net/BrendanEich/value-objects
    https://bugzilla.mozilla.org/show_bug.cgi?id=749786
    ## Value proxies
    http://wiki.ecmascript.org/doku.php?id=strawman:value_proxies
    ## ES next
    https://mail.mozilla.org/pipermail/es-discuss/2013-April/029949.html
    

    View Slide