Data-centric Metaprogramming @ ICOOOLPS 2015

The data-centric metaprogramming presentation at the ICOOOLPS workshop (Implementation, Compilation, Optimization of Object-Oriented Languages, Programs and Systems Workshop) in Prague, Czech Republic. Website: http://2015.ecoop.org/track/ICOOOLPS-2015-papers#program

Project website: http://scala-ildl.org

Transcript

1. scala-ildl.org
   Data-centric Metaprogramming
   in Object-Oriented Languages
   6th of July 2015
   ICOOLPS Invited Talk
   Prague, Czech Republic
   

   View Slide

2. scala-ildl.org
   STOP
   Please ask if things
   are not clear!
   

   View Slide

3. scala-ildl.org
   STOP
   Please ask if things
   are not clear!
   Really! I mean it!
   

   View Slide

4. scala-ildl.org
   Vlad URECHE
   PhD student in the Scala Team @ EPFL
   Working on program transformations
   focusing on data representation.
   Miniboxing guy.
   @
   @VladUreche
   @VladUreche
   vlad.ureche@epfl.ch
   

   View Slide

5. scala-ildl.org
   Data-centric Metaprogramming
   in Object-Oriented Languages
   

   View Slide

6. scala-ildl.org
   Data-centric Metaprogramming
   in Object-Oriented Languages
   Joint work with
   ●
   Martin Odersky
   ●
   Aggelos Biboudis
   ●
   Yannis Smaragdakis
   

   View Slide

7. scala-ildl.org
   infoscience.epfl.ch/record/207050
   

   View Slide

8. scala-ildl.org
   github.com/miniboxing/ildl-plugin
   

   View Slide

9. scala-ildl.org
   Data-centric Metaprogramming
   in Object-Oriented Languages
   

   View Slide

10. scala-ildl.org
    Data-centric Metaprogramming
    in Object-Oriented Languages
    

    View Slide

11. scala-ildl.org
    Motivation
    Transformation
    Applications
    Challenges
    Conclusion
    Related Work
    

    View Slide

12. scala-ildl.org
    Object Composition
    Object Composition
    class Vector[T] { … }
    

    View Slide

13. scala-ildl.org
    Object Composition
    Object Composition
    class Vector[T] { … } The Vector collection
    in the Scala library
    

    View Slide

14. scala-ildl.org
    Object Composition
    Object Composition
    class Employee(...)
    ID NAME SALARY
    class Vector[T] { … } The Vector collection
    in the Scala library
    

    View Slide

15. scala-ildl.org
    Object Composition
    Object Composition
    class Employee(...)
    ID NAME SALARY
    Auto-generated,
    corresponds to a
    table row
    class Vector[T] { … } The Vector collection
    in the Scala library
    

    View Slide

16. scala-ildl.org
    Object Composition
    Object Composition
    class Vector[T] { … }
    class Employee(...)
    ID NAME SALARY
    

    View Slide

17. scala-ildl.org
    Object Composition
    Object Composition
    class Vector[T] { … }
    class Employee(...)
    ID NAME SALARY
    

    View Slide

18. scala-ildl.org
    Object Composition
    Object Composition
    class Employee(...)
    ID NAME SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    class Vector[T] { … }
    

    View Slide

19. scala-ildl.org
    Object Composition
    Object Composition
    class Employee(...)
    ID NAME SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    class Vector[T] { … }
    Traversal requires
    dereferencing a pointer
    for each employee.
    

    View Slide

20. scala-ildl.org
    A Better Representation
    A Better Representation
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    

    View Slide

21. scala-ildl.org
    A Better Representation
    A Better Representation
    NAME ...
    NAME
    EmployeeVector
    ID ID ...
    ...
    SALARY SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    

    View Slide

22. scala-ildl.org
    A Better Representation
    A Better Representation
    NAME ...
    NAME
    EmployeeVector
    ID ID ...
    ...
    SALARY SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    iteration is 5x faster
    

    View Slide

23. scala-ildl.org
    A Better Representation
    A Better Representation
    NAME ...
    NAME
    EmployeeVector
    ID ID ...
    ...
    SALARY SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    iteration is 5x faster
    C++ would produce a
    better representation here,
    but there are still cases where
    the C++ representation could
    be improved over.
    

    View Slide

24. scala-ildl.org
    A Better Representation
    A Better Representation
    ●
    In isolation Vector[T] and Employee are optimal
    

    View Slide

25. scala-ildl.org
    A Better Representation
    A Better Representation
    ●
    In isolation Vector[T] and Employee are optimal
    ●
    Together, Vector[Employee] can be optimized
    

    View Slide

26. scala-ildl.org
    A Better Representation
    A Better Representation
    ●
    In isolation Vector[T] and Employee are optimal
    ●
    Together, Vector[Employee] can be optimized
    Challenge: No means of
    communicating this
    to the compiler
    

    View Slide

27. scala-ildl.org
    A Better Representation
    A Better Representation
    ●
    In isolation Vector[T] and Employee are optimal
    ●
    Together, Vector[Employee] can be optimized
    Challenge: No means of
    communicating this
    to the compiler
    You may disagree. We'll have
    a related work section later.
    

    View Slide

28. scala-ildl.org
    

    View Slide

29. scala-ildl.org
    ●
    Transforming the code by hand
    – Makes maintenance difficult
    – Changes are not contained
    

    View Slide

30. scala-ildl.org
    ●
    Transforming the code by hand
    – Makes maintenance difficult
    – Changes are not contained
    Can we
    automate this?
    

    View Slide

31. scala-ildl.org
    Motivation
    Transformation
    Applications
    Challenges
    Conclusion
    Related Work
    

    View Slide

32. scala-ildl.org
    Transformation
    Transformation
    Definition Application
    

    View Slide

33. scala-ildl.org
    Transformation
    Transformation
    Definition Application
    ●
    can't be automated
    ●
    based on experience
    ●
    based on speculation
    ●
    one-time effort
    

    View Slide

34. scala-ildl.org
    Transformation
    Transformation
    Definition Application
    ●
    can't be automated
    ●
    based on experience
    ●
    based on speculation
    ●
    one-time effort
    programmer
    

    View Slide

35. scala-ildl.org
    Transformation
    Transformation
    Definition Application
    ●
    can't be automated
    ●
    based on experience
    ●
    based on speculation
    ●
    one-time effort
    ●
    repetitive and simple
    ●
    affects code readability
    ●
    is verbose
    ●
    is error-prone
    programmer
    

    View Slide

36. scala-ildl.org
    Transformation
    Transformation
    programmer
    Definition Application
    ●
    can't be automated
    ●
    based on experience
    ●
    based on speculation
    ●
    one-time effort
    ●
    repetitive and simple
    ●
    affects code readability
    ●
    is verbose
    ●
    is error-prone
    compiler
    (automated)
    

    View Slide

37. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    ...
    }
    

    View Slide

38. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    ...
    }
    An object that
    describes a
    Transformation.
    

    View Slide

39. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    ...
    }
    An object that
    describes a
    Transformation.
    A marker trait for
    transformations.
    

    View Slide

40. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    ...
    }
    

    View Slide

41. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    ...
    }
    What does the
    compiler need to
    know?
    

    View Slide

42. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    ...
    }
    What does the
    compiler need to
    know?
    The target of the
    transformation and
    its representation.
    

    View Slide

43. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    type Target = Vector[Employee]
    type Result = EmployeeVector
    ...
    }
    

    View Slide

44. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    type Target = Vector[Employee]
    type Result = EmployeeVector
    ...
    } The transformation is
    type-driven we indicate
    →
    the type of the target and
    of the representation.
    

    View Slide

45. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    type Target = Vector[Employee]
    type Result = EmployeeVector
    ...
    } The transformation is
    type-driven we indicate
    →
    the type of the target and
    of the representation.
    The improved representation
    is defined in the host language.
    

    View Slide

46. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    type Target = Vector[Employee]
    type Result = EmployeeVector
    ...
    }
    

    View Slide

47. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    type Target = Vector[Employee]
    type Result = EmployeeVector
    ...
    }
    How to transform
    Vector[Employee]
    into an
    EmployeeVector?
    

    View Slide

48. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    type Target = Vector[Employee]
    type Result = EmployeeVector
    def toResult(t: Target): Result = ...
    def toTarget(t: Result): Target = ...
    ...
    }
    

    View Slide

49. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    type Target = Vector[Employee]
    type Result = EmployeeVector
    def toResult(t: Target): Result = ...
    def toTarget(t: Result): Target = ...
    ...
    }
    Conversions to/from
    Vector[Employee]
    that consume/produce a
    EmployeeVector?
    

    View Slide

50. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    type Target = Vector[Employee]
    type Result = EmployeeVector
    def toResult(t: Target): Result = ...
    def toTarget(t: Result): Target = ...
    ...
    }
    

    View Slide

51. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    type Target = Vector[Employee]
    type Result = EmployeeVector
    def toResult(t: Target): Result = ...
    def toTarget(t: Result): Target = ...
    ...
    }
    So far so good, but how
    to execute Vector[Employee]
    operations on EmployeeVector?
    

    View Slide

52. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    object VectorOfEmployeeOpt extends Transformation {
    type Target = Vector[Employee]
    type Result = EmployeeVector
    def toResult(t: Target): Result = ...
    def toTarget(t: Result): Target = ...
    def bypass_length: Int = ...
    def bypass_apply(i: Int): Employee = ...
    def bypass_update(i: Int, v: Employee) = ...
    def bypass_toString: String = …
    ...
    }
    

    View Slide

53. scala-ildl.org
    Data-centric Metaprogramming
    Data-centric Metaprogramming
    programmer
    Definition Application
    ●
    can't be automated
    ●
    based on experience
    ●
    based on speculation
    ●
    one-time effort
    ●
    repetitive and simple
    ●
    affects code readability
    ●
    is verbose
    ●
    is error-prone
    compiler
    (automated)
    

    View Slide

54. scala-ildl.org
    

    View Slide

55. scala-ildl.org
    Motivation
    Transformation
    Applications
    Challenges
    Conclusion
    Related Work
    

    View Slide

56. scala-ildl.org
    Motivation
    Transformation
    Applications
    Challenges
    Conclusion
    Related Work
    Open-world
    Best Represenation?
    Composition
    

    View Slide

57. scala-ildl.org
    Motivation
    Transformation
    Applications
    Challenges
    Conclusion
    Related Work
    Open-world
    Best Represenation?
    Composition
    

    View Slide

58. scala-ildl.org
    Open World Assumption
    Open World Assumption
    class Vector[T] { … }
    class Employee(...)
    ID NAME SALARY
    

    View Slide

59. scala-ildl.org
    Open World Assumption
    Open World Assumption
    class Vector[T] { … }
    class Employee(...)
    ID NAME SALARY
    

    View Slide

60. scala-ildl.org
    Open World Assumption
    Open World Assumption
    class Employee(...)
    ID NAME SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    class Vector[T] { … }
    

    View Slide

61. scala-ildl.org
    Open World Assumption
    Open World Assumption
    class Employee(...)
    ID NAME SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    class Vector[T] { … }
    NAME ...
    NAME
    EmployeeVector
    ID ID ...
    ...
    SALARY SALARY
    

    View Slide

62. scala-ildl.org
    Open World Assumption
    Open World Assumption
    class Employee(...)
    ID NAME SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    class Vector[T] { … }
    NAME ...
    NAME
    EmployeeVector
    ID ID ...
    ...
    SALARY SALARY
    class NewEmployee(...)
    extends Employee(...)
    ID NAME SALARY DEPT
    

    View Slide

63. scala-ildl.org
    Open World Assumption
    Open World Assumption
    class Employee(...)
    ID NAME SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    class Vector[T] { … }
    NAME ...
    NAME
    EmployeeVector
    ID ID ...
    ...
    SALARY SALARY
    class NewEmployee(...)
    extends Employee(...)
    ID NAME SALARY DEPT
    Oooops...
    

    View Slide

64. scala-ildl.org
    Open World Assumption
    Open World Assumption
    ●
    Globally anything can happen
    

    View Slide

65. scala-ildl.org
    Open World Assumption
    Open World Assumption
    ●
    Globally anything can happen
    ●
    Locally the programmer has full control:
    – Knows the values that will be used
    – Can reject non-conforming values
    

    View Slide

66. scala-ildl.org
    Open World Assumption
    Open World Assumption
    ●
    Globally anything can happen
    ●
    Locally the programmer has full control:
    – Knows the values that will be used
    – Can reject non-conforming values How to
    use this
    information?
    

    View Slide

67. scala-ildl.org
    Open World Assumption
    Open World Assumption
    ●
    Globally anything can happen
    ●
    Locally the programmer has full control:
    – Knows the values that will be used
    – Can reject non-conforming values How to
    use this
    information?
    Scopes
    

    View Slide

68. scala-ildl.org
    Scopes
    Scopes
    def indexSalary(employees: Vector[Employee],
    by: Float): Vector[Employee] =
    for (employee ← employees)
    yield employee.copy(
    salary = (1 + by) * employee.salary
    )
    

    View Slide

69. scala-ildl.org
    Scopes
    Scopes
    def indexSalary(employees: Vector[Employee],
    by: Float): Vector[Employee] =
    for (employee ← employees)
    yield employee.copy(
    salary = (1 + by) * employee.salary
    )
    Method operating on
    Scala collections
    (familiar to programmers)
    

    View Slide

70. scala-ildl.org
    Scopes
    Scopes
    def indexSalary(employees: Vector[Employee],
    by: Float): Vector[Employee] =
    for (employee ← employees)
    yield employee.copy(
    salary = (1 + by) * employee.salary
    )
    

    View Slide

71. scala-ildl.org
    Scopes
    Scopes
    transform(VectorOfEmployeeOpt) {
    def indexSalary(employees: Vector[Employee],
    by: Float): Vector[Employee] =
    for (employee ← employees)
    yield employee.copy(
    salary = (1 + by) * employee.salary
    )
    }
    

    View Slide

72. scala-ildl.org
    Scopes
    Scopes
    transform(VectorOfEmployeeOpt) {
    def indexSalary(employees: Vector[Employee],
    by: Float): Vector[Employee] =
    for (employee ← employees)
    yield employee.copy(
    salary = (1 + by) * employee.salary
    )
    }
    Now the method operates
    on the EmployeeVector
    representation.
    

    View Slide

73. scala-ildl.org
    Scopes
    Scopes
    transform(VectorOfEmployeeOpt) {
    def indexSalary(employees: Vector[Employee],
    by: Float): Vector[Employee] =
    for (employee ← employees)
    yield employee.copy(
    salary = (1 + by) * employee.salary
    )
    }
    Now the method operates
    on the EmployeeVector
    representation.
    Programmers can freely choose
    which parts of their code to transform.
    

    View Slide

74. scala-ildl.org
    Scopes
    Scopes
    ●
    Can wrap statements, methods even entire classes
    – Inlined immediately after the parser
    – Visible outside the "scope"
    

    View Slide

75. scala-ildl.org
    Scopes
    Scopes
    ●
    Can wrap statements, methods even entire classes
    – Inlined immediately after the parser
    – Visible outside the "scope"
    ●
    Locally closed world
    – Incoming/outgoing values go through conversions
    – Programmer can reject unexpected values
    

    View Slide

76. scala-ildl.org
    Research Challenges
    Research Challenges
    ●
    No way to say "@only Employee" in the type system
    – Currently producing a runtime error
    – Ideally we should generate a compile-time error
    

    View Slide

77. scala-ildl.org
    Research Challenges
    Research Challenges
    ●
    No way to say "@only Employee" in the type system
    – Currently producing a runtime error
    – Ideally we should generate a compile-time error
    ●
    No way to undo variance (e.g. List[+T])
    – We have to convert back to List[Employee]
    – Still correct, but inefficient
    ●
    Could we restrict this in the type system?
    ●
    Or prevent the transformation?
    val v: List[Any] = listOfEmployees
    

    View Slide

78. scala-ildl.org
    Motivation
    Transformation
    Applications
    Challenges
    Conclusion
    Related Work
    Open-world
    Best Represenation?
    Composition
    

    View Slide

79. scala-ildl.org
    Best Representation
    Best Representation
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    

    View Slide

80. scala-ildl.org
    Best ...?
    Best ...?
    NAME ...
    NAME
    EmployeeVector
    ID ID ...
    ...
    SALARY SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    

    View Slide

81. scala-ildl.org
    Best ...?
    Best ...?
    NAME ...
    NAME
    EmployeeVector
    ID ID ...
    ...
    SALARY SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    EmployeeJSON
    {
    id: 123,
    name: “John Doe”
    salary: 100
    }
    

    View Slide

82. scala-ildl.org
    Best ...?
    Best ...?
    NAME ...
    NAME
    EmployeeVector
    ID ID ...
    ...
    SALARY SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    EmployeeJSON
    {
    id: 123,
    name: “John Doe”
    salary: 100
    }
    CompactVector
    
    

    View Slide

83. scala-ildl.org
    Best ...?
    Best ...?
    NAME ...
    NAME
    EmployeeVector
    ID ID ...
    ...
    SALARY SALARY
    Vector[Employee]
    ID NAME SALARY
    ID NAME SALARY
    EmployeeJSON
    {
    id: 123,
    name: “John Doe”
    salary: 100
    }
    CompactVector
    
    Scopes!
    

    View Slide

84. scala-ildl.org
    Scopes
    Scopes
    transform(VectorOfEmployeeOpt) {
    def indexSalary(employees: Vector[Employee],
    by: Float): Vector[Employee] =
    for (employee ← employees)
    yield employee.copy(
    salary = (1 + by) * employee.salary
    )
    }
    

    View Slide

85. scala-ildl.org
    Scopes
    Scopes
    transform(VectorOfEmployeeOpt) {
    def indexSalary(employees: Vector[Employee],
    by: Float): Vector[Employee] =
    for (employee ← employees)
    yield employee.copy(
    salary = (1 + by) * employee.salary
    )
    }
    Method operating on
    column-based storage
    

    View Slide

86. scala-ildl.org
    Scopes
    Scopes
    transform(VectorOfEmployeeJSON) {
    def indexSalary(employees: Vector[Employee],
    by: Float): Vector[Employee] =
    for (employee ← employees)
    yield employee.copy(
    salary = (1 + by) * employee.salary
    )
    }
    Method operating on
    JSON data
    

    View Slide

87. scala-ildl.org
    Scopes
    Scopes
    transform(VectorOfEmployeeBinary) {
    def indexSalary(employees: Vector[Employee],
    by: Float): Vector[Employee] =
    for (employee ← employees)
    yield employee.copy(
    salary = (1 + by) * employee.salary
    )
    }
    Method operating on
    binary data
    

    View Slide

88. scala-ildl.org
    Motivation
    Transformation
    Applications
    Challenges
    Conclusion
    Related Work
    Open-world
    Best Represenation?
    Composition
    

    View Slide

89. scala-ildl.org
    Scope Composition
    Scope Composition
    ●
    Code can be
    – Left untransformed (using the original repr.)
    – Transformed using different representations
    

    View Slide

90. scala-ildl.org
    Scope Composition
    Scope Composition
    calling
    ●
    Original code
    ●
    Transformed code
    ●
    Original code
    ●
    Transformed code
    ●
    Same transformation
    ●
    Different transformation
    ●
    Code can be
    – Left untransformed (using the original repr.)
    – Transformed using different representations
    

    View Slide

91. scala-ildl.org
    Scope Composition
    Scope Composition
    calling
    ●
    Original code
    ●
    Transformed code
    ●
    Original code
    ●
    Transformed code
    ●
    Same transformation
    ●
    Different transformation
    ●
    Code can be
    – Left untransformed (using the original repr.)
    – Transformed using different representations
    

    View Slide

92. scala-ildl.org
    Scope Composition – Case 1
    Scope Composition – Case 1
    transform(VectorOfEmployeeOpt) {
    def indexSalary(vector: Vector[Employee]) = ...
    }
    

    View Slide

93. scala-ildl.org
    Scope Composition – Case 1
    Scope Composition – Case 1
    transform(VectorOfEmployeeOpt) {
    def indexSalary(vector: Vector[Employee]) = ...
    }
    val employees = loadEmployees(...)
    indexSalary(employees)
    No transformation
    scope applied here.
    

    View Slide

94. scala-ildl.org
    Scope Composition – Case 1
    Scope Composition – Case 1
    transform(VectorOfEmployeeOpt) {
    def indexSalary(vector: Vector[Employee]) = ...
    }
    val employees = loadEmployees(...)
    indexSalary(employees)
    No transformation
    scope applied here.
    

    View Slide

95. scala-ildl.org
    Scope Composition – Case 1
    Scope Composition – Case 1
    transform(VectorOfEmployeeOpt) {
    def indexSalary(vector: Vector[Employee]) = ...
    }
    val employees = loadEmployees(...)
    indexSalary(employees)
    No transformation
    scope applied here.
    Different representation:
    ●
    vector: EmployeeVector
    ●
    employees: Vector[Employee]
    

    View Slide

96. scala-ildl.org
    Scope Composition – Case 1
    Scope Composition – Case 1
    transform(VectorOfEmployeeOpt) {
    def indexSalary(vector: Vector[Employee]) = ...
    }
    val employees = loadEmployees(...)
    indexSalary(employees)
    No transformation
    scope applied here.
    Different representation:
    ●
    vector: EmployeeVector
    ●
    employees: Vector[Employee]
    The compiler adds conversions
    

    View Slide

97. scala-ildl.org
    Scope Composition – Case 1
    Scope Composition – Case 1
    ●
    Values using different representations
    – Are automatically converted
    – Even in different files
    – Even across separate compilation
    

    View Slide

98. scala-ildl.org
    Scope Composition – Case 1
    Scope Composition – Case 1
    ●
    Values using different representations
    – Are automatically converted
    – Even in different files
    – Even across separate compilation
    ●
    This may not always be what you want
    – aliasing mutable objects
    – expensive coercions
    

    View Slide

99. scala-ildl.org
    Scope Composition – Case 1
    Scope Composition – Case 1
    ●
    Values using different representations
    – Are automatically converted
    – Even in different files
    – Even across separate compilation
    ●
    This may not always be what you want
    – aliasing mutable objects
    – expensive coercions
    ●
    The compiler can warn about this
    

    View Slide

100. scala-ildl.org
     Scope Composition – Case 1
     Scope Composition – Case 1
     transform(VectorOfEmployeeOpt) {
     def indexSalary(vector: Vector[Employee]) = ...
     }
     val employees = loadEmployees(...)
     indexSalary(employees)
     

     View Slide

101. scala-ildl.org
     Scope Composition – Case 1
     Scope Composition – Case 1
     transform(VectorOfEmployeeOpt) {
     def indexSalary(vector: Vector[Employee]) = ...
     }
     val employees = loadEmployees(...)
     indexSalary(employees)
     warning: When calling method indexSalary, the
     argument 'employees' needs to be converted to the
     'EmployeeVector' representation, which may incur an
     overhead:
     indexSalary(employees)
     ˆ
     

     View Slide

102. scala-ildl.org
     Scope Composition
     Scope Composition
     calling
     ●
     Original code
     ●
     Transformed code
     ●
     Original code
     ●
     Transformed code
     ●
     Same transformation
     ●
     Different transformation
     ●
     Code can be
     – Left untransformed (using the original repr.)
     – Transformed using different representations
     

     View Slide

103. scala-ildl.org
     Scope Composition
     Scope Composition
     calling
     ●
     Original code
     ●
     Transformed code
     ●
     Original code
     ●
     Transformed code
     ●
     Same transformation
     ●
     Different transformation
     ●
     Code can be
     – Left untransformed (using the original repr.)
     – Transformed using different representations
     

     View Slide

104. scala-ildl.org
     Scope Composition – Case 2
     Scope Composition – Case 2
     transform(VectorOfEmployeeOpt) {
     def indexSalary(vector: Vector[Employee]) = ...
     }
     

     View Slide

105. scala-ildl.org
     // in another file, compiled separately:
     Scope Composition – Case 2
     Scope Composition – Case 2
     transform(VectorOfEmployeeOpt) {
     def indexSalary(vector: Vector[Employee]) = ...
     }
     transform(VectorOfEmployeeOpt) {
     val employees = loadEmployees(...)
     indexSalary(employees)
     }
     

     View Slide

106. scala-ildl.org
     // in another file, compiled separately:
     Scope Composition – Case 2
     Scope Composition – Case 2
     transform(VectorOfEmployeeOpt) {
     def indexSalary(vector: Vector[Employee]) = ...
     }
     transform(VectorOfEmployeeOpt) {
     val employees = loadEmployees(...)
     indexSalary(employees)
     }
     

     View Slide

107. scala-ildl.org
     // in another file, compiled separately:
     Scope Composition – Case 2
     Scope Composition – Case 2
     transform(VectorOfEmployeeOpt) {
     def indexSalary(vector: Vector[Employee]) = ...
     }
     Scopes compose across
     separate compilation
     transform(VectorOfEmployeeOpt) {
     val employees = loadEmployees(...)
     indexSalary(employees)
     }
     

     View Slide

108. scala-ildl.org
     Scope Composition – Case 2
     Scope Composition – Case 2
     ●
     Values transformed identically
     – Are passed without conversions
     – Even in different files
     – Even across separate compilation
     

     View Slide

109. scala-ildl.org
     Scope Composition
     Scope Composition
     calling
     ●
     Original code
     ●
     Transformed code
     ●
     Original code
     ●
     Transformed code
     ●
     Same transformation
     ●
     Different transformation
     ●
     Code can be
     – Left untransformed (using the original repr.)
     – Transformed using different representations
     

     View Slide

110. scala-ildl.org
     Scope Composition
     Scope Composition
     calling
     ●
     Original code
     ●
     Transformed code
     ●
     Original code
     ●
     Transformed code
     ●
     Same transformation
     ●
     Different transformation
     ●
     Code can be
     – Left untransformed (using the original repr.)
     – Transformed using different representations
     

     View Slide

111. scala-ildl.org
     Scope Composition
     Scope Composition
     calling
     ●
     Original code
     ●
     Transformed code
     ●
     Original code
     ●
     Transformed code
     ●
     Same transformation
     ●
     Different transformation
     ●
     Code can be
     – Left untransformed (using the original repr.)
     – Transformed using different representations
     double conversions
     

     View Slide

112. scala-ildl.org
     Scope Composition
     Scope Composition
     calling
     ●
     Original code
     ●
     Transformed code
     ●
     Original code
     ●
     Transformed code
     ●
     Same transformation
     ●
     Different transformation
     ●
     Code can be
     – Left untransformed (using the original repr.)
     – Transformed using different representations
     

     View Slide

113. scala-ildl.org
     Scope Composition
     Scope Composition
     calling
     overriding
     ●
     Original code
     ●
     Transformed code
     ●
     Original code
     ●
     Transformed code
     ●
     Same transformation
     ●
     Different transformation
     ●
     Code can be
     – Left untransformed (using the original repr.)
     – Transformed using different representations
     

     View Slide

114. scala-ildl.org
     Scope Composition
     Scope Composition
     calling
     overriding
     ●
     Original code
     ●
     Transformed code
     ●
     Original code
     ●
     Transformed code
     ●
     Same transformation
     ●
     Different transformation
     ●
     Code can be
     – Left untransformed (using the original repr.)
     – Transformed using different representations
     The transformation has to
     preserve the object model.
     

     View Slide

115. scala-ildl.org
     Scope Composition
     Scope Composition
     calling
     overriding
     ●
     Original code
     ●
     Transformed code
     ●
     Original code
     ●
     Transformed code
     ●
     Same transformation
     ●
     Different transformation
     ●
     Code can be
     – Left untransformed (using the original repr.)
     – Transformed using different representations
     The transformation has to
     preserve the object model.
     All these cases are handled automatically
     by the ildl transformation!
     

     View Slide

116. scala-ildl.org
     Research Challenges
     Research Challenges
     ●
     Chaining transformations
     – A B
     → + B C
     → = A C
     →
     

     View Slide

117. scala-ildl.org
     Research Challenges
     Research Challenges
     ●
     Chaining transformations
     – A B
     → + B C
     → = A C
     →
     ●
     Interaction with Generics
     – We can transform List[(Int, Int)] to List[Long]
     – We can transform List[T] for any T
     

     View Slide

118. scala-ildl.org
     Research Challenges
     Research Challenges
     ●
     Chaining transformations
     – A B
     → + B C
     → = A C
     →
     ●
     Interaction with Generics
     – We can transform List[(Int, Int)] to List[Long]
     – We can transform List[T] for any T
     – But only transforming (Int, Int) to Long
     ●
     Doesn't transform List[(Int, Int)] to List[Long]
     ●
     We have to do it separately
     

     View Slide

119. scala-ildl.org
     Motivation
     Transformation
     Applications
     Challenges
     Conclusion
     Related Work
     Open-world
     Best Represenation?
     Composition
     

     View Slide

120. scala-ildl.org
     Array of Stuct
     Array of Stuct (Column-oriented)
     (Column-oriented)
     NAME ...
     NAME
     EmployeeVector
     ID ID ...
     ...
     SALARY SALARY
     Vector[Employee]
     ID NAME SALARY
     ID NAME SALARY
     

     View Slide

121. scala-ildl.org
     Array of Stuct
     Array of Stuct (Column-oriented)
     (Column-oriented)
     NAME ...
     NAME
     EmployeeVector
     ID ID ...
     ...
     SALARY SALARY
     Vector[Employee]
     ID NAME SALARY
     ID NAME SALARY
     5x faster
     

     View Slide

122. scala-ildl.org
     Specialization
     Specialization and stack allocation
     and stack allocation
     3 5
     (3,5)
     Tuples in Scala
     are generic so
     they need to use
     pointers and objects
     

     View Slide

123. scala-ildl.org
     Specialization
     Specialization and stack allocation
     and stack allocation
     3 5
     3 5
     (3,5) (3,5)
     Tuples in Scala
     are generic so
     they need to use
     pointers and objects
     + stack
     allocation
     

     View Slide

124. scala-ildl.org
     + stack
     allocation
     Specialization
     Specialization and stack allocation
     and stack allocation
     14x faster
     reduced memory footprint
     3 5
     3 5
     (3,5) (3,5)
     Tuples in Scala
     are generic so
     they need to use
     pointers and objects
     

     View Slide

125. scala-ildl.org
     Deforestation
     Deforestation
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     

     View Slide

126. scala-ildl.org
     Deforestation
     Deforestation
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     List(2,3,4)
     

     View Slide

127. scala-ildl.org
     Deforestation
     Deforestation
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     List(2,3,4) List(4,6,8)
     

     View Slide

128. scala-ildl.org
     Deforestation
     Deforestation
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     List(2,3,4) List(4,6,8) 18
     

     View Slide

129. scala-ildl.org
     Deforestation
     Deforestation
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     List(2,3,4) List(4,6,8) 18
     

     View Slide

130. scala-ildl.org
     Deforestation
     Deforestation
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     List(2,3,4) List(4,6,8) 18
     transform(ListDeforestation) {
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     }
     

     View Slide

131. scala-ildl.org
     Deforestation
     Deforestation
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     List(2,3,4) List(4,6,8) 18
     transform(ListDeforestation) {
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     }
     accumulate
     function
     

     View Slide

132. scala-ildl.org
     Deforestation
     Deforestation
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     List(2,3,4) List(4,6,8) 18
     transform(ListDeforestation) {
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     }
     accumulate
     function
     accumulate
     function
     

     View Slide

133. scala-ildl.org
     Deforestation
     Deforestation
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     List(2,3,4) List(4,6,8) 18
     transform(ListDeforestation) {
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     }
     accumulate
     function
     accumulate
     function
     compute:
     18
     

     View Slide

134. scala-ildl.org
     Deforestation
     Deforestation
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     List(2,3,4) List(4,6,8) 18
     transform(ListDeforestation) {
     List(1,2,3).map(_ + 1).map(_ * 2).sum
     }
     accumulate
     function
     accumulate
     function
     compute:
     18
     6x faster
     

     View Slide

135. scala-ildl.org
     Research Challenge
     Research Challenge
     ●
     Scopes make speculations constraints
     →
     ●
     Programmers need to make sure the code conforms
     – Deforestation requires pure functions
     – Otherwise it reorders effects → broken semantics
     ●
     We need a static language
     – For specifying the constraints
     – For verifying these on the actual scope
     

     View Slide

136. scala-ildl.org
     Motivation
     Transformation
     Applications
     Challenges
     Conclusion
     Related Work
     Open-world
     Best Represenation?
     Composition
     

     View Slide

137. scala-ildl.org
     Multi-Stage Programming
     Multi-Stage Programming
     ●
     Multi-Stage Programming
     – “Abstraction without regret” - Tiark Rompf
     – DSLs small enough to be staged
     →
     ●
     10000x speed improvements
     

     View Slide

138. scala-ildl.org
     Multi-Stage Programming
     Multi-Stage Programming
     ●
     Multi-Stage Programming
     – “Abstraction without regret” - Tiark Rompf
     – DSLs small enough to be staged
     →
     ●
     10000x speed improvements
     – Scala many features not supported by LMS:
     →
     ●
     Separate compilation/modularization
     ●
     Dynamic dispatch
     ●
     Aliasing
     ●
     Reflection
     

     View Slide

139. scala-ildl.org
     Multi-Stage Programming
     Multi-Stage Programming
     ●
     Multi-Stage Programming
     – “Abstraction without regret” - Tiark Rompf
     – DSLs small enough to be staged
     →
     ●
     10000x speed improvements
     – Scala many features not supported by LMS:
     →
     ●
     Separate compilation/modularization
     ●
     Dynamic dispatch
     ●
     Aliasing
     ●
     Reflection
     If we add support,
     we lose the ability
     to optimize code :(
     

     View Slide

140. scala-ildl.org
     Low-level Optimizers
     Low-level Optimizers
     ●
     JIT optimizers with virtual machine support
     – Access to the low-level code
     – Can assume a (local) closed world
     – Can speculate based on profiles
     – On the critical path – limited analysis
     

     View Slide

141. scala-ildl.org
     Low-level Optimizers
     Low-level Optimizers
     ●
     JIT optimizers with virtual machine support
     – Access to the low-level code
     – Can assume a (local) closed world
     – Can speculate based on profiles
     – On the critical path – limited analysis
     – Biggest opportunities are high-level - O(n2) O(n)
     →
     ●
     Incoming code is low-level
     ●
     Rarely possible to recover opportunities
     

     View Slide

142. scala-ildl.org
     Low-level Optimizers
     Low-level Optimizers
     ●
     JIT optimizers with virtual machine support
     – Access to the low-level code
     – Can assume a (local) closed world
     – Can speculate based on profiles
     – On the critical path – limited analysis
     – Biggest opportunities are high-level - O(n2) O(n)
     →
     ●
     Incoming code is low-level
     ●
     Rarely possible to recover opportunities
     Typical solution:
     Metaprogramming
     

     View Slide

143. scala-ildl.org
     Metaprogramming
     Metaprogramming
     ●
     Not your grandpa's C preprocessor
     def optimize(tree: AST): AST = {
     ...
     }
     

     View Slide

144. scala-ildl.org
     Metaprogramming
     Metaprogramming
     ●
     Not your grandpa's C preprocessor
     ●
     Full-fledged program transformers
     – :) Lots of power
     def optimize(tree: AST): AST = {
     ...
     }
     

     View Slide

145. scala-ildl.org
     Metaprogramming
     Metaprogramming
     ●
     Not your grandpa's C preprocessor
     ●
     Full-fledged program transformers
     – :) Lots of power
     – :( Lots of responsibility
     def optimize(tree: AST): AST = {
     ...
     }
     

     View Slide

146. scala-ildl.org
     Metaprogramming
     Metaprogramming
     ●
     Not your grandpa's C preprocessor
     ●
     Full-fledged program transformers
     – :) Lots of power
     – :( Lots of responsibility
     ●
     Compiler invariants
     ●
     Object-oriented model
     ●
     Modularity
     def optimize(tree: AST): AST = {
     ...
     }
     

     View Slide

147. scala-ildl.org
     Metaprogramming
     Metaprogramming
     ●
     Not your grandpa's C preprocessor
     ●
     Full-fledged program transformers
     – :) Lots of power
     – :( Lots of responsibility
     ●
     Compiler invariants
     ●
     Object-oriented model
     ●
     Modularity
     def optimize(tree: AST): AST = {
     ...
     }
     Can we make
     metaprogramming
     “high-level”?
     

     View Slide

148. scala-ildl.org
     Motivation
     Transformation
     Applications
     Challenges
     Conclusion
     Related Work
     Open-world
     Best Represenation?
     Composition
     

     View Slide

149. scala-ildl.org
     Conclusion
     Conclusion
     ●
     Object-oriented composition inefcient representation
     →
     ●
     Solution: data-centric metaprogramming
     – Splitting the responsibility:
     ●
     Defining the Transformation programmer
     →
     ●
     Applying the Transformation compiler
     →
     – Scopes
     ●
     Adapt the data representation to the operation
     ●
     Allow speculating properties of the scope
     ●
     We've just begun to scratch the surface
     – Many interesting research questions lie ahead
     

     View Slide

150. scala-ildl.org
     

     View Slide

151. scala-ildl.org
     Thank you!
     

     View Slide