Code as data (RubyConfBY 2019 edition)

Algorithms are typically encoded in code. Sometimes, however, letting non-developers modify algorithms can be beneficial — but for that, we'll have to move the implementation of the algorithm from code into data. Doing so yields a bunch of interesting advantages.

Imagine that you're implementing a complex algorithm that encapsulates some business process at your company. The business stakeholders are pleased, but sometimes come to you with questions, such as

Why is this calculation result so high?
Can you please tweak some factors in the algorithm?

One-off requests like these are probably fine, but occasionally they can be so numerous that you end up spending a significant amount of your time dealing with incoming requests. (You have more interesting stuff to do!) Ideally, business stakeholders themselves would be able to figure out why that calculation result is so high, and would be able to change the factors themselves.

The implementation of the algorithm is in code – and it is typically not feasible to let business stakeholders handle code. (They have more interesting stuff to do!) We can move the algorithm's implementation out of code and into data… and that yields us a bunch of interesting advantages.

Transcript

1. CODE AS DATA
   DENIS DEFREYNE (PRONOUNS: HE/HIM)
   @DDFREYNE / DENIS.WS
   RUBYCONFBY 2019
   2019-04-06
   

   View Slide

2. nanoc / cri / ddmemoize / ddmetrics / …
   

   View Slide

3. I work at
   

   View Slide

4. $$$
   

   View Slide

5. 
   

   View Slide

6. Why do we need a pricing algorithm?

   

   

   

   

   View Slide

7. Why do we need a pricing algorithm?

   

   * fixed price?

   

   

   View Slide

8. Why do we need a pricing algorithm?

   

   * fixed price? unfair.

   

   

   View Slide

9. Why do we need a pricing algorithm?

   

   * fixed price? unfair.

   * price per hour?

   

   View Slide

10. Why do we need a pricing algorithm?

    

    * fixed price? unfair.

    * price per hour? hard to calculate.

    

    View Slide

11. Why do we need a pricing algorithm?

    

    * fixed price? unfair.

    * price per hour? hard to calculate.

    * wait and see what the price turns out to be?
    

    View Slide

12. Why do we need a pricing algorithm?

    

    * fixed price? unfair.

    * price per hour? hard to calculate.

    * wait and see what the price turns out to be? risky.
    

    View Slide

13. Why do we need a pricing algorithm?

    

    * fixed price? unfair.

    * price per hour? hard to calculate.
    * wait and see what the price turns out to be? risky.

    * algorithm?
    

    View Slide

14. Why do we need a pricing algorithm?

    

    * fixed price? unfair.

    * price per hour? hard to calculate.
    * wait and see what the price turns out to be? risky.

    * algorithm? yes!
    

    View Slide

15. $$$
    

    View Slide

16. $$$
    * Business need to know

    why prices are the way they are

    

    

    

    

    

    

    View Slide

17. $$$
    * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    

    

    

    View Slide

18. $$$
    * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    

    

    

    View Slide

19. $$$
    * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas
    

    View Slide

20. def total_price
    200 + volume * distance
    end
    

    View Slide

21. THE IDEA
    

    View Slide

22. def total_price
    200 + volume * distance
    end
    

    View Slide

23. def total_price
    200 + volume * distance
    end
    sum
    

    View Slide

24. def total_price
    200 + volume * distance
    end
    sum
    product
    

    View Slide

25. def total_price
    200 + volume * distance
    end
    sum
    product
    variable
    

    View Slide

26. def total_price
    200 + volume * distance
    end
    sum
    product
    constant
    variable
    

    View Slide

27. View Slide

28. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    

    View Slide

29. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    class ProductExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    

    View Slide

30. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    class ProductExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    class ConstantExpr
    def initialize(value)
    @value = value
    end
    end

    

    View Slide

31. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    class ProductExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    class ConstantExpr
    def initialize(value)
    @value = value
    end
    end

    class VariableExpr
    def initialize(name)
    @name = name
    end
    end

    

    View Slide

32. def price_formula
    SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Tvolume),
    VariableExpr.new(Tdistance),
    )
    )
    end
    

    View Slide

33. params = {
    volume: 20,
    distance: 200,
    }
    price_formula.evaluate(params)
    

    View Slide

34. 

    

    params = {
    volume: 20,
    distance: 200,
    }
    price_formula.evaluate(params)

    

    4200
    

    View Slide

35. def evaluate(params)
    

    View Slide

36. class VariableExpr
    def initialize(name)
    @name = name
    end
    def evaluate(params)
    params.fetch(@name)
    end
    end
    

    View Slide

37. class ConstantExpr
    def initialize(value)
    @value = value
    end
    def evaluate(_params)
    @value
    end
    end
    

    View Slide

38. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    def evaluate(params)
    @left.evaluate(params) + @right.evaluate(params)
    end
    end

    

    View Slide

39. class ProductExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    def evaluate(params)
    @left.evaluate(params) * @right.evaluate(params)
    end
    end

    

    View Slide

40. 

    

    params = {
    volume: 20,
    distance: 200,
    }
    

    

    

    View Slide

41. 

    

    params = {
    volume: 20,
    distance: 200,
    }
    price_formula.evaluate(params)

    

    

    View Slide

42. 

    

    params = {
    volume: 20,
    distance: 200,
    }
    price_formula.evaluate(params)

    

    4200
    

    View Slide

43. THE POINT
    

    View Slide

44. SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Tvolume),
    VariableExpr.new(Tdistance),
    )
    )

    

    

    

    

    

    

    View Slide

45. SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Tvolume),
    VariableExpr.new(Tdistance),
    )
    )

    

    + Z[> 4200
    const Z[> 200
    * Z[> 4000
    var volume Z[> 20
    var distance Z[> 200
    

    View Slide

46. class Node
    def initialize(name, value, children)
    @name = name
    @value = value
    @children = children
    end
    end
    

    View Slide

47. + Z[> 4200
    const Z[> 200
    * Z[> 4000
    var volume Z[> 20
    var distance Z[> 200
    

    View Slide

48. + Z[> 4200
    const Z[> 200
    * Z[> 4000
    var volume Z[> 20
    var distance Z[> 200
    Node.new("+", 4200, [const_node, product_node])
    

    View Slide

49. + Z[> 4200
    const Z[> 200
    * Z[> 4000
    var volume Z[> 20
    var distance Z[> 200
    Node.new("+", 4200, [const_node, product_node])
    Node.new("var volume", 20, [])
    

    View Slide

50. View Slide

51. class VariableExpr
    def initialize(name)
    @name = name
    end
    def evaluate(params)
    params.fetch(@name)
    end
    end
    

    View Slide

52. class VariableExpr
    def initialize(name)
    @name = name
    end
    def evaluate(params)
    Node.new(
    "var #{@name}",
    params.fetch(@name),
    [],
    )
    end
    end
    

    View Slide

53. class ConstantExpr
    def initialize(value)
    @value = value
    end
    def evaluate(_params)
    @value
    end
    end
    

    View Slide

54. class ConstantExpr
    def initialize(value)
    @value = value
    end
    def evaluate(_params)
    Node.new(
    'const',
    @value,
    [],
    )
    end
    end
    

    View Slide

55. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    def evaluate(params)
    left_node = @left.evaluate(params)
    right_node = @right.evaluate(params)
    Node.new(
    '+',
    left_node.value + right_node.value,
    [left_node, right_node],
    )
    end
    end
    

    View Slide

56. class ProductExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    def evaluate(params)
    left_node = @left.evaluate(params)
    right_node = @right.evaluate(params)
    Node.new(
    '*',
    left_node.value * right_node.value,
    [left_node, right_node],
    )
    end
    end
    

    View Slide

57. class Node
    def to_s
    # … left to your imagination …
    end
    end
    

    View Slide

58. puts price_formula.evaluate(params)

    

    

    

    

    

    

    View Slide

59. puts price_formula.evaluate(params)
    + >?> 4200
    const >?> 200
    * >?> 4000
    var volume >?> 20
    var distance >?> 200
    

    View Slide

60. puts price_formula.evaluate(params).value

    

    

    

    

    

    

    View Slide

61. puts price_formula.evaluate(params).value

    

    4200

    

    

    

    

    View Slide

62. THE BEST PART:
    The price formula did not change!
    

    View Slide

63. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas
    

    View Slide

64. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas
    

    View Slide

65. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas
    

    View Slide

66. THE POINT (PART II): THE GUI
    

    View Slide

67. 
    

    View Slide

68. "200 + volume * distance"
    

    

    

    

    

    

    

    View Slide

69. "200 + volume * distance"
    C> SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Nvolume),
    VariableExpr.new(Ndistance),
    )
    )
    

    View Slide

70. parse("200 + volume * distance")
    C> SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Nvolume),
    VariableExpr.new(Ndistance),
    )
    )
    

    View Slide

71. require 'd-parse'
    

    View Slide

72. module Grammar
    extend DParseijDSL
    # …
    end
    

    View Slide

73. DIGIT =
    char_in('0'..'9')
    

    View Slide

74. CONSTANT_EXPR =
    repeat1(DIGIT)
    .capture
    .map { |d| ConstantExpr.new(d.to_i(10)) }
    

    View Slide

75. VARIABLE_EXPR =
    repeat1(char_in('a'..'z'))
    .capture
    .map { |d| VariableExpr.new(d) }
    

    View Slide

76. OPERAND =
    alt(
    CONSTANT_EXPR,
    VARIABLE_EXPR,
    )
    

    View Slide

77. OPERATOR =
    alt(
    char('+'),
    char('*'),
    ).capture
    

    View Slide

78. OP_SEQ =
    intersperse(
    OPERAND,
    OPERATOR,
    ).map { |d| OpSeqExpr.new(d) }
    

    View Slide

79. ROOT =
    seq(
    OP_SEQ,
    eof,
    ).first
    

    View Slide

80. input =
    "200 + volume * distance"
    

    View Slide

81. input =
    "200 + volume * distance"
    raw_expr =
    GrammarijROOT.apply(
    input_string,
    )
    

    View Slide

82. OpSeqExpr.new(
    [
    ConstantExpr.new(200),
    "+",
    VariableExpr.new(Tvolume),
    "*",
    VariableExpr.new(Tdistance),
    ]
    )
    

    View Slide

83. def resolve_op_seq_exprs(expr)
    # … apply shunting yard or so …
    end
    

    View Slide

84. resolve_op_seq_exprs(raw_expr)
    

    View Slide

85. SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Tvolume),
    VariableExpr.new(Tdistance),
    )
    )
    

    View Slide

86. input =
    "200 + volume * distance"
    

    View Slide

87. input =
    "200 + volume * distance"
    GrammarijROOT.apply(
    input,
    )
    

    View Slide

88. input =
    "200 + volume * distance"
    resolve_op_seq_exprs(
    GrammarijROOT.apply(
    input,
    ),
    )
    

    View Slide

89. input =
    "200 + volume * distance"
    price_formula =
    resolve_op_seq_exprs(
    GrammarijROOT.apply(
    input,
    ),
    )
    

    View Slide

90. params = {
    volume: 20,
    distance: 200,
    }
    

    

    

    View Slide

91. params = {
    volume: 20,
    distance: 200,
    }
    puts price_formula.evaluate(params).value

    

    

    View Slide

92. params = {
    volume: 20,
    distance: 200,
    }
    puts price_formula.evaluate(params).value

    

    4200
    

    View Slide

93. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas
    

    View Slide

94. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas
    

    View Slide

95. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas
    

    View Slide

96. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas
    

    View Slide

97. THE POINT (PART III): MORE BENEFITS
    

    View Slide

98. MORE BENEFITS:
    

    

    

    View Slide

99. MORE BENEFITS:
    * Versioning & locking

    

    

    View Slide

100. MORE BENEFITS:
     * Versioning & locking

     * Translate to Ruby
     

     

     View Slide

101. MORE BENEFITS:
     * Versioning & locking

     * Translate to Ruby
     * Translate to SQL

     

     View Slide

102. MORE BENEFITS:
     * Versioning & locking

     * Translate to Ruby
     * Translate to SQL

     * Compile with LLVM
     

     View Slide

103. Is this over-engineering?
     

     View Slide

104. TAKE-AWAYS
     

     View Slide

105. * It’s much easier to reason about data

     than to reason about code.
     

     

     

     View Slide

106. * It’s much easier to reason about data

     than to reason about code.
     * Try novel approaches; they can

     reveal new opportunities.
     

     View Slide

107. DENIS DEFREYNE
     @DDFREYNE / DENIS.WS
     

     View Slide