$30 off During Our Annual Pro Sale. View Details »

Code as data (RubyConfBY 2019 edition)

Code as data (RubyConfBY 2019 edition)

Algorithms are typically encoded in code. Sometimes, however, letting non-developers modify algorithms can be beneficial — but for that, we'll have to move the implementation of the algorithm from code into data. Doing so yields a bunch of interesting advantages.

Imagine that you're implementing a complex algorithm that encapsulates some business process at your company. The business stakeholders are pleased, but sometimes come to you with questions, such as

Why is this calculation result so high?
Can you please tweak some factors in the algorithm?

One-off requests like these are probably fine, but occasionally they can be so numerous that you end up spending a significant amount of your time dealing with incoming requests. (You have more interesting stuff to do!) Ideally, business stakeholders themselves would be able to figure out why that calculation result is so high, and would be able to change the factors themselves.

The implementation of the algorithm is in code – and it is typically not feasible to let business stakeholders handle code. (They have more interesting stuff to do!) We can move the algorithm's implementation out of code and into data… and that yields us a bunch of interesting advantages.

Denis Defreyne

April 06, 2019
Tweet

More Decks by Denis Defreyne

Other Decks in Technology

Transcript

  1. CODE AS DATA
    DENIS DEFREYNE (PRONOUNS: HE/HIM)
    @DDFREYNE / DENIS.WS
    RUBYCONFBY 2019
    2019-04-06

    View Slide

  2. nanoc / cri / ddmemoize / ddmetrics / …

    View Slide

  3. I work at

    View Slide

  4. $$$

    View Slide


  5. View Slide

  6. Why do we need a pricing algorithm?




    View Slide

  7. Why do we need a pricing algorithm?


    * fixed price?


    View Slide

  8. Why do we need a pricing algorithm?


    * fixed price? unfair.


    View Slide

  9. Why do we need a pricing algorithm?


    * fixed price? unfair.

    * price per hour?


    View Slide

  10. Why do we need a pricing algorithm?


    * fixed price? unfair.

    * price per hour? hard to calculate.


    View Slide

  11. Why do we need a pricing algorithm?


    * fixed price? unfair.

    * price per hour? hard to calculate.

    * wait and see what the price turns out to be?

    View Slide

  12. Why do we need a pricing algorithm?


    * fixed price? unfair.

    * price per hour? hard to calculate.

    * wait and see what the price turns out to be? risky.

    View Slide

  13. Why do we need a pricing algorithm?


    * fixed price? unfair.

    * price per hour? hard to calculate.
    * wait and see what the price turns out to be? risky.

    * algorithm?

    View Slide

  14. Why do we need a pricing algorithm?


    * fixed price? unfair.

    * price per hour? hard to calculate.
    * wait and see what the price turns out to be? risky.

    * algorithm? yes!

    View Slide

  15. $$$

    View Slide

  16. $$$
    * Business need to know

    why prices are the way they are






    View Slide

  17. $$$
    * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas



    View Slide

  18. $$$
    * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas



    View Slide

  19. $$$
    * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas

    View Slide

  20. def total_price
    200 + volume * distance
    end

    View Slide

  21. THE IDEA

    View Slide

  22. def total_price
    200 + volume * distance
    end

    View Slide

  23. def total_price
    200 + volume * distance
    end
    sum

    View Slide

  24. def total_price
    200 + volume * distance
    end
    sum
    product

    View Slide

  25. def total_price
    200 + volume * distance
    end
    sum
    product
    variable

    View Slide

  26. def total_price
    200 + volume * distance
    end
    sum
    product
    constant
    variable

    View Slide

  27. View Slide

  28. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end

    View Slide

  29. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    class ProductExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end

    View Slide

  30. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    class ProductExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    class ConstantExpr
    def initialize(value)
    @value = value
    end
    end


    View Slide

  31. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    class ProductExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    end
    class ConstantExpr
    def initialize(value)
    @value = value
    end
    end

    class VariableExpr
    def initialize(name)
    @name = name
    end
    end


    View Slide

  32. def price_formula
    SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Tvolume),
    VariableExpr.new(Tdistance),
    )
    )
    end

    View Slide

  33. params = {
    volume: 20,
    distance: 200,
    }
    price_formula.evaluate(params)

    View Slide



  34. params = {
    volume: 20,
    distance: 200,
    }
    price_formula.evaluate(params)


    4200

    View Slide

  35. def evaluate(params)

    View Slide

  36. class VariableExpr
    def initialize(name)
    @name = name
    end
    def evaluate(params)
    params.fetch(@name)
    end
    end

    View Slide

  37. class ConstantExpr
    def initialize(value)
    @value = value
    end
    def evaluate(_params)
    @value
    end
    end

    View Slide

  38. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    def evaluate(params)
    @left.evaluate(params) + @right.evaluate(params)
    end
    end


    View Slide

  39. class ProductExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    def evaluate(params)
    @left.evaluate(params) * @right.evaluate(params)
    end
    end


    View Slide



  40. params = {
    volume: 20,
    distance: 200,
    }


    View Slide



  41. params = {
    volume: 20,
    distance: 200,
    }
    price_formula.evaluate(params)


    View Slide



  42. params = {
    volume: 20,
    distance: 200,
    }
    price_formula.evaluate(params)


    4200

    View Slide

  43. THE POINT

    View Slide

  44. SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Tvolume),
    VariableExpr.new(Tdistance),
    )
    )






    View Slide

  45. SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Tvolume),
    VariableExpr.new(Tdistance),
    )
    )


    + Z[> 4200
    const Z[> 200
    * Z[> 4000
    var volume Z[> 20
    var distance Z[> 200

    View Slide

  46. class Node
    def initialize(name, value, children)
    @name = name
    @value = value
    @children = children
    end
    end

    View Slide

  47. + Z[> 4200
    const Z[> 200
    * Z[> 4000
    var volume Z[> 20
    var distance Z[> 200

    View Slide

  48. + Z[> 4200
    const Z[> 200
    * Z[> 4000
    var volume Z[> 20
    var distance Z[> 200
    Node.new("+", 4200, [const_node, product_node])

    View Slide

  49. + Z[> 4200
    const Z[> 200
    * Z[> 4000
    var volume Z[> 20
    var distance Z[> 200
    Node.new("+", 4200, [const_node, product_node])
    Node.new("var volume", 20, [])

    View Slide

  50. View Slide

  51. class VariableExpr
    def initialize(name)
    @name = name
    end
    def evaluate(params)
    params.fetch(@name)
    end
    end

    View Slide

  52. class VariableExpr
    def initialize(name)
    @name = name
    end
    def evaluate(params)
    Node.new(
    "var #{@name}",
    params.fetch(@name),
    [],
    )
    end
    end

    View Slide

  53. class ConstantExpr
    def initialize(value)
    @value = value
    end
    def evaluate(_params)
    @value
    end
    end

    View Slide

  54. class ConstantExpr
    def initialize(value)
    @value = value
    end
    def evaluate(_params)
    Node.new(
    'const',
    @value,
    [],
    )
    end
    end

    View Slide

  55. class SumExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    def evaluate(params)
    left_node = @left.evaluate(params)
    right_node = @right.evaluate(params)
    Node.new(
    '+',
    left_node.value + right_node.value,
    [left_node, right_node],
    )
    end
    end

    View Slide

  56. class ProductExpr
    def initialize(left, right)
    @left = left
    @right = right
    end
    def evaluate(params)
    left_node = @left.evaluate(params)
    right_node = @right.evaluate(params)
    Node.new(
    '*',
    left_node.value * right_node.value,
    [left_node, right_node],
    )
    end
    end

    View Slide

  57. class Node
    def to_s
    # … left to your imagination …
    end
    end

    View Slide

  58. puts price_formula.evaluate(params)






    View Slide

  59. puts price_formula.evaluate(params)
    + >?> 4200
    const >?> 200
    * >?> 4000
    var volume >?> 20
    var distance >?> 200

    View Slide

  60. puts price_formula.evaluate(params).value






    View Slide

  61. puts price_formula.evaluate(params).value


    4200




    View Slide

  62. THE BEST PART:
    The price formula did not change!

    View Slide

  63. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas

    View Slide

  64. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas

    View Slide

  65. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas

    View Slide

  66. THE POINT (PART II): THE GUI

    View Slide


  67. View Slide

  68. "200 + volume * distance"






    View Slide

  69. "200 + volume * distance"
    C> SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Nvolume),
    VariableExpr.new(Ndistance),
    )
    )

    View Slide

  70. parse("200 + volume * distance")
    C> SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Nvolume),
    VariableExpr.new(Ndistance),
    )
    )

    View Slide

  71. require 'd-parse'

    View Slide

  72. module Grammar
    extend DParseijDSL
    # …
    end

    View Slide

  73. DIGIT =
    char_in('0'..'9')

    View Slide

  74. CONSTANT_EXPR =
    repeat1(DIGIT)
    .capture
    .map { |d| ConstantExpr.new(d.to_i(10)) }

    View Slide

  75. VARIABLE_EXPR =
    repeat1(char_in('a'..'z'))
    .capture
    .map { |d| VariableExpr.new(d) }

    View Slide

  76. OPERAND =
    alt(
    CONSTANT_EXPR,
    VARIABLE_EXPR,
    )

    View Slide

  77. OPERATOR =
    alt(
    char('+'),
    char('*'),
    ).capture

    View Slide

  78. OP_SEQ =
    intersperse(
    OPERAND,
    OPERATOR,
    ).map { |d| OpSeqExpr.new(d) }

    View Slide

  79. ROOT =
    seq(
    OP_SEQ,
    eof,
    ).first

    View Slide

  80. input =
    "200 + volume * distance"

    View Slide

  81. input =
    "200 + volume * distance"
    raw_expr =
    GrammarijROOT.apply(
    input_string,
    )

    View Slide

  82. OpSeqExpr.new(
    [
    ConstantExpr.new(200),
    "+",
    VariableExpr.new(Tvolume),
    "*",
    VariableExpr.new(Tdistance),
    ]
    )

    View Slide

  83. def resolve_op_seq_exprs(expr)
    # … apply shunting yard or so …
    end

    View Slide

  84. resolve_op_seq_exprs(raw_expr)

    View Slide

  85. SumExpr.new(
    ConstantExpr.new(200),
    ProductExpr.new(
    VariableExpr.new(Tvolume),
    VariableExpr.new(Tdistance),
    )
    )

    View Slide

  86. input =
    "200 + volume * distance"

    View Slide

  87. input =
    "200 + volume * distance"
    GrammarijROOT.apply(
    input,
    )

    View Slide

  88. input =
    "200 + volume * distance"
    resolve_op_seq_exprs(
    GrammarijROOT.apply(
    input,
    ),
    )

    View Slide

  89. input =
    "200 + volume * distance"
    price_formula =
    resolve_op_seq_exprs(
    GrammarijROOT.apply(
    input,
    ),
    )

    View Slide

  90. params = {
    volume: 20,
    distance: 200,
    }


    View Slide

  91. params = {
    volume: 20,
    distance: 200,
    }
    puts price_formula.evaluate(params).value


    View Slide

  92. params = {
    volume: 20,
    distance: 200,
    }
    puts price_formula.evaluate(params).value


    4200

    View Slide

  93. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas

    View Slide

  94. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas

    View Slide

  95. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas

    View Slide

  96. * Business need to know

    why prices are the way they are

    * Developers need to be involved

    to implement new formulas

    * Developers have no good way

    to verify new formulas

    * It is difficult

    to dry-run new formulas

    View Slide

  97. THE POINT (PART III): MORE BENEFITS

    View Slide

  98. MORE BENEFITS:


    View Slide

  99. MORE BENEFITS:
    * Versioning & locking


    View Slide

  100. MORE BENEFITS:
    * Versioning & locking

    * Translate to Ruby

    View Slide

  101. MORE BENEFITS:
    * Versioning & locking

    * Translate to Ruby
    * Translate to SQL


    View Slide

  102. MORE BENEFITS:
    * Versioning & locking

    * Translate to Ruby
    * Translate to SQL

    * Compile with LLVM

    View Slide

  103. Is this over-engineering?

    View Slide

  104. TAKE-AWAYS

    View Slide

  105. * It’s much easier to reason about data

    than to reason about code.


    View Slide

  106. * It’s much easier to reason about data

    than to reason about code.
    * Try novel approaches; they can

    reveal new opportunities.

    View Slide

  107. DENIS DEFREYNE
    @DDFREYNE / DENIS.WS

    View Slide