Code as data (RubyConfBY 2019 edition)

Code as data (RubyConfBY 2019 edition)

Algorithms are typically encoded in code. Sometimes, however, letting non-developers modify algorithms can be beneficial — but for that, we'll have to move the implementation of the algorithm from code into data. Doing so yields a bunch of interesting advantages.

Imagine that you're implementing a complex algorithm that encapsulates some business process at your company. The business stakeholders are pleased, but sometimes come to you with questions, such as

Why is this calculation result so high?
Can you please tweak some factors in the algorithm?

One-off requests like these are probably fine, but occasionally they can be so numerous that you end up spending a significant amount of your time dealing with incoming requests. (You have more interesting stuff to do!) Ideally, business stakeholders themselves would be able to figure out why that calculation result is so high, and would be able to change the factors themselves.

The implementation of the algorithm is in code – and it is typically not feasible to let business stakeholders handle code. (They have more interesting stuff to do!) We can move the algorithm's implementation out of code and into data… and that yields us a bunch of interesting advantages.

Be732ee41fd3038aa98a0a7e7b7be081?s=128

Denis Defreyne

April 06, 2019
Tweet

Transcript

  1. CODE AS DATA DENIS DEFREYNE (PRONOUNS: HE/HIM) @DDFREYNE / DENIS.WS

    RUBYCONFBY 2019 2019-04-06
  2. nanoc / cri / ddmemoize / ddmetrics / …

  3. I work at

  4. $$$

  5. <disclaimer>

  6. Why do we need a pricing algorithm?
 
 
 


  7. Why do we need a pricing algorithm?
 
 * fixed

    price?
 

  8. Why do we need a pricing algorithm?
 
 * fixed

    price? unfair.
 

  9. Why do we need a pricing algorithm?
 
 * fixed

    price? unfair.
 * price per hour?

  10. Why do we need a pricing algorithm?
 
 * fixed

    price? unfair.
 * price per hour? hard to calculate.

  11. Why do we need a pricing algorithm?
 
 * fixed

    price? unfair.
 * price per hour? hard to calculate.
 * wait and see what the price turns out to be?
  12. Why do we need a pricing algorithm?
 
 * fixed

    price? unfair.
 * price per hour? hard to calculate.
 * wait and see what the price turns out to be? risky.
  13. Why do we need a pricing algorithm?
 
 * fixed

    price? unfair.
 * price per hour? hard to calculate. * wait and see what the price turns out to be? risky.
 * algorithm?
  14. Why do we need a pricing algorithm?
 
 * fixed

    price? unfair.
 * price per hour? hard to calculate. * wait and see what the price turns out to be? risky.
 * algorithm? yes!
  15. $$$

  16. $$$ * Business need to know
 why prices are the

    way they are
 
 
 
 
 

  17. $$$ * Business need to know
 why prices are the

    way they are
 * Developers need to be involved
 to implement new formulas
 
 

  18. $$$ * Business need to know
 why prices are the

    way they are
 * Developers need to be involved
 to implement new formulas
 * Developers have no good way
 to verify new formulas
 
 

  19. $$$ * Business need to know
 why prices are the

    way they are
 * Developers need to be involved
 to implement new formulas
 * Developers have no good way
 to verify new formulas
 * It is difficult
 to dry-run new formulas
  20. def total_price 200 + volume * distance end

  21. THE IDEA

  22. def total_price 200 + volume * distance end

  23. def total_price 200 + volume * distance end sum

  24. def total_price 200 + volume * distance end sum product

  25. def total_price 200 + volume * distance end sum product

    variable
  26. def total_price 200 + volume * distance end sum product

    constant variable
  27. None
  28. class SumExpr def initialize(left, right) @left = left @right =

    right end end
  29. class SumExpr def initialize(left, right) @left = left @right =

    right end end class ProductExpr def initialize(left, right) @left = left @right = right end end
  30. class SumExpr def initialize(left, right) @left = left @right =

    right end end class ProductExpr def initialize(left, right) @left = left @right = right end end class ConstantExpr def initialize(value) @value = value end end

  31. class SumExpr def initialize(left, right) @left = left @right =

    right end end class ProductExpr def initialize(left, right) @left = left @right = right end end class ConstantExpr def initialize(value) @value = value end end
 class VariableExpr def initialize(name) @name = name end end

  32. def price_formula SumExpr.new( ConstantExpr.new(200), ProductExpr.new( VariableExpr.new(Tvolume), VariableExpr.new(Tdistance), ) ) end

  33. params = { volume: 20, distance: 200, } price_formula.evaluate(params)

  34. 
 
 params = { volume: 20, distance: 200, }

    price_formula.evaluate(params)
 
 4200
  35. def evaluate(params)

  36. class VariableExpr def initialize(name) @name = name end def evaluate(params)

    params.fetch(@name) end end
  37. class ConstantExpr def initialize(value) @value = value end def evaluate(_params)

    @value end end
  38. class SumExpr def initialize(left, right) @left = left @right =

    right end def evaluate(params) @left.evaluate(params) + @right.evaluate(params) end end

  39. class ProductExpr def initialize(left, right) @left = left @right =

    right end def evaluate(params) @left.evaluate(params) * @right.evaluate(params) end end

  40. 
 
 params = { volume: 20, distance: 200, }

    
 

  41. 
 
 params = { volume: 20, distance: 200, }

    price_formula.evaluate(params)
 

  42. 
 
 params = { volume: 20, distance: 200, }

    price_formula.evaluate(params)
 
 4200
  43. THE POINT

  44. SumExpr.new( ConstantExpr.new(200), ProductExpr.new( VariableExpr.new(Tvolume), VariableExpr.new(Tdistance), ) )
 
 
 


    
 

  45. SumExpr.new( ConstantExpr.new(200), ProductExpr.new( VariableExpr.new(Tvolume), VariableExpr.new(Tdistance), ) )
 
 + Z[>

    4200 const Z[> 200 * Z[> 4000 var volume Z[> 20 var distance Z[> 200
  46. class Node def initialize(name, value, children) @name = name @value

    = value @children = children end end
  47. + Z[> 4200 const Z[> 200 * Z[> 4000 var

    volume Z[> 20 var distance Z[> 200
  48. + Z[> 4200 const Z[> 200 * Z[> 4000 var

    volume Z[> 20 var distance Z[> 200 Node.new("+", 4200, [const_node, product_node])
  49. + Z[> 4200 const Z[> 200 * Z[> 4000 var

    volume Z[> 20 var distance Z[> 200 Node.new("+", 4200, [const_node, product_node]) Node.new("var volume", 20, [])
  50. None
  51. class VariableExpr def initialize(name) @name = name end def evaluate(params)

    params.fetch(@name) end end
  52. class VariableExpr def initialize(name) @name = name end def evaluate(params)

    Node.new( "var #{@name}", params.fetch(@name), [], ) end end
  53. class ConstantExpr def initialize(value) @value = value end def evaluate(_params)

    @value end end
  54. class ConstantExpr def initialize(value) @value = value end def evaluate(_params)

    Node.new( 'const', @value, [], ) end end
  55. class SumExpr def initialize(left, right) @left = left @right =

    right end def evaluate(params) left_node = @left.evaluate(params) right_node = @right.evaluate(params) Node.new( '+', left_node.value + right_node.value, [left_node, right_node], ) end end
  56. class ProductExpr def initialize(left, right) @left = left @right =

    right end def evaluate(params) left_node = @left.evaluate(params) right_node = @right.evaluate(params) Node.new( '*', left_node.value * right_node.value, [left_node, right_node], ) end end
  57. class Node def to_s # … left to your imagination

    … end end
  58. puts price_formula.evaluate(params)
 
 
 
 
 


  59. puts price_formula.evaluate(params) + >?> 4200 const >?> 200 * >?>

    4000 var volume >?> 20 var distance >?> 200
  60. puts price_formula.evaluate(params).value
 
 
 
 
 


  61. puts price_formula.evaluate(params).value
 
 4200
 
 
 


  62. THE BEST PART: The price formula did not change!

  63. * Business need to know
 why prices are the way

    they are
 * Developers need to be involved
 to implement new formulas
 * Developers have no good way
 to verify new formulas
 * It is difficult
 to dry-run new formulas
  64. * Business need to know
 why prices are the way

    they are
 * Developers need to be involved
 to implement new formulas
 * Developers have no good way
 to verify new formulas
 * It is difficult
 to dry-run new formulas
  65. * Business need to know
 why prices are the way

    they are
 * Developers need to be involved
 to implement new formulas
 * Developers have no good way
 to verify new formulas
 * It is difficult
 to dry-run new formulas
  66. THE POINT (PART II): THE GUI

  67. <demozeit>

  68. "200 + volume * distance" 
 
 
 
 


  69. "200 + volume * distance" C> SumExpr.new( ConstantExpr.new(200), ProductExpr.new( VariableExpr.new(Nvolume),

    VariableExpr.new(Ndistance), ) )
  70. parse("200 + volume * distance") C> SumExpr.new( ConstantExpr.new(200), ProductExpr.new( VariableExpr.new(Nvolume),

    VariableExpr.new(Ndistance), ) )
  71. require 'd-parse'

  72. module Grammar extend DParseijDSL # … end

  73. DIGIT = char_in('0'..'9')

  74. CONSTANT_EXPR = repeat1(DIGIT) .capture .map { |d| ConstantExpr.new(d.to_i(10)) }

  75. VARIABLE_EXPR = repeat1(char_in('a'..'z')) .capture .map { |d| VariableExpr.new(d) }

  76. OPERAND = alt( CONSTANT_EXPR, VARIABLE_EXPR, )

  77. OPERATOR = alt( char('+'), char('*'), ).capture

  78. OP_SEQ = intersperse( OPERAND, OPERATOR, ).map { |d| OpSeqExpr.new(d) }

  79. ROOT = seq( OP_SEQ, eof, ).first

  80. input = "200 + volume * distance"

  81. input = "200 + volume * distance" raw_expr = GrammarijROOT.apply(

    input_string, )
  82. OpSeqExpr.new( [ ConstantExpr.new(200), "+", VariableExpr.new(Tvolume), "*", VariableExpr.new(Tdistance), ] )

  83. def resolve_op_seq_exprs(expr) # … apply shunting yard or so …

    end
  84. resolve_op_seq_exprs(raw_expr)

  85. SumExpr.new( ConstantExpr.new(200), ProductExpr.new( VariableExpr.new(Tvolume), VariableExpr.new(Tdistance), ) )

  86. input = "200 + volume * distance"

  87. input = "200 + volume * distance" GrammarijROOT.apply( input, )

  88. input = "200 + volume * distance" resolve_op_seq_exprs( GrammarijROOT.apply( input,

    ), )
  89. input = "200 + volume * distance" price_formula = resolve_op_seq_exprs(

    GrammarijROOT.apply( input, ), )
  90. params = { volume: 20, distance: 200, } 
 


  91. params = { volume: 20, distance: 200, } puts price_formula.evaluate(params).value


  92. params = { volume: 20, distance: 200, } puts price_formula.evaluate(params).value


    
 4200
  93. * Business need to know
 why prices are the way

    they are
 * Developers need to be involved
 to implement new formulas
 * Developers have no good way
 to verify new formulas
 * It is difficult
 to dry-run new formulas
  94. * Business need to know
 why prices are the way

    they are
 * Developers need to be involved
 to implement new formulas
 * Developers have no good way
 to verify new formulas
 * It is difficult
 to dry-run new formulas
  95. * Business need to know
 why prices are the way

    they are
 * Developers need to be involved
 to implement new formulas
 * Developers have no good way
 to verify new formulas
 * It is difficult
 to dry-run new formulas
  96. * Business need to know
 why prices are the way

    they are
 * Developers need to be involved
 to implement new formulas
 * Developers have no good way
 to verify new formulas
 * It is difficult
 to dry-run new formulas
  97. THE POINT (PART III): MORE BENEFITS

  98. MORE BENEFITS: 
 


  99. MORE BENEFITS: * Versioning & locking
 


  100. MORE BENEFITS: * Versioning & locking
 * Translate to Ruby

  101. MORE BENEFITS: * Versioning & locking
 * Translate to Ruby

    * Translate to SQL

  102. MORE BENEFITS: * Versioning & locking
 * Translate to Ruby

    * Translate to SQL
 * Compile with LLVM
  103. Is this over-engineering?

  104. TAKE-AWAYS

  105. * It’s much easier to reason about data
 than to

    reason about code. 
 

  106. * It’s much easier to reason about data
 than to

    reason about code. * Try novel approaches; they can
 reveal new opportunities.
  107. DENIS DEFREYNE @DDFREYNE / DENIS.WS