Pragmatic Monadic Programming in Ruby

Slide 1

Slide 1 text

Pragmatic Monadic Programming in Ruby @joker1007 (Repro inc.) RubyKaigi 2019

Slide 2

Slide 2 text

self.inspect id: @joker1007 name: Tomohiro Hashidate Repro inc. CTO I familiar with ... Ruby/Rails Ruby Black Magic (TracePoint) Speaker of RubyKaigi2018, RubyConf2018

Slide 3

Slide 3 text

We provide a web service as ... Analytics of Mobile Apps, Web Apps. Marketing Automation. Of course, We're hiring!!

Slide 4

Slide 4 text

I am a member of an amazing Ruby community.

Slide 5

Slide 5 text

In ruby-2.6, Proc is very interesting!!

Slide 6

Slide 6 text

Composing proc Proc#>> Proc#<<

Slide 7

Slide 7 text

RubyVM::AST.of can receive a Proc pr = proc { puts :foo } ast = RubyVM::AbstractSyntaxTree.of(pr) pp ast (SCOPE@1:10-1:23 tbl: [] args: nil body: (FCALL@1:12-1:21 :puts (ARRAY@1:17-1:21 (LIT@1:17-1:21 :foo) nil)))

Slide 8

Slide 8 text

TracePoint#enable can receive a Proc pr = proc do puts :foo puts :bar end trace = TracePoint.new(:line) do |tp| p tp.lineno end trace.enable(target: pr) pr.call 2 foo 3 bar

Slide 9

Slide 9 text

To begin with, What is "Proc"?

Slide 10

Slide 10 text

"Proc" is object of Procedure In other words, Function object. pr = proc { puts :hello_proc } p pr #

Slide 11

Slide 11 text

"Proc" is closure. def counter x = 0 proc { x += 1 } end c = counter counter.call # => 1 counter.call # => 2 Proc keeps the environment of the scope where it is created.

Slide 12

Slide 12 text

Method can receive a "Proc" as block def with_retry(exception, &block) block.call rescue exception retry end

Slide 13

Slide 13 text

In fact, Ruby ... has function object as First-class object can receive functions as return value can pass functions as method arguments In other words, Ruby has a factor of functional programming.

Slide 14

Slide 14 text

Some FP languages have a very interesting feature.

Slide 15

Slide 15 text

Monad In functional programming, a monad is a design pattern[1] that allows structuring programs generically while automating away boilerplate code needed by the program logic. Monads achieve this by providing their own data type, which represents a specific form of computation, along with one procedure to wrap values of any basic type within the monad (yielding a monadic value) and another to compose functions that output monadic values (called monadic functions). from Wikipedia https://en.wikipedia.org/wiki/Monad_(functional_programming).

Slide 16

Slide 16 text

It seems difficult!!

Slide 17

Slide 17 text

But, Monad is very simple and useful pattern actually. And syntax sugar is very important for monad.

Slide 18

Slide 18 text

I thinked. "I can implement true Monad syntax sugar by Ruby black magics."

Slide 19

Slide 19 text

Let's get down to the main topic.

Slide 20

Slide 20 text

Agenda Functor in Ruby Applicative Functor in Ruby Monad in Ruby Syntax sugar for monad Implement monadic syntax in Ruby Examples, DEMO Today, I will not explain mathematics. I will talk about only the programming technique. ここまでで5 〜6 分ぐらいだと良いなあ

Slide 21

Slide 21 text

Functor Functor ... is a container has a context for specific purpose. is a object that can be mapped by any function. The most popular functor in Ruby is "Array". [1,2,3].map { |i| i.to_s } In Haskell, map for Functor is called fmap .

Slide 22

Slide 22 text

Why is Functor useful Without functor and map, a container needs to implement all methods for any object that it may contain. Functor makes a container enable to collaborate with any methods.

Slide 23

Slide 23 text

Functor requires some laws functor.fmap(&:itself) == functor functor.fmap(&(proc_a >> proc_b)) == functor.fmap(&proc_a).fmap(&proc_b) These laws ensure that the behaviors of a functor are proper.

Slide 24

Slide 24 text

Applicative Functor (1) If you want to process with more than two functor objects, But fmap cannot handle more than two functors. [1,2,3].map do |i| [5,6,7].map do |j| i + j end end This sample outputs nested array. Of course, we can use flatten and flat_map . But we have a more functional approach.

Slide 25

Slide 25 text

Applicative Functor (2) Applicative Functor can contain functions. And contained functions apply other objects contained by each functor. array_plus = [:+.to_proc] array_plus.ap([2, 3], [4]) # => [6, 7] array_plus.ap([2, 3], [5, 6, 7]) # => [7, 8, 9, 8, 9, 10] array_plus.ap([2, 3], []) # => []

Slide 26

Slide 26 text

Applicative Functor (3) def ap(*targets) curried = ->(*extracted) do fmap { |pr| pr.call(*extracted) } end.curry(targets.size) applied = targets.inject(pure(curried)) do |pured, t| pured.flat_map do |pr| t.fmap { |v| pr.call(v) } end end applied.flat_map(&:itself) end

Slide 27

Slide 27 text

Applicative also requires some laws But laws of Applicative Functor are more complicated than ones of Functor. Sorry, I omit explaining details.

Slide 28

Slide 28 text

What is difference between Applicative and Monad Applicative Functor cannot express multiple dependent effects. For example, a calculation that may fail depends on whether previous calculations succeeded or failed. In such cases, Monad is useful.

Slide 29

Slide 29 text

Monad Monad is a container like Functor and Applicative. In Haskell, Monad requires some implementations. class Monad m where (>>=) :: m a -> (a -> m b) -> m b (>>) :: m a -> m b -> m b return :: a -> m a fail :: String -> m a Especially, (>>=) is most important. it is called "bind operator".

Slide 30

Slide 30 text

What is >>= (bind operator) ? In the case of Array, Array#bind receives a function that receives an item contained by the array and outputs new array. Example. ["foo","bar"].bind { |s| s.split(//) } # => ["f","o","o","b","a","r"] In fact, it's flat_map

Slide 31

Slide 31 text

Monad in Scala Scala has a syntax sugar for monad. for { x <- Some(10) y <- functionMaybeFailed() } yield x + y // Return Some(10 + y) or None Scala transforms this code to flat_map style internally. Like below. Some(10).flatMap { x => functionMaybeFailed().flatMap { y => x + y } }

Slide 32

Slide 32 text

Monad is flat_map !

Slide 33

Slide 33 text

Syntax sugar for monad Some functional languages have syntax sugar for monad. Haskell has do-syntax, Scala has for-syntax. Because the main purpose of monad is a chain of contextual computation, and syntax sugar is very effective to use it more easily.

Slide 34

Slide 34 text

Nested flatMap is not readable Some(10).flatMap { x => functionA(x).flatMap { y => fuctionB.flatMap { z => z.process } } } The frequent appearance of flatMap is very noisy. { , } is the same. indent++.

Slide 35

Slide 35 text

Scala is hybrid paradigm language Ruby is similar to Scala in a sense I copied code transformation from Scala

Slide 36

Slide 36 text

Monad syntax in Ruby https://github.com/joker1007/monar calc = ->(*val) do val.monadic_eval do |x| a = x.odd? ? x : x / 2 y <<= [a + 14] z <<= [y, y + 1, y + 2] end end expect(calc.call(7, 8)).to \ eq([21, 22, 23, 18, 19, 20]) This code is valid syntax!! There is no warning.

Slide 37

Slide 37 text

My idea is very simple. a <<= transform to flat_map do |a| It's all. <<= is important discovery!!

Slide 38

Slide 38 text

<<= is assignment with operator a <<= foo equals a = a << foo This is valid ruby code. And Ruby treats a as assigned local variable! Moreover, most ruby programmers have not written such codes. In other words, I can change the behavior freely!!

Slide 39

Slide 39 text

Review the code transformation [1,2,3].monadic_eval do |i| j <<= [i, i*2] j.select(&:odd?) end transforms to [1,2,3].flat_map do |i| [i, i*2].flat_map do |j| j.select(&:odd?) end end

Slide 40

Slide 40 text

It is difficult to resolve nested do-end by method chain. OK, AST Transformation!! We already have RubyVM::AST.of . It is a new feature of Ruby-2.6 and very useful for handling AST.

Slide 41

Slide 41 text

Breakdown of my implementation Extract AST from given block by RubyVM::AST.of Detect a pattern like a <<= foo Extract fragments of source code Reconstruct source code Wrap into new proc (to cache reconstructed code) instance_eval new source code ここまでで20 分弱だと良いなあ

Slide 42

Slide 42 text

Detect the pattern def __is_bind_statement?(node) (node.type == :DASGN || node.type == :DASGN_CURR) && node.children[1].type == :CALL && node.children[1].children[1] == :<< end

Slide 43

Slide 43 text

Extract fragments of codes # @param source [Array] lines of source code def extract_source(source, first_lineno, first_column, last_lineno, last_column) lines = source[(first_lineno - 1)..(last_lineno-1)] first_line = lines.shift last_line = lines.pop if last_line.nil? first_line[first_column..last_column] else first_line[first_column..-1] + lines.join + last_line[0..last_column] end end

Slide 44

Slide 44 text

Reconstruct codes def __transform_node(source, buf, node, last_stmt: false) if __is_bind_statement?(node) lvar = node.children[0] rhv = node.children[1].children[2] origin = Monad.extract_source( source, rhv.first_lineno, rhv.first_column, rhv.last_lineno, rhv.last_column).chomp buf[0].concat( "(#{origin}).flat_map do |#{lvar}|\n" ) buf[1] += 1 else buf[0].concat("(#{Monad.extract_source( source, node.first_lineno, node.first_column, node.last_lineno, node.last_column ).chomp})\n") end end

Slide 45

Slide 45 text

Wrap into a new proc gen = "proc { |#{caller_local_variables.map(&:to_s).join(",")}| begin; " + buf[0] + "rescue => ex; rescue_in_monad(ex); end; }\n" pr = instance_eval(gen, caller_location.path, caller_location.lineno) Monad.proc_cache["#{block_location[0]}:#{block_location[1]}"] = pr instance_eval outputs a proc object which contains transformed process. And I cached it to avoid source code transformation repeatedly.

Slide 46

Slide 46 text

Handling local variables Reconstructing codes lose local variables, because the environment contained by block is lost. Binding is required to handle local variables.

Slide 47

Slide 47 text

That is when TracePoint is effective! I found a technique to get Binding of a given block

Slide 48

Slide 48 text

proc_binding = nil trace = TracePoint.new(:line) do |tp| # this block is called Just before evaluating proc's first line proc_binding = tp.binding throw :escape end catch(:escape) do # In Ruby-2.6, TracePoint can limit tracking target. trace.enable(target: block) block.call ensure trace.disable end

Slide 49

Slide 49 text

I got a Binding of Proc!! And Binding has everything for black magic.

Slide 50

Slide 50 text

ast = RubyVM::AbstractSyntaxTree.of(block); # SCOPE args_tbl = ast.children[0] args_node = ast.children[1] caller_local_variables = proc_binding.local_variables - args_tbl gen = "proc { |#{caller_local_variables.map(&:to_s).join(",")}| ...." I got local_variables and copy into generated proc.

Slide 51

Slide 51 text

At last, instance_exec generates proc with local variables of the caller. instance_exec( *(caller_local_variables.map { |lvar| proc_binding.local_variable_get(lvar) }), &generated_pr )

Slide 52

Slide 52 text

Show powers of monad syntax by some examples. Maybe Either Future State ParserCombinator ここまでで30 分ぐらい

Slide 53

Slide 53 text

Maybe (Just) class Just include Monad include Monar::Maybe def initialize(value) @value = value; end def flat_map(&pr) pr.call(@value); end def monad_class Monar::Maybe; end end

Slide 54

Slide 54 text

Maybe (Nothing) class Nothing include Monad include Monar::Maybe def initialize(*value); end def flat_map(&pr) self; end def monad_class Monar::Maybe; end end

Slide 55

Slide 55 text

Maybe example key1 = "users/#{user.id}/posts/#{post.id}" key2 = "posts/#{post.id}/comments" # lookup_cache returns Maybe lookup_cache(key1).monadic_eval do |cached_post| cached_comments <<= lookup_cache(key2) # If lookup_cache(key2) fails, # below processes are not executed post = Oj.load(cached_post) comments = Oj.load(cached_comments) post.comments = comments pure post end # return Just(post) or Nothing

Slide 56

Slide 56 text

Either (Right) Ritht is the same as Just .

Slide 57

Slide 57 text

Either (Left) Left is the same as Nothing . But Left has a exception.

Slide 58

Slide 58 text

Either (helper function) def either(&pr) Monar::Either.right(pr.call) rescue => ex Monar::Either.left.new(ex) end

Slide 59

Slide 59 text

Either example either { Balance.find_by!(user: user_a) }.monadic_eval do |balance_a| _ <<= balance_a.ensure_amount!(amount) # return Either balance_b <<= either { Balance.find_by!(user: user_b) } TransferMoneyService.new(from: user_a, to: user_b, amount: amount) .process # return Either end # return Right(result) or Left(exception) Either hides error handling behind monad syntax. And it has a high affinity with pattern matching. Pattern matching is very hot topic.

Slide 60

Slide 60 text

Future (extend concurrent-ruby) Concurrent::Promises::Future.include(Monad) class Concurrent::Promises::Future def flat_map(&pr) yield value! rescue => ex if rejected? self else Concurrent::Promises.rejected_future(ex) end end end

Slide 61

Slide 61 text

Future example Concurrent::Promises.make_future(5).monadic_eval do |x| a = x.odd? ? x : x / 2 b = Concurrent::Promises.future { sleep 2; 11 } c = Concurrent::Promises.future { sleep 1; 3 } d <<= b # wait b e <<= c # wait c pure(a + d + e) end # Return Future(19) like async syntax of JavaScript.

Slide 62

Slide 62 text

State (constructor) State has a proc that receives current state and outputs value and next state. class State include Monad # @param next_state [Proc (s -> [a, s])] def initialize(next_state) raise ArgumentError.new("need to respond to :call") unless next_state.respond_to?(:call) @next_state = next_state end end

Slide 63

Slide 63 text

State (flat_map) def flat_map(&pr0) self.class.new( proc do |s0| x, s1 = run_state(s0) new_state_monad = pr0.call(x) new_state_monad.next_state.call(s1) end ) end run_state(s0) -> [a, s2] -> block.call(a) -> new state monad -> run_state(s1) These processes are wrapped by the new State class.

Slide 64

Slide 64 text

State (helper function) def get State.new(proc { |s| [s, s] }) end def put(st) State.new(proc { |_| [nil, st] }) end

Slide 65

Slide 65 text

State example state = State.pure(5).monadic_eval do |n| status <<= get # Return initial state _ <<= if status != :saved result = n * 10 put(:saved) # Update state else pure(nil) end pure(result) end val, st = state.run_state(:not_saved) # val == 50, st == :saved val, st = state.run_state(:saved) # val == nil, st == :saved

Slide 66

Slide 66 text

Mental model of State State handles 2 pipelines.

Slide 67

Slide 67 text

ParserCombinator Parser also contains a proc. that proc receives String and returns [[Object, String]] . Object is a result of parsing. String is remained characters. flat_map expresses parser combination. | expresses selective parser.

Slide 68

Slide 68 text

Parser#flat_map def flat_map(&pr) self.class.new( proc do |str0| result0 = run_parser(str0) # return Array result0.flat_map do |output, remained| next_parser = pr.call(output) next_parser.run_parser(remained) end end ) end Below processes is wrapped by Proc . Get result -> given proc processes result -> Get New Parser -> parse remained chars.

Slide 69

Slide 69 text

Parser#| def |(other) self.class.new( proc do |str| result0 = run_parser(str) result0.empty? other.run_parser(str) : result0 end ) end

Slide 70

Slide 70 text

Parser examples 1 def anychar new( proc do |string| string == "" ? [] : [[string[0], string[1..-1]]] end ) end # @param cond [Proc (Char -> TrueClass | FalseClass)] def satisfy(cond) anychar.flat_map do |char| if cond.call(char) new(proc { |string| [[char, string]] }) else new(proc { |string| [] }) end end end

Slide 71

Slide 71 text

Parser examples 2 def string(str) return pure "" if str == "" c, tail = str[0], str[1..-1] char(c).monadic_eval do |c1| cs <<= self.class.string(tail) pure [c1, *cs].join end end

Slide 72

Slide 72 text

Parser examples 3 def one_of(chars) satisfy(->(char) { chars.include?(char) }) end def many(parser, level = 1) combined = parser | pure(nil) combined.monadic_eval do |result| result2 <<= if result.nil? pure(nil) else self.class.many(parser, level + 1) end pure([result, result2].compact.flatten(level)) end end

Slide 73

Slide 73 text

DEMO (Arithmetic Parser)

Slide 74

Slide 74 text

Conclusion Monad is not difficult, It's flat_map The abstraction of monad is very powerful Syntax is very important I want other representations for nested block By this implementation, I recognized the fun of monadic programming again.

Slide 75

Slide 75 text

At last, I never recommend using this gem on production now!! If you are interested in TracePoint and AST, Examples of the gem's code are helpful for you, maybe. And, let's enjoy darkness programming!!