Enumerator - Enumerable's Ugly Cousin

Why don't more Rubyists use Enumerator? Learn more about this beautiful Ruby concept that could use a little more love.

Transcript

1. Enumerator
   Enumerable’s Ugly Cousin
   

   View Slide

2. 2
   @rossta
   NYC
   

   View Slide

3. Pretend
   everything
   is normal
   NYC
   

   View Slide

4. Choose one of two speeds:
   Move fast or…
   NYC
   

   View Slide

5. GET OUT OF THE WAY
   NYC
   

   View Slide

6. Question yourself
   at every turn…
   NYC
   

   View Slide

7. NYC
   

   View Slide

8. View Slide

9. What we say about
   Enumerable
   

   View Slide

10. Powerful, simple,
    and elegant.
    How modules
    should be made!
    Learn it. Use it.
    

    View Slide

11. Why I fell in love
    with Ruby
    

    View Slide

12. What we say about
    Enumerator
    

    View Slide

13. I don’t get it I would never write
    code this way
    That’s so ugly
    

    View Slide

14. This seems like
    a big hack
    Why would I ever
    use this?
    

    View Slide

15. OMFG
    

    View Slide

16. View Slide

17. PRACTICAL
    EXPRESSIVE
    CLEAN
    ELEGANT
    MINASWAN
    READABLE
    SIMPLE
    SIMPLE
    HAPPINESS
    BEAUTIFUL
    CONCISE
    

    View Slide

18. Conventions are great
    because Productivity!
    

    View Slide

19. Conventions can
    limit our growth
    But…
    

    View Slide

20. for loops are useful!
    C++
    MATLAB
    Perl
    Algol
    BASIC
    PostScript
    C
    Pascal
    Ada
    Bash
    Haskell
    Python Lua
    Java
    JavaScript
    PHP
    ActionScript
    Go
    

    View Slide

21. for n in [1, 2, 3]
    # ...
    end
    

    View Slide

22. “Real” Rubyists
    use each, right?
    [1, 2, 3].each { |x| # ... }
    

    View Slide

23. – Zed Shaw
    “When you teach people
    social norms as if they are
    universal truths you are
    actually indoctrinating
    them not educating them. .”
    

    View Slide

24. “Code Morality”
    The “right” way and
    everything else
    

    View Slide

25. View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. Social Norms + Conventions
    !==
    Universal truth
    

    View Slide

31. Yes, follow conventions!
    but…
    

    View Slide

32. Also explore
    unconventional ideas
    

    View Slide

33. Unconventional
    Ugly
    

    View Slide

34. What is Enumerator?
    

    View Slide

35. Enumerator
    – Ruby Docs
    “A class which allows
    both internal and
    external iteration.”
    

    View Slide

36. enum = [1, 2, 3].to_enum
    # => #
    enum.next # => 1
    enum.next # => 2
    enum.next # => 3
    enum.next # => StopIteration
    enum.each { |x| # ... }
    

    View Slide

37. Let’s explore
    

    View Slide

38. Java
    What I learned from
    

    View Slide

39. Iterator
    

    View Slide

40. ArrayList list =
    list.add(
    list.add(
    list.add(
    ArrayList list = new ArrayList();
    list.add(1);
    list.add(2);
    list.add(3);
    Iterator it = list.iterator();
    while(it.hasNext()) {
    Object element = it.next();
    System.out.println(element);
    };
    

    View Slide

41. ArrayList list =
    list.add(
    list.add(
    list.add(
    Iterator it = list.iterator();
    while
    Object element = it.next();
    System.out.println(element);
    };
    it.next();
    

    View Slide

42. Iterator
    an object that can suspend
    iteration and pass control
    back to its caller
    

    View Slide

43. Iterator
    Like yield
    turned inside out
    

    View Slide

44. def each
    yield 1 # to block
    yield 2 # to block
    yield 3 # to block
    end
    enum = [1, 2, 3].to_enum
    enum.next # yield 1
    enum.next # yield 2
    enum.next # yield 3
    

    View Slide

45. Use case:
    Replace nested loops
    

    View Slide

46. 
    <% projects.each do |project| %>
    
    <%= project.name %>
    
    <% end %>
    
    <% colors = %w[aliceblue ghostwhite].to_enum(:cycle) %>
    

    View Slide

47. <% colors =
    
    <%
    <%
    
    colors.next
    

    View Slide

48. What I learned from
    JavaScript
    & Python
    

    View Slide

49. Generator
    

    View Slide

50. Iterators
    Generators
    

    View Slide

51. Generators #next
    

    View Slide

52. Generators
    produce
    data
    on the
    fly
    

    View Slide

53. async
    infinite sequences
    fetching data
    list comprehensions
    concurrency
    lazy evaluation
    coroutines
    

    View Slide

54. JavaScript
    Generators in
    

    View Slide

55. View Slide

56. –AirBnB JavaScript Style Guide
    “Don't use generators.”
    

    View Slide

57. View Slide

58. View Slide

59. View Slide

60. View Slide

61. Python
    Generators in
    

    View Slide

62. Awesome
    Simple, expressive.
    Love them.
    List comprehensions…
    insanely readable and
    easy to maintain
    

    View Slide

63. gen = python_generator()
    gen.next() # => 1
    gen.next() # => 2
    gen.next() # => 3
    def python_generator():
    yield 1
    yield 2
    yield 3
    

    View Slide

64. def fibonacci(n):
    result = []
    

    View Slide

65. def fibonacci(n):
    result = []
    a = b = 1
    for i in xrange(n):
    result.append(a)
    a, b = b, a + b
    return result
    

    View Slide

66. def fibonacci(n):
    result = []
    a = b = 1
    for i in xrange(n):
    result.append(a)
    a, b = b, a + b
    return result
    for a in fibonacci(20):
    print a
    

    View Slide

67. def fibonacci(n):
    a = b = 1
    for i in xrange(n):
    a, b = b, a + b
    for a in fibonacci(20):
    print a
    

    View Slide

68. def fibonacci(n):
    a = b = 1
    for i in xrange(n):
    yield a
    a, b = b, a + b
    for a in fibonacci(20):
    print a
    

    View Slide

69. gen = fibonacci(20)
    gen.next()
    gen.next()
    gen.next()
    def
    a = b =
    a, b = b, a + b
    

    View Slide

70. Generator
    a function that can suspend
    its execution and pass
    control back to its caller
    

    View Slide

71. Generator
    Provides ability to treat
    algorithms as collections
    

    View Slide

72. Can we write
    generators in Ruby?*
    *yes, we can!
    Q:
    

    View Slide

73. def fibonacci(n):
    a = b = 1
    for i in xrange(n):
    yield a
    a, b = b, a + b
    for a in fibonacci(20):
    print a
    

    View Slide

74. def fibonacci(n)
    a = b = 1
    n.times do
    yield a
    a, b = b, a + b
    end
    end
    fibonacci(20) { |a| puts a }
    Not enumerable!
    

    View Slide

75. Enumeratorize it!
    

    View Slide

76. def fibonacci(n)
    a = b = 1
    n.times do
    yield a
    a, b = b, a + b
    end
    end
    def
    a = b =
    n.times
    a, b = b, a + b
    end
    return to_enum(__method__, n) unless block_given?
    

    View Slide

77. return to_enum(__method__) unless block_given?
    This seems like a big hack
    Too magical
    That’s ugly
    

    View Slide

78. def fibonacci(n)
    a = b = 1
    n.times do
    yield a
    a, b = b, a + b
    end
    end
    def
    a = b =
    n.times
    a, b = b, a + b
    end
    fibonacci(7).each { |a| puts a }
    fibonacci(25).map { |a| a * 3 }.select(&:odd?)
    fibonacci(100).find { |a| a > 50 }
    return to_enum(__method__, n) unless block_given?
    ]
    [ Enumerable!
    

    View Slide

79. Enumerable
    +
    Generator
    ==
    Enumerator
    

    View Slide

80. to_enum
    

    View Slide

81. to_enum is everywhere
    

    View Slide

82. [1, 2, 3].to_enum(:each)
    # => #
    [1, 2, 3].to_enum(:map)
    # => #
    [1, 2, 3].to_enum(:select)
    # => #
    [1, 2, 3].to_enum(:group_by)
    # => #
    [1, 2, 3].to_enum(:cycle)
    # => #
    

    View Slide

83. [1, 2, 3].each
    # => #
    [1, 2, 3].map
    # => #
    [1, 2, 3].select
    # => #
    [1, 2, 3].group_by
    # => #
    [1, 2, 3].cycle
    # => #
    

    View Slide

84. Fixnum
    4.times
    1.upto(10)
    10.upto(1)
    Range
    (1..10).each
    (1..10).step
    Struct
    struct.each
    struct.each_pair
    String
    "123".each_byte
    "123".each_char
    "123".each_codepoint
    "123".each_line
    "123".gsub
    File
    File.foreach
    file.each
    file.each_line
    file.each_byte
    file.each_codepoint
    ObjectSpace
    ObjectSpace.each_object
    loop
    

    View Slide

85. Object#to_enum
    Not magic!
    

    View Slide

86. We can re-implement
    to_enum in Ruby
    

    View Slide

87. obj_to_enum(int argc, VALUE *argv, VALUE obj)
    {
    VALUE enumerator, meth = sym_each;
    if (argc > 0) {
    --argc;
    meth = *argv++;
    }
    enumerator = rb_enumeratorize_with_size(obj, meth,
    argc, argv, 0);
    if (rb_block_given_p()) {
    enumerator_ptr(enumerator)->size =
    rb_block_proc();
    }
    return enumerator;
    }
    static VALUE
    enumerator.c
    

    View Slide

88. obj_to_enum(
    {
    VALUE enumerator, meth = sym_each;
    --argc;
    meth = *argv++;
    }
    enumerator = rb_enumeratorize_with_size(obj, meth,
    argc, argv,
    enumerator_ptr(enumerator)->size =
    rb_block_proc();
    }
    return enumerator;
    }
    rb_enumeratorize_with_size(obj, meth,
    static VALUE
    enumerator.c
    

    View Slide

89. VALUE
    rb_enumeratorize_with_size(VALUE obj, VALUE meth,
    int
    *size_fn)
    {
    dispatching to either
    return lazy_to_enum_i(obj, meth, argc, argv, size_fn);
    return enumerator_init(
    enumerator_allocate(rb_cEnumerator),
    }
    return enumerator_init(
    enumerator_allocate(rb_cEnumerator),
    obj, meth, argc, argv, size_fn, Qnil);
    rb_enumeratorize_with_size
    

    View Slide

90. def to_enum(method, *args)
    Enumerator.new(self, method, *args)
    end
    

    View Slide

91. Enumerator
    Collection
    Enumerable
    method
    

    View Slide

92. View Slide

93. static
    next_i
    {
    VALUE nil
    VALUE result;
    result
    next_ii
    e
    "iteration reached an end"
    return
    }
    next_i(VALUE curr, VALUE obj)
    return rb_fiber_yield(1, &nil);
    enumerator.c
    Fiber!
    

    View Slide

94. We can re-implement
    Enumerator with Fiber
    

    View Slide

95. Fiber.new
    fiber = Fiber.new do
    Fiber.yield 1
    Fiber.yield 2
    Fiber.yield 3
    end
    fiber.resume # => 1
    fiber.resume # => 2
    fiber.resume # => 3
    

    View Slide

96. Look familiar?
    Q:
    

    View Slide

97. gen = python_generator()
    gen.next() # => 1
    gen.next() # => 2
    gen.next() # => 3
    def python_generator():
    yield 1
    yield 2
    yield 3
    

    View Slide

98. Fiber.new
    fiber = Fiber.new do
    Fiber.yield 1
    Fiber.yield 2
    Fiber.yield 3
    end
    fiber.resume # => 1
    fiber.resume # => 2
    fiber.resume # => 3
    

    View Slide

99. a block that can suspend its
    execution and pass control
    back to its caller
    Fiber
    

    View Slide

100. Fiber =~ Generator
     

     View Slide

101. def to_enum(method, *args)
     Enumerator.new(self, method, *args)
     end
     

     View Slide

102. class CustomEnumerator
     def initialize(collection, meth, *args)
     @collection = collection
     @fiber = Fiber.new do
     @collection.send(meth, *args) do |n|
     Fiber.yield(n)
     end
     raise StopIteration
     end
     end
     end
     class CustomEnumerator
     def initialize(collection, meth, *args)
     @collection = collection
     class CustomEnumerator
     

     View Slide

103. @fiber = Fiber.new do
     @collection.send(meth, *args) do |n|
     Fiber.yield(n)
     end
     raise StopIteration
     end
     class
     def
     end
     

     View Slide

104. Enumerator
     Fiber
     Collection
     Enumerable
     method
     

     View Slide

105. class CustomEnumerator
     def next
     @fiber.resume
     end
     end
     

     View Slide

106. class CustomEnumerator
     include Enumerable
     def next
     @fiber.resume
     end
     def each
     loop { yield self.next }
     end
     end
     # => StopIteration
     

     View Slide

107. Enumerable
     +
     Fiber
     ==
     Enumerator
     

     View Slide

108. Should we
     “enumeratorize”
     by convention?
     Q:
     

     View Slide

109. class PaginatedApiClient
     def posts(page=0)
     return to_enum(:posts, page) unless bg?
     # fetch page
     # yield each post
     # call posts(page+1)
     end
     end
     

     View Slide

110. class Document
     include Enumerable
     def each
     # yield each
     end
     end
     paragraph
     word
     line
     class Document
     

     View Slide

111. class Document
     def each_line
     return to_enum(:each_line) unless block_given?
     # yield each line
     end
     def each_word
     return to_enum(:each_word) unless block_given?
     # yield each word
     end
     def each_para
     return to_enum(:each_para) unless block_given?
     # yield each paragraph
     end
     end
     class Document
     def each_line
     return to_enum(:each_line) unless block_given?
     # yield each line
     end
     class Document
     def each_line
     return to_enum(:each_line) unless block_given?
     # yield each line
     end
     def each_word
     return to_enum(:each_word) unless block_given?
     # yield each word
     end
     class Document
     

     View Slide

112. class BinaryTree
     def breadth_first
     return to_enum(__method__) unless block_given?
     # yield values in “breadth first” order
     end
     def pre_order
     # etc…
     end
     def post_order
     end
     def in_order
     end
     end
     

     View Slide

113. tree =
     tree.breadth_first.
     with_index.
     partition { |n, i| i.odd? }.
     flat_map(&:join)
     # => “b1d3f5”, “a0c2e4”
     

     View Slide

114. Ugly but effective
     return to_enum(__method__) unless block_given?
     tree.breadth_first.
     with_index.
     partition { #… }
     flat_map { #… }
     

     View Slide

115. What I learned from
     Clojure
     & Elixir
     

     View Slide

116. Infinite
     Sequence
     

     View Slide

117. (take 2
     (filter odd? [0 1 2 3 4 5]))
     [0, 1, 2, 3, 4, 5]
     |> Enum.filter(&Integer.is_odd/1)
     |> Enum.take(2)
     

     View Slide

118. (take 2
     (filter odd? (iterate inc 0)))
     Stream.iterate(0, &(&1 + 1))
     |> Stream.filter(&Integer.is_odd/1)
     |> Enum.take(2)
     

     View Slide

119. We don’t have infinite
     sequences in Ruby…
     or do we?
     

     View Slide

120. def fibonacci
     Enumerator.new do |y|
     a = b = 1
     loop do
     y.yield a
     a, b = b, a + b
     end
     end
     end
     like Fiber.yield!
     

     View Slide

121. def
     a = b =
     y.yield a
     a, b = b, a + b
     end fibonacci.lazy.
     select(&:odd?).
     take(3)
     

     View Slide

122. lazy augments how
     data is processed in an
     enumerator chain
     

     View Slide

123. Eager Pipeline
     [] map filter sum
     

     View Slide

124. Lazy Pipeline
     [] map filter sum
     

     View Slide

125. Failed Project
     What I learned from a
     

     View Slide

126. Recurrence
     

     View Slide

127. View Slide

128. What’s the estimate?
     Oh, about 4 days
     

     View Slide

129. 4 days
     weeks
     

     View Slide

130. Modeling Recurrence
     

     View Slide

131. { every: :month, on: { friday: 13 } }
     

     View Slide

132. Present Day: Redemption
     

     View Slide

133. Wish list
     enumerable
     lazily generate events
     infinite recurrence
     queryable
     

     View Slide

134. i.e., an algorithm for an
     enumerable infinite
     recurring events
     

     View Slide

135. Enumeratorize it!
     

     View Slide

136. $ gem install montrose
     

     View Slide

137. Recurrence
     Enumerator
     find next event
     yield
     stop or continue
     

     View Slide

138. # Expressive
     Montrose.weekly(on: :monday, at: "9 am")
     # => #
     # Flexible
     Montrose.hourly.interval(3)
     Montrose.every(3.hours)
     Montrose.r(every: 3.hours)
     # Chainable
     Montrose.monthly.
     starting(1.year.from_now).
     on(friday: 13).
     repeat(5)
     

     View Slide

139. # Enumerable
     r = Montrose.monthly
     r.until(1.year.from_now).each do |event|
     # …
     end
     r.lazy.chunk(&:month).take(8).to_a
     # => [
     [4, 2016-04-06 12:00:00 -0500],
     [5, 2016-05—06 12:00:00 -0500],
     [6, 2016-06—06 12:00:00 -0500],
     ...]
     

     View Slide

140. Enumerator is beautiful
     

     View Slide

141. fibers
     infinite sequences
     generators
     iterators
     lazy evaluation
     Enumerator
     Enumerable
     

     View Slide

142. This wasn’t just a talk
     about Enumerator
     

     View Slide

143. –Me
     “I wasn’t capable of writing
     Montrose a few years go. I
     needed to get out of my
     comfort zone.”
     

     View Slide

144. MY RUBY GROWTH CYCLE
     Uninformed Optimism
     Informed Pessimism
     Informed Optimism
     

     View Slide

145. View Slide

146. “WTF?”
     Instead of
     

     View Slide

147. “What can I learn from this?”
     Try
     

     View Slide

148. Sometimes, ugly code can
     teach us something
     

     View Slide

149. Go forth and be curious
     

     View Slide

150. @rossta
     rossta.net/talks
     Ross Kaffenberger
     

     View Slide