$30 off During Our Annual Pro Sale. View Details »

Enumerator - Enumerable's Ugly Cousin

Enumerator - Enumerable's Ugly Cousin

Why don't more Rubyists use Enumerator? Learn more about this beautiful Ruby concept that could use a little more love.

Ross Kaffenberger

March 31, 2016
Tweet

More Decks by Ross Kaffenberger

Other Decks in Technology

Transcript

  1. Enumerator
    Enumerable’s Ugly Cousin

    View Slide

  2. 2
    @rossta
    NYC

    View Slide

  3. Pretend
    everything
    is normal
    NYC

    View Slide

  4. Choose one of two speeds:
    Move fast or…
    NYC

    View Slide

  5. GET OUT OF THE WAY
    NYC

    View Slide

  6. Question yourself
    at every turn…
    NYC

    View Slide

  7. NYC

    View Slide

  8. View Slide

  9. What we say about
    Enumerable

    View Slide

  10. Powerful, simple,
    and elegant.
    How modules
    should be made!
    Learn it. Use it.

    View Slide

  11. Why I fell in love
    with Ruby

    View Slide

  12. What we say about
    Enumerator

    View Slide

  13. I don’t get it I would never write
    code this way
    That’s so ugly

    View Slide

  14. This seems like
    a big hack
    Why would I ever
    use this?

    View Slide

  15. OMFG

    View Slide

  16. View Slide

  17. PRACTICAL
    EXPRESSIVE
    CLEAN
    ELEGANT
    MINASWAN
    READABLE
    SIMPLE
    SIMPLE
    HAPPINESS
    BEAUTIFUL
    CONCISE

    View Slide

  18. Conventions are great
    because Productivity!

    View Slide

  19. Conventions can
    limit our growth
    But…

    View Slide

  20. for loops are useful!
    C++
    MATLAB
    Perl
    Algol
    BASIC
    PostScript
    C
    Pascal
    Ada
    Bash
    Haskell
    Python Lua
    Java
    JavaScript
    PHP
    ActionScript
    Go

    View Slide

  21. for n in [1, 2, 3]
    # ...
    end

    View Slide

  22. “Real” Rubyists
    use each, right?
    [1, 2, 3].each { |x| # ... }

    View Slide

  23. – Zed Shaw
    “When you teach people
    social norms as if they are
    universal truths you are
    actually indoctrinating
    them not educating them. .”

    View Slide

  24. “Code Morality”
    The “right” way and
    everything else

    View Slide

  25. View Slide

  26. View Slide

  27. View Slide

  28. View Slide

  29. View Slide

  30. Social Norms + Conventions
    !==
    Universal truth

    View Slide

  31. Yes, follow conventions!
    but…

    View Slide

  32. Also explore
    unconventional ideas

    View Slide

  33. Unconventional
    Ugly

    View Slide

  34. What is Enumerator?

    View Slide

  35. Enumerator
    – Ruby Docs
    “A class which allows
    both internal and
    external iteration.”

    View Slide

  36. enum = [1, 2, 3].to_enum
    # => #
    enum.next # => 1
    enum.next # => 2
    enum.next # => 3
    enum.next # => StopIteration
    enum.each { |x| # ... }

    View Slide

  37. Let’s explore

    View Slide

  38. Java
    What I learned from

    View Slide

  39. Iterator

    View Slide

  40. ArrayList list =
    list.add(
    list.add(
    list.add(
    ArrayList list = new ArrayList();
    list.add(1);
    list.add(2);
    list.add(3);
    Iterator it = list.iterator();
    while(it.hasNext()) {
    Object element = it.next();
    System.out.println(element);
    };

    View Slide

  41. ArrayList list =
    list.add(
    list.add(
    list.add(
    Iterator it = list.iterator();
    while
    Object element = it.next();
    System.out.println(element);
    };
    it.next();

    View Slide

  42. Iterator
    an object that can suspend
    iteration and pass control
    back to its caller

    View Slide

  43. Iterator
    Like yield
    turned inside out

    View Slide

  44. def each
    yield 1 # to block
    yield 2 # to block
    yield 3 # to block
    end
    enum = [1, 2, 3].to_enum
    enum.next # yield 1
    enum.next # yield 2
    enum.next # yield 3

    View Slide

  45. Use case:
    Replace nested loops

    View Slide


  46. <% projects.each do |project| %>

    <%= project.name %>

    <% end %>

    <% colors = %w[aliceblue ghostwhite].to_enum(:cycle) %>

    View Slide

  47. <% colors =

    <%
    <%

    colors.next

    View Slide

  48. What I learned from
    JavaScript
    & Python

    View Slide

  49. Generator

    View Slide

  50. Iterators
    Generators

    View Slide

  51. Generators #next

    View Slide

  52. Generators
    produce
    data
    on the
    fly

    View Slide

  53. async
    infinite sequences
    fetching data
    list comprehensions
    concurrency
    lazy evaluation
    coroutines

    View Slide

  54. JavaScript
    Generators in

    View Slide

  55. View Slide

  56. –AirBnB JavaScript Style Guide
    “Don't use generators.”

    View Slide

  57. View Slide

  58. View Slide

  59. View Slide

  60. View Slide

  61. Python
    Generators in

    View Slide

  62. Awesome
    Simple, expressive.
    Love them.
    List comprehensions…
    insanely readable and
    easy to maintain

    View Slide

  63. gen = python_generator()
    gen.next() # => 1
    gen.next() # => 2
    gen.next() # => 3
    def python_generator():
    yield 1
    yield 2
    yield 3

    View Slide

  64. def fibonacci(n):
    result = []

    View Slide

  65. def fibonacci(n):
    result = []
    a = b = 1
    for i in xrange(n):
    result.append(a)
    a, b = b, a + b
    return result

    View Slide

  66. def fibonacci(n):
    result = []
    a = b = 1
    for i in xrange(n):
    result.append(a)
    a, b = b, a + b
    return result
    for a in fibonacci(20):
    print a

    View Slide

  67. def fibonacci(n):
    a = b = 1
    for i in xrange(n):
    a, b = b, a + b
    for a in fibonacci(20):
    print a

    View Slide

  68. def fibonacci(n):
    a = b = 1
    for i in xrange(n):
    yield a
    a, b = b, a + b
    for a in fibonacci(20):
    print a

    View Slide

  69. gen = fibonacci(20)
    gen.next()
    gen.next()
    gen.next()
    def
    a = b =
    a, b = b, a + b

    View Slide

  70. Generator
    a function that can suspend
    its execution and pass
    control back to its caller

    View Slide

  71. Generator
    Provides ability to treat
    algorithms as collections

    View Slide

  72. Can we write
    generators in Ruby?*
    *yes, we can!
    Q:

    View Slide

  73. def fibonacci(n):
    a = b = 1
    for i in xrange(n):
    yield a
    a, b = b, a + b
    for a in fibonacci(20):
    print a

    View Slide

  74. def fibonacci(n)
    a = b = 1
    n.times do
    yield a
    a, b = b, a + b
    end
    end
    fibonacci(20) { |a| puts a }
    Not enumerable!

    View Slide

  75. Enumeratorize it!

    View Slide

  76. def fibonacci(n)
    a = b = 1
    n.times do
    yield a
    a, b = b, a + b
    end
    end
    def
    a = b =
    n.times
    a, b = b, a + b
    end
    return to_enum(__method__, n) unless block_given?

    View Slide

  77. return to_enum(__method__) unless block_given?
    This seems like a big hack
    Too magical
    That’s ugly

    View Slide

  78. def fibonacci(n)
    a = b = 1
    n.times do
    yield a
    a, b = b, a + b
    end
    end
    def
    a = b =
    n.times
    a, b = b, a + b
    end
    fibonacci(7).each { |a| puts a }
    fibonacci(25).map { |a| a * 3 }.select(&:odd?)
    fibonacci(100).find { |a| a > 50 }
    return to_enum(__method__, n) unless block_given?
    ]
    [ Enumerable!

    View Slide

  79. Enumerable
    +
    Generator
    ==
    Enumerator

    View Slide

  80. to_enum

    View Slide

  81. to_enum is everywhere

    View Slide

  82. [1, 2, 3].to_enum(:each)
    # => #
    [1, 2, 3].to_enum(:map)
    # => #
    [1, 2, 3].to_enum(:select)
    # => #
    [1, 2, 3].to_enum(:group_by)
    # => #
    [1, 2, 3].to_enum(:cycle)
    # => #

    View Slide

  83. [1, 2, 3].each
    # => #
    [1, 2, 3].map
    # => #
    [1, 2, 3].select
    # => #
    [1, 2, 3].group_by
    # => #
    [1, 2, 3].cycle
    # => #

    View Slide

  84. Fixnum
    4.times
    1.upto(10)
    10.upto(1)
    Range
    (1..10).each
    (1..10).step
    Struct
    struct.each
    struct.each_pair
    String
    "123".each_byte
    "123".each_char
    "123".each_codepoint
    "123".each_line
    "123".gsub
    File
    File.foreach
    file.each
    file.each_line
    file.each_byte
    file.each_codepoint
    ObjectSpace
    ObjectSpace.each_object
    loop

    View Slide

  85. Object#to_enum
    Not magic!

    View Slide

  86. We can re-implement
    to_enum in Ruby

    View Slide

  87. obj_to_enum(int argc, VALUE *argv, VALUE obj)
    {
    VALUE enumerator, meth = sym_each;
    if (argc > 0) {
    --argc;
    meth = *argv++;
    }
    enumerator = rb_enumeratorize_with_size(obj, meth,
    argc, argv, 0);
    if (rb_block_given_p()) {
    enumerator_ptr(enumerator)->size =
    rb_block_proc();
    }
    return enumerator;
    }
    static VALUE
    enumerator.c

    View Slide

  88. obj_to_enum(
    {
    VALUE enumerator, meth = sym_each;
    --argc;
    meth = *argv++;
    }
    enumerator = rb_enumeratorize_with_size(obj, meth,
    argc, argv,
    enumerator_ptr(enumerator)->size =
    rb_block_proc();
    }
    return enumerator;
    }
    rb_enumeratorize_with_size(obj, meth,
    static VALUE
    enumerator.c

    View Slide

  89. VALUE
    rb_enumeratorize_with_size(VALUE obj, VALUE meth,
    int
    *size_fn)
    {
    dispatching to either
    return lazy_to_enum_i(obj, meth, argc, argv, size_fn);
    return enumerator_init(
    enumerator_allocate(rb_cEnumerator),
    }
    return enumerator_init(
    enumerator_allocate(rb_cEnumerator),
    obj, meth, argc, argv, size_fn, Qnil);
    rb_enumeratorize_with_size

    View Slide

  90. def to_enum(method, *args)
    Enumerator.new(self, method, *args)
    end

    View Slide

  91. Enumerator
    Collection
    Enumerable
    method

    View Slide

  92. View Slide

  93. static
    next_i
    {
    VALUE nil
    VALUE result;
    result
    next_ii
    e
    "iteration reached an end"
    return
    }
    next_i(VALUE curr, VALUE obj)
    return rb_fiber_yield(1, &nil);
    enumerator.c
    Fiber!

    View Slide

  94. We can re-implement
    Enumerator with Fiber

    View Slide

  95. Fiber.new
    fiber = Fiber.new do
    Fiber.yield 1
    Fiber.yield 2
    Fiber.yield 3
    end
    fiber.resume # => 1
    fiber.resume # => 2
    fiber.resume # => 3

    View Slide

  96. Look familiar?
    Q:

    View Slide

  97. gen = python_generator()
    gen.next() # => 1
    gen.next() # => 2
    gen.next() # => 3
    def python_generator():
    yield 1
    yield 2
    yield 3

    View Slide

  98. Fiber.new
    fiber = Fiber.new do
    Fiber.yield 1
    Fiber.yield 2
    Fiber.yield 3
    end
    fiber.resume # => 1
    fiber.resume # => 2
    fiber.resume # => 3

    View Slide

  99. a block that can suspend its
    execution and pass control
    back to its caller
    Fiber

    View Slide

  100. Fiber =~ Generator

    View Slide

  101. def to_enum(method, *args)
    Enumerator.new(self, method, *args)
    end

    View Slide

  102. class CustomEnumerator
    def initialize(collection, meth, *args)
    @collection = collection
    @fiber = Fiber.new do
    @collection.send(meth, *args) do |n|
    Fiber.yield(n)
    end
    raise StopIteration
    end
    end
    end
    class CustomEnumerator
    def initialize(collection, meth, *args)
    @collection = collection
    class CustomEnumerator

    View Slide

  103. @fiber = Fiber.new do
    @collection.send(meth, *args) do |n|
    Fiber.yield(n)
    end
    raise StopIteration
    end
    class
    def
    end

    View Slide

  104. Enumerator
    Fiber
    Collection
    Enumerable
    method

    View Slide

  105. class CustomEnumerator
    def next
    @fiber.resume
    end
    end

    View Slide

  106. class CustomEnumerator
    include Enumerable
    def next
    @fiber.resume
    end
    def each
    loop { yield self.next }
    end
    end
    # => StopIteration

    View Slide

  107. Enumerable
    +
    Fiber
    ==
    Enumerator

    View Slide

  108. Should we
    “enumeratorize”
    by convention?
    Q:

    View Slide

  109. class PaginatedApiClient
    def posts(page=0)
    return to_enum(:posts, page) unless bg?
    # fetch page
    # yield each post
    # call posts(page+1)
    end
    end

    View Slide

  110. class Document
    include Enumerable
    def each
    # yield each
    end
    end
    paragraph
    word
    line
    class Document

    View Slide

  111. class Document
    def each_line
    return to_enum(:each_line) unless block_given?
    # yield each line
    end
    def each_word
    return to_enum(:each_word) unless block_given?
    # yield each word
    end
    def each_para
    return to_enum(:each_para) unless block_given?
    # yield each paragraph
    end
    end
    class Document
    def each_line
    return to_enum(:each_line) unless block_given?
    # yield each line
    end
    class Document
    def each_line
    return to_enum(:each_line) unless block_given?
    # yield each line
    end
    def each_word
    return to_enum(:each_word) unless block_given?
    # yield each word
    end
    class Document

    View Slide

  112. class BinaryTree
    def breadth_first
    return to_enum(__method__) unless block_given?
    # yield values in “breadth first” order
    end
    def pre_order
    # etc…
    end
    def post_order
    end
    def in_order
    end
    end

    View Slide

  113. tree =
    tree.breadth_first.
    with_index.
    partition { |n, i| i.odd? }.
    flat_map(&:join)
    # => “b1d3f5”, “a0c2e4”

    View Slide

  114. Ugly but effective
    return to_enum(__method__) unless block_given?
    tree.breadth_first.
    with_index.
    partition { #… }
    flat_map { #… }

    View Slide

  115. What I learned from
    Clojure
    & Elixir

    View Slide

  116. Infinite
    Sequence

    View Slide

  117. (take 2
    (filter odd? [0 1 2 3 4 5]))
    [0, 1, 2, 3, 4, 5]
    |> Enum.filter(&Integer.is_odd/1)
    |> Enum.take(2)

    View Slide

  118. (take 2
    (filter odd? (iterate inc 0)))
    Stream.iterate(0, &(&1 + 1))
    |> Stream.filter(&Integer.is_odd/1)
    |> Enum.take(2)

    View Slide

  119. We don’t have infinite
    sequences in Ruby…
    or do we?

    View Slide

  120. def fibonacci
    Enumerator.new do |y|
    a = b = 1
    loop do
    y.yield a
    a, b = b, a + b
    end
    end
    end
    like Fiber.yield!

    View Slide

  121. def
    a = b =
    y.yield a
    a, b = b, a + b
    end fibonacci.lazy.
    select(&:odd?).
    take(3)

    View Slide

  122. lazy augments how
    data is processed in an
    enumerator chain

    View Slide

  123. Eager Pipeline
    [] map filter sum

    View Slide

  124. Lazy Pipeline
    [] map filter sum

    View Slide

  125. Failed Project
    What I learned from a

    View Slide

  126. Recurrence

    View Slide

  127. 2008

    View Slide

  128. What’s the estimate?
    Oh, about 4 days

    View Slide

  129. 4 days
    weeks

    View Slide

  130. Modeling Recurrence

    View Slide

  131. { every: :month, on: { friday: 13 } }

    View Slide

  132. Present Day: Redemption

    View Slide

  133. Wish list
    enumerable
    lazily generate events
    infinite recurrence
    queryable

    View Slide

  134. i.e., an algorithm for an
    enumerable infinite
    recurring events

    View Slide

  135. Enumeratorize it!

    View Slide

  136. $ gem install montrose

    View Slide

  137. Recurrence
    Enumerator
    find next event
    yield
    stop or continue

    View Slide

  138. # Expressive
    Montrose.weekly(on: :monday, at: "9 am")
    # => #
    # Flexible
    Montrose.hourly.interval(3)
    Montrose.every(3.hours)
    Montrose.r(every: 3.hours)
    # Chainable
    Montrose.monthly.
    starting(1.year.from_now).
    on(friday: 13).
    repeat(5)

    View Slide

  139. # Enumerable
    r = Montrose.monthly
    r.until(1.year.from_now).each do |event|
    # …
    end
    r.lazy.chunk(&:month).take(8).to_a
    # => [
    [4, 2016-04-06 12:00:00 -0500],
    [5, 2016-05—06 12:00:00 -0500],
    [6, 2016-06—06 12:00:00 -0500],
    ...]

    View Slide

  140. Enumerator is beautiful

    View Slide

  141. fibers
    infinite sequences
    generators
    iterators
    lazy evaluation
    Enumerator
    Enumerable

    View Slide

  142. This wasn’t just a talk
    about Enumerator

    View Slide

  143. –Me
    “I wasn’t capable of writing
    Montrose a few years go. I
    needed to get out of my
    comfort zone.”

    View Slide

  144. MY RUBY GROWTH CYCLE
    Uninformed Optimism
    Informed Pessimism
    Informed Optimism

    View Slide

  145. View Slide

  146. “WTF?”
    Instead of

    View Slide

  147. “What can I learn from this?”
    Try

    View Slide

  148. Sometimes, ugly code can
    teach us something

    View Slide

  149. Go forth and be curious

    View Slide

  150. @rossta
    rossta.net/talks
    Ross Kaffenberger

    View Slide