Enumerator - Enumerable's Ugly Cousin

Slide 1

Slide 1 text

Enumerator Enumerable’s Ugly Cousin

Slide 2

Slide 2 text

2 @rossta NYC

Slide 3

Slide 3 text

Pretend everything is normal NYC

Slide 4

Slide 4 text

Choose one of two speeds: Move fast or… NYC

Slide 5

Slide 5 text

GET OUT OF THE WAY NYC

Slide 6

Slide 6 text

Question yourself at every turn… NYC

Slide 7

Slide 7 text

NYC

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

What we say about Enumerable

Slide 10

Slide 10 text

Powerful, simple, and elegant. How modules should be made! Learn it. Use it.

Slide 11

Slide 11 text

Why I fell in love with Ruby

Slide 12

Slide 12 text

What we say about Enumerator

Slide 13

Slide 13 text

I don’t get it I would never write code this way That’s so ugly

Slide 14

Slide 14 text

This seems like a big hack Why would I ever use this?

Slide 15

Slide 15 text

OMFG

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

PRACTICAL EXPRESSIVE CLEAN ELEGANT MINASWAN READABLE SIMPLE SIMPLE HAPPINESS BEAUTIFUL CONCISE

Slide 18

Slide 18 text

Conventions are great because Productivity!

Slide 19

Slide 19 text

Conventions can limit our growth But…

Slide 20

Slide 20 text

for loops are useful! C++ MATLAB Perl Algol BASIC PostScript C Pascal Ada Bash Haskell Python Lua Java JavaScript PHP ActionScript Go

Slide 21

Slide 21 text

for n in [1, 2, 3] # ... end

Slide 22

Slide 22 text

“Real” Rubyists use each, right? [1, 2, 3].each { |x| # ... }

Slide 23

Slide 23 text

– Zed Shaw “When you teach people social norms as if they are universal truths you are actually indoctrinating them not educating them. .”

Slide 24

Slide 24 text

“Code Morality” The “right” way and everything else

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

Social Norms + Conventions !== Universal truth

Slide 31

Slide 31 text

Yes, follow conventions! but…

Slide 32

Slide 32 text

Also explore unconventional ideas

Slide 33

Slide 33 text

Unconventional Ugly

Slide 34

Slide 34 text

What is Enumerator?

Slide 35

Slide 35 text

Enumerator – Ruby Docs “A class which allows both internal and external iteration.”

Slide 36

Slide 36 text

enum = [1, 2, 3].to_enum # => # enum.next # => 1 enum.next # => 2 enum.next # => 3 enum.next # => StopIteration enum.each { |x| # ... }

Slide 37

Slide 37 text

Let’s explore

Slide 38

Slide 38 text

Java What I learned from

Slide 39

Slide 39 text

Iterator

Slide 40

Slide 40 text

ArrayList list = list.add( list.add( list.add( ArrayList list = new ArrayList(); list.add(1); list.add(2); list.add(3); Iterator it = list.iterator(); while(it.hasNext()) { Object element = it.next(); System.out.println(element); };

Slide 41

Slide 41 text

ArrayList list = list.add( list.add( list.add( Iterator it = list.iterator(); while Object element = it.next(); System.out.println(element); }; it.next();

Slide 42

Slide 42 text

Iterator an object that can suspend iteration and pass control back to its caller

Slide 43

Slide 43 text

Iterator Like yield turned inside out

Slide 44

Slide 44 text

def each yield 1 # to block yield 2 # to block yield 3 # to block end enum = [1, 2, 3].to_enum enum.next # yield 1 enum.next # yield 2 enum.next # yield 3

Slide 45

Slide 45 text

Use case: Replace nested loops

Slide 46

Slide 46 text

<% projects.each do |project| %> <% end %> <%= project.name %> <% colors = %w[aliceblue ghostwhite].to_enum(:cycle) %>

Slide 47

Slide 47 text

<% colors = <% <% colors.next

Slide 48

Slide 48 text

What I learned from JavaScript & Python

Slide 49

Slide 49 text

Generator

Slide 50

Slide 50 text

Iterators Generators

Slide 51

Slide 51 text

Generators #next

Slide 52

Slide 52 text

Generators produce data on the ﬂy

Slide 53

Slide 53 text

async inﬁnite sequences fetching data list comprehensions concurrency lazy evaluation coroutines

Slide 54

Slide 54 text

JavaScript Generators in

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

–AirBnB JavaScript Style Guide “Don't use generators.”

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

Python Generators in

Slide 62

Slide 62 text

Awesome Simple, expressive. Love them. List comprehensions… insanely readable and easy to maintain

Slide 63

Slide 63 text

gen = python_generator() gen.next() # => 1 gen.next() # => 2 gen.next() # => 3 def python_generator(): yield 1 yield 2 yield 3

Slide 64

Slide 64 text

def fibonacci(n): result = []

Slide 65

Slide 65 text

def fibonacci(n): result = [] a = b = 1 for i in xrange(n): result.append(a) a, b = b, a + b return result

Slide 66

Slide 66 text

def fibonacci(n): result = [] a = b = 1 for i in xrange(n): result.append(a) a, b = b, a + b return result for a in fibonacci(20): print a

Slide 67

Slide 67 text

def fibonacci(n): a = b = 1 for i in xrange(n): a, b = b, a + b for a in fibonacci(20): print a

Slide 68

Slide 68 text

def fibonacci(n): a = b = 1 for i in xrange(n): yield a a, b = b, a + b for a in fibonacci(20): print a

Slide 69

Slide 69 text

gen = fibonacci(20) gen.next() gen.next() gen.next() def a = b = a, b = b, a + b

Slide 70

Slide 70 text

Generator a function that can suspend its execution and pass control back to its caller

Slide 71

Slide 71 text

Generator Provides ability to treat algorithms as collections

Slide 72

Slide 72 text

Can we write generators in Ruby?* *yes, we can! Q:

Slide 73

Slide 73 text

def fibonacci(n): a = b = 1 for i in xrange(n): yield a a, b = b, a + b for a in fibonacci(20): print a

Slide 74

Slide 74 text

def fibonacci(n) a = b = 1 n.times do yield a a, b = b, a + b end end fibonacci(20) { |a| puts a } Not enumerable!

Slide 75

Slide 75 text

Enumeratorize it!

Slide 76

Slide 76 text

def fibonacci(n) a = b = 1 n.times do yield a a, b = b, a + b end end def a = b = n.times a, b = b, a + b end return to_enum(__method__, n) unless block_given?

Slide 77

Slide 77 text

return to_enum(__method__) unless block_given? This seems like a big hack Too magical That’s ugly

Slide 78

Slide 78 text

def fibonacci(n) a = b = 1 n.times do yield a a, b = b, a + b end end def a = b = n.times a, b = b, a + b end fibonacci(7).each { |a| puts a } fibonacci(25).map { |a| a * 3 }.select(&:odd?) fibonacci(100).find { |a| a > 50 } return to_enum(__method__, n) unless block_given? ] [ Enumerable!

Slide 79

Slide 79 text

Enumerable + Generator == Enumerator

Slide 80

Slide 80 text

to_enum

Slide 81

Slide 81 text

to_enum is everywhere

Slide 82

Slide 82 text

[1, 2, 3].to_enum(:each) # => # [1, 2, 3].to_enum(:map) # => # [1, 2, 3].to_enum(:select) # => # [1, 2, 3].to_enum(:group_by) # => # [1, 2, 3].to_enum(:cycle) # => #

Slide 83

Slide 83 text

[1, 2, 3].each # => # [1, 2, 3].map # => # [1, 2, 3].select # => # [1, 2, 3].group_by # => # [1, 2, 3].cycle # => #

Slide 84

Slide 84 text

Fixnum 4.times 1.upto(10) 10.upto(1) Range (1..10).each (1..10).step Struct struct.each struct.each_pair String "123".each_byte "123".each_char "123".each_codepoint "123".each_line "123".gsub File File.foreach file.each file.each_line file.each_byte file.each_codepoint ObjectSpace ObjectSpace.each_object loop

Slide 85

Slide 85 text

Object#to_enum Not magic!

Slide 86

Slide 86 text

We can re-implement to_enum in Ruby

Slide 87

Slide 87 text

obj_to_enum(int argc, VALUE *argv, VALUE obj) { VALUE enumerator, meth = sym_each; if (argc > 0) { --argc; meth = *argv++; } enumerator = rb_enumeratorize_with_size(obj, meth, argc, argv, 0); if (rb_block_given_p()) { enumerator_ptr(enumerator)->size = rb_block_proc(); } return enumerator; } static VALUE enumerator.c

Slide 88

Slide 88 text

obj_to_enum( { VALUE enumerator, meth = sym_each; --argc; meth = *argv++; } enumerator = rb_enumeratorize_with_size(obj, meth, argc, argv, enumerator_ptr(enumerator)->size = rb_block_proc(); } return enumerator; } rb_enumeratorize_with_size(obj, meth, static VALUE enumerator.c

Slide 89

Slide 89 text

VALUE rb_enumeratorize_with_size(VALUE obj, VALUE meth, int *size_fn) { dispatching to either return lazy_to_enum_i(obj, meth, argc, argv, size_fn); return enumerator_init( enumerator_allocate(rb_cEnumerator), } return enumerator_init( enumerator_allocate(rb_cEnumerator), obj, meth, argc, argv, size_fn, Qnil); rb_enumeratorize_with_size

Slide 90

Slide 90 text

def to_enum(method, *args) Enumerator.new(self, method, *args) end

Slide 91

Slide 91 text

Enumerator Collection Enumerable method

Slide 92

Slide 92 text

No content

Slide 93

Slide 93 text

static next_i { VALUE nil VALUE result; result next_ii e "iteration reached an end" return } next_i(VALUE curr, VALUE obj) return rb_fiber_yield(1, &nil); enumerator.c Fiber!

Slide 94

Slide 94 text

We can re-implement Enumerator with Fiber

Slide 95

Slide 95 text

Fiber.new fiber = Fiber.new do Fiber.yield 1 Fiber.yield 2 Fiber.yield 3 end fiber.resume # => 1 fiber.resume # => 2 fiber.resume # => 3

Slide 96

Slide 96 text

Look familiar? Q:

Slide 97

Slide 97 text

gen = python_generator() gen.next() # => 1 gen.next() # => 2 gen.next() # => 3 def python_generator(): yield 1 yield 2 yield 3

Slide 98

Slide 98 text

Fiber.new fiber = Fiber.new do Fiber.yield 1 Fiber.yield 2 Fiber.yield 3 end fiber.resume # => 1 fiber.resume # => 2 fiber.resume # => 3

Slide 99

Slide 99 text

a block that can suspend its execution and pass control back to its caller Fiber

Slide 100

Slide 100 text

Fiber =~ Generator

Slide 101

Slide 101 text

def to_enum(method, *args) Enumerator.new(self, method, *args) end

Slide 102

Slide 102 text

class CustomEnumerator def initialize(collection, meth, *args) @collection = collection @fiber = Fiber.new do @collection.send(meth, *args) do |n| Fiber.yield(n) end raise StopIteration end end end class CustomEnumerator def initialize(collection, meth, *args) @collection = collection class CustomEnumerator

Slide 103

Slide 103 text

@fiber = Fiber.new do @collection.send(meth, *args) do |n| Fiber.yield(n) end raise StopIteration end class def end

Slide 104

Slide 104 text

Enumerator Fiber Collection Enumerable method

Slide 105

Slide 105 text

class CustomEnumerator def next @fiber.resume end end

Slide 106

Slide 106 text

class CustomEnumerator include Enumerable def next @fiber.resume end def each loop { yield self.next } end end # => StopIteration

Slide 107

Slide 107 text

Enumerable + Fiber == Enumerator

Slide 108

Slide 108 text

Should we “enumeratorize” by convention? Q:

Slide 109

Slide 109 text

class PaginatedApiClient def posts(page=0) return to_enum(:posts, page) unless bg? # fetch page # yield each post # call posts(page+1) end end

Slide 110

Slide 110 text

class Document include Enumerable def each # yield each end end paragraph word line class Document

Slide 111

Slide 111 text

class Document def each_line return to_enum(:each_line) unless block_given? # yield each line end def each_word return to_enum(:each_word) unless block_given? # yield each word end def each_para return to_enum(:each_para) unless block_given? # yield each paragraph end end class Document def each_line return to_enum(:each_line) unless block_given? # yield each line end class Document def each_line return to_enum(:each_line) unless block_given? # yield each line end def each_word return to_enum(:each_word) unless block_given? # yield each word end class Document

Slide 112

Slide 112 text

class BinaryTree def breadth_first return to_enum(__method__) unless block_given? # yield values in “breadth first” order end def pre_order # etc… end def post_order end def in_order end end

Slide 113

Slide 113 text

tree = tree.breadth_first. with_index. partition { |n, i| i.odd? }. flat_map(&:join) # => “b1d3f5”, “a0c2e4”

Slide 114

Slide 114 text

Ugly but eﬀective return to_enum(__method__) unless block_given? tree.breadth_first. with_index. partition { #… } flat_map { #… }

Slide 115

Slide 115 text

What I learned from Clojure & Elixir

Slide 116

Slide 116 text

Inﬁnite Sequence

Slide 117

Slide 117 text

(take 2 (filter odd? [0 1 2 3 4 5])) [0, 1, 2, 3, 4, 5] |> Enum.filter(&Integer.is_odd/1) |> Enum.take(2)

Slide 118

Slide 118 text

(take 2 (filter odd? (iterate inc 0))) Stream.iterate(0, &(&1 + 1)) |> Stream.filter(&Integer.is_odd/1) |> Enum.take(2)

Slide 119

Slide 119 text

We don’t have inﬁnite sequences in Ruby… or do we?

Slide 120

Slide 120 text

def fibonacci Enumerator.new do |y| a = b = 1 loop do y.yield a a, b = b, a + b end end end like Fiber.yield!

Slide 121

Slide 121 text

def a = b = y.yield a a, b = b, a + b end fibonacci.lazy. select(&:odd?). take(3)

Slide 122

Slide 122 text

lazy augments how data is processed in an enumerator chain

Slide 123

Slide 123 text

Eager Pipeline [] map filter sum

Slide 124

Slide 124 text

Lazy Pipeline [] map filter sum

Slide 125

Slide 125 text

Failed Project What I learned from a

Slide 126

Slide 126 text

Recurrence

Slide 127

Slide 127 text

2008

Slide 128

Slide 128 text

What’s the estimate? Oh, about 4 days

Slide 129

Slide 129 text

4 days weeks

Slide 130

Slide 130 text

Modeling Recurrence

Slide 131

Slide 131 text

{ every: :month, on: { friday: 13 } }

Slide 132

Slide 132 text

Present Day: Redemption

Slide 133

Slide 133 text

Wish list enumerable lazily generate events infinite recurrence queryable

Slide 134

Slide 134 text

i.e., an algorithm for an enumerable inﬁnite recurring events

Slide 135

Slide 135 text

Enumeratorize it!

Slide 136

Slide 136 text

$ gem install montrose

Slide 137

Slide 137 text

Recurrence Enumerator ﬁnd next event yield stop or continue

Slide 138

Slide 138 text

# Expressive Montrose.weekly(on: :monday, at: "9 am") # => # # Flexible Montrose.hourly.interval(3) Montrose.every(3.hours) Montrose.r(every: 3.hours) # Chainable Montrose.monthly. starting(1.year.from_now). on(friday: 13). repeat(5)

Slide 139

Slide 139 text

# Enumerable r = Montrose.monthly r.until(1.year.from_now).each do |event| # … end r.lazy.chunk(&:month).take(8).to_a # => [ [4, 2016-04-06 12:00:00 -0500], [5, 2016-05—06 12:00:00 -0500], [6, 2016-06—06 12:00:00 -0500], ...]

Slide 140

Slide 140 text

Enumerator is beautiful

Slide 141

Slide 141 text

ﬁbers inﬁnite sequences generators iterators lazy evaluation Enumerator Enumerable

Slide 142

Slide 142 text

This wasn’t just a talk about Enumerator

Slide 143

Slide 143 text

–Me “I wasn’t capable of writing Montrose a few years go. I needed to get out of my comfort zone.”