Lazy Enumeration

An introduction to lazy enumeration in ruby

Transcript

1. Lazy Enumera,on Eric Hodel — drbrain@segment7.net
2. Loop values = [1, 2, 3, 4] doubles = [] index = 0 while index < values.length do doubles << values[index] * 2 index += 1 end
3. Enumera,ng Where am I? values[index] Am I done? index < values.length What’s next? index += 1
4. Enumerator API Where am I? next #=> nil or exception if done Am I done? nil, StopIteration What’s next? handled for you
5. External Enumerator result = db_conn.exec ‘SELECT * FROM orders’ while order = result.next do # … end
6. Internal Enumerator result = db_conn.exec ‘SELECT * FROM orders’ result.each_row do |order| # … end
7. External vs Internal You write loop Impera,ve C, Ruby Loop built-in Func,onal Scheme, Ruby
8. Eager Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value = orders.map { |order| # 10,000 values order[‘total’] }.reduce { |sum, order_total| sum + order_total }
9. Eager Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value = orders.map { |order| # 100,000,000 values order[‘total’] }.reduce { |sum, order_total| sum + order_total }
10. 100,000,000 Objects >> a = Array.new 100_000_000 >> ObjectSpace.memsize_of a => 800000040 800MB 400ms
11. Lazy Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value = orders.lazy.map { |order| order[‘total’] }.reduce { |sum, order_total| sum + order_total } 10MB similar ,me
12. Eager Processing 100M 100M .map .map 100M .map 100M
13. Lazy Processing 1 1 .map .map 1 .map 1 2 2 2 2 3 3 3 3 … … … … 100M 100M 100M 100M
14. How does lazy work? Fibers!
15. Fiber? •Story line for a program •One Fiber runs at a ,me •Scheduled by author •“Corou,ne”
16. Hierarchy Process ↳Thread ↳Fiber OS scheduled Manually scheduled
17. Scheduling Fibers resume(input) #=> output Run a specific Fiber Fiber.yield(value) Return output to #resume
18. Ac,ve Fiber 1 1 .map .map 1 .map 1 2 2 2 2 3 3 3 3 … … … … 100M 100M 100M 100M Fiber Fiber Fiber Fiber
19. Example of Fiber ⃠
20. Lazy Enumera,on •Reduces memory used •Great for huge data sets •Processes one at a ,me •Uses Fiber (corou,ne)