Lazy Enumeration

Transcript

1. Lazy Enumera,on Eric Hodel — drbrain@segment7.net

2. Loop values = [1, 2, 3, 4] doubles = []

   index = 0 while index < values.length do doubles << values[index] * 2 index += 1 end

3. Enumera,ng Where am I? values[index] Am I done? index <

   values.length What’s next? index += 1

4. Enumerator API Where am I? next #=> nil or exception

   if done Am I done? nil, StopIteration What’s next? handled for you

5. External Enumerator result = db_conn.exec ‘SELECT * FROM orders’ while

   order = result.next do # … end

6. Internal Enumerator result = db_conn.exec ‘SELECT * FROM orders’ result.each_row

   do |order| # … end

7. External vs Internal You write loop Impera,ve C, Ruby Loop

   built-in Func,onal Scheme, Ruby

8. Eager Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value

   = orders.map { |order| # 10,000 values order[‘total’] }.reduce { |sum, order_total| sum + order_total }

9. Eager Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value

   = orders.map { |order| # 100,000,000 values order[‘total’] }.reduce { |sum, order_total| sum + order_total }

10. 100,000,000 Objects >> a = Array.new 100_000_000 >> ObjectSpace.memsize_of a

    => 800000040 800MB 400ms

11. Lazy Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value

    = orders.lazy.map { |order| order[‘total’] }.reduce { |sum, order_total| sum + order_total } 10MB similar ,me

12. Eager Processing 100M 100M .map .map 100M .map 100M

13. Lazy Processing 1 1 .map .map 1 .map 1 2

    2 2 2 3 3 3 3 … … … … 100M 100M 100M 100M

14. How does lazy work? Fibers!

15. Fiber? •Story line for a program •One Fiber runs at

    a ,me •Scheduled by author •“Corou,ne”

16. Hierarchy Process ↳Thread ↳Fiber OS scheduled Manually scheduled

17. Scheduling Fibers resume(input) #=> output Run a specific Fiber Fiber.yield(value)

    Return output to #resume

18. Ac,ve Fiber 1 1 .map .map 1 .map 1 2

    2 2 2 3 3 3 3 … … … … 100M 100M 100M 100M Fiber Fiber Fiber Fiber

19. Example of Fiber ⃠

20. Lazy Enumera,on •Reduces memory used •Great for huge data sets

    •Processes one at a ,me •Uses Fiber (corou,ne)