Save 37% off PRO during our Black Friday Sale! »

Lazy Enumeration

Lazy Enumeration

An introduction to lazy enumeration in ruby

58479f76374a3ba3c69b9804163f39f4?s=128

Eric Hodel

April 27, 2016
Tweet

Transcript

  1. Lazy Enumera,on Eric Hodel — drbrain@segment7.net

  2. Loop values = [1, 2, 3, 4] doubles = []

    index = 0 while index < values.length do doubles << values[index] * 2 index += 1 end
  3. Enumera,ng Where am I? values[index] Am I done? index <

    values.length What’s next? index += 1
  4. Enumerator API Where am I? next #=> nil or exception

    if done Am I done? nil, StopIteration What’s next? handled for you
  5. External Enumerator result = db_conn.exec ‘SELECT * FROM orders’ while

    order = result.next do # … end
  6. Internal Enumerator result = db_conn.exec ‘SELECT * FROM orders’ result.each_row

    do |order| # … end
  7. External vs Internal You write loop Impera,ve C, Ruby Loop

    built-in Func,onal Scheme, Ruby
  8. Eager Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value

    = orders.map { |order| # 10,000 values order[‘total’] }.reduce { |sum, order_total| sum + order_total }
  9. Eager Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value

    = orders.map { |order| # 100,000,000 values order[‘total’] }.reduce { |sum, order_total| sum + order_total }
  10. 100,000,000 Objects >> a = Array.new 100_000_000 >> ObjectSpace.memsize_of a

    => 800000040 800MB 400ms
  11. Lazy Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value

    = orders.lazy.map { |order| order[‘total’] }.reduce { |sum, order_total| sum + order_total } 10MB similar ,me
  12. Eager Processing 100M 100M .map .map 100M .map 100M

  13. Lazy Processing 1 1 .map .map 1 .map 1 2

    2 2 2 3 3 3 3 … … … … 100M 100M 100M 100M
  14. How does lazy work? Fibers!

  15. Fiber? •Story line for a program •One Fiber runs at

    a ,me •Scheduled by author •“Corou,ne”
  16. Hierarchy Process ↳Thread ↳Fiber OS scheduled Manually scheduled

  17. Scheduling Fibers resume(input) #=> output Run a specific Fiber Fiber.yield(value)

    Return output to #resume
  18. Ac,ve Fiber 1 1 .map .map 1 .map 1 2

    2 2 2 3 3 3 3 … … … … 100M 100M 100M 100M Fiber Fiber Fiber Fiber
  19. Example of Fiber ⃠

  20. Lazy Enumera,on •Reduces memory used •Great for huge data sets

    •Processes one at a ,me •Uses Fiber (corou,ne)