Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Lazy Enumeration
Search
Eric Hodel
April 27, 2016
Programming
0
110
Lazy Enumeration
An introduction to lazy enumeration in ruby
Eric Hodel
April 27, 2016
Tweet
Share
More Decks by Eric Hodel
See All by Eric Hodel
Building maintainable command-line tools with MRuby
drbrain
0
560
Introduction to Rake
drbrain
0
260
Lessons in Mentorship
drbrain
1
160
Open Source Maintenance — Ruby on Ales 2014
drbrain
1
83
Open Source Maintenance — RailsClub Moscow
drbrain
1
140
drbdump
drbrain
2
430
Other Decks in Programming
See All in Programming
WEBアプリケーションにおけるAWS Lambdaを用いた大規模な非同期処理の実践
delhi09
PRO
7
4.2k
Subclassing, Composition, Python, and You
hynek
3
170
AWS CDKを用いたセキュアなCI/CDパイプラインの構築 / Build a secure CI/CD pipeline using AWS CDK
seike460
PRO
3
610
RemixとCloudflare Stack におけるFile Upload
ossamoon
1
130
ACES Meet におけるリリース作業改善の取り組み
fukucheee
0
130
Competitionsだけじゃない! Kaggle Notebooks Grandmasterのすすめ
corochann
1
290
Memory API: Patterns, Use Cases, and Performance
josepaumard
1
170
実践Dash - 手を抜きながら本気で作るデータApplicationの基本と応用 / Dash for Python and Baseball
shinyorke
2
440
UnJSで簡単に始めるCLIツール開発 / cli-tool-development-with-unjs
aoseyuu
2
300
Pydantic x Database API:turu-pyの開発
yassun7010
1
660
色んなオートローダーを覗き見る #phpcon_okinawa
o0h
PRO
5
400
"Swarming" をコンセプトに掲げるアジャイルチームのベストプラクティス
boykush
2
250
Featured
See All Featured
The Art of Programming - Codeland 2020
erikaheidi
51
13k
VelocityConf: Rendering Performance Case Studies
addyosmani
325
23k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
159
15k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
92
16k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
GraphQLとの向き合い方2022年版
quramy
43
13k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
28
7.5k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
231
17k
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.8k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
4
120
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
37
1.7k
How to train your dragon (web standard)
notwaldorf
87
5.6k
Transcript
Lazy Enumera,on Eric Hodel —
[email protected]
Loop values = [1, 2, 3, 4] doubles = []
index = 0 while index < values.length do doubles << values[index] * 2 index += 1 end
Enumera,ng Where am I? values[index] Am I done? index <
values.length What’s next? index += 1
Enumerator API Where am I? next #=> nil or exception
if done Am I done? nil, StopIteration What’s next? handled for you
External Enumerator result = db_conn.exec ‘SELECT * FROM orders’ while
order = result.next do # … end
Internal Enumerator result = db_conn.exec ‘SELECT * FROM orders’ result.each_row
do |order| # … end
External vs Internal You write loop Impera,ve C, Ruby Loop
built-in Func,onal Scheme, Ruby
Eager Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value
= orders.map { |order| # 10,000 values order[‘total’] }.reduce { |sum, order_total| sum + order_total }
Eager Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value
= orders.map { |order| # 100,000,000 values order[‘total’] }.reduce { |sum, order_total| sum + order_total }
100,000,000 Objects >> a = Array.new 100_000_000 >> ObjectSpace.memsize_of a
=> 800000040 800MB 400ms
Lazy Enumera,on orders = db_conn.exec ‘SELECT total FROM orders’ total_order_value
= orders.lazy.map { |order| order[‘total’] }.reduce { |sum, order_total| sum + order_total } 10MB similar ,me
Eager Processing 100M 100M .map .map 100M .map 100M
Lazy Processing 1 1 .map .map 1 .map 1 2
2 2 2 3 3 3 3 … … … … 100M 100M 100M 100M
How does lazy work? Fibers!
Fiber? •Story line for a program •One Fiber runs at
a ,me •Scheduled by author •“Corou,ne”
Hierarchy Process ↳Thread ↳Fiber OS scheduled Manually scheduled
Scheduling Fibers resume(input) #=> output Run a specific Fiber Fiber.yield(value)
Return output to #resume
Ac,ve Fiber 1 1 .map .map 1 .map 1 2
2 2 2 3 3 3 3 … … … … 100M 100M 100M 100M Fiber Fiber Fiber Fiber
Example of Fiber ⃠
Lazy Enumera,on •Reduces memory used •Great for huge data sets
•Processes one at a ,me •Uses Fiber (corou,ne)