Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elixir Stream by Examples
Search
Wu Qing
March 18, 2018
Programming
0
49
Elixir Stream by Examples
Presented at ElixirCamp 3 (March 16–19, 2018) in Lord Somers Camp, Somers, VIC
Wu Qing
March 18, 2018
Tweet
Share
More Decks by Wu Qing
See All by Wu Qing
Building a Slack app with Phoenix
wiserfirst
0
56
Journey of Upgrading an App to Phoenix 1.6
wiserfirst
0
67
Bitwise Fun with Elixir
wiserfirst
0
180
Tower of Hanoi with Phoenix LiveView
wiserfirst
0
620
What is telemetry all about
wiserfirst
0
74
What's New In Ecto 3
wiserfirst
1
280
Practical tips about with (in Elixir)
wiserfirst
0
36
Gentle Introduction to Elixir Macros
wiserfirst
0
38
Fight flaky specs with RSpec
wiserfirst
0
64
Other Decks in Programming
See All in Programming
Flutterで備える!Accessibility Nutrition Labels完全ガイド
yuukiw00w
0
170
AI時代の『改訂新版 良いコード/悪いコードで学ぶ設計入門』 / ai-good-code-bad-code
minodriven
21
8.7k
20250704_教育事業におけるアジャイルなデータ基盤構築
hanon52_
5
880
PHPでWebSocketサーバーを実装しよう2025
kubotak
0
300
Composerが「依存解決」のためにどんな工夫をしているか #phpcon
o0h
PRO
1
320
PicoRuby on Rails
makicamel
2
140
ペアプロ × 生成AI 現場での実践と課題について / generative-ai-in-pair-programming
codmoninc
2
20k
High-Level Programming Languages in AI Era -Human Thought and Mind-
hayat01sh1da
PRO
0
830
The Evolution of Enterprise Java with Jakarta EE 11 and Beyond
ivargrimstad
0
150
iOS 26にアップデートすると実機でのHot Reloadができない?
umigishiaoi
0
130
GPUを計算資源として使おう!
primenumber
1
190
設計やレビューに悩んでいるPHPerに贈る、クリーンなオブジェクト設計の指針たち
panda_program
6
2.3k
Featured
See All Featured
BBQ
matthewcrist
89
9.7k
What's in a price? How to price your products and services
michaelherold
246
12k
How to train your dragon (web standard)
notwaldorf
96
6.1k
The Cost Of JavaScript in 2023
addyosmani
51
8.5k
Statistics for Hackers
jakevdp
799
220k
GitHub's CSS Performance
jonrohan
1031
460k
Site-Speed That Sticks
csswizardry
10
690
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Documentation Writing (for coders)
carmenintech
72
4.9k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Transcript
Elixir Stream by Examples Qing Wu
None
Service S Integration
Data source: API endpoint • millions of records involved •
pagination that maps to id ranges
Data source @spec fetch_data(integer) :: [Map.t] def fetch_data(page_number) do ...
end
Sending to Service S • Generate CSV files • Upload
to an FTP server
Sending data @spec handle_data([Map.t]) :: :ok | {:error, String.t} def
handle_data(data) do ... end
Solution page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |> fetch_data
|> handle_data end)
None
Enum Module • Provide functions to work with collections that
can be enumerated over • Functions in Enum always produce immediate results
Enum examples iex(1)> result1 = 1..10 |> Enum.map(&(&1 + 20))
[21, 22, 23, 24, 25, 26, 27, 28, 29, 30] iex(2)> result2 = result1 |> Enum.filter(&(rem(&1, 2) == 0)) [22, 24, 26, 28, 30] iex(3)> result2 |> Enum.reduce(0, &+/2) 130
Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |>
fetch_data |> handle_data end)
None
Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |>
fetch_data |> handle_data end)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
None
Stream module iex(1)> result1 = 1..10 |> Stream.map(&(&1 + 20))
#Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>]]> iex(2)> result2 = result1 |> Stream.filter(&(rem(&1, 2) == 0)) #Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>, #Function<39.51599720/1 in Stream.filter/2>]]> iex(3)> result2 |> Enum.reduce(0, &+/2) 130
Stream Solution page_range |> Stream.map(&fetch_data/1) |> Stream.concat |> Stream.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Summary • Stream module is very handy for lazy evaluation
• saves unnecessary iterations and memory • determine when to trigger evaluation
Further reading http://elixir-lang.org/getting-started/ enumerables-and-streams.html Offical Docs https://hexdocs.pm/elixir/Stream.html