Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elixir Stream by Examples
Search
Wu Qing
March 18, 2018
Programming
0
49
Elixir Stream by Examples
Presented at ElixirCamp 3 (March 16–19, 2018) in Lord Somers Camp, Somers, VIC
Wu Qing
March 18, 2018
Tweet
Share
More Decks by Wu Qing
See All by Wu Qing
Building a Slack app with Phoenix
wiserfirst
0
58
Journey of Upgrading an App to Phoenix 1.6
wiserfirst
0
67
Bitwise Fun with Elixir
wiserfirst
0
190
Tower of Hanoi with Phoenix LiveView
wiserfirst
0
620
What is telemetry all about
wiserfirst
0
74
What's New In Ecto 3
wiserfirst
1
290
Practical tips about with (in Elixir)
wiserfirst
0
36
Gentle Introduction to Elixir Macros
wiserfirst
0
38
Fight flaky specs with RSpec
wiserfirst
0
64
Other Decks in Programming
See All in Programming
AI OCR API on Lambdaを Datadogで可視化してみた
nealle
0
230
AIコーディングAgentとの向き合い方
eycjur
0
250
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
170
CJK and Unicode From a PHP Committer
youkidearitai
PRO
0
100
複雑なドメインに挑む.pdf
yukisakai1225
5
950
KessokuでDIでもgoroutineを活用する / Go Connect #6
mazrean
0
140
Zendeskのチケットを Amazon Bedrockで 解析した
ryokosuge
3
250
コンテキストエンジニアリング Cursor編
kinopeee
1
750
2025 年のコーディングエージェントの現在地とエンジニアの仕事の変化について
azukiazusa1
16
7.4k
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
310
Vue・React マルチプロダクト開発を支える Vite
andpad
0
110
More Approvers for Greater OSS and Japan Community
tkikuc
1
110
Featured
See All Featured
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
How GitHub (no longer) Works
holman
315
140k
A better future with KSS
kneath
239
17k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Git: the NoSQL Database
bkeepers
PRO
431
66k
Typedesign – Prime Four
hannesfritz
42
2.8k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.5k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
49
3k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
358
30k
Transcript
Elixir Stream by Examples Qing Wu
None
Service S Integration
Data source: API endpoint • millions of records involved •
pagination that maps to id ranges
Data source @spec fetch_data(integer) :: [Map.t] def fetch_data(page_number) do ...
end
Sending to Service S • Generate CSV files • Upload
to an FTP server
Sending data @spec handle_data([Map.t]) :: :ok | {:error, String.t} def
handle_data(data) do ... end
Solution page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |> fetch_data
|> handle_data end)
None
Enum Module • Provide functions to work with collections that
can be enumerated over • Functions in Enum always produce immediate results
Enum examples iex(1)> result1 = 1..10 |> Enum.map(&(&1 + 20))
[21, 22, 23, 24, 25, 26, 27, 28, 29, 30] iex(2)> result2 = result1 |> Enum.filter(&(rem(&1, 2) == 0)) [22, 24, 26, 28, 30] iex(3)> result2 |> Enum.reduce(0, &+/2) 130
Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |>
fetch_data |> handle_data end)
None
Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |>
fetch_data |> handle_data end)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
None
Stream module iex(1)> result1 = 1..10 |> Stream.map(&(&1 + 20))
#Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>]]> iex(2)> result2 = result1 |> Stream.filter(&(rem(&1, 2) == 0)) #Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>, #Function<39.51599720/1 in Stream.filter/2>]]> iex(3)> result2 |> Enum.reduce(0, &+/2) 130
Stream Solution page_range |> Stream.map(&fetch_data/1) |> Stream.concat |> Stream.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Summary • Stream module is very handy for lazy evaluation
• saves unnecessary iterations and memory • determine when to trigger evaluation
Further reading http://elixir-lang.org/getting-started/ enumerables-and-streams.html Offical Docs https://hexdocs.pm/elixir/Stream.html