Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elixir Stream by Examples
Search
Wu Qing
March 18, 2018
Programming
0
49
Elixir Stream by Examples
Presented at ElixirCamp 3 (March 16–19, 2018) in Lord Somers Camp, Somers, VIC
Wu Qing
March 18, 2018
Tweet
Share
More Decks by Wu Qing
See All by Wu Qing
Building a Slack app with Phoenix
wiserfirst
0
58
Journey of Upgrading an App to Phoenix 1.6
wiserfirst
0
67
Bitwise Fun with Elixir
wiserfirst
0
190
Tower of Hanoi with Phoenix LiveView
wiserfirst
0
620
What is telemetry all about
wiserfirst
0
75
What's New In Ecto 3
wiserfirst
1
290
Practical tips about with (in Elixir)
wiserfirst
0
36
Gentle Introduction to Elixir Macros
wiserfirst
0
38
Fight flaky specs with RSpec
wiserfirst
0
64
Other Decks in Programming
See All in Programming
Ruby Parser progress report 2025
yui_knk
1
460
Testing Trophyは叫ばない
toms74209200
0
890
Cache Me If You Can
ryunen344
2
4k
そのAPI、誰のため? Androidライブラリ設計における利用者目線の実践テクニック
mkeeda
2
2.8k
為你自己學 Python - 冷知識篇
eddie
1
350
1から理解するWeb Push
dora1998
7
1.9k
go test -json そして testing.T.Attr / Kyoto.go #63
utgwkk
3
320
Platformに“ちょうどいい”責務ってどこ? 関心の熱さにあわせて考える、責務分担のプラクティス
estie
1
140
「手軽で便利」に潜む罠。 Popover API を WCAG 2.2の視点で安全に使うには
taitotnk
0
870
より安全で効率的な Go コードへ: Protocol Buffers Opaque API の導入
shwatanap
2
790
Kiroで始めるAI-DLC
kaonash
2
630
AIでLINEスタンプを作ってみた
eycjur
1
230
Featured
See All Featured
Designing for Performance
lara
610
69k
Building Applications with DynamoDB
mza
96
6.6k
Faster Mobile Websites
deanohume
309
31k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
Balancing Empowerment & Direction
lara
3
630
BBQ
matthewcrist
89
9.8k
How to Ace a Technical Interview
jacobian
279
23k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.9k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Into the Great Unknown - MozCon
thekraken
40
2k
Java REST API Framework Comparison - PWX 2021
mraible
33
8.8k
Become a Pro
speakerdeck
PRO
29
5.5k
Transcript
Elixir Stream by Examples Qing Wu
None
Service S Integration
Data source: API endpoint • millions of records involved •
pagination that maps to id ranges
Data source @spec fetch_data(integer) :: [Map.t] def fetch_data(page_number) do ...
end
Sending to Service S • Generate CSV files • Upload
to an FTP server
Sending data @spec handle_data([Map.t]) :: :ok | {:error, String.t} def
handle_data(data) do ... end
Solution page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |> fetch_data
|> handle_data end)
None
Enum Module • Provide functions to work with collections that
can be enumerated over • Functions in Enum always produce immediate results
Enum examples iex(1)> result1 = 1..10 |> Enum.map(&(&1 + 20))
[21, 22, 23, 24, 25, 26, 27, 28, 29, 30] iex(2)> result2 = result1 |> Enum.filter(&(rem(&1, 2) == 0)) [22, 24, 26, 28, 30] iex(3)> result2 |> Enum.reduce(0, &+/2) 130
Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |>
fetch_data |> handle_data end)
None
Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |>
fetch_data |> handle_data end)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
None
Stream module iex(1)> result1 = 1..10 |> Stream.map(&(&1 + 20))
#Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>]]> iex(2)> result2 = result1 |> Stream.filter(&(rem(&1, 2) == 0)) #Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>, #Function<39.51599720/1 in Stream.filter/2>]]> iex(3)> result2 |> Enum.reduce(0, &+/2) 130
Stream Solution page_range |> Stream.map(&fetch_data/1) |> Stream.concat |> Stream.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Summary • Stream module is very handy for lazy evaluation
• saves unnecessary iterations and memory • determine when to trigger evaluation
Further reading http://elixir-lang.org/getting-started/ enumerables-and-streams.html Offical Docs https://hexdocs.pm/elixir/Stream.html