Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elixir Stream by Examples
Search
Wu Qing
March 18, 2018
Programming
0
38
Elixir Stream by Examples
Presented at ElixirCamp 3 (March 16–19, 2018) in Lord Somers Camp, Somers, VIC
Wu Qing
March 18, 2018
Tweet
Share
More Decks by Wu Qing
See All by Wu Qing
Building a Slack app with Phoenix
wiserfirst
0
34
Journey of Upgrading an App to Phoenix 1.6
wiserfirst
0
64
Bitwise Fun with Elixir
wiserfirst
0
160
Tower of Hanoi with Phoenix LiveView
wiserfirst
0
570
What is telemetry all about
wiserfirst
0
60
What's New In Ecto 3
wiserfirst
1
270
Practical tips about with (in Elixir)
wiserfirst
0
31
Gentle Introduction to Elixir Macros
wiserfirst
0
35
Fight flaky specs with RSpec
wiserfirst
0
57
Other Decks in Programming
See All in Programming
Folding Cheat Sheet #1
philipschwarz
PRO
0
210
Javaエンジニアのための Nodejs/Nuxt3入門
hidekatsu_izuno
0
280
puregoの活用例
aethiopicuschan
0
220
OpenAPIを中心に考えるAPI開発入門 / Introduction to API Development with a Focus on OpenAPI
seike460
PRO
2
120
ログラスを支える設計標準について / loglass-design-standards
urmot
10
2.1k
⼤規模⾔語モデルの拡張(RAG)が 終わったかも知れない件について
nearme_tech
22
15k
Java 22 Overview
kishida
1
170
Blue/Greenデプロイの導入による 運用フローの改善
kudoas
1
360
#phpcon_odawara オープン・クローズドなテストフィクスチャを求めて / open closed test fixtures
77web
3
220
VSCodeでのDatabricks開発もお勧めしたい/I would also recommend Databricks development with VSCode.
kazumain
0
240
TYPO3 v13 – The road to LTS: What's new and new APIs
luisasofie_xoxo
0
180
コードレビューで学ぶ!Kotlinオブジェクト指向デザインパターン
akkie76
2
180
Featured
See All Featured
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
1
1.3k
It's Worth the Effort
3n
180
27k
Teambox: Starting and Learning
jrom
128
8.4k
From Idea to $5000 a Month in 5 Months
shpigford
377
45k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
13
1.5k
GitHub's CSS Performance
jonrohan
1023
450k
Stop Working from a Prison Cell
hatefulcrawdad
266
19k
StorybookのUI Testing Handbookを読んだ
zakiyama
11
4.6k
Art, The Web, and Tiny UX
lynnandtonic
288
19k
Scaling GitHub
holman
457
140k
Designing Experiences People Love
moore
136
23k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
18
6.9k
Transcript
Elixir Stream by Examples Qing Wu
None
Service S Integration
Data source: API endpoint • millions of records involved •
pagination that maps to id ranges
Data source @spec fetch_data(integer) :: [Map.t] def fetch_data(page_number) do ...
end
Sending to Service S • Generate CSV files • Upload
to an FTP server
Sending data @spec handle_data([Map.t]) :: :ok | {:error, String.t} def
handle_data(data) do ... end
Solution page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |> fetch_data
|> handle_data end)
None
Enum Module • Provide functions to work with collections that
can be enumerated over • Functions in Enum always produce immediate results
Enum examples iex(1)> result1 = 1..10 |> Enum.map(&(&1 + 20))
[21, 22, 23, 24, 25, 26, 27, 28, 29, 30] iex(2)> result2 = result1 |> Enum.filter(&(rem(&1, 2) == 0)) [22, 24, 26, 28, 30] iex(3)> result2 |> Enum.reduce(0, &+/2) 130
Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |>
fetch_data |> handle_data end)
None
Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |>
fetch_data |> handle_data end)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
None
Stream module iex(1)> result1 = 1..10 |> Stream.map(&(&1 + 20))
#Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>]]> iex(2)> result2 = result1 |> Stream.filter(&(rem(&1, 2) == 0)) #Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>, #Function<39.51599720/1 in Stream.filter/2>]]> iex(3)> result2 |> Enum.reduce(0, &+/2) 130
Stream Solution page_range |> Stream.map(&fetch_data/1) |> Stream.concat |> Stream.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Summary • Stream module is very handy for lazy evaluation
• saves unnecessary iterations and memory • determine when to trigger evaluation
Further reading http://elixir-lang.org/getting-started/ enumerables-and-streams.html Offical Docs https://hexdocs.pm/elixir/Stream.html