Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elixir Stream by Examples
Search
Wu Qing
March 18, 2018
Programming
0
49
Elixir Stream by Examples
Presented at ElixirCamp 3 (March 16–19, 2018) in Lord Somers Camp, Somers, VIC
Wu Qing
March 18, 2018
Tweet
Share
More Decks by Wu Qing
See All by Wu Qing
Building a Slack app with Phoenix
wiserfirst
0
56
Journey of Upgrading an App to Phoenix 1.6
wiserfirst
0
67
Bitwise Fun with Elixir
wiserfirst
0
180
Tower of Hanoi with Phoenix LiveView
wiserfirst
0
620
What is telemetry all about
wiserfirst
0
74
What's New In Ecto 3
wiserfirst
1
280
Practical tips about with (in Elixir)
wiserfirst
0
36
Gentle Introduction to Elixir Macros
wiserfirst
0
38
Fight flaky specs with RSpec
wiserfirst
0
64
Other Decks in Programming
See All in Programming
Claude Code + Container Use と Cursor で作る ローカル並列開発環境のススメ / ccc local dev
kaelaela
10
5.8k
Startups on Rails in Past, Present and Future–Irina Nazarova, RailsConf 2025
irinanazarova
0
130
Flutterで備える!Accessibility Nutrition Labels完全ガイド
yuukiw00w
0
160
Node-RED を(HTTP で)つなげる MCP サーバーを作ってみた
highu
0
120
ISUCON研修おかわり会 講義スライド
arfes0e2b3c
1
450
たった 1 枚の PHP ファイルで実装する MCP サーバ / MCP Server with Vanilla PHP
okashoi
1
260
Team operations that are not burdened by SRE
kazatohiei
1
310
Rails Frontend Evolution: It Was a Setup All Along
skryukov
0
160
AI コーディングエージェントの時代へ:JetBrains が描く開発の未来
masaruhr
1
180
技術同人誌をMCP Serverにしてみた
74th
1
650
AI時代の『改訂新版 良いコード/悪いコードで学ぶ設計入門』 / ai-good-code-bad-code
minodriven
20
7.7k
Is Xcode slowly dying out in 2025?
uetyo
1
280
Featured
See All Featured
Thoughts on Productivity
jonyablonski
69
4.7k
Art, The Web, and Tiny UX
lynnandtonic
299
21k
GraphQLとの向き合い方2022年版
quramy
49
14k
Designing for Performance
lara
610
69k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.1k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.9k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
Raft: Consensus for Rubyists
vanstee
140
7k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
The Language of Interfaces
destraynor
158
25k
Transcript
Elixir Stream by Examples Qing Wu
None
Service S Integration
Data source: API endpoint • millions of records involved •
pagination that maps to id ranges
Data source @spec fetch_data(integer) :: [Map.t] def fetch_data(page_number) do ...
end
Sending to Service S • Generate CSV files • Upload
to an FTP server
Sending data @spec handle_data([Map.t]) :: :ok | {:error, String.t} def
handle_data(data) do ... end
Solution page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |> fetch_data
|> handle_data end)
None
Enum Module • Provide functions to work with collections that
can be enumerated over • Functions in Enum always produce immediate results
Enum examples iex(1)> result1 = 1..10 |> Enum.map(&(&1 + 20))
[21, 22, 23, 24, 25, 26, 27, 28, 29, 30] iex(2)> result2 = result1 |> Enum.filter(&(rem(&1, 2) == 0)) [22, 24, 26, 28, 30] iex(3)> result2 |> Enum.reduce(0, &+/2) 130
Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |>
fetch_data |> handle_data end)
None
Solution (Naive) page_range = 0..999 Enum.each(page_range, fn(page_number) -> page_number |>
fetch_data |> handle_data end)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Enum Solution page_range |> Enum.map(&fetch_data/1) |> Enum.concat |> Enum.chunk_every(10_000) |>
Enum.each(&handle_data/1)
None
Stream module iex(1)> result1 = 1..10 |> Stream.map(&(&1 + 20))
#Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>]]> iex(2)> result2 = result1 |> Stream.filter(&(rem(&1, 2) == 0)) #Stream<[enum: 1..10, funs: [#Function<46.51599720/1 in Stream.map/2>, #Function<39.51599720/1 in Stream.filter/2>]]> iex(3)> result2 |> Enum.reduce(0, &+/2) 130
Stream Solution page_range |> Stream.map(&fetch_data/1) |> Stream.concat |> Stream.chunk_every(10_000) |>
Enum.each(&handle_data/1)
Summary • Stream module is very handy for lazy evaluation
• saves unnecessary iterations and memory • determine when to trigger evaluation
Further reading http://elixir-lang.org/getting-started/ enumerables-and-streams.html Offical Docs https://hexdocs.pm/elixir/Stream.html