Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How to Use In-Memory Streams
Search
HayaoSuzuki
August 29, 2020
Technology
1
4.4k
How to Use In-Memory Streams
PyCon JP 2020
HayaoSuzuki
August 29, 2020
Tweet
Share
More Decks by HayaoSuzuki
See All by HayaoSuzuki
Tasting "Python Distilled"
hayaosuzuki
0
250
Let's implement useless Python objects
hayaosuzuki
0
1.7k
How to Write Robust Python Code
hayaosuzuki
5
4.1k
Unknown Evolution of the Built-in Function pow
hayaosuzuki
0
1.4k
Python for Everyday
hayaosuzuki
1
1.9k
Do you know cmath module?
hayaosuzuki
0
3.2k
Elementary Number Theory with Python
hayaosuzuki
1
3.5k
Django QuerySet "ARE" Patterns
hayaosuzuki
0
3.3k
A Modernization of Legacy Django Based Applications
hayaosuzuki
1
7.7k
Other Decks in Technology
See All in Technology
Datadogとともにオブザーバビリティを布教しよう
mego2221
0
130
Nekko Cloud、 これまでとこれから ~学生サークルが作る、 小さなクラウド
logica0419
2
730
5分で紹介する生成AIエージェントとAmazon Bedrock Agents / 5-minutes introduction to generative AI agents and Amazon Bedrock Agents
hideakiaoyagi
0
220
Postmanを使いこなす!2025年ぜひとも押さえておきたいPostmanの10の機能
nagix
2
120
7日間でハッキングをはじめる本をはじめてみませんか?_ITエンジニア本大賞2025
nomizone
2
1.4k
依存関係があるコンポーネントは Barrel ファイルでまとめよう
azukiazusa1
3
530
個人開発から公式機能へ: PlaywrightとRailsをつなげた3年の軌跡
yusukeiwaki
10
2.7k
管理者しか知らないOutlookの裏側のAIを覗く#AzureTravelers
hirotomotaguchi
1
240
Classmethod AI Talks(CATs) #15 司会進行スライド(2025.02.06) / classmethod-ai-talks-aka-cats_moderator-slides_vol15_2025-02-06
shinyaa31
0
170
飲食店予約台帳を支えるインタラクティブ UI 設計と実装
siropaca
6
1.4k
RSNA2024振り返り
nanachi
0
500
スタートアップ1人目QAエンジニアが QAチームを立ち上げ、“個”からチーム、 そして“組織”に成長するまで / How to set up QA team at reiwatravel
mii3king
1
1.1k
Featured
See All Featured
The Cost Of JavaScript in 2023
addyosmani
47
7.3k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
330
21k
How to Ace a Technical Interview
jacobian
276
23k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.7k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.4k
Thoughts on Productivity
jonyablonski
69
4.5k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
4
400
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
132
33k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
27
1.5k
Unsuck your backbone
ammeep
669
57k
Transcript
ΠϯϝϞϦʔετϦʔϜ׆༻ज़ How to Use In-Memory Streams Hayao Suzuki PyCon JP
2020 August 29, 2020
ൃදʹࡍͯ͠ GitHub ʹࢿྉ͕͋Γ·͢ › https://github.com/HayaoSuzuki/pyconjp2020 Twitter ͷϋογϡλά › #pyconjp_1 PyCon
JP Fellow Slack › #jp-2020-track-1 2 / 27
Who am I ? ͓લ୭Α Name Hayao Suzukiʢླɹॣʣ Twitter @CardinalXaro
Work Python Programmer at iRidge, Inc. 3 / 27
Who am I ? Technical Reviewer › Effective Python ୈ
2 ൛ (O’Reilly Japan) › ಈֶ͔ͯ͠Ϳྔࢠίϯϐϡʔλϓϩάϥϛϯά (O’Reilly Japan) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 4 / 27
Who am I ? Selected Talks › ϨΨγʔ Django ΞϓϦέʔγϣϯͷݱԽ
(DjangoCongress JP 2018) › SymPy ʹΑΔࣜॲཧ (PyCon JP 2018) › Python ͱָ͠Ήॳ (PyCon mini Hiroshima 2019) › ܅ cmath Λ͍ͬͯΔ͔ (PyCon mini Shizuoka 2020) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 5 / 27
ࠓͷඪ ͜Μͳ՝Λղܾ͍ͨ͠ʂ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͠ɺCSV ϑΝΠϧʹՃ͢Δ › Ϋϥυ্ʹߏஙͨ͠طଘͷγεςϜʹՃ͢ΔܗͰ࣮͢Δ ›
ຖ࣮ߦ͢Δ ΫϥυαʔϏεैྔ՝ۚ ͳΔ͘ਝʹॲཧ͍ͨ͠ʂ 6 / 27
ࠓͷඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͢Δ › GB αΠζͷσʔλΛ
CSV ϑΝΠϧʹՃ͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥυετϨʔδʹΞοϓϩʔυ͢Δ ੳ › σʔλαΠζ͕େ͖͍ › σʔλͷՃ୯७ͳॲཧ 7 / 27
ࠓͷඪ ϘτϧωοΫͲ͔͜ › ZIP ѹॖͦΕ΄ͲେมͰͳ͍ › σʔλՃ୯७ͳॲཧ › ϘτϧωοΫ I/O
ॲཧʹ͋Γͦ͏ Կͱ͔ͯ͠ I/O ॲཧΛਝʹॲཧ͍ͨ͠ʂʂʂ 8 / 27
Today’s Theme In-Memory Streams 9 / 27
Stream? ͦͦετϦʔϜͬͯԿʁ ετϦʔϜϑΝΠϧΦϒδΣΫτͰ͋Δɻ 10 / 27
File Object? ϑΝΠϧΦϒδΣΫτͬͯԿʁ › read() write() ͳͲͷϝιουΛ࣋ͭΦϒδΣΫτ › σΟεΫ্ͷϑΝΠϧผͷॴʹ͋ΔετϨʔδɺೖग़ྗػثͱ
ΓͱΓ͕Ͱ͖Δ 11 / 27
File Object? ϑΝΠϧΦϒδΣΫτͨͪ › ੜόΠφϦϑΝΠϧ › όοϑΝ͖όΠφϦϑΝΠϧ › ςΩετϑΝΠϧ 12
/ 27
͍ํ ςΩετϑΝΠϧ f = open("myfile.txt", "r") όοϑΝ͖όΠφϦ f = open("myfile.jpg",
"rb") 13 / 27
open ؔͷཪଆ open ԿΛ͍ͯ͠Δͷ͔ʁ OS ͷγεςϜίʔϧ API ΛݺͿ 14 /
27
open ؔͷཪଆ ྫɿCSV ʹՃ͢Δ with open("events.csv", "w") as csv_file: fieldnames
= ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 15 / 27
open ؔͷཪଆ ྫɿWindows › CreateFileʢϑΝΠϧͷΞΫηεݖऔಘʣ › QueryAllInformationFileʢϑΝΠϧใͷऔಘʣ › WriteFileʢϑΝΠϧॻ͖ࠐΉʣ ›
CloseFileʢϑΝΠϧΛด͡Δʣ Process Monitor ܦ༝Ͱ֬ೝͨ͠ɻ 16 / 27
open ؔͷཪଆ ྫɿUbuntu on WSL › openat ʢϑΝΠϧͷΦʔϓϯʣ › fstatʢϑΝΠϧใͷऔಘʣ
› ioctlʢσόΠε੍ޚʣ › lseekʢϑΝΠϧͷγʔΫʣ › writeʢϑΝΠϧॻ͖ࠐΉʣ › closeʢϑΝΠϧΛด͡Δʣ strace ܦ༝Ͱ֬ೝͨ͠ɻ 17 / 27
࠷ޙʹস͏ͷ୭ͩ ࠷ऴతͳՌͲ͜ʹஔ͘ʁ › ϑΝΠϧΛϩʔΧϧʹอଘ͢Δͷ͕ΰʔϧͰͳ͍ › ϑΝΠϧΛ AWS S3 ͳͲͷ֎෦ʹஔ͖͍ͨ ϩʔΧϧσόΠεʹϑΝΠϧΛॻ͖ࠐΈͨ͘ͳ͍ʂ
18 / 27
Today’s Theme In-Memory Streams 19 / 27
ΠϯϝϞϦʔετϦʔϜ ΠϯϝϞϦʔετϦʔϜͱ › str bytes ΛϑΝΠϧΦϒδΣΫτͷΑ͏ʹѻ͑Δ › ಡΈॻ͖ՄೳɺϥϯμϜΞΫηεՄೳ 20
/ 27
StringIO StringIO ςΩετϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿCSV Λ StringIO ͰऔΓѻ͏ import io with
io.StringIO() as csv_file: fieldnames = ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 21 / 27
BytesIO BytesIO όοϑΝ͖όΠφϦϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿPNG Λ BytesIO ͰऔΓѻ͏ import io with
io.BytesIO(png_bytes) as f: png_header = f.read(8) print(png_header) # b'\x89PNG\r\n\x1a\n' 22 / 27
෮शɿࠓͷඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͢Δ › GB αΠζͷσʔλΛ
CSV ϑΝΠϧʹՃ͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥυετϨʔδʹΞοϓϩʔυ͢Δ 23 / 27
σʔλΛΠϯλʔωοτܦ༝Ͱऔಘ͢Δ ྫɿConnpass API Λίʔϧ͢Δ with urllib.request.urlopen(url) as response: events =
json.load(response)["events"] 24 / 27
σʔλΛՃ͢Δ ྫɿAPI ͷऔಘ݁ՌΛ CSV ʹ͢Δ with io.StringIO() as ts: header
= ["title", "started_at", "ended_at"] writer = csv.DictWriter(ts, fieldnames=header) writer.writeheader() writer.writerows(events) 25 / 27
σʔλΛѹॖ&Ξοϓϩʔυ ྫɿZIP ʹѹॖͯ͠ AWS S3 ʹΞοϓϩʔυ with io.BytesIO() as bs:
with zipfile.ZipFile(bytes_stream, "w") as zf: zf.writestr("events.csv", ts.getvalue()) bs.seek(0) # ϑΝΠϧγʔΫ͕ϙΠϯτ s3.upload_fileobj(bs, "bucket", "events.zip") 26 / 27
Conclusion ·ͱΊ › io ϞδϡʔϧʹΠϯϝϞϦʔετϦʔϜؚ͕·ΕΔɻ › str bytes ΛϑΝΠϧΦϒδΣΫτͷΑ͏ʹѻ͏͜ͱ͕Ͱ͖Δɻ
› ௨ৗͷ open ͱҟͳΓγεςϜίʔϧ͕ݺΕͳ͍ɻ › σΟεΫͷ I/O ΛݮΒ͍ͨ͠ɺ·ͨͰ͖ͳ͍ঢ়گԼͰͷར༻ ͕࠷దͰ͋Δɻ io ϞδϡʔϧΛօ༷ͷಓ۩ശʹೖΕ͍ͯͩ͘͞ʂ 27 / 27