Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How to Use In-Memory Streams
Search
HayaoSuzuki
August 29, 2020
Technology
1
4.4k
How to Use In-Memory Streams
PyCon JP 2020
HayaoSuzuki
August 29, 2020
Tweet
Share
More Decks by HayaoSuzuki
See All by HayaoSuzuki
Tasting "Python Distilled"
hayaosuzuki
0
250
Let's implement useless Python objects
hayaosuzuki
0
1.7k
How to Write Robust Python Code
hayaosuzuki
5
4.1k
Unknown Evolution of the Built-in Function pow
hayaosuzuki
0
1.4k
Python for Everyday
hayaosuzuki
1
1.9k
Do you know cmath module?
hayaosuzuki
0
3.2k
Elementary Number Theory with Python
hayaosuzuki
1
3.5k
Django QuerySet "ARE" Patterns
hayaosuzuki
0
3.3k
A Modernization of Legacy Django Based Applications
hayaosuzuki
1
7.7k
Other Decks in Technology
See All in Technology
リアルタイム分析データベースで実現する SQLベースのオブザーバビリティ
mikimatsumoto
0
950
サーバーレスアーキテクチャと生成AIの融合 / Serverless Meets Generative AI
_kensh
12
3k
SA Night #2 FinatextのSA思想/SA Night #2 Finatext session
satoshiimai
1
100
開発者が自律的に AWS Security Hub findings に 対応する仕組みと AWS re:Invent 2024 登壇体験談 / Developers autonomously report AWS Security Hub findings Corresponding mechanism and AWS re:Invent 2024 presentation experience
kaminashi
0
190
30分でわかる『アジャイルデータモデリング』
hanon52_
9
2.2k
10分で紹介するAmazon Bedrock利用時のセキュリティ対策 / 10-minutes introduction to security measures when using Amazon Bedrock
hideakiaoyagi
0
170
ビジネスと現場活動をつなぐソフトウェアエンジニアリング~とあるスタートアッププロダクトの成長記録より~
mizunori
0
210
日経電子版 x AIエージェントの可能性とAgentic RAGによって提案書生成を行う技術
masahiro_nishimi
1
290
APIファーストで実現する運用性の高い IoT プラットフォーム: SORACOMのアプローチ
soracom
PRO
0
240
『衛星データ利用の方々にとって近いようで触れる機会のなさそうな小話 ~ 衛星搭載ソフトウェアと衛星運用ソフトウェア (実物) を動かしながらわいわいする編 ~』 @日本衛星データコミニティ勉強会
meltingrabbit
0
120
Googleマップ/Earthが一般化した 地図タイルのイマ
mapconcierge4agu
1
200
テストアーキテクチャ設計で実現する高品質で高スピードな開発の実践 / Test Architecture Design in Practice
ropqa
3
710
Featured
See All Featured
Building Your Own Lightsaber
phodgson
104
6.2k
We Have a Design System, Now What?
morganepeng
51
7.4k
Fontdeck: Realign not Redesign
paulrobertlloyd
82
5.4k
Music & Morning Musume
bryan
46
6.3k
Building an army of robots
kneath
302
45k
Rebuilding a faster, lazier Slack
samanthasiow
79
8.8k
Testing 201, or: Great Expectations
jmmastey
41
7.2k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.4k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.2k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
8
270
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
40
2k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Transcript
ΠϯϝϞϦʔετϦʔϜ׆༻ज़ How to Use In-Memory Streams Hayao Suzuki PyCon JP
2020 August 29, 2020
ൃදʹࡍͯ͠ GitHub ʹࢿྉ͕͋Γ·͢ › https://github.com/HayaoSuzuki/pyconjp2020 Twitter ͷϋογϡλά › #pyconjp_1 PyCon
JP Fellow Slack › #jp-2020-track-1 2 / 27
Who am I ? ͓લ୭Α Name Hayao Suzukiʢླɹॣʣ Twitter @CardinalXaro
Work Python Programmer at iRidge, Inc. 3 / 27
Who am I ? Technical Reviewer › Effective Python ୈ
2 ൛ (O’Reilly Japan) › ಈֶ͔ͯ͠Ϳྔࢠίϯϐϡʔλϓϩάϥϛϯά (O’Reilly Japan) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 4 / 27
Who am I ? Selected Talks › ϨΨγʔ Django ΞϓϦέʔγϣϯͷݱԽ
(DjangoCongress JP 2018) › SymPy ʹΑΔࣜॲཧ (PyCon JP 2018) › Python ͱָ͠Ήॳ (PyCon mini Hiroshima 2019) › ܅ cmath Λ͍ͬͯΔ͔ (PyCon mini Shizuoka 2020) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 5 / 27
ࠓͷඪ ͜Μͳ՝Λղܾ͍ͨ͠ʂ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͠ɺCSV ϑΝΠϧʹՃ͢Δ › Ϋϥυ্ʹߏஙͨ͠طଘͷγεςϜʹՃ͢ΔܗͰ࣮͢Δ ›
ຖ࣮ߦ͢Δ ΫϥυαʔϏεैྔ՝ۚ ͳΔ͘ਝʹॲཧ͍ͨ͠ʂ 6 / 27
ࠓͷඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͢Δ › GB αΠζͷσʔλΛ
CSV ϑΝΠϧʹՃ͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥυετϨʔδʹΞοϓϩʔυ͢Δ ੳ › σʔλαΠζ͕େ͖͍ › σʔλͷՃ୯७ͳॲཧ 7 / 27
ࠓͷඪ ϘτϧωοΫͲ͔͜ › ZIP ѹॖͦΕ΄ͲେมͰͳ͍ › σʔλՃ୯७ͳॲཧ › ϘτϧωοΫ I/O
ॲཧʹ͋Γͦ͏ Կͱ͔ͯ͠ I/O ॲཧΛਝʹॲཧ͍ͨ͠ʂʂʂ 8 / 27
Today’s Theme In-Memory Streams 9 / 27
Stream? ͦͦετϦʔϜͬͯԿʁ ετϦʔϜϑΝΠϧΦϒδΣΫτͰ͋Δɻ 10 / 27
File Object? ϑΝΠϧΦϒδΣΫτͬͯԿʁ › read() write() ͳͲͷϝιουΛ࣋ͭΦϒδΣΫτ › σΟεΫ্ͷϑΝΠϧผͷॴʹ͋ΔετϨʔδɺೖग़ྗػثͱ
ΓͱΓ͕Ͱ͖Δ 11 / 27
File Object? ϑΝΠϧΦϒδΣΫτͨͪ › ੜόΠφϦϑΝΠϧ › όοϑΝ͖όΠφϦϑΝΠϧ › ςΩετϑΝΠϧ 12
/ 27
͍ํ ςΩετϑΝΠϧ f = open("myfile.txt", "r") όοϑΝ͖όΠφϦ f = open("myfile.jpg",
"rb") 13 / 27
open ؔͷཪଆ open ԿΛ͍ͯ͠Δͷ͔ʁ OS ͷγεςϜίʔϧ API ΛݺͿ 14 /
27
open ؔͷཪଆ ྫɿCSV ʹՃ͢Δ with open("events.csv", "w") as csv_file: fieldnames
= ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 15 / 27
open ؔͷཪଆ ྫɿWindows › CreateFileʢϑΝΠϧͷΞΫηεݖऔಘʣ › QueryAllInformationFileʢϑΝΠϧใͷऔಘʣ › WriteFileʢϑΝΠϧॻ͖ࠐΉʣ ›
CloseFileʢϑΝΠϧΛด͡Δʣ Process Monitor ܦ༝Ͱ֬ೝͨ͠ɻ 16 / 27
open ؔͷཪଆ ྫɿUbuntu on WSL › openat ʢϑΝΠϧͷΦʔϓϯʣ › fstatʢϑΝΠϧใͷऔಘʣ
› ioctlʢσόΠε੍ޚʣ › lseekʢϑΝΠϧͷγʔΫʣ › writeʢϑΝΠϧॻ͖ࠐΉʣ › closeʢϑΝΠϧΛด͡Δʣ strace ܦ༝Ͱ֬ೝͨ͠ɻ 17 / 27
࠷ޙʹস͏ͷ୭ͩ ࠷ऴతͳՌͲ͜ʹஔ͘ʁ › ϑΝΠϧΛϩʔΧϧʹอଘ͢Δͷ͕ΰʔϧͰͳ͍ › ϑΝΠϧΛ AWS S3 ͳͲͷ֎෦ʹஔ͖͍ͨ ϩʔΧϧσόΠεʹϑΝΠϧΛॻ͖ࠐΈͨ͘ͳ͍ʂ
18 / 27
Today’s Theme In-Memory Streams 19 / 27
ΠϯϝϞϦʔετϦʔϜ ΠϯϝϞϦʔετϦʔϜͱ › str bytes ΛϑΝΠϧΦϒδΣΫτͷΑ͏ʹѻ͑Δ › ಡΈॻ͖ՄೳɺϥϯμϜΞΫηεՄೳ 20
/ 27
StringIO StringIO ςΩετϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿCSV Λ StringIO ͰऔΓѻ͏ import io with
io.StringIO() as csv_file: fieldnames = ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 21 / 27
BytesIO BytesIO όοϑΝ͖όΠφϦϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿPNG Λ BytesIO ͰऔΓѻ͏ import io with
io.BytesIO(png_bytes) as f: png_header = f.read(8) print(png_header) # b'\x89PNG\r\n\x1a\n' 22 / 27
෮शɿࠓͷඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͢Δ › GB αΠζͷσʔλΛ
CSV ϑΝΠϧʹՃ͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥυετϨʔδʹΞοϓϩʔυ͢Δ 23 / 27
σʔλΛΠϯλʔωοτܦ༝Ͱऔಘ͢Δ ྫɿConnpass API Λίʔϧ͢Δ with urllib.request.urlopen(url) as response: events =
json.load(response)["events"] 24 / 27
σʔλΛՃ͢Δ ྫɿAPI ͷऔಘ݁ՌΛ CSV ʹ͢Δ with io.StringIO() as ts: header
= ["title", "started_at", "ended_at"] writer = csv.DictWriter(ts, fieldnames=header) writer.writeheader() writer.writerows(events) 25 / 27
σʔλΛѹॖ&Ξοϓϩʔυ ྫɿZIP ʹѹॖͯ͠ AWS S3 ʹΞοϓϩʔυ with io.BytesIO() as bs:
with zipfile.ZipFile(bytes_stream, "w") as zf: zf.writestr("events.csv", ts.getvalue()) bs.seek(0) # ϑΝΠϧγʔΫ͕ϙΠϯτ s3.upload_fileobj(bs, "bucket", "events.zip") 26 / 27
Conclusion ·ͱΊ › io ϞδϡʔϧʹΠϯϝϞϦʔετϦʔϜؚ͕·ΕΔɻ › str bytes ΛϑΝΠϧΦϒδΣΫτͷΑ͏ʹѻ͏͜ͱ͕Ͱ͖Δɻ
› ௨ৗͷ open ͱҟͳΓγεςϜίʔϧ͕ݺΕͳ͍ɻ › σΟεΫͷ I/O ΛݮΒ͍ͨ͠ɺ·ͨͰ͖ͳ͍ঢ়گԼͰͷར༻ ͕࠷దͰ͋Δɻ io ϞδϡʔϧΛօ༷ͷಓ۩ശʹೖΕ͍ͯͩ͘͞ʂ 27 / 27