Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How to Use In-Memory Streams
Search
HayaoSuzuki
August 29, 2020
Technology
1
3.7k
How to Use In-Memory Streams
PyCon JP 2020
HayaoSuzuki
August 29, 2020
Tweet
Share
More Decks by HayaoSuzuki
See All by HayaoSuzuki
Tasting "Python Distilled"
hayaosuzuki
0
69
Let's implement useless Python objects
hayaosuzuki
0
1.3k
How to Write Robust Python Code
hayaosuzuki
5
3.5k
Unknown Evolution of the Built-in Function pow
hayaosuzuki
0
1.2k
Python for Everyday
hayaosuzuki
1
1.7k
Do you know cmath module?
hayaosuzuki
0
2.8k
Elementary Number Theory with Python
hayaosuzuki
1
3.1k
Django QuerySet "ARE" Patterns
hayaosuzuki
0
2.9k
A Modernization of Legacy Django Based Applications
hayaosuzuki
1
6.7k
Other Decks in Technology
See All in Technology
Hyperledger Fabricの成長、成熟を振り返る / Looking back history of Hyperledger Fabric
gakumura
0
180
中規模・ミドルTier開発組織におけるDevRelの戦略と実行と成果 - DevRel Guild Conference Mini -
leveragestech
2
270
OCI コスト管理
ocise
1
120
開発者体験を意識した開発チームの生産性向上の取り組み
ham0215
3
660
四国のあのイベントの〇〇システムを45日間で構築した話 / cloudohenro2024_tachibana
biatunky
0
160
セキュリティ監視の内製化 効率とリスク
mixi_engineers
PRO
6
690
ログラスが面白いと思う理由をマネージャーがエモく語ってみる / 20240829 vs LT
yoshikiiida
1
520
夏休みの(最後の)宿題 for JuliaTokyo #12
antimon2
0
130
Datadog を使ったプロダクトとクラウドの セキュリティモニタリング
mrtc0
0
490
Chunk computing
matyo91
0
270
Javaにおける関数型プログラミンへの取り組み
skrb
6
220
PlaywrightによるE2Eテスト入門 / Introduction to E2E Testing with Playwright
rhumie
3
940
Featured
See All Featured
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
34
1.9k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
27
8.9k
RailsConf 2023
tenderlove
26
790
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
Building Better People: How to give real-time feedback that sticks.
wjessup
359
18k
4 Signs Your Business is Dying
shpigford
179
21k
Bash Introduction
62gerente
608
210k
Visualization
eitanlees
142
15k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
246
1.3M
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
28
2.2k
Embracing the Ebb and Flow
colly
83
4.4k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
278
13k
Transcript
ΠϯϝϞϦʔετϦʔϜ׆༻ज़ How to Use In-Memory Streams Hayao Suzuki PyCon JP
2020 August 29, 2020
ൃදʹࡍͯ͠ GitHub ʹࢿྉ͕͋Γ·͢ › https://github.com/HayaoSuzuki/pyconjp2020 Twitter ͷϋογϡλά › #pyconjp_1 PyCon
JP Fellow Slack › #jp-2020-track-1 2 / 27
Who am I ? ͓લ୭Α Name Hayao Suzukiʢླɹॣʣ Twitter @CardinalXaro
Work Python Programmer at iRidge, Inc. 3 / 27
Who am I ? Technical Reviewer › Effective Python ୈ
2 ൛ (O’Reilly Japan) › ಈֶ͔ͯ͠Ϳྔࢠίϯϐϡʔλϓϩάϥϛϯά (O’Reilly Japan) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 4 / 27
Who am I ? Selected Talks › ϨΨγʔ Django ΞϓϦέʔγϣϯͷݱԽ
(DjangoCongress JP 2018) › SymPy ʹΑΔࣜॲཧ (PyCon JP 2018) › Python ͱָ͠Ήॳ (PyCon mini Hiroshima 2019) › ܅ cmath Λ͍ͬͯΔ͔ (PyCon mini Shizuoka 2020) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 5 / 27
ࠓͷඪ ͜Μͳ՝Λղܾ͍ͨ͠ʂ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͠ɺCSV ϑΝΠϧʹՃ͢Δ › Ϋϥυ্ʹߏஙͨ͠طଘͷγεςϜʹՃ͢ΔܗͰ࣮͢Δ ›
ຖ࣮ߦ͢Δ ΫϥυαʔϏεैྔ՝ۚ ͳΔ͘ਝʹॲཧ͍ͨ͠ʂ 6 / 27
ࠓͷඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͢Δ › GB αΠζͷσʔλΛ
CSV ϑΝΠϧʹՃ͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥυετϨʔδʹΞοϓϩʔυ͢Δ ੳ › σʔλαΠζ͕େ͖͍ › σʔλͷՃ୯७ͳॲཧ 7 / 27
ࠓͷඪ ϘτϧωοΫͲ͔͜ › ZIP ѹॖͦΕ΄ͲେมͰͳ͍ › σʔλՃ୯७ͳॲཧ › ϘτϧωοΫ I/O
ॲཧʹ͋Γͦ͏ Կͱ͔ͯ͠ I/O ॲཧΛਝʹॲཧ͍ͨ͠ʂʂʂ 8 / 27
Today’s Theme In-Memory Streams 9 / 27
Stream? ͦͦετϦʔϜͬͯԿʁ ετϦʔϜϑΝΠϧΦϒδΣΫτͰ͋Δɻ 10 / 27
File Object? ϑΝΠϧΦϒδΣΫτͬͯԿʁ › read() write() ͳͲͷϝιουΛ࣋ͭΦϒδΣΫτ › σΟεΫ্ͷϑΝΠϧผͷॴʹ͋ΔετϨʔδɺೖग़ྗػثͱ
ΓͱΓ͕Ͱ͖Δ 11 / 27
File Object? ϑΝΠϧΦϒδΣΫτͨͪ › ੜόΠφϦϑΝΠϧ › όοϑΝ͖όΠφϦϑΝΠϧ › ςΩετϑΝΠϧ 12
/ 27
͍ํ ςΩετϑΝΠϧ f = open("myfile.txt", "r") όοϑΝ͖όΠφϦ f = open("myfile.jpg",
"rb") 13 / 27
open ؔͷཪଆ open ԿΛ͍ͯ͠Δͷ͔ʁ OS ͷγεςϜίʔϧ API ΛݺͿ 14 /
27
open ؔͷཪଆ ྫɿCSV ʹՃ͢Δ with open("events.csv", "w") as csv_file: fieldnames
= ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 15 / 27
open ؔͷཪଆ ྫɿWindows › CreateFileʢϑΝΠϧͷΞΫηεݖऔಘʣ › QueryAllInformationFileʢϑΝΠϧใͷऔಘʣ › WriteFileʢϑΝΠϧॻ͖ࠐΉʣ ›
CloseFileʢϑΝΠϧΛด͡Δʣ Process Monitor ܦ༝Ͱ֬ೝͨ͠ɻ 16 / 27
open ؔͷཪଆ ྫɿUbuntu on WSL › openat ʢϑΝΠϧͷΦʔϓϯʣ › fstatʢϑΝΠϧใͷऔಘʣ
› ioctlʢσόΠε੍ޚʣ › lseekʢϑΝΠϧͷγʔΫʣ › writeʢϑΝΠϧॻ͖ࠐΉʣ › closeʢϑΝΠϧΛด͡Δʣ strace ܦ༝Ͱ֬ೝͨ͠ɻ 17 / 27
࠷ޙʹস͏ͷ୭ͩ ࠷ऴతͳՌͲ͜ʹஔ͘ʁ › ϑΝΠϧΛϩʔΧϧʹอଘ͢Δͷ͕ΰʔϧͰͳ͍ › ϑΝΠϧΛ AWS S3 ͳͲͷ֎෦ʹஔ͖͍ͨ ϩʔΧϧσόΠεʹϑΝΠϧΛॻ͖ࠐΈͨ͘ͳ͍ʂ
18 / 27
Today’s Theme In-Memory Streams 19 / 27
ΠϯϝϞϦʔετϦʔϜ ΠϯϝϞϦʔετϦʔϜͱ › str bytes ΛϑΝΠϧΦϒδΣΫτͷΑ͏ʹѻ͑Δ › ಡΈॻ͖ՄೳɺϥϯμϜΞΫηεՄೳ 20
/ 27
StringIO StringIO ςΩετϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿCSV Λ StringIO ͰऔΓѻ͏ import io with
io.StringIO() as csv_file: fieldnames = ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 21 / 27
BytesIO BytesIO όοϑΝ͖όΠφϦϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿPNG Λ BytesIO ͰऔΓѻ͏ import io with
io.BytesIO(png_bytes) as f: png_header = f.read(8) print(png_header) # b'\x89PNG\r\n\x1a\n' 22 / 27
෮शɿࠓͷඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ GB αΠζͷσʔλΛऔಘ͢Δ › GB αΠζͷσʔλΛ
CSV ϑΝΠϧʹՃ͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥυετϨʔδʹΞοϓϩʔυ͢Δ 23 / 27
σʔλΛΠϯλʔωοτܦ༝Ͱऔಘ͢Δ ྫɿConnpass API Λίʔϧ͢Δ with urllib.request.urlopen(url) as response: events =
json.load(response)["events"] 24 / 27
σʔλΛՃ͢Δ ྫɿAPI ͷऔಘ݁ՌΛ CSV ʹ͢Δ with io.StringIO() as ts: header
= ["title", "started_at", "ended_at"] writer = csv.DictWriter(ts, fieldnames=header) writer.writeheader() writer.writerows(events) 25 / 27
σʔλΛѹॖ&Ξοϓϩʔυ ྫɿZIP ʹѹॖͯ͠ AWS S3 ʹΞοϓϩʔυ with io.BytesIO() as bs:
with zipfile.ZipFile(bytes_stream, "w") as zf: zf.writestr("events.csv", ts.getvalue()) bs.seek(0) # ϑΝΠϧγʔΫ͕ϙΠϯτ s3.upload_fileobj(bs, "bucket", "events.zip") 26 / 27
Conclusion ·ͱΊ › io ϞδϡʔϧʹΠϯϝϞϦʔετϦʔϜؚ͕·ΕΔɻ › str bytes ΛϑΝΠϧΦϒδΣΫτͷΑ͏ʹѻ͏͜ͱ͕Ͱ͖Δɻ
› ௨ৗͷ open ͱҟͳΓγεςϜίʔϧ͕ݺΕͳ͍ɻ › σΟεΫͷ I/O ΛݮΒ͍ͨ͠ɺ·ͨͰ͖ͳ͍ঢ়گԼͰͷར༻ ͕࠷దͰ͋Δɻ io ϞδϡʔϧΛօ༷ͷಓ۩ശʹೖΕ͍ͯͩ͘͞ʂ 27 / 27