Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to Use In-Memory Streams

HayaoSuzuki
August 29, 2020

How to Use In-Memory Streams

PyCon JP 2020

HayaoSuzuki

August 29, 2020
Tweet

More Decks by HayaoSuzuki

Other Decks in Technology

Transcript

  1. Who am I ? Technical Reviewer › Effective Python ୈ

    2 ൛ (O’Reilly Japan) › ಈֶ͔ͯ͠Ϳྔࢠίϯϐϡʔλϓϩάϥϛϯά (O’Reilly Japan) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 4 / 27
  2. Who am I ? Selected Talks › ϨΨγʔ Django ΞϓϦέʔγϣϯͷݱ୅Խ

    (DjangoCongress JP 2018) › SymPy ʹΑΔ਺ࣜॲཧ (PyCon JP 2018) › Python ͱָ͠Ήॳ౳੔਺࿦ (PyCon mini Hiroshima 2019) › ܅͸ cmath Λ஌͍ͬͯΔ͔ (PyCon mini Shizuoka 2020) https://xaro.hatenablog.jp/ ʹϦετ͕͋Γ·͢ɻ 5 / 27
  3. ࠓ೔ͷ໨ඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ਺ GB αΠζͷσʔλΛऔಘ͢Δ › ਺ GB αΠζͷσʔλΛ

    CSV ϑΝΠϧʹՃ޻͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥ΢υετϨʔδʹΞοϓϩʔυ͢Δ ෼ੳ › σʔλαΠζ͕େ͖͍ › σʔλͷՃ޻͸୯७ͳॲཧ 7 / 27
  4. open ؔ਺ͷཪଆ ྫɿCSV ʹՃ޻͢Δ with open("events.csv", "w") as csv_file: fieldnames

    = ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 15 / 27
  5. open ؔ਺ͷཪଆ ྫɿUbuntu on WSL › openat ʢϑΝΠϧͷΦʔϓϯʣ › fstatʢϑΝΠϧ৘ใͷऔಘʣ

    › ioctlʢσόΠε੍ޚʣ › lseekʢϑΝΠϧͷγʔΫʣ › writeʢϑΝΠϧ΁ॻ͖ࠐΉʣ › closeʢϑΝΠϧΛด͡Δʣ strace ܦ༝Ͱ֬ೝͨ͠ɻ 17 / 27
  6. StringIO StringIO ςΩετϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿCSV Λ StringIO ͰऔΓѻ͏ import io with

    io.StringIO() as csv_file: fieldnames = ["title", "started_at", "ended_at"] writer = csv.DictWriter(csv_file, fieldnames) writer.writeheader() writer.writerows(events) 21 / 27
  7. BytesIO BytesIO όοϑΝ෇͖όΠφϦϑΝΠϧͷͨΊͷΠϯϝϞϦετϦʔϜ ྫɿPNG Λ BytesIO ͰऔΓѻ͏ import io with

    io.BytesIO(png_bytes) as f: png_header = f.read(8) print(png_header) # b'\x89PNG\r\n\x1a\n' 22 / 27
  8. ෮शɿࠓ೔ͷ໨ඪ ॲཧͷྲྀΕ › Πϯλʔωοτܦ༝Ͱ਺ GB αΠζͷσʔλΛऔಘ͢Δ › ਺ GB αΠζͷσʔλΛ

    CSV ϑΝΠϧʹՃ޻͢Δ › CSV ϑΝΠϧΛ ZIP ѹॖ͢Δ › ZIP ѹॖσʔλΛΫϥ΢υετϨʔδʹΞοϓϩʔυ͢Δ 23 / 27
  9. σʔλΛՃ޻͢Δ ྫɿAPI ͷऔಘ݁ՌΛ CSV ʹ͢Δ with io.StringIO() as ts: header

    = ["title", "started_at", "ended_at"] writer = csv.DictWriter(ts, fieldnames=header) writer.writeheader() writer.writerows(events) 25 / 27
  10. σʔλΛѹॖ&Ξοϓϩʔυ ྫɿZIP ʹѹॖͯ͠ AWS S3 ʹΞοϓϩʔυ with io.BytesIO() as bs:

    with zipfile.ZipFile(bytes_stream, "w") as zf: zf.writestr("events.csv", ts.getvalue()) bs.seek(0) # ϑΝΠϧγʔΫ͕ϙΠϯτ s3.upload_fileobj(bs, "bucket", "events.zip") 26 / 27
  11. Conclusion ·ͱΊ › io Ϟδϡʔϧʹ͸ΠϯϝϞϦʔετϦʔϜؚ͕·ΕΔɻ › str ΍ bytes ΛϑΝΠϧΦϒδΣΫτͷΑ͏ʹѻ͏͜ͱ͕Ͱ͖Δɻ

    › ௨ৗͷ open ͱҟͳΓγεςϜίʔϧ͕ݺ͹Εͳ͍ɻ › σΟεΫ΁ͷ I/O ΛݮΒ͍ͨ͠ɺ·ͨ͸Ͱ͖ͳ͍ঢ়گԼͰͷར༻ ͕࠷దͰ͋Δɻ io ϞδϡʔϧΛօ༷ͷಓ۩ശʹೖΕ͍ͯͩ͘͞ʂ 27 / 27