$30 off During Our Annual Pro Sale. View Details »

io/fs.FS for testability. io/fs.FS for abstraction. / Go Conference 2021 Autumn (Online)

io/fs.FS for testability. io/fs.FS for abstraction. / Go Conference 2021 Autumn (Online)

https://gocon.jp/2021autumn/sessions/io-fs-fs-testability-abstraction/

Go 1.16から追加されたio/fsパッケージが提供しているインターフェースについて、皆さんはどのような目的で活用していますか?

私は、主に2つの目的で活用しています。1つはテスト容易性(テスタビリティ)、もう1つはファイルシステムとしての抽象化です。

本発表では、具体的な事例を元に上記の2つの目的での使い方を紹介します。
また、fs.FSインターフェースを使ったことで失敗した事例も紹介したいと思います。

本発表を通じてio/fsのインターフェースの利用イメージを持ってもらえれば嬉しいです。

Ken’ichiro Oyama

November 13, 2021
Tweet

More Decks by Ken’ichiro Oyama

Other Decks in Technology

Transcript

  1. io/fs.FS for testability
    .

    io/fs.FS for abstraction.
    খࢁ݈Ұ࿠ / GMO PEPABO inc.
    2021.11.13 Go Conference 2021 Autumn (Online)
    1

    View Slide

  2. ࣗݾ঺հ
    ٕज़෦ɹٕज़ج൫νʔϜ
    2018೥ த్ೖࣾ
    খࢁ ݈Ұ࿠ Ken’ichiro Oyama
    গ࣮͠༻తͰখ͞ͳOSSΛॻ͘ͷ͕झຯɻ
    ● GitHub : k1Lo
    W

    ● Twitter : @k1LoW
    2

    View Slide

  3. ΞδΣϯμ
    1. io/fs ύοέʔδ
    2. io/fs.FS for testabilit
    y

    3. io/fs.FS for abstractio
    n

    4. ·ͱΊ
    3

    View Slide

  4. io/fs ύοέʔδ
    4

    View Slide

  5. fs package
    Package fs defines basic interfaces to a file system.


    A file system can be provided by the host operating
    system but also by other packages.


    https://pkg.go.dev/io/fs


    fsύοέʔδ͸ɺϑΝΠϧγεςϜͷجຊతͳinterfaceΛ
    ఆ͍ٛͯ͠·͢ɻϑΝΠϧγεςϜ͸ɺϗετOS͔Βఏ
    ڙ͞ΕΔ͜ͱ΋͋Ε͹ɺଞͷύοέʔδ͔Βఏڙ͞ΕΔ
    ͜ͱ΋͋Γ·͢ɻ


    5

    View Slide

  6. “ϑΝΠϧγεςϜͷinterfaceΛఏڙ͍ͯ͠Δ”
    • interfac
    e

    • io.Reader΍io.Writerͱಉ͡ɻGopherʹ͸͓ೃછΈ
    • ϑΝΠϧγεςϜ
    • ϑΝΠϧγεςϜ?
    6
    io/fs ύοέʔδ

    View Slide

  7. LinuxͷϑΝΠϧγεςϜ
    “ϑΝΠϧΛΧςΰϦผʹ੔ཧͰ͖ΔΑ͏ʹ͢ΔͨΊ
    ʹɺLinuxʹ͓͚ΔϑΝΠϧγεςϜʹ͸ɺσΟϨΫ
    τϦͱ͍͏ɺϑΝΠϧΛ֨ೲ͢ΔͨΊͷಛघͳϑΝΠ
    ϧ͕ଘࡏ͠·͢ɻ


    σΟϨΫτϦͷதʹ͸ɺ௨ৗͷϑΝΠϧɺ͋Δ͍͸͞
    ΒʹσΟϨΫτϦΛอ࣋͢Δ͜ͱ͕Ͱ͖·͢ɻผͷ
    σΟϨΫτϦҎԼͳΒɺෳ਺ͷϑΝΠϧʹಉ໊͡લΛ
    ෇͚ΒΕ·͢ɻ͜ͷ͘͠ΈʹΑͬͯɺLinuxͷϑΝΠ
    ϧγεςϜ͸πϦʔߏ଄Λ͍ͯ͠·͢”
    ෢಺ ֮ “ʦࢼͯ͠ཧղʧLinuxͷ͘͠Έ ʙ࣮ݧͱਤղͰֶͿOSͱϋʔυ΢ΣΞͷجૅ஌ࣝ”


    io/fs ύοέʔδ
    7
    7

    View Slide

  8. ϑΝΠϧγεςϜ


    = ϑΝΠϧͱσΟϨΫτϦ


    …ͱߟ͑Δ
    8
    io/fs ύοέʔδ

    View Slide

  9. ͞Βʹ
    “Linux͕ѻ͑ΔϑΝΠϧγεςϜ͸1͚ͭͩͰ͸͋Γ
    ·ͤΜɻʮext4ʯʮXFSʯʮBtrfsʯͳͲɺෳ਺ͷϑΝ
    ΠϧγεςϜΛѻ͏͜ͱ͕Ͱ͖·͢ɻʢதུʣͨͩ
    ͠ɺͲͷϑΝΠϧγεςϜͰ͋ͬͯ΋ɺϢʔβ͔Β͸
    ࣍ͷΑ͏ͳγεςϜίʔϧͷൃߦͱ͍͏౷Ұͨ͠Πϯ
    λʔϑΣʔεʹΑͬͯΞΫηεͰ͖·͢ɻ”
    ෢಺ ֮ “ʦࢼͯ͠ཧղʧLinuxͷ͘͠Έ ʙ࣮ݧͱਤղͰֶͿOSͱϋʔυ΢ΣΞͷجૅ஌ࣝ”


    9
    9
    io/fs ύοέʔδ

    View Slide

  10. • io/fs ύοέʔδ͕ఏڙ͍ͯ͠Δinterface΋ϑΝΠϧγεςϜͷ࣮૷Λ੾Γସ͑ΔͨΊ
    ͷ΋ͷ
    • ͦΕ͸ͦ͏
    • ΍ͬͺΓio.Reader΍io.Writerͱಉ͡ɻಛʹਂ͘ߟ͑Δ͜ͱ͸ͳ͔ͬͨ
    ීஈ࢖͍ͬͯΔϑΝΠϧγεςϜ΋ΠϯλʔϑΣʔεΛఏڙ͍ͯ͠Δ
    10
    io/fs ύοέʔδ

    View Slide

  11. io/fs.FS for testability
    11

    View Slide

  12. io/fs.FS for testability
    io.Reader / io.Writer for testability
    • ςετΛॻ͖΍͘͢͢ΔͨΊʹೖग़ྗ͕͋Δؔ਺ͷҾ਺Λ
    io.Reader / io.Writer interfaceʹ͢Δͱ͍͏࣮૷
    • ޷͖ͳ࢖͍ํ
    12

    View Slide

  13. io/fs.FS for testability
    io.Reader / io.Writer for testability
    13
    ࣮ࡍͷར༻
    ςετίʔυ
    ςετ࣌ʹҾ਺ͷ stdin, stdout ʹ౉͢ΠϯελϯεΛม͑Δ

    View Slide

  14. io/fs.FS for testability
    • minilsͱ͍͏ΧϨϯτσΟϨΫτϦͷϑΝΠϧҰཡΛग़ྗ͢Δ͚ͩͷ࣮૷
    • https://github.com/k1LoW/minils
    • io/fs ύοέʔδ͕ఏڙ͢ΔinterfaceͷReadDirFSΛҾ਺ʹड͚औΔ࣮૷ʹ͍ͯ͠Δ
    io/fs.FS io/fs.ReadDirFS for testability
    14

    View Slide

  15. io/fs.FS for testability
    • MapͰϑΝΠϧͷ৘ใΛཏྻ͢Δ͚ͩͰio/fs͕ఏڙ͢ΔinterfaceͷશͯΛຬͨ͢Πϯ
    ελϯεΛฦͯ͘͠ΕΔ
    • ࣮ࡍͷϩʔΧϧϑΝΠϧγεςϜʹςετ༻ϑΝΠϧΛஔ͘͜ͱͳ͘ςετΛॻ͘͜
    ͱ͕ՄೳʹͳΔ
    testing/fstest.MapFS
    15

    View Slide

  16. io/fs.FS for testability
    “io/fsύοέʔδʹΑΓOSͷϑΝΠϧγεςϜʹґଘ͠ͳ͍ॲཧΛॻ͚·͢ɻ”
    Go Conference 2021 Autumn (Online) ʹ͓͚Δ tenntennࢯͷൃදͷΞϒετϥΫτ


    16
    16
    io/fs.FS for testability

    View Slide

  17. io/fs.FS for abstraction
    17

    View Slide

  18. io/fs.FS for abstraction
    The proc file system is a pseudo-file system
    which is used as an interface to kernel data
    structures. It is commonly mounted at /proc.


    procϑΝΠϧγεςϜ͸ɺΧʔωϧͷσʔλߏ
    ଄΁ͷΠϯλʔϑΣΠεͱͯ͠࢖༻͞ΕΔٙࣅ
    ϑΝΠϧγεςϜͰ͢ɻ ௨ৗɺ/procʹϚ΢ϯ
    τ͞Ε·͢ɻ
    Linuxͷprocfs
    18
    man 5 proc ͷ DESCRIPTIONΑΓҾ༻

    View Slide

  19. ϑΝΠϧγεςϜͱ͍͏σʔλߏ଄΁ͷந৅Խ
    • ϑΝΠϧγεςϜ͸ʮϑΝΠϧͱσΟϨΫτϦʯͰɺʮϑΝΠ
    ϧͱσΟϨΫτϦʯ͸πϦʔߏ଄ʹͳ͍ͬͯΔɻ
    • ͭ·Γσʔλ͕ʢ΋͘͠͸ͦͷσʔλͷඞཁͱߟ͑ΒΕΔҰ෦
    ෼͕ʣ͢ͰʹπϦʔߏ଄ʹͳ͍ͬͯͨΓɺπϦʔߏ଄ʹ഑ஔ͢
    Δ͜ͱ͕ͰͰ͖Δ৔߹ɺٖࣅతͳϑΝΠϧγεςϜͱͯ͠ѻ͏
    ͜ͱ͕Մೳ
    19
    io/fs.FS for abstraction

    View Slide

  20. ϑΝΠϧγεςϜͱͯ͠ѻ͑ΔͱԿ͕ศརͳͷ͔
    • ϑΝΠϧγεςϜͱ͍͏ܗͰΠϯλʔϑΣʔε͕ἧ͏ͷͰɺ
    grep΍sortͳͲ༷ʑͳϑΝΠϧγεςϜΛѻ͏͜ͱʹ௕͚ͨ
    πʔϧ΍࢓૊Έ͕ͦͷ··࢖͑Δ
    • ԾʹϑΝΠϧγεςϜͱ͍͏ΠϯλʔϑΣʔεΛѻ͏ڧྗͳ
    πʔϧΛ࡞Ε͹ɺσʔλߏ଄ΛϑΝΠϧγεςϜͱͯ͠දݱͰ
    ͖Δ΋ͷશͯʹରͯ͠ޮՌΛൃشͤ͞Δ͜ͱ͕Ͱ͖Δ
    20
    io/fs.FS for abstraction

    View Slide

  21. io/fs.FS for abstraction
    21
    ʢLinuxͱಉ͡Α͏ʹʣ༷ʑͳσʔλΛ
    ϑΝΠϧγεςϜͱ͍͏σʔλߏ଄΁
    ந৅Խͨ͠͏͑Ͱɺio/fsͷinterfaceΛ
    ఏڙ͢Δ
    io/fs.FS for abstraction

    View Slide

  22. gh-grep
    • https://github.com/k1LoW/gh-grep


    • GitHub APIΛ࢖ͬͯGitHub্ͷϦϞʔτϦϙδτϦͷίʔυʹ
    ରͯ͠grep͢ΔπʔϧʢGitHub CLI ΤΫεςϯγϣϯʣ
    22
    io/fs.FS for abstraction

    View Slide

  23. io/fs.FS for abstraction
    • gh-grepͷ࣮૷͸2ͭͷύοέʔδͷ૊Έ߹Θͤ
    • github.com/bmatcuk/doublestar
    • globϥΠΫΑΓ΋͏গ͠Ϧονͳݕࡧ৚݅ͱio/fs.FS interfaceΛड͚औͬͯ৚
    ݅ʹϚονͨ͠ϑΝΠϧͷྻڍ͕Մೳ
    • github.com/k1LoW/ghfs
    • GitHubͷϦϙδτϦΛϑΝΠϧγεςϜͱͯ͠ந৅Խͯ͠ io/fs.FS interface
    Λఏڙ
    • ࣮૷࣌ɺio/fs.FS interfaceΛհ͢Δ͜ͱͰGitHub APIΛͲ͏ୟ͍ͯϑΝΠϧΛݕࡧ͢Δ͔
    ͱ͍͏͜ͱΛҰ੾ߟ͑Δඞཁ͕ͳ͔ͬͨ
    • ؔ৺ͷ෼཭͕Ͱ͖͍ͯͨ
    bmatcuk/doublestar ← io/fs.FS ← k1LoW/ghfs
    23

    View Slide

  24. io/fs.FS for abstraction
    24

    View Slide

  25. • https://github.com/k1LoW/octocov
    • ίʔυϝτϦΫεπʔϧ…ίʔυΧόϨοδɺCode to Test Ratioɺςετ࣮ߦ࣌ؒ
    • ଟ਺ͷϓϩάϥϛϯάݴޠͱɺओཁͳΧόϨοδϨϙʔτϑΥʔϚοτʹରԠ
    • GitHub Actionsͷactionͱͯ͠ίʔυϝτϦΫεΛूܭɺP/RίϝϯτɺϨϙʔτ
    • CLIπʔϧͱͯ͠ίʔυΧόϨοδΛूܭɻදࣔ
    • ΧόϨοδόοδੜ੒
    octocov
    25
    25
    io/fs.FS for abstraction

    View Slide

  26. io/fs.FS for abstraction
    • octocov͸ίʔυϝτϦΫεΛࢦఆͷσʔλετΞʹૹ৴͢Δ͜ͱ͕Ͱ͖Δ
    • ผ్octocovΛ “CentralϞʔυ” ͱͯ͠ಈ͔͢͜ͱͰσʔλετΞʹ͋Δ֤Ϧϙδτ
    Ϧ͔Βૹ৴͞ΕͨίʔυϝτϦΫεΛऔಘ͠ɺҰཡදࣔ΍Ұׅͨ͠όοδੜ੒ͳͲ
    ͕Մೳʹͳ͍ͬͯΔ
    ίʔυϝτϦΫεσʔλͷૹ৴ɾऔಘ(1/2)
    26

    View Slide

  27. io/fs.FS for abstraction
    • octocov͸༷ʑͳछྨͷσʔλετΞʹରԠ͍ͯ͠Δ
    • શͯͷσʔλετΞͷػೳΛಉ͡Α͏ʹ࣮૷͍ͨ͠
    • ڞ௨ͷio/fs.FS interfaceΛ࣋ͨͤΔ͜ͱͰશͯϑΝΠϧγεςϜͷΑ͏ʹ૸ࠪͰ͖Δ
    • ϩʔΧϧϑΝΠϧγεςϜ … github.com/k1LoW/osf
    s

    • GitHubϦϙδτϦ… github.com/k1LoW/ghf
    s

    • S3 … github.com/jszwec/s3f
    s

    • GCS … github.com/mauri870/gcsf
    s

    • BigQuery … octocov಺෦Ͱಠࣗʹ࣮૷
    ίʔυϝτϦΫεσʔλͷऔಘ(2/2)
    27

    View Slide

  28. io/fs.FS for abstraction
    • octocovͰఆ͍ٛͯ͠Δσʔληοτͷ
    ςʔϒϧʹରͯ͠ΫΤϦΛൃߦ
    • SELECTͨ݁͠ՌΛ fstest.MapFS ʹ٧
    ΊࠐΜͰฦ͢
    • ͜Ε͚ͩͰ io/fs.FS interfaceΛຬͨͤΔ
    BigQueryͷσʔληοτͷ io/fs.FS interface
    28

    View Slide

  29. io/fs.FS for abstraction
    Linuxʢͱ͍͏͔OSʣͷϑΝΠϧγεςϜͱio/fsύοέʔδͷࢥ૝͸͍ۙ?
    29
    • ڞ௨ͨ͠ΠϯλʔϑΣʔεΛఏڙ͢ΔͨΊͷ΋ͷͩͬͨΓɺϑΝΠϧγεςϜͱ͍͏
    σʔλߏ଄ʹந৅Խͯ͠ͳΜͰ΋ϑΝΠϧγεςϜͱͯ͠ѻ͏ͱ͍͏໘Ͱ΋ɺ͍ۙ΋
    ͷΛײ͡Δ
    • OSͷ։ൃऀͱͯ͠༗໊ͳ͋ͷํ͕͍Βͬ͠ΌͨΓ͢Δ͔Β͔ͳ…ʁ
    • ݸਓతʹ͸ʮඪ४ύοέʔδͰఏڙ͢Δ΋ͷͳͷ͔ͳʁʯͱࢥ͍͕ͬͯͨ࢖ͬͯΈ
    ͨΒศརͩͬͨ
    • LinuxϑΝΠϧγεςϜͰ࣮ݱ͍ͯ͠Δ͞·͟·ͳϞϊ͕ࢀߟʹͳΔͷ͸͔֬

    View Slide

  30. Package fs defines “basic” interfaces to a file system.
    • ͱ͸͍͑ɺجຊతͳinterface͚͕ͩఏڙ͞Ε͍ͯΔͱ͍͏͜ͱ͸
    ๨Εͯ͸͍͚ͳ͍
    • v1.17࣌఺Ͱ͸Read OnlyͳinterfaceͷΈ͕ఏڙ͞Ε͍ͯΔ
    • WritableͳϑΝΠϧγεςϜ interface͕ඞཁͳ৔߹͸αʔυ
    ύʔςΟύοέʔδͱͯ͠ https://github.com/spf13/afero ͕
    ༗໊
    • ྫ͑͹ʮγϯϘϦοΫϦϯΫʯͱ͍͏Α͏ͳ֓೦͸ͳ͍
    • ಛघͳܗͰ೚ҙͷσΟϨΫτϦΛtarͳͲʹΞʔΧΠϒΛ͢Δ
    ࣾ಺πʔϧΛ࡞੒ͨ͠ࡍʹɺԿ΋ߟ͑ͣ io/fs.FS interface Λ
    ࠾༻ͯ͠όάΛൃੜͤͯ͞͠·ͬͨ
    30
    io/fs.FS for abstraction

    View Slide

  31. ·ͱΊ
    31

    View Slide

  32. ·ͱΊ
    • io/fs ύοέʔδʹ͍ͭͯɺൃදऀͷ࣮૷ܦݧΛ΋ͱʹ2ͭͷ໨త͔Βͷར༻ྫΛ঺հ
    ͨ͠
    1. OSͷϑΝΠϧγεςϜʹґଘͤͣʹػೳΛ࣮૷͢Δ͜ͱͰςελϏϦςΟΛ্͛
    Δ໨త
    2. ϑΝΠϧγεςϜͱ͍͏σʔλߏ଄ʹந৅Խ͢Δ͜ͱͰɺ༷ʑͳϦιʔε૸ࠪʹ
    ڞ௨ͷΠϯλʔϑΣʔεΛ࣋ͨͤΔ໨త
    • ͲͪΒ΋LinuxͷϑΝΠϧγεςϜ΍ͦͷपลͷࢥ૝ͱ͍ۙͱ͍͏ײ૝
    • ·ͩ·ͩൃݟ͕͋Γͦ͏ͳύοέʔδ
    ·ͱΊ
    32

    View Slide

  33. Thank You!
    Thank You!
    33

    View Slide