io/fs.FS for testability. io/fs.FS for abstraction. / Go Conference 2021 Autumn (Online)

https://gocon.jp/2021autumn/sessions/io-fs-fs-testability-abstraction/

Go 1.16から追加されたio/fsパッケージが提供しているインターフェースについて、皆さんはどのような目的で活用していますか？

私は、主に2つの目的で活用しています。1つはテスト容易性（テスタビリティ）、もう1つはファイルシステムとしての抽象化です。

本発表では、具体的な事例を元に上記の2つの目的での使い方を紹介します。
また、fs.FSインターフェースを使ったことで失敗した事例も紹介したいと思います。

本発表を通じてio/fsのインターフェースの利用イメージを持ってもらえれば嬉しいです。

Transcript

1. io/fs.FS for testability
   .
   
   io/fs.FS for abstraction.
   খࢁ݈Ұ࿠ / GMO PEPABO inc.
   2021.11.13 Go Conference 2021 Autumn (Online)
   1
   

   View Slide

2. ࣗݾ঺հ
   ٕज़෦ɹٕज़ج൫νʔϜ
   2018೥ த్ೖࣾ
   খࢁ ݈Ұ࿠ Ken’ichiro Oyama
   গ࣮͠༻తͰখ͞ͳOSSΛॻ͘ͷ͕झຯɻ
   ● GitHub : k1Lo
   W
   
   ● Twitter : @k1LoW
   2
   

   View Slide

3. ΞδΣϯμ
   1. io/fs ύοέʔδ
   2. io/fs.FS for testabilit
   y
   
   3. io/fs.FS for abstractio
   n
   
   4. ·ͱΊ
   3
   

   View Slide

4. io/fs ύοέʔδ
   4
   

   View Slide

5. fs package
   Package fs defines basic interfaces to a file system.
   
   
   A file system can be provided by the host operating
   system but also by other packages.
   
   
   https://pkg.go.dev/io/fs
   
   
   fsύοέʔδ͸ɺϑΝΠϧγεςϜͷجຊతͳinterfaceΛ
   ఆ͍ٛͯ͠·͢ɻϑΝΠϧγεςϜ͸ɺϗετOS͔Βఏ
   ڙ͞ΕΔ͜ͱ΋͋Ε͹ɺଞͷύοέʔδ͔Βఏڙ͞ΕΔ
   ͜ͱ΋͋Γ·͢ɻ
   
   
   5
   

   View Slide

6. “ϑΝΠϧγεςϜͷinterfaceΛఏڙ͍ͯ͠Δ”
   • interfac
   e
   
   • io.Reader΍io.Writerͱಉ͡ɻGopherʹ͸͓ೃછΈ
   • ϑΝΠϧγεςϜ
   • ϑΝΠϧγεςϜ?
   6
   io/fs ύοέʔδ
   

   View Slide

7. LinuxͷϑΝΠϧγεςϜ
   “ϑΝΠϧΛΧςΰϦผʹ੔ཧͰ͖ΔΑ͏ʹ͢ΔͨΊ
   ʹɺLinuxʹ͓͚ΔϑΝΠϧγεςϜʹ͸ɺσΟϨΫ
   τϦͱ͍͏ɺϑΝΠϧΛ֨ೲ͢ΔͨΊͷಛघͳϑΝΠ
   ϧ͕ଘࡏ͠·͢ɻ
   
   
   σΟϨΫτϦͷதʹ͸ɺ௨ৗͷϑΝΠϧɺ͋Δ͍͸͞
   ΒʹσΟϨΫτϦΛอ࣋͢Δ͜ͱ͕Ͱ͖·͢ɻผͷ
   σΟϨΫτϦҎԼͳΒɺෳ਺ͷϑΝΠϧʹಉ໊͡લΛ
   ෇͚ΒΕ·͢ɻ͜ͷ͘͠ΈʹΑͬͯɺLinuxͷϑΝΠ
   ϧγεςϜ͸πϦʔߏ଄Λ͍ͯ͠·͢”
   ෢಺ ֮ “ʦࢼͯ͠ཧղʧLinuxͷ͘͠Έ ʙ࣮ݧͱਤղͰֶͿOSͱϋʔυ΢ΣΞͷجૅ஌ࣝ”
   
   
   io/fs ύοέʔδ
   7
   7
   

   View Slide

8. ϑΝΠϧγεςϜ
   
   
   = ϑΝΠϧͱσΟϨΫτϦ
   
   
   …ͱߟ͑Δ
   8
   io/fs ύοέʔδ
   

   View Slide

9. ͞Βʹ
   “Linux͕ѻ͑ΔϑΝΠϧγεςϜ͸1͚ͭͩͰ͸͋Γ
   ·ͤΜɻʮext4ʯʮXFSʯʮBtrfsʯͳͲɺෳ਺ͷϑΝ
   ΠϧγεςϜΛѻ͏͜ͱ͕Ͱ͖·͢ɻʢதུʣͨͩ
   ͠ɺͲͷϑΝΠϧγεςϜͰ͋ͬͯ΋ɺϢʔβ͔Β͸
   ࣍ͷΑ͏ͳγεςϜίʔϧͷൃߦͱ͍͏౷Ұͨ͠Πϯ
   λʔϑΣʔεʹΑͬͯΞΫηεͰ͖·͢ɻ”
   ෢಺ ֮ “ʦࢼͯ͠ཧղʧLinuxͷ͘͠Έ ʙ࣮ݧͱਤղͰֶͿOSͱϋʔυ΢ΣΞͷجૅ஌ࣝ”
   
   
   9
   9
   io/fs ύοέʔδ
   

   View Slide

10. • io/fs ύοέʔδ͕ఏڙ͍ͯ͠Δinterface΋ϑΝΠϧγεςϜͷ࣮૷Λ੾Γସ͑ΔͨΊ
    ͷ΋ͷ
    • ͦΕ͸ͦ͏
    • ΍ͬͺΓio.Reader΍io.Writerͱಉ͡ɻಛʹਂ͘ߟ͑Δ͜ͱ͸ͳ͔ͬͨ
    ීஈ࢖͍ͬͯΔϑΝΠϧγεςϜ΋ΠϯλʔϑΣʔεΛఏڙ͍ͯ͠Δ
    10
    io/fs ύοέʔδ
    

    View Slide

11. io/fs.FS for testability
    11
    

    View Slide

12. io/fs.FS for testability
    io.Reader / io.Writer for testability
    • ςετΛॻ͖΍͘͢͢ΔͨΊʹೖग़ྗ͕͋Δؔ਺ͷҾ਺Λ
    io.Reader / io.Writer interfaceʹ͢Δͱ͍͏࣮૷
    • ޷͖ͳ࢖͍ํ
    12
    

    View Slide

13. io/fs.FS for testability
    io.Reader / io.Writer for testability
    13
    ࣮ࡍͷར༻
    ςετίʔυ
    ςετ࣌ʹҾ਺ͷ stdin, stdout ʹ౉͢ΠϯελϯεΛม͑Δ
    

    View Slide

14. io/fs.FS for testability
    • minilsͱ͍͏ΧϨϯτσΟϨΫτϦͷϑΝΠϧҰཡΛग़ྗ͢Δ͚ͩͷ࣮૷
    • https://github.com/k1LoW/minils
    • io/fs ύοέʔδ͕ఏڙ͢ΔinterfaceͷReadDirFSΛҾ਺ʹड͚औΔ࣮૷ʹ͍ͯ͠Δ
    io/fs.FS io/fs.ReadDirFS for testability
    14
    

    View Slide

15. io/fs.FS for testability
    • MapͰϑΝΠϧͷ৘ใΛཏྻ͢Δ͚ͩͰio/fs͕ఏڙ͢ΔinterfaceͷશͯΛຬͨ͢Πϯ
    ελϯεΛฦͯ͘͠ΕΔ
    • ࣮ࡍͷϩʔΧϧϑΝΠϧγεςϜʹςετ༻ϑΝΠϧΛஔ͘͜ͱͳ͘ςετΛॻ͘͜
    ͱ͕ՄೳʹͳΔ
    testing/fstest.MapFS
    15
    

    View Slide

16. io/fs.FS for testability
    “io/fsύοέʔδʹΑΓOSͷϑΝΠϧγεςϜʹґଘ͠ͳ͍ॲཧΛॻ͚·͢ɻ”
    Go Conference 2021 Autumn (Online) ʹ͓͚Δ tenntennࢯͷൃදͷΞϒετϥΫτ
    
    
    16
    16
    io/fs.FS for testability
    

    View Slide

17. io/fs.FS for abstraction
    17
    

    View Slide

18. io/fs.FS for abstraction
    The proc file system is a pseudo-file system
    which is used as an interface to kernel data
    structures. It is commonly mounted at /proc.
    
    
    procϑΝΠϧγεςϜ͸ɺΧʔωϧͷσʔλߏ
    ଄΁ͷΠϯλʔϑΣΠεͱͯ͠࢖༻͞ΕΔٙࣅ
    ϑΝΠϧγεςϜͰ͢ɻ ௨ৗɺ/procʹϚ΢ϯ
    τ͞Ε·͢ɻ
    Linuxͷprocfs
    18
    man 5 proc ͷ DESCRIPTIONΑΓҾ༻
    

    View Slide

19. ϑΝΠϧγεςϜͱ͍͏σʔλߏ଄΁ͷந৅Խ
    • ϑΝΠϧγεςϜ͸ʮϑΝΠϧͱσΟϨΫτϦʯͰɺʮϑΝΠ
    ϧͱσΟϨΫτϦʯ͸πϦʔߏ଄ʹͳ͍ͬͯΔɻ
    • ͭ·Γσʔλ͕ʢ΋͘͠͸ͦͷσʔλͷඞཁͱߟ͑ΒΕΔҰ෦
    ෼͕ʣ͢ͰʹπϦʔߏ଄ʹͳ͍ͬͯͨΓɺπϦʔߏ଄ʹ഑ஔ͢
    Δ͜ͱ͕ͰͰ͖Δ৔߹ɺٖࣅతͳϑΝΠϧγεςϜͱͯ͠ѻ͏
    ͜ͱ͕Մೳ
    19
    io/fs.FS for abstraction
    

    View Slide

20. ϑΝΠϧγεςϜͱͯ͠ѻ͑ΔͱԿ͕ศརͳͷ͔
    • ϑΝΠϧγεςϜͱ͍͏ܗͰΠϯλʔϑΣʔε͕ἧ͏ͷͰɺ
    grep΍sortͳͲ༷ʑͳϑΝΠϧγεςϜΛѻ͏͜ͱʹ௕͚ͨ
    πʔϧ΍࢓૊Έ͕ͦͷ··࢖͑Δ
    • ԾʹϑΝΠϧγεςϜͱ͍͏ΠϯλʔϑΣʔεΛѻ͏ڧྗͳ
    πʔϧΛ࡞Ε͹ɺσʔλߏ଄ΛϑΝΠϧγεςϜͱͯ͠දݱͰ
    ͖Δ΋ͷશͯʹରͯ͠ޮՌΛൃشͤ͞Δ͜ͱ͕Ͱ͖Δ
    20
    io/fs.FS for abstraction
    

    View Slide

21. io/fs.FS for abstraction
    21
    ʢLinuxͱಉ͡Α͏ʹʣ༷ʑͳσʔλΛ
    ϑΝΠϧγεςϜͱ͍͏σʔλߏ଄΁
    ந৅Խͨ͠͏͑Ͱɺio/fsͷinterfaceΛ
    ఏڙ͢Δ
    io/fs.FS for abstraction
    

    View Slide

22. gh-grep
    • https://github.com/k1LoW/gh-grep
    
    
    • GitHub APIΛ࢖ͬͯGitHub্ͷϦϞʔτϦϙδτϦͷίʔυʹ
    ରͯ͠grep͢ΔπʔϧʢGitHub CLI ΤΫεςϯγϣϯʣ
    22
    io/fs.FS for abstraction
    

    View Slide

23. io/fs.FS for abstraction
    • gh-grepͷ࣮૷͸2ͭͷύοέʔδͷ૊Έ߹Θͤ
    • github.com/bmatcuk/doublestar
    • globϥΠΫΑΓ΋͏গ͠Ϧονͳݕࡧ৚݅ͱio/fs.FS interfaceΛड͚औͬͯ৚
    ݅ʹϚονͨ͠ϑΝΠϧͷྻڍ͕Մೳ
    • github.com/k1LoW/ghfs
    • GitHubͷϦϙδτϦΛϑΝΠϧγεςϜͱͯ͠ந৅Խͯ͠ io/fs.FS interface
    Λఏڙ
    • ࣮૷࣌ɺio/fs.FS interfaceΛհ͢Δ͜ͱͰGitHub APIΛͲ͏ୟ͍ͯϑΝΠϧΛݕࡧ͢Δ͔
    ͱ͍͏͜ͱΛҰ੾ߟ͑Δඞཁ͕ͳ͔ͬͨ
    • ؔ৺ͷ෼཭͕Ͱ͖͍ͯͨ
    bmatcuk/doublestar ← io/fs.FS ← k1LoW/ghfs
    23
    

    View Slide

24. io/fs.FS for abstraction
    24
    

    View Slide

25. • https://github.com/k1LoW/octocov
    • ίʔυϝτϦΫεπʔϧ…ίʔυΧόϨοδɺCode to Test Ratioɺςετ࣮ߦ࣌ؒ
    • ଟ਺ͷϓϩάϥϛϯάݴޠͱɺओཁͳΧόϨοδϨϙʔτϑΥʔϚοτʹରԠ
    • GitHub Actionsͷactionͱͯ͠ίʔυϝτϦΫεΛूܭɺP/RίϝϯτɺϨϙʔτ
    • CLIπʔϧͱͯ͠ίʔυΧόϨοδΛूܭɻදࣔ
    • ΧόϨοδόοδੜ੒
    octocov
    25
    25
    io/fs.FS for abstraction
    

    View Slide

26. io/fs.FS for abstraction
    • octocov͸ίʔυϝτϦΫεΛࢦఆͷσʔλετΞʹૹ৴͢Δ͜ͱ͕Ͱ͖Δ
    • ผ్octocovΛ “CentralϞʔυ” ͱͯ͠ಈ͔͢͜ͱͰσʔλετΞʹ͋Δ֤Ϧϙδτ
    Ϧ͔Βૹ৴͞ΕͨίʔυϝτϦΫεΛऔಘ͠ɺҰཡදࣔ΍Ұׅͨ͠όοδੜ੒ͳͲ
    ͕Մೳʹͳ͍ͬͯΔ
    ίʔυϝτϦΫεσʔλͷૹ৴ɾऔಘ(1/2)
    26
    

    View Slide

27. io/fs.FS for abstraction
    • octocov͸༷ʑͳछྨͷσʔλετΞʹରԠ͍ͯ͠Δ
    • શͯͷσʔλετΞͷػೳΛಉ͡Α͏ʹ࣮૷͍ͨ͠
    • ڞ௨ͷio/fs.FS interfaceΛ࣋ͨͤΔ͜ͱͰશͯϑΝΠϧγεςϜͷΑ͏ʹ૸ࠪͰ͖Δ
    • ϩʔΧϧϑΝΠϧγεςϜ … github.com/k1LoW/osf
    s
    
    • GitHubϦϙδτϦ… github.com/k1LoW/ghf
    s
    
    • S3 … github.com/jszwec/s3f
    s
    
    • GCS … github.com/mauri870/gcsf
    s
    
    • BigQuery … octocov಺෦Ͱಠࣗʹ࣮૷
    ίʔυϝτϦΫεσʔλͷऔಘ(2/2)
    27
    

    View Slide

28. io/fs.FS for abstraction
    • octocovͰఆ͍ٛͯ͠Δσʔληοτͷ
    ςʔϒϧʹରͯ͠ΫΤϦΛൃߦ
    • SELECTͨ݁͠ՌΛ fstest.MapFS ʹ٧
    ΊࠐΜͰฦ͢
    • ͜Ε͚ͩͰ io/fs.FS interfaceΛຬͨͤΔ
    BigQueryͷσʔληοτͷ io/fs.FS interface
    28
    

    View Slide

29. io/fs.FS for abstraction
    Linuxʢͱ͍͏͔OSʣͷϑΝΠϧγεςϜͱio/fsύοέʔδͷࢥ૝͸͍ۙ?
    29
    • ڞ௨ͨ͠ΠϯλʔϑΣʔεΛఏڙ͢ΔͨΊͷ΋ͷͩͬͨΓɺϑΝΠϧγεςϜͱ͍͏
    σʔλߏ଄ʹந৅Խͯ͠ͳΜͰ΋ϑΝΠϧγεςϜͱͯ͠ѻ͏ͱ͍͏໘Ͱ΋ɺ͍ۙ΋
    ͷΛײ͡Δ
    • OSͷ։ൃऀͱͯ͠༗໊ͳ͋ͷํ͕͍Βͬ͠ΌͨΓ͢Δ͔Β͔ͳ…ʁ
    • ݸਓతʹ͸ʮඪ४ύοέʔδͰఏڙ͢Δ΋ͷͳͷ͔ͳʁʯͱࢥ͍͕ͬͯͨ࢖ͬͯΈ
    ͨΒศརͩͬͨ
    • LinuxϑΝΠϧγεςϜͰ࣮ݱ͍ͯ͠Δ͞·͟·ͳϞϊ͕ࢀߟʹͳΔͷ͸͔֬
    

    View Slide

30. Package fs defines “basic” interfaces to a file system.
    • ͱ͸͍͑ɺجຊతͳinterface͚͕ͩఏڙ͞Ε͍ͯΔͱ͍͏͜ͱ͸
    ๨Εͯ͸͍͚ͳ͍
    • v1.17࣌఺Ͱ͸Read OnlyͳinterfaceͷΈ͕ఏڙ͞Ε͍ͯΔ
    • WritableͳϑΝΠϧγεςϜ interface͕ඞཁͳ৔߹͸αʔυ
    ύʔςΟύοέʔδͱͯ͠ https://github.com/spf13/afero ͕
    ༗໊
    • ྫ͑͹ʮγϯϘϦοΫϦϯΫʯͱ͍͏Α͏ͳ֓೦͸ͳ͍
    • ಛघͳܗͰ೚ҙͷσΟϨΫτϦΛtarͳͲʹΞʔΧΠϒΛ͢Δ
    ࣾ಺πʔϧΛ࡞੒ͨ͠ࡍʹɺԿ΋ߟ͑ͣ io/fs.FS interface Λ
    ࠾༻ͯ͠όάΛൃੜͤͯ͞͠·ͬͨ
    30
    io/fs.FS for abstraction
    

    View Slide

31. ·ͱΊ
    31
    

    View Slide

32. ·ͱΊ
    • io/fs ύοέʔδʹ͍ͭͯɺൃදऀͷ࣮૷ܦݧΛ΋ͱʹ2ͭͷ໨త͔Βͷར༻ྫΛ঺հ
    ͨ͠
    1. OSͷϑΝΠϧγεςϜʹґଘͤͣʹػೳΛ࣮૷͢Δ͜ͱͰςελϏϦςΟΛ্͛
    Δ໨త
    2. ϑΝΠϧγεςϜͱ͍͏σʔλߏ଄ʹந৅Խ͢Δ͜ͱͰɺ༷ʑͳϦιʔε૸ࠪʹ
    ڞ௨ͷΠϯλʔϑΣʔεΛ࣋ͨͤΔ໨త
    • ͲͪΒ΋LinuxͷϑΝΠϧγεςϜ΍ͦͷपลͷࢥ૝ͱ͍ۙͱ͍͏ײ૝
    • ·ͩ·ͩൃݟ͕͋Γͦ͏ͳύοέʔδ
    ·ͱΊ
    32
    

    View Slide

33. Thank You!
    Thank You!
    33
    

    View Slide