io/fs.FS for testability. io/fs.FS for abstraction. / Go Conference 2021 Autumn (Online)

https://gocon.jp/2021autumn/sessions/io-fs-fs-testability-abstraction/

Go 1.16から追加されたio/fsパッケージが提供しているインターフェースについて、皆さんはどのような目的で活用していますか？

私は、主に2つの目的で活用しています。1つはテスト容易性（テスタビリティ）、もう1つはファイルシステムとしての抽象化です。

本発表では、具体的な事例を元に上記の2つの目的での使い方を紹介します。
また、fs.FSインターフェースを使ったことで失敗した事例も紹介したいと思います。

本発表を通じてio/fsのインターフェースの利用イメージを持ってもらえれば嬉しいです。

Transcript

1. io/fs.FS for testability . io/fs.FS for abstraction. খࢁ݈Ұ࿠ / GMO PEPABO inc. 2021.11.13 Go Conference 2021 Autumn (Online) 1
2. ࣗݾ঺հ ٕज़෦ɹٕज़ج൫νʔϜ 2018೥ த్ೖࣾ খࢁ ݈Ұ࿠ Ken’ichiro Oyama গ࣮͠༻తͰখ͞ͳOSSΛॻ͘ͷ͕झຯɻ • GitHub : k1Lo W • Twitter : @k1LoW 2
3. ΞδΣϯμ 1. io/fs ύοέʔδ 2. io/fs.FS for testabilit y 3. io/fs.FS for abstractio n 4. ·ͱΊ 3
4. io/fs ύοέʔδ 4
5. fs package Package fs defines basic interfaces to a file system. A file system can be provided by the host operating system but also by other packages. https://pkg.go.dev/io/fs fsύοέʔδ͸ɺϑΝΠϧγεςϜͷجຊతͳinterfaceΛ ఆ͍ٛͯ͠·͢ɻϑΝΠϧγεςϜ͸ɺϗετOS͔Βఏ ڙ͞ΕΔ͜ͱ΋͋Ε͹ɺଞͷύοέʔδ͔Βఏڙ͞ΕΔ ͜ͱ΋͋Γ·͢ɻ 5
6. “ϑΝΠϧγεςϜͷinterfaceΛఏڙ͍ͯ͠Δ” • interfac e • io.Reader΍io.Writerͱಉ͡ɻGopherʹ͸͓ೃછΈ • ϑΝΠϧγεςϜ • ϑΝΠϧγεςϜ? 6 io/fs ύοέʔδ
7. LinuxͷϑΝΠϧγεςϜ “ϑΝΠϧΛΧςΰϦผʹ੔ཧͰ͖ΔΑ͏ʹ͢ΔͨΊ ʹɺLinuxʹ͓͚ΔϑΝΠϧγεςϜʹ͸ɺσΟϨΫ τϦͱ͍͏ɺϑΝΠϧΛ֨ೲ͢ΔͨΊͷಛघͳϑΝΠ ϧ͕ଘࡏ͠·͢ɻ σΟϨΫτϦͷதʹ͸ɺ௨ৗͷϑΝΠϧɺ͋Δ͍͸͞ ΒʹσΟϨΫτϦΛอ࣋͢Δ͜ͱ͕Ͱ͖·͢ɻผͷ σΟϨΫτϦҎԼͳΒɺෳ਺ͷϑΝΠϧʹಉ໊͡લΛ ෇͚ΒΕ·͢ɻ͜ͷ͘͠ΈʹΑͬͯɺLinuxͷϑΝΠ ϧγεςϜ͸πϦʔߏ଄Λ͍ͯ͠·͢” ෢಺ ֮ “ʦࢼͯ͠ཧղʧLinuxͷ͘͠Έ ʙ࣮ݧͱਤղͰֶͿOSͱϋʔυ΢ΣΞͷجૅ஌ࣝ” io/fs ύοέʔδ 7 7
8. ϑΝΠϧγεςϜ = ϑΝΠϧͱσΟϨΫτϦ …ͱߟ͑Δ 8 io/fs ύοέʔδ
9. ͞Βʹ “Linux͕ѻ͑ΔϑΝΠϧγεςϜ͸1͚ͭͩͰ͸͋Γ ·ͤΜɻʮext4ʯʮXFSʯʮBtrfsʯͳͲɺෳ਺ͷϑΝ ΠϧγεςϜΛѻ͏͜ͱ͕Ͱ͖·͢ɻʢதུʣͨͩ ͠ɺͲͷϑΝΠϧγεςϜͰ͋ͬͯ΋ɺϢʔβ͔Β͸ ࣍ͷΑ͏ͳγεςϜίʔϧͷൃߦͱ͍͏౷Ұͨ͠Πϯ λʔϑΣʔεʹΑͬͯΞΫηεͰ͖·͢ɻ” ෢಺ ֮ “ʦࢼͯ͠ཧղʧLinuxͷ͘͠Έ ʙ࣮ݧͱਤղͰֶͿOSͱϋʔυ΢ΣΞͷجૅ஌ࣝ” 9 9 io/fs ύοέʔδ
10. • io/fs ύοέʔδ͕ఏڙ͍ͯ͠Δinterface΋ϑΝΠϧγεςϜͷ࣮૷Λ੾Γସ͑ΔͨΊ ͷ΋ͷ • ͦΕ͸ͦ͏ • ΍ͬͺΓio.Reader΍io.Writerͱಉ͡ɻಛʹਂ͘ߟ͑Δ͜ͱ͸ͳ͔ͬͨ ීஈ࢖͍ͬͯΔϑΝΠϧγεςϜ΋ΠϯλʔϑΣʔεΛఏڙ͍ͯ͠Δ 10 io/fs ύοέʔδ
11. io/fs.FS for testability 11
12. io/fs.FS for testability io.Reader / io.Writer for testability • ςετΛॻ͖΍͘͢͢ΔͨΊʹೖग़ྗ͕͋Δؔ਺ͷҾ਺Λ io.Reader / io.Writer interfaceʹ͢Δͱ͍͏࣮૷ • ޷͖ͳ࢖͍ํ 12
13. io/fs.FS for testability io.Reader / io.Writer for testability 13 ࣮ࡍͷར༻ ςετίʔυ ςετ࣌ʹҾ਺ͷ stdin, stdout ʹ౉͢ΠϯελϯεΛม͑Δ
14. io/fs.FS for testability • minilsͱ͍͏ΧϨϯτσΟϨΫτϦͷϑΝΠϧҰཡΛग़ྗ͢Δ͚ͩͷ࣮૷ • https://github.com/k1LoW/minils • io/fs ύοέʔδ͕ఏڙ͢ΔinterfaceͷReadDirFSΛҾ਺ʹड͚औΔ࣮૷ʹ͍ͯ͠Δ io/fs.FS io/fs.ReadDirFS for testability 14
15. io/fs.FS for testability • MapͰϑΝΠϧͷ৘ใΛཏྻ͢Δ͚ͩͰio/fs͕ఏڙ͢ΔinterfaceͷશͯΛຬͨ͢Πϯ ελϯεΛฦͯ͘͠ΕΔ • ࣮ࡍͷϩʔΧϧϑΝΠϧγεςϜʹςετ༻ϑΝΠϧΛஔ͘͜ͱͳ͘ςετΛॻ͘͜ ͱ͕ՄೳʹͳΔ testing/fstest.MapFS 15
16. io/fs.FS for testability “io/fsύοέʔδʹΑΓOSͷϑΝΠϧγεςϜʹґଘ͠ͳ͍ॲཧΛॻ͚·͢ɻ” Go Conference 2021 Autumn (Online) ʹ͓͚Δ tenntennࢯͷൃදͷΞϒετϥΫτ 16 16 io/fs.FS for testability
17. io/fs.FS for abstraction 17
18. io/fs.FS for abstraction The proc file system is a pseudo-file system which is used as an interface to kernel data structures. It is commonly mounted at /proc. procϑΝΠϧγεςϜ͸ɺΧʔωϧͷσʔλߏ ଄΁ͷΠϯλʔϑΣΠεͱͯ͠࢖༻͞ΕΔٙࣅ ϑΝΠϧγεςϜͰ͢ɻ ௨ৗɺ/procʹϚ΢ϯ τ͞Ε·͢ɻ Linuxͷprocfs 18 man 5 proc ͷ DESCRIPTIONΑΓҾ༻
19. ϑΝΠϧγεςϜͱ͍͏σʔλߏ଄΁ͷந৅Խ • ϑΝΠϧγεςϜ͸ʮϑΝΠϧͱσΟϨΫτϦʯͰɺʮϑΝΠ ϧͱσΟϨΫτϦʯ͸πϦʔߏ଄ʹͳ͍ͬͯΔɻ • ͭ·Γσʔλ͕ʢ΋͘͠͸ͦͷσʔλͷඞཁͱߟ͑ΒΕΔҰ෦ ෼͕ʣ͢ͰʹπϦʔߏ଄ʹͳ͍ͬͯͨΓɺπϦʔߏ଄ʹ഑ஔ͢ Δ͜ͱ͕ͰͰ͖Δ৔߹ɺٖࣅతͳϑΝΠϧγεςϜͱͯ͠ѻ͏ ͜ͱ͕Մೳ 19 io/fs.FS for abstraction
20. ϑΝΠϧγεςϜͱͯ͠ѻ͑ΔͱԿ͕ศརͳͷ͔ • ϑΝΠϧγεςϜͱ͍͏ܗͰΠϯλʔϑΣʔε͕ἧ͏ͷͰɺ grep΍sortͳͲ༷ʑͳϑΝΠϧγεςϜΛѻ͏͜ͱʹ௕͚ͨ πʔϧ΍࢓૊Έ͕ͦͷ··࢖͑Δ • ԾʹϑΝΠϧγεςϜͱ͍͏ΠϯλʔϑΣʔεΛѻ͏ڧྗͳ πʔϧΛ࡞Ε͹ɺσʔλߏ଄ΛϑΝΠϧγεςϜͱͯ͠දݱͰ ͖Δ΋ͷશͯʹରͯ͠ޮՌΛൃشͤ͞Δ͜ͱ͕Ͱ͖Δ 20 io/fs.FS for abstraction
21. io/fs.FS for abstraction 21 ʢLinuxͱಉ͡Α͏ʹʣ༷ʑͳσʔλΛ ϑΝΠϧγεςϜͱ͍͏σʔλߏ଄΁ ந৅Խͨ͠͏͑Ͱɺio/fsͷinterfaceΛ ఏڙ͢Δ io/fs.FS for abstraction
22. gh-grep • https://github.com/k1LoW/gh-grep • GitHub APIΛ࢖ͬͯGitHub্ͷϦϞʔτϦϙδτϦͷίʔυʹ ରͯ͠grep͢ΔπʔϧʢGitHub CLI ΤΫεςϯγϣϯʣ 22 io/fs.FS for abstraction
23. io/fs.FS for abstraction • gh-grepͷ࣮૷͸2ͭͷύοέʔδͷ૊Έ߹Θͤ • github.com/bmatcuk/doublestar • globϥΠΫΑΓ΋͏গ͠Ϧονͳݕࡧ৚݅ͱio/fs.FS interfaceΛड͚औͬͯ৚ ݅ʹϚονͨ͠ϑΝΠϧͷྻڍ͕Մೳ • github.com/k1LoW/ghfs • GitHubͷϦϙδτϦΛϑΝΠϧγεςϜͱͯ͠ந৅Խͯ͠ io/fs.FS interface Λఏڙ • ࣮૷࣌ɺio/fs.FS interfaceΛհ͢Δ͜ͱͰGitHub APIΛͲ͏ୟ͍ͯϑΝΠϧΛݕࡧ͢Δ͔ ͱ͍͏͜ͱΛҰ੾ߟ͑Δඞཁ͕ͳ͔ͬͨ • ؔ৺ͷ෼཭͕Ͱ͖͍ͯͨ bmatcuk/doublestar ← io/fs.FS ← k1LoW/ghfs 23
24. io/fs.FS for abstraction 24
25. • https://github.com/k1LoW/octocov • ίʔυϝτϦΫεπʔϧ…ίʔυΧόϨοδɺCode to Test Ratioɺςετ࣮ߦ࣌ؒ • ଟ਺ͷϓϩάϥϛϯάݴޠͱɺओཁͳΧόϨοδϨϙʔτϑΥʔϚοτʹରԠ • GitHub Actionsͷactionͱͯ͠ίʔυϝτϦΫεΛूܭɺP/RίϝϯτɺϨϙʔτ • CLIπʔϧͱͯ͠ίʔυΧόϨοδΛूܭɻදࣔ • ΧόϨοδόοδੜ੒ octocov 25 25 io/fs.FS for abstraction
26. io/fs.FS for abstraction • octocov͸ίʔυϝτϦΫεΛࢦఆͷσʔλετΞʹૹ৴͢Δ͜ͱ͕Ͱ͖Δ • ผ్octocovΛ “CentralϞʔυ” ͱͯ͠ಈ͔͢͜ͱͰσʔλετΞʹ͋Δ֤Ϧϙδτ Ϧ͔Βૹ৴͞ΕͨίʔυϝτϦΫεΛऔಘ͠ɺҰཡදࣔ΍Ұׅͨ͠όοδੜ੒ͳͲ ͕Մೳʹͳ͍ͬͯΔ ίʔυϝτϦΫεσʔλͷૹ৴ɾऔಘ(1/2) 26
27. io/fs.FS for abstraction • octocov͸༷ʑͳछྨͷσʔλετΞʹରԠ͍ͯ͠Δ • શͯͷσʔλετΞͷػೳΛಉ͡Α͏ʹ࣮૷͍ͨ͠ • ڞ௨ͷio/fs.FS interfaceΛ࣋ͨͤΔ͜ͱͰશͯϑΝΠϧγεςϜͷΑ͏ʹ૸ࠪͰ͖Δ • ϩʔΧϧϑΝΠϧγεςϜ … github.com/k1LoW/osf s • GitHubϦϙδτϦ… github.com/k1LoW/ghf s • S3 … github.com/jszwec/s3f s • GCS … github.com/mauri870/gcsf s • BigQuery … octocov಺෦Ͱಠࣗʹ࣮૷ ίʔυϝτϦΫεσʔλͷऔಘ(2/2) 27
28. io/fs.FS for abstraction • octocovͰఆ͍ٛͯ͠Δσʔληοτͷ ςʔϒϧʹରͯ͠ΫΤϦΛൃߦ • SELECTͨ݁͠ՌΛ fstest.MapFS ʹ٧ ΊࠐΜͰฦ͢ • ͜Ε͚ͩͰ io/fs.FS interfaceΛຬͨͤΔ BigQueryͷσʔληοτͷ io/fs.FS interface 28
29. io/fs.FS for abstraction Linuxʢͱ͍͏͔OSʣͷϑΝΠϧγεςϜͱio/fsύοέʔδͷࢥ૝͸͍ۙ? 29 • ڞ௨ͨ͠ΠϯλʔϑΣʔεΛఏڙ͢ΔͨΊͷ΋ͷͩͬͨΓɺϑΝΠϧγεςϜͱ͍͏ σʔλߏ଄ʹந৅Խͯ͠ͳΜͰ΋ϑΝΠϧγεςϜͱͯ͠ѻ͏ͱ͍͏໘Ͱ΋ɺ͍ۙ΋ ͷΛײ͡Δ • OSͷ։ൃऀͱͯ͠༗໊ͳ͋ͷํ͕͍Βͬ͠ΌͨΓ͢Δ͔Β͔ͳ…ʁ • ݸਓతʹ͸ʮඪ४ύοέʔδͰఏڙ͢Δ΋ͷͳͷ͔ͳʁʯͱࢥ͍͕ͬͯͨ࢖ͬͯΈ ͨΒศརͩͬͨ • LinuxϑΝΠϧγεςϜͰ࣮ݱ͍ͯ͠Δ͞·͟·ͳϞϊ͕ࢀߟʹͳΔͷ͸͔֬
30. Package fs defines “basic” interfaces to a file system. • ͱ͸͍͑ɺجຊతͳinterface͚͕ͩఏڙ͞Ε͍ͯΔͱ͍͏͜ͱ͸ ๨Εͯ͸͍͚ͳ͍ • v1.17࣌఺Ͱ͸Read OnlyͳinterfaceͷΈ͕ఏڙ͞Ε͍ͯΔ • WritableͳϑΝΠϧγεςϜ interface͕ඞཁͳ৔߹͸αʔυ ύʔςΟύοέʔδͱͯ͠ https://github.com/spf13/afero ͕ ༗໊ • ྫ͑͹ʮγϯϘϦοΫϦϯΫʯͱ͍͏Α͏ͳ֓೦͸ͳ͍ • ಛघͳܗͰ೚ҙͷσΟϨΫτϦΛtarͳͲʹΞʔΧΠϒΛ͢Δ ࣾ಺πʔϧΛ࡞੒ͨ͠ࡍʹɺԿ΋ߟ͑ͣ io/fs.FS interface Λ ࠾༻ͯ͠όάΛൃੜͤͯ͞͠·ͬͨ 30 io/fs.FS for abstraction
31. ·ͱΊ 31
32. ·ͱΊ • io/fs ύοέʔδʹ͍ͭͯɺൃදऀͷ࣮૷ܦݧΛ΋ͱʹ2ͭͷ໨త͔Βͷར༻ྫΛ঺հ ͨ͠ 1. OSͷϑΝΠϧγεςϜʹґଘͤͣʹػೳΛ࣮૷͢Δ͜ͱͰςελϏϦςΟΛ্͛ Δ໨త 2. ϑΝΠϧγεςϜͱ͍͏σʔλߏ଄ʹந৅Խ͢Δ͜ͱͰɺ༷ʑͳϦιʔε૸ࠪʹ ڞ௨ͷΠϯλʔϑΣʔεΛ࣋ͨͤΔ໨త • ͲͪΒ΋LinuxͷϑΝΠϧγεςϜ΍ͦͷपลͷࢥ૝ͱ͍ۙͱ͍͏ײ૝ • ·ͩ·ͩൃݟ͕͋Γͦ͏ͳύοέʔδ ·ͱΊ 32
33. Thank You! Thank You! 33