Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding Filesystem using go-fuse, from scratch

Understanding Filesystem using go-fuse, from scratch


Jumpei Takiyasu

November 17, 2017

More Decks by Jumpei Takiyasu

Other Decks in Technology


  1. Understanding Filesystem using go-fuse, from scratch Jumpei Takiyasu M3, Inc.

  2. = &Me{ Name: "Jumpei Takiyasu", Company: "M3, Inc.", // ex-Filesystem

    engineer Github: "juntaki", Twitter: "juntaki", Web: "https://juntaki.com", }
  3. Contents 1. Filesystem basics 2. How to make Filesystem 3.

    Mounting S3 using go-fuse
  4. Filesystem basics

  5. Filesystem OS abstraction for data I/O.

  6. Filesystem of Linux Implementation of the VFS(Virtual File System) interfaces.

    Source of data are various: • Memory: procfs, ramfs • Block device: XFS, ext4, btrfs • Network: NFS, CIFS
  7. VFS operation example read(2) System call: vfs_read() User Kernel file->f_op->read()

    struct file_operations { struct module *owner; loff_t (*llseek) (struct file *, loff_t, int); ssize_t (*read) (struct file *, char __user *, size_t, loff_t *); ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); …. Application glibc VFS xfs, ext4, etc. Block layer
  8. Data structure of VFS on-memory structure • inode (index node)

    • dentry (directory entry) http://www.coins.tsukuba.ac.jp/~yas/classes/os2-2012/2013-02-19/index.html
  9. Rough example of on-disk data structure Regular file On-disk inode

    Directory “Filename1” : <inode number> “Filename2” : <inode number> On-disk inode data[0] data[1] Root inode number Super block On-disk inode Directory [...]
  10. How to make Filesystem

  11. FUSE Easy to implement filesystem Slow performance due to context

    switch and memory copy User Kernel Application glibc VFS FUSE Daemon
  12. hanwen/go-fuse https://github.com/hanwen/go-fuse native bindings for the FUSE kernel module.

  13. pathfs.FileSystem interface (Omitted) type FileSystem interface { // These should

    update the file's ctime too. Chmod(name string, mode uint32, context *fuse.Context) (code fuse.Status) Chown(name string, uid uint32, gid uint32, context *fuse.Context) (code fuse.Status) Utimens(name string, Atime *time.Time, Mtime *time.Time, context *fuse.Context) (code fuse.Status) Truncate(name string, size uint64, context *fuse.Context) (code fuse.Status) Access(name string, mode uint32, context *fuse.Context) (code fuse.Status) // Tree structure Mkdir(name string, mode uint32, context *fuse.Context) fuse.Status Rename(oldName string, newName string, context *fuse.Context) (code fuse.Status) Rmdir(name string, context *fuse.Context) (code fuse.Status) Unlink(name string, context *fuse.Context) (code fuse.Status) // File handling. If opening for writing, the file's mtime // should be updated too. Open(name string, flags uint32, context *fuse.Context) (file nodefs.File, code fuse.Status) Create(name string, flags uint32, mode uint32, context *fuse.Context) (file nodefs.File, code fuse.Status) // Directory handling OpenDir(name string, context *fuse.Context) (stream []fuse.DirEntry, code fuse.Status) // Symlinks. Symlink(value string, linkName string, context *fuse.Context) (code fuse.Status) Readlink(name string, context *fuse.Context) (string, fuse.Status) }
  14. Mounting S3 using go-fuse

  15. juntaki/bucketsync ★ S3 backed FUSE Filesystem written in Go with

    dedup, compression and client-side encryption. https://github.com/juntaki/bucketsync TODO :p
  16. S3 backed Filesystem, most simple way S3 on AWS console

    looks like Filesystem! • Full path = Object key • Data = Object contents Problem: • Directory rename performance • Random read/write with compression/encryption
  17. Bucketsync data structure overview Directory Meta obj. Directory Meta obj.

    File Meta obj. DataChunk object root File Meta obj. Data chunk’s key is Hash of content. No upload needed for existing key Random generated key Meta object is cached in memory (LRU)
  18. Demo

  19. Conclusion Go is productive! Let’s try your idea with go-fuse!