Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding Filesystem using go-fuse, from sc...

Understanding Filesystem using go-fuse, from scratch

Jumpei Takiyasu

November 17, 2017
Tweet

More Decks by Jumpei Takiyasu

Other Decks in Technology

Transcript

  1. = &Me{ Name: "Jumpei Takiyasu", Company: "M3, Inc.", // ex-Filesystem

    engineer Github: "juntaki", Twitter: "juntaki", Web: "https://juntaki.com", }
  2. Filesystem of Linux Implementation of the VFS(Virtual File System) interfaces.

    Source of data are various: • Memory: procfs, ramfs • Block device: XFS, ext4, btrfs • Network: NFS, CIFS
  3. VFS operation example read(2) System call: vfs_read() User Kernel file->f_op->read()

    struct file_operations { struct module *owner; loff_t (*llseek) (struct file *, loff_t, int); ssize_t (*read) (struct file *, char __user *, size_t, loff_t *); ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); …. Application glibc VFS xfs, ext4, etc. Block layer
  4. Data structure of VFS on-memory structure • inode (index node)

    • dentry (directory entry) http://www.coins.tsukuba.ac.jp/~yas/classes/os2-2012/2013-02-19/index.html
  5. Rough example of on-disk data structure Regular file On-disk inode

    Directory “Filename1” : <inode number> “Filename2” : <inode number> On-disk inode data[0] data[1] Root inode number Super block On-disk inode Directory [...]
  6. FUSE Easy to implement filesystem Slow performance due to context

    switch and memory copy User Kernel Application glibc VFS FUSE Daemon
  7. pathfs.FileSystem interface (Omitted) type FileSystem interface { // These should

    update the file's ctime too. Chmod(name string, mode uint32, context *fuse.Context) (code fuse.Status) Chown(name string, uid uint32, gid uint32, context *fuse.Context) (code fuse.Status) Utimens(name string, Atime *time.Time, Mtime *time.Time, context *fuse.Context) (code fuse.Status) Truncate(name string, size uint64, context *fuse.Context) (code fuse.Status) Access(name string, mode uint32, context *fuse.Context) (code fuse.Status) // Tree structure Mkdir(name string, mode uint32, context *fuse.Context) fuse.Status Rename(oldName string, newName string, context *fuse.Context) (code fuse.Status) Rmdir(name string, context *fuse.Context) (code fuse.Status) Unlink(name string, context *fuse.Context) (code fuse.Status) // File handling. If opening for writing, the file's mtime // should be updated too. Open(name string, flags uint32, context *fuse.Context) (file nodefs.File, code fuse.Status) Create(name string, flags uint32, mode uint32, context *fuse.Context) (file nodefs.File, code fuse.Status) // Directory handling OpenDir(name string, context *fuse.Context) (stream []fuse.DirEntry, code fuse.Status) // Symlinks. Symlink(value string, linkName string, context *fuse.Context) (code fuse.Status) Readlink(name string, context *fuse.Context) (string, fuse.Status) }
  8. juntaki/bucketsync ★ S3 backed FUSE Filesystem written in Go with

    dedup, compression and client-side encryption. https://github.com/juntaki/bucketsync TODO :p
  9. S3 backed Filesystem, most simple way S3 on AWS console

    looks like Filesystem! • Full path = Object key • Data = Object contents Problem: • Directory rename performance • Random read/write with compression/encryption
  10. Bucketsync data structure overview Directory Meta obj. Directory Meta obj.

    File Meta obj. DataChunk object root File Meta obj. Data chunk’s key is Hash of content. No upload needed for existing key Random generated key Meta object is cached in memory (LRU)