Share
Embed
Understanding Filesystem using go-fuse, from scratch
Jumpei Takiyasu
November 17, 2017
Transcript
19 slides
-
= &Me{ Name: "Jumpei Takiyasu", Company: "M3, Inc.", // ex-Filesystem
engineer Github: "juntaki", Twitter: "juntaki", Web: "https://juntaki.com", } -
Contents 1. Filesystem basics 2. How to make Filesystem 3.
Mounting S3 using go-fuse -
Filesystem of Linux Implementation of the VFS(Virtual File System) interfaces.
Source of data are various: • Memory: procfs, ramfs • Block device: XFS, ext4, btrfs • Network: NFS, CIFS -
VFS operation example read(2) System call: vfs_read() User Kernel file->f_op->read()
struct file_operations { struct module *owner; loff_t (*llseek) (struct file *, loff_t, int); ssize_t (*read) (struct file *, char __user *, size_t, loff_t *); ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *); …. Application glibc VFS xfs, ext4, etc. Block layer -
Data structure of VFS on-memory structure • inode (index node)
• dentry (directory entry) http://www.coins.tsukuba.ac.jp/~yas/classes/os2-2012/2013-02-19/index.html -
Rough example of on-disk data structure Regular file On-disk inode
Directory “Filename1” : <inode number> “Filename2” : <inode number> On-disk inode data[0] data[1] Root inode number Super block On-disk inode Directory [...] -
FUSE Easy to implement filesystem Slow performance due to context
switch and memory copy User Kernel Application glibc VFS FUSE Daemon -
pathfs.FileSystem interface (Omitted) type FileSystem interface { // These should
update the file's ctime too. Chmod(name string, mode uint32, context *fuse.Context) (code fuse.Status) Chown(name string, uid uint32, gid uint32, context *fuse.Context) (code fuse.Status) Utimens(name string, Atime *time.Time, Mtime *time.Time, context *fuse.Context) (code fuse.Status) Truncate(name string, size uint64, context *fuse.Context) (code fuse.Status) Access(name string, mode uint32, context *fuse.Context) (code fuse.Status) // Tree structure Mkdir(name string, mode uint32, context *fuse.Context) fuse.Status Rename(oldName string, newName string, context *fuse.Context) (code fuse.Status) Rmdir(name string, context *fuse.Context) (code fuse.Status) Unlink(name string, context *fuse.Context) (code fuse.Status) // File handling. If opening for writing, the file's mtime // should be updated too. Open(name string, flags uint32, context *fuse.Context) (file nodefs.File, code fuse.Status) Create(name string, flags uint32, mode uint32, context *fuse.Context) (file nodefs.File, code fuse.Status) // Directory handling OpenDir(name string, context *fuse.Context) (stream []fuse.DirEntry, code fuse.Status) // Symlinks. Symlink(value string, linkName string, context *fuse.Context) (code fuse.Status) Readlink(name string, context *fuse.Context) (string, fuse.Status) } -
juntaki/bucketsync ★ S3 backed FUSE Filesystem written in Go with
dedup, compression and client-side encryption. https://github.com/juntaki/bucketsync TODO :p -
S3 backed Filesystem, most simple way S3 on AWS console
looks like Filesystem! • Full path = Object key • Data = Object contents Problem: • Directory rename performance • Random read/write with compression/encryption -
Bucketsync data structure overview Directory Meta obj. Directory Meta obj.
File Meta obj. DataChunk object root File Meta obj. Data chunk’s key is Hash of content. No upload needed for existing key Random generated key Meta object is cached in memory (LRU)