Creating a userspace filesystem in node.js

How to build a filesystem using the open-source FUSE library in node.js.

Transcript

1. FrontendFS
   Creating a userspace filesystem in node.js
   Clay Smith, New Relic
   

   View Slide

2. A file system (or filesystem) is a way
   of storing all data on a data storage
   device.
   BUILDING A FILESYSTEM IN NODE.JS
   Simple English Wikipedia
   

   View Slide

3. BUILDING A FILESYSTEM IN NODE.JS
   Kernel
   Virtual  File  System
   EXT3
   Hardware
   NFS
   userspace
   kernel
   a  process
   https://en.wikipedia.org/wiki/Protection_ring  
   (thanks  @maykaynwd!)
   

   View Slide

4. BUILDING A FILESYSTEM IN NODE.JS
   Kernel
   Virtual  File  System
   EXT3
   Hardware
   NFS FUSE
   userspace
   kernel
   a  process fuse  client
   

   View Slide

5. BUILDING A FILESYSTEM IN NODE.JS
   Kernel
   Virtual  File  System
   EXT3
   Hardware
   NFS
   a  process
   FUSE
   fuse  client
   userspace
   kernel
   libfuse
   

   View Slide

6. BUILDING A FILESYSTEM IN NODE.JS
   fuse  client.js
   good news: node is great at I/O
   

   View Slide

7. BUILDING A FILESYSTEM IN NODE.JS
   How to write a
   (virtual) filesystem
   in node.js
   CC  A  SA  https://www.flickr.com/photos/ewedistrict/25323526
   

   View Slide

8. BUILDING A FILESYSTEM IN NODE.JS
   • Use the fuse-bindings github project
   • https://github.com/mafintosh/fuse-bindings
   • Implement required filesystem operations
   • Run your node.js code and mount on an
   existing directory.
   • Success?!?!
   

   View Slide

9. BUILDING A FILESYSTEM IN NODE.JS
   read(filePath, fd, buf, len, pos, cb) { }
   write(filePath, fd, buf, len, pos, cb) { }
   create(filePath, mode, cb) { }
   open(filePath, flags, cb) { }
   readdir(filePath, cb) { }
   Good  list  for  reference:  https://www.cs.hmc.edu/~geoff/classes/hmc.cs135.201001/homework/fuse/fuse_doc.html#function-­‐
   purposes  
   getattr(filePath, cb) { }
   

   View Slide

10. BUILDING A FILESYSTEM IN NODE.JS
    getattr(/)
    getattr(/cat.jpg)
    create(/cat.jpg, 33188)
    getattr(/cat.jpg)
    write(/cat.jpg, 0, len: 65536, buf.length: 65536, pos: 0
    write(/cat.jpg, 0, len: 65536, buf.length: 65536, pos:
    65536)
    // ...
    getattr(/cat.jpg)
    release(/cat.jpg, 0)
    Example: What does a file copy look like?
    

    View Slide

11. BUILDING A FILESYSTEM IN NODE.JS
    In-memory virtual filesystem backed
    by JS array (yikes)
    

    View Slide

12. BUILDING A FILESYSTEM IN NODE.JS
    var files = [
    {
    name: 'readme.md',
    data: new Buffer('# test file\n'),
    }
    ];
    A 'file' is just an item in this array: Buffer()-backed
    

    View Slide

13. BUILDING A FILESYSTEM IN NODE.JS
    readdir() {
    // ...
    for (var i = 0; i < files.length; i++) {
    var buffer = files[i].data;
    var hash = hasha(buffer).substring(0, 12);
    files[i].name = `${hash}-${files[i].name}`;
    files[i].hasFingerprint = true;
    }
    }
    Let's rename the filename to be a hash!
    

    View Slide

14. BUILDING A FILESYSTEM IN NODE.JS
    releaseFile(filePath, fd) {
    // Buffer is a WritableStream()
    var buffer = this.files[fileIndex].data.getContents();
    zlib.gzip(buffer, function(error, result) {
    // create file if compression succeeded.
    });
    }
    Like gzip?
    

    View Slide

15. BUILDING A FILESYSTEM IN NODE.JS
    Performance profiling
    CC  A  SA  www.flickr.com/photos/libertinus/9231731038  
    

    View Slide

16. Benchmarking disk or file system
    IO performance can be tricky at
    best.
    BUILDING A FILESYSTEM IN NODE.JS
    James Coyle
    

    View Slide

17. BUILDING A FILESYSTEM IN NODE.JS
    dd: "The Easy Route"
    •time sh -c "dd if=/dev/zero of=./in-
    memory/tst.tmp bs=4k count=1000 && sync"
    •409600 bytes written 1000 times...
    •Result: 3 mb/s (node, FuSE fs)
    

    View Slide

18. BUILDING A FILESYSTEM IN NODE.JS
    tl;dr - use for fun dev tools,
    exporting APIs, system learnings
    

    View Slide

19. BUILDING A FILESYSTEM IN NODE.JS
    see also...
    • https://github.com/Munter/fusile
    • ^ the "real" frontendFS
    • https://www.npmjs.com/package/mount-url
    • github.com/mafintosh/torrent-mount
    

    View Slide

20. BUILDING A FILESYSTEM IN NODE.JS
    thanks!
    @smithclay
    require('afterparty') - tomorrow at New Relic PDX.
    https://bit.ly/nodeparty
    

    View Slide