$30 off During Our Annual Pro Sale. View Details »

Creating a userspace filesystem in node.js

Clay Smith
December 08, 2015

Creating a userspace filesystem in node.js

How to build a filesystem using the open-source FUSE library in node.js.

Clay Smith

December 08, 2015
Tweet

More Decks by Clay Smith

Other Decks in Programming

Transcript

  1. FrontendFS
    Creating a userspace filesystem in node.js
    Clay Smith, New Relic

    View Slide

  2. A file system (or filesystem) is a way
    of storing all data on a data storage
    device.
    BUILDING A FILESYSTEM IN NODE.JS
    Simple English Wikipedia

    View Slide

  3. BUILDING A FILESYSTEM IN NODE.JS
    Kernel
    Virtual  File  System
    EXT3
    Hardware
    NFS
    userspace
    kernel
    a  process
    https://en.wikipedia.org/wiki/Protection_ring  
    (thanks  @maykaynwd!)

    View Slide

  4. BUILDING A FILESYSTEM IN NODE.JS
    Kernel
    Virtual  File  System
    EXT3
    Hardware
    NFS FUSE
    userspace
    kernel
    a  process fuse  client

    View Slide

  5. BUILDING A FILESYSTEM IN NODE.JS
    Kernel
    Virtual  File  System
    EXT3
    Hardware
    NFS
    a  process
    FUSE
    fuse  client
    userspace
    kernel
    libfuse

    View Slide

  6. BUILDING A FILESYSTEM IN NODE.JS
    fuse  client.js
    good news: node is great at I/O

    View Slide

  7. BUILDING A FILESYSTEM IN NODE.JS
    How to write a
    (virtual) filesystem
    in node.js
    CC  A  SA  https://www.flickr.com/photos/ewedistrict/25323526

    View Slide

  8. BUILDING A FILESYSTEM IN NODE.JS
    • Use the fuse-bindings github project
    • https://github.com/mafintosh/fuse-bindings
    • Implement required filesystem operations
    • Run your node.js code and mount on an
    existing directory.
    • Success?!?!

    View Slide

  9. BUILDING A FILESYSTEM IN NODE.JS
    read(filePath, fd, buf, len, pos, cb) { }
    write(filePath, fd, buf, len, pos, cb) { }
    create(filePath, mode, cb) { }
    open(filePath, flags, cb) { }
    readdir(filePath, cb) { }
    Good  list  for  reference:  https://www.cs.hmc.edu/~geoff/classes/hmc.cs135.201001/homework/fuse/fuse_doc.html#function-­‐
    purposes  
    getattr(filePath, cb) { }

    View Slide

  10. BUILDING A FILESYSTEM IN NODE.JS
    getattr(/)
    getattr(/cat.jpg)
    create(/cat.jpg, 33188)
    getattr(/cat.jpg)
    write(/cat.jpg, 0, len: 65536, buf.length: 65536, pos: 0
    write(/cat.jpg, 0, len: 65536, buf.length: 65536, pos:
    65536)
    // ...
    getattr(/cat.jpg)
    release(/cat.jpg, 0)
    Example: What does a file copy look like?

    View Slide

  11. BUILDING A FILESYSTEM IN NODE.JS
    In-memory virtual filesystem backed
    by JS array (yikes)

    View Slide

  12. BUILDING A FILESYSTEM IN NODE.JS
    var files = [
    {
    name: 'readme.md',
    data: new Buffer('# test file\n'),
    }
    ];
    A 'file' is just an item in this array: Buffer()-backed

    View Slide

  13. BUILDING A FILESYSTEM IN NODE.JS
    readdir() {
    // ...
    for (var i = 0; i < files.length; i++) {
    var buffer = files[i].data;
    var hash = hasha(buffer).substring(0, 12);
    files[i].name = `${hash}-${files[i].name}`;
    files[i].hasFingerprint = true;
    }
    }
    Let's rename the filename to be a hash!

    View Slide

  14. BUILDING A FILESYSTEM IN NODE.JS
    releaseFile(filePath, fd) {
    // Buffer is a WritableStream()
    var buffer = this.files[fileIndex].data.getContents();
    zlib.gzip(buffer, function(error, result) {
    // create file if compression succeeded.
    });
    }
    Like gzip?

    View Slide

  15. BUILDING A FILESYSTEM IN NODE.JS
    Performance profiling
    CC  A  SA  www.flickr.com/photos/libertinus/9231731038  

    View Slide

  16. Benchmarking disk or file system
    IO performance can be tricky at
    best.
    BUILDING A FILESYSTEM IN NODE.JS
    James Coyle

    View Slide

  17. BUILDING A FILESYSTEM IN NODE.JS
    dd: "The Easy Route"
    •time sh -c "dd if=/dev/zero of=./in-
    memory/tst.tmp bs=4k count=1000 && sync"
    •409600 bytes written 1000 times...
    •Result: 3 mb/s (node, FuSE fs)

    View Slide

  18. BUILDING A FILESYSTEM IN NODE.JS
    tl;dr - use for fun dev tools,
    exporting APIs, system learnings

    View Slide

  19. BUILDING A FILESYSTEM IN NODE.JS
    see also...
    • https://github.com/Munter/fusile
    • ^ the "real" frontendFS
    • https://www.npmjs.com/package/mount-url
    • github.com/mafintosh/torrent-mount

    View Slide

  20. BUILDING A FILESYSTEM IN NODE.JS
    thanks!
    @smithclay
    require('afterparty') - tomorrow at New Relic PDX.
    https://bit.ly/nodeparty

    View Slide