Creating a userspace filesystem in node.js

2ad55a66b8325eb097612c069ab76231?s=47 Clay Smith
December 08, 2015

Creating a userspace filesystem in node.js

How to build a filesystem using the open-source FUSE library in node.js.

2ad55a66b8325eb097612c069ab76231?s=128

Clay Smith

December 08, 2015
Tweet

Transcript

  1. FrontendFS Creating a userspace filesystem in node.js Clay Smith, New

    Relic
  2. A file system (or filesystem) is a way of storing

    all data on a data storage device. BUILDING A FILESYSTEM IN NODE.JS Simple English Wikipedia
  3. BUILDING A FILESYSTEM IN NODE.JS Kernel Virtual  File  System EXT3

    Hardware NFS userspace kernel a  process https://en.wikipedia.org/wiki/Protection_ring   (thanks  @maykaynwd!)
  4. BUILDING A FILESYSTEM IN NODE.JS Kernel Virtual  File  System EXT3

    Hardware NFS FUSE userspace kernel a  process fuse  client
  5. BUILDING A FILESYSTEM IN NODE.JS Kernel Virtual  File  System EXT3

    Hardware NFS a  process FUSE fuse  client userspace kernel libfuse
  6. BUILDING A FILESYSTEM IN NODE.JS fuse  client.js good news: node

    is great at I/O
  7. BUILDING A FILESYSTEM IN NODE.JS How to write a (virtual)

    filesystem in node.js CC  A  SA  https://www.flickr.com/photos/ewedistrict/25323526
  8. BUILDING A FILESYSTEM IN NODE.JS • Use the fuse-bindings github

    project • https://github.com/mafintosh/fuse-bindings • Implement required filesystem operations • Run your node.js code and mount on an existing directory. • Success?!?!
  9. BUILDING A FILESYSTEM IN NODE.JS read(filePath, fd, buf, len, pos,

    cb) { } write(filePath, fd, buf, len, pos, cb) { } create(filePath, mode, cb) { } open(filePath, flags, cb) { } readdir(filePath, cb) { } Good  list  for  reference:  https://www.cs.hmc.edu/~geoff/classes/hmc.cs135.201001/homework/fuse/fuse_doc.html#function-­‐ purposes   getattr(filePath, cb) { }
  10. BUILDING A FILESYSTEM IN NODE.JS getattr(/) getattr(/cat.jpg) create(/cat.jpg, 33188) getattr(/cat.jpg)

    write(/cat.jpg, 0, len: 65536, buf.length: 65536, pos: 0 write(/cat.jpg, 0, len: 65536, buf.length: 65536, pos: 65536) // ... getattr(/cat.jpg) release(/cat.jpg, 0) Example: What does a file copy look like?
  11. BUILDING A FILESYSTEM IN NODE.JS In-memory virtual filesystem backed by

    JS array (yikes)
  12. BUILDING A FILESYSTEM IN NODE.JS var files = [ {

    name: 'readme.md', data: new Buffer('# test file\n'), } ]; A 'file' is just an item in this array: Buffer()-backed
  13. BUILDING A FILESYSTEM IN NODE.JS readdir() { // ... for

    (var i = 0; i < files.length; i++) { var buffer = files[i].data; var hash = hasha(buffer).substring(0, 12); files[i].name = `${hash}-${files[i].name}`; files[i].hasFingerprint = true; } } Let's rename the filename to be a hash!
  14. BUILDING A FILESYSTEM IN NODE.JS releaseFile(filePath, fd) { // Buffer

    is a WritableStream() var buffer = this.files[fileIndex].data.getContents(); zlib.gzip(buffer, function(error, result) { // create file if compression succeeded. }); } Like gzip?
  15. BUILDING A FILESYSTEM IN NODE.JS Performance profiling CC  A  SA

     www.flickr.com/photos/libertinus/9231731038  
  16. Benchmarking disk or file system IO performance can be tricky

    at best. BUILDING A FILESYSTEM IN NODE.JS James Coyle
  17. BUILDING A FILESYSTEM IN NODE.JS dd: "The Easy Route" •time

    sh -c "dd if=/dev/zero of=./in- memory/tst.tmp bs=4k count=1000 && sync" •409600 bytes written 1000 times... •Result: 3 mb/s (node, FuSE fs)
  18. BUILDING A FILESYSTEM IN NODE.JS tl;dr - use for fun

    dev tools, exporting APIs, system learnings
  19. BUILDING A FILESYSTEM IN NODE.JS see also... • https://github.com/Munter/fusile •

    ^ the "real" frontendFS • https://www.npmjs.com/package/mount-url • github.com/mafintosh/torrent-mount
  20. BUILDING A FILESYSTEM IN NODE.JS thanks! @smithclay require('afterparty') - tomorrow

    at New Relic PDX. https://bit.ly/nodeparty