Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Enclose.IO: current cutting-edges and the future work

8002c84eb4c18170632f8fb7efb09288?s=47 Minqi Pan
November 24, 2016

Enclose.IO: current cutting-edges and the future work

8002c84eb4c18170632f8fb7efb09288?s=128

Minqi Pan

November 24, 2016
Tweet

Transcript

  1. Enclose.IO: current cutting- edges and the future work Minqi Pan

  2. I’m Minqi Pan github.com/pmq20 twitter @psvr

  3. We made some modifications to … • Node.js Compile-time •

    Node.js Link-time / Test-time • Node.js Run-time
  4. Compile-time

  5. • Serialize project files into structs of {path, *source, source_len}

    • Serialize raw file contents into array of bytes • Bonus of Compiler hints: the compiler can tell us the offsets of those JS scripts • (TODO) better usage of resource handling of the compiler
  6. • TODO: maybe {path, *source, source_len, atime, mtime, ctime, birthtime}

    for fs.stat()? Anybody uses that info? • TODO: better record dir, for fs.readdir
  7. • skip ParseArgs • (cf. in Start(), ParseArgs is called

    before StartNodeInstance and CreateEnvironment)
  8. • checks for memfs in process.argv[1] • customized entrance via

    automatically setting process.argv[1] • e.g. process.argv[1]="/__enclose_io_memfs__/ node_modules/coffee-script/bin/coffee" • (cf. ExecuteString called on bootstrap_node.js when LoadEnvironment)
  9. • special treatment of __enclose_io_fork__ • special treatment of ENCLOSE_IO_USE_ORIGINAL_NODE

  10. • adds MEMFS::stat_file • adds MEMFS::stat_dir • adds MEMFS::resolve

  11. • adds MEMFS::exists? • adds MEMFS::read_file • adds MEMFS::read_dir

  12. • (TODO) simulate file descriptors • (TODO) Libmemfs: a portable,

    pure C implementation of a Unix-style, read-only, in- memory filesystem.
  13. Link-time / Test-time

  14. • Enforces static linking • copies routines of openssl, libuv,

    … into the executable image • (TODO) copy C extensions like fsevents used by the project into the executable image • problem of GLIBC and GLIBCXX
  15. passes original tests

  16. add Enclose-specific tests

  17. Runtime

  18. • the program is loaded into memory • your serialized

    project files are initially stored in .text, are then “copied” by the OS into the data segment (.data) of the program virtual address space during start-up • (cf.) text segment, the “REAL” executable instructions, read-only and has a fixed size • (TODO) read-only data segment (.rodata)
  19. – Wikipedia: Loader (computing) “In the case of operating systems

    that support virtual memory, the loader may not actually copy the contents of executable files into memory, but rather may simply declare to the virtual memory subsystem that there is a mapping between a region of memory allocated to contain the running program's code and the contents of the associated executable file.”
  20. – Wikipedia: Loader (computing) “The virtual memory subsystem is then

    made aware that pages with that region of memory need to be filled on demand if and when program execution actually hits those areas of unfilled memory. This may mean parts of a program's code are not actually copied into memory until they are actually used, and unused code may never be loaded into memory at all.”
  21. • String::NewFromUtf8 during start-up • need to copy memory when

    passed to V8 • from data segment to the heap, what a PAIN!
  22. • (TODO) to speed up startup, use zero-copy external string

    resources, V8 no longer requires that strings are aligned if they are one-byte strings (string data must be immutable and that the data must be Latin-1 and not UTF-8, which would require special treatment internally in the engine and do not allow efficient indexing)
 
 V8 3.20.9 enforces that external pointers are aligned on a two-byte boundary (cf. Tagged pointers, objects are always at least 4-byte aligned, and we can never have a pointer to the middle of an object in JavaScript) • (TODO) maybe zero-copy buffers?
  23. • (TODO) to speed up startup, delay String::NewExternalOneByte only when

    accessed, not running literally millions of String::NewFromUtf8/ String::NewExternalOneByte at start up
  24. • require first finds in memfs • require.resolve first resolves

    in memfs
  25. • fs.readFile, fs.readFileSync: first checks for memfs path • fs.readdir:

    first checks for memfs path
  26. • fs.fstat, fs.lstat, fs.lstatSync, fs.statSync: first checks for memfs path

    • fs.watch, fs.watchFile: does not watch for memfs • fs.realpathSync: first checks for memfs path
  27. • child_process.fork needs the original argv[1] semantic • will set

    __enclose_io_fork__ as argv[1] in child_process.fork
  28. • child_process.spawn(…process.execPath…) needs the original node semantic • will set

    ENCLOSE_IO_USE_ORIGINAL_NODE
  29. Future Work • see previous TODO’s • only include JS

    scripts that can be referenced • only include symbols that can be referenced • generate unoptimized machine code in compile- time via v8, no sources required in runtime
  30. Thank you! https://github.com/enclose-io