The Dark Art Of Bundlers

Chain React 2017

Transcript

1. The dark art of bundlers
   Hello everyone,
   
   I am Mike and I am the Co-Founder and CTO at Callstack. I am also one of the core contributors to React Native.
   
   Today I’ll be demystifying Javascript bundlers, specifically in React Native context.
   

   View Slide

2. What is a bundler?
   Let’s start with the most fundamental question - do you know what a bundler is? Please raise your hand!
   
   > (Quite good) Nice! I can see a lot of hands! Please bear with me as we get through those slides
   
   > (Not good) Great! This might be the most educational presentation I have ever made then haha
   
   

   View Slide

3. a.js b.js
   d.js
   c.js
   e.js f.js
   bundle.js
   eg. Webpack
   So bundler is a tool that simply takes all your Javascript source files and puts them into one file. That process is called bundling. There are many of them these days, with
   Webpack, Browserify or Metro being most popular.
   
   

   View Slide

4. To bundle or not to bundle?
   You are probably now wondering why do we even need that - good question. To better understand that, let’s step back to dark days of jQuery and early web.
   
   

   View Slide

5. Early web
   Historically, Javascript hasn’t had a standard for requiring dependencies from your code. There was no `import` or `require` statements.
   
   So how did we make functions of our code visible to the outer world and how did we import functions from other people’s code?
   

   View Slide

6. Globals, yea!
   The only way was through global variables…
   

   View Slide

7. Globals, yea!
   For example, if you wanted to use jQuery… you would go ahead and add a script tag that loads it from external resource.
   
   As a result, your global scope would get populated with its bindings, in this case a dollar sign.
   

   View Slide

8. Where’s the problem?
   Okay, one could say that this is actually neat. All set with just a few lines of code, right?
   
   Imagine this scenario: Your prototype works out, your application grows and you end up looking into splitting your file into smaller, reusable pieces.
   
   

   View Slide

9. Multiple files
   You would end up with something like this, on slide.
   
   Now, the code that will get executed in the script tag should have access to all globals defined in the preceding files.
   
   
   
   The problem here is the order. What if foo.js wanted to use bar.js code. Manipulating the order files are loaded and executed is a design flaw. I call it script hell. From
   callback hell we know it’s bad!
   
   

   View Slide

10. How do we solve it?
    So how do we solve this?
    
    Turns out we already did this in other environments. Take Node.js for example.
    
    

    View Slide

11. Node.js
    It implements its own module system using `require` function and `exports` object amongst other things based on the Common.js module spec.
    

    View Slide

12. Web
    What if we could do the same in our frontend code? That would great, huh?
    
    There’s one technical limitation though.
    
    

    View Slide

13. `require` is sync
    `fetch` is async
    `require` itself is synchronous. In Node.js environment, when you want to load a file, it’s physically accessible on the disc, in most cases next to existing file.
    
    In our context, however, we need to make an HTTP request to load a file that has never been loaded by the browser. And that request is asynchronous.
    
    

    View Slide

14. To bundle or not to bundle?
    The solution is simple. We can bundle all the files into one, so that we have all required dependencies in memory at the time of `require` function being called. And that’s
    the primary reason for bundling explained in 5 minutes.
    

    View Slide

15. Wait… is that all a bundler can do?
    Bundlers can be perceived as a natural evolution of task runners, like Gulp or Grunt that happened over the course of past few years. There are lots of responsibilities in
    scope for them rather than just getting your code into one chunk for sake of organising dependencies.
    
    

    View Slide

16. Transpilation
    One of them is transpilation. A typical bundler can, or has support, for transpiling your code.
    

    View Slide

17. Transpilation
    That means for reading input source in particular format (e.g. newest Javascript spec) and outputting it in an older standard (e.g. ES5). The most popular out there is
    Babel and most bundlers out here provide a first-class integration with it by default.
    
    

    View Slide

18. Code splitting
    We mentioned already that in order for `require` to work, entire application has to be loaded into memory. That causes your bundle size to increase and ship with modules
    that don’t necessarily need to be loaded on the initial page run.
    

    View Slide

19. Code splitting
    Code splitting is a feature that allows you to split your code into various bundles which can then be loaded on demand or in parallel.
    
    In ES2015 compatible environment for example, you can use `import` function to obtain an instance of a module. The `Promise` is resolved as soon as the loading
    process is finished.
    

    View Slide

20. Code minification
    Bundlers also enable code minification. For example, Webpack 2 ships with a built-in support for tree shaking. That means it can determine what parts of your modules
    are not used (e.g. module foo is never required) and effectively remove those parts. Combined with regular minifiers, like Uglify, it contributes to smaller bundle size that
    gets delivered to users.
    
    

    View Slide

21. How does that all relate to
    React Native?
    So that’s a bit of theory.
    
    Now, how does that all relate to React Native?
    
    

    View Slide

22. Quick look at infrastructure
    With React Native, we express our native mobile applications by writing Javascript and layouts - by writing React components.
    
    However, since both iOS and Android are distinctive platforms with their own SDKs, languages and tools, there is no way to run the Javascript in the simple way.
    

    View Slide

23. Javascript VM
    What those all platforms have in common is the ability to spin a Javascript VM that can process and execute an arbitrary application bundle.
    
    React Native then sets up a bi-directional data flow by using a “bridge” - layer that can accept calls from Javascript and pass them through to a given native module
    

    View Slide

24. Same constraints
    The use of Javascript VM is what makes constraints we just talked in context of web, applicable here as well.
    
    Just to give you a great example: On iOS, React Native uses built-in Safari to execute your application logic. Its features and performance changes on per system basis.
    You will find a different set of ES6 supported on your testing Mac than on iOS8, 9 or 10. So you definitely want to transpile your code to lowest common subset
    supported.
    

    View Slide

25. Is React Native web alike?
    Sounds like React Native is pretty similar to web, right? We have access to some of the same polyfills, globals, APIs. The language and spec is also the same. We bundle
    code the same way. There are even modules on npm that work in both environments.
    
    However, there’s one thing that is fundamentally different.
    
    

    View Slide

26. react-native start
    It is the react-native start command we run every day.
    
    At first glance, all it does is it serves your entire application as a single bundle that can be accessed on your localhost.
    
    Pretty simple, right? Then, why don’t we spawn Webpack and configure it to serve the bundle at same URL?
    
    I like experiments, so I thought - why not try doing it.
    

    View Slide

27. Webpack config!
    I started with creating an empty webpack config where I could just point to an entry point and define some other settings for React Native to load the bundle.
    
    

    Then, I thought of all React Native specific features I knew I have to support for the first implementation to work.
    

    View Slide

28. One of them was of course dev server - React Native works by loading bundle from an arbitrary URL. In most cases, it’s the name of entry point with platform.bundle
    extension.
    

    View Slide

29. Then, I had to integrate Babel to transpile ES6 and other Javascript code to run on my iOS device.
    
    By default, React Native does that to your entire node_modules folder, including even modules that don’t need it. To make my experiment backwards compatible, I
    decided to conditionally whitelist all modules start with “react” keyword (like react-native itself or any react-native plugin).
    

    View Slide

30. Last but not least, to make my platform-specific code to run, I used `resolve.extensions` to define order files should be resolved.
    
    In this case, I wanted my implementation to load first, platform specific file, then native file and end up with loading just a js file.
    

    View Slide

31. Haste module system
    Finally, the most entertaining part was supporting Facebook proprietary `node-haste` system.

    

    It’s pretty simple - any file annotated with @providesModule, can be globally required within the project by its name. That way, you can have files spread across entire
    repository and require them w/o knowing their exact location.
    
    Of course, Webpack has no idea how to support this and what it will try is to look inside node_modules.
    

    View Slide

32. But it has a great `resolve.alias` feature. It is a map where keys are names of modules that can be required globally and values - absolute path to a file that should get
    loaded.
    
    My solution turned out to be quite dumb, but perfect in terms of performance. I decided to glob for `js` files in provided array of folders and just parse doc blocks of every
    file that had it.
    

    View Slide

33. Prototype
    After hacking with it for few nights, I finally got it running. In less than 7 seconds Webpack created and served a bundle on port and URL I wanted.
    
    

    Great, I thought - nothing but declarative Webpack config and I rule the world.
    
    My excitement went away as soon as I opened a native client. I have noticed lots of features were missing.
    
    Example: “Live reload” wasn’t working at all.
    
    

    View Slide

34. Debug in Chrome
    (screen) Attempting to “Debug in Chrome” started producing red screens. By that time, I didn’t really know what “Runtime is not ready” meant.
    

    View Slide

35. No more 

    nice errors
    (screen) More importantly, red screen messages started looking weird - the stack trace was unreadable and it was almost impossible to locate the original issue.
    
    

    View Slide

36. There must be more…
    I started to realise that in context of React Native, there has to be more than just a bundler that is running.
    
    Turns out, running `react-native start` spins up entire set of tools focused on providing a better developer experience. At its core, there’s a bundler.
    
    All the rest are just complementary features built on top of it.
    

    View Slide

37. Introducing: Packager
    I call it a packager - tool used to develop, debug and package an application. It builds on top of an arbitrary bundler, like Webpack and ships with pre-configured set of
    transforms required for the code to run in particular environment.
    

    View Slide

38. Digging in deeper!
    So I decided to keep my experiment going and see if I can make Webpack work with React Native. The plan was to support all its features, including symlinks, that never
    worked with default packager yet.
    
    I was in particular interested where’s the boundary, that is - how many features I’ll have to write on top in order to make it a drop-in replacement.
    

    View Slide

39. Looking for connection
    The first thing I started looking into was the Chrome Debugger. On standard React Native app, turning that mode opens a new tab in the browser. However, there wasn’t
    any API on iOS or Android that I was aware of to open an arbitrary app on a Mac.
    
    I quickly realised that it has to be packager that opens a Chrome tab, not React Native. Since it is a process running on a host machine, it has access to all required APIs.
    
    Now how do those two pieces talk to each other?
    
    

    View Slide

40. Packager is http server
    Turns out packager is an http server.
    
    

    View Slide

41. Http server
    It is composed out of standalone middlewares, each, having different responsibility. Together, they form a tool that provides the developer experience we know from
    React Native. It is a really cool and powerful concept - being able to plug & play certain parts as a middleware makes it really hackable and customisable.
    
    As you can see, the last middleware here in the list is the `metro-bundler`, the one that serves your bundle by default.
    

    View Slide

42. “RPC over HTTP”
    React Native Packager
    GET /launch-devtools
    status
    The communication is established by React Native, calling Packager endpoint, e.g. launch-devtools.
    
    Then, the handler is executing procedure on a host machine and reports back to React Native client.
    
    It is very similar to a remote procedure call, in distributed computing, with a difference that everything happens on your localhost!
    
    

    View Slide

43. Bundler middleware
    So in order to get my project rolling, instead of spawning webpack from command line and relying purely on webpack config, I decided to use express server and
    webpack-dev-middleware.

    

    That way, I could still get an instance of express server and get the ability to extend it with the features standard packager has by default.
    
    

    View Slide

44. Live reload middleware
    Let’s take a look at another middleware - live reload middleware. As the name indicates, it’s responsibility is to provide a “Live reload” function in our dev environment.
    

    View Slide

45. Long-lasting request
    Once activated, React Native sends a long-lasting GET request to the Packager.
    

    View Slide

46. On each such request, the middleware stores a response object associated with a particular request.

    

    As soon as change to files happen (in case of webpack, it can be done by listening to `done` event), all connections are closed with `changed: true` indicating that live
    reload is required on the client.
    

    View Slide

47. Dev tools middleware
    Now that we have a better understanding of how the communication between Packager and React Native is done, it’s finally the time to look at the dev tools middleware.
    
    

    I am quite excited about that one - the ability to use an arbitrary browser to run and debug our Javascript was pure magic to me the first time I saw it.

    

    View Slide

48. Launch Dev Tools
    As soon as you press Debug with Chrome, React Native executes a remote procedure to launch dev tools inside Chrome tab when necessary.
    
    Right after, both device and debugger connect with each other by using Websocket connection. Since there’s no way for them to connect directly, they do it through
    Packager, which proxies all messages.
    
    If for some reason the connection cannot be established, we end up with “Runtime not ready for debugging” message we already saw on past slides.
    
    

    View Slide

49. Once all’s well and good, WebSocket executor on the device sends the JSON message to the Chrome tab.
    
    It has to contain at least a `method` property - it tells debugger which handler to execute.
    
    The rest of the keys varies on handler-basis.
    

    View Slide

50. The `executeApplicationScript` is a so-called special handler. It is hardcoded inside the debugger.
    
    First, it attaches the `inject` JSON onto the global object. One of them is `__fbBatchedBridge`.
    
    Later, it uses WebWorker `importScripts` to synchronously load Javascript at given URL into the worker scope.
    

    View Slide

51. If given method is not a special case handler…
    

    View Slide

52. … its handler will be looked up at __fbBatchedBridge.
    
    Oh, and by the way - this communication pattern is pretty much the same when you are using device Javascript Core instead of Chrome Debugger. The difference is that
    instead of Websocket connection, iOS specific API is used to evaluate scripts.
    

    View Slide

53. Symbolicate middleware
    Once I got debugger up and running, I started looking into understanding why error messages didn’t have proper stack traces.
    
    

    Now I didn’t include this in my initial prototype - Steve Kellock from Infinite Red contributed it later - thank you! However, I find it very interesting so decided to describe it
    as if it was there from the very beginning.
    

    View Slide

54. Symbolicate frames
    What happens is that as soon as there’s an error - React Native asks packager to symbolicate the error stack trace.
    
    

    It does that by passing an entire callstack and expects the same structure, but with appropriate lines, columns and files to be returned as a response.
    

    View Slide

55. As soon as we get such request, we create an instance of `SourceMapConsumer`. We pass it a raw source map (as for example read from file system).

    

    SourceMapConsumer represents a parsed source map which we can query for information about the original file positions by giving it a file position in the generated
    source.
    

    View Slide

56. We then map request body (which is an array of frames received from the device) to the original source, line and column information we got by querying
    SourceMapConsumer.
    
    Such array is then sent back to React Native which quickly replaces unsymbolicated stack trace with a better one.
    
    

    View Slide

57. Bundler should be pluggable
    The reason I am telling you this is I believe the bundler implementation should be pluggable. The packager in React Native does way more things than just compiling our
    code. And those things are not trivial - some of them were carefully designed over the course of months.
    
    From my experience, it took me two weeks to implement Webpack to work with React Native. If the entire platform was pluggable, it would’ve been a breeze.
    
    

    View Slide

58. Do we really need that?
    But… do we really need an alternative bundler?
    
    Let me answer this question with another one.
    
    

    View Slide

59. Do you want symlinks?
    Lack of support for symlinks was for long one of the most common issues with React Native development, especially for library developers. React Native promised the
    workflow we know from Web and Node.js in the world of mobile development. Breaking that expectation was painful for many.
    
    The default Metro Bundler uses Watchman to listen to file changes and reacts accordingly. Now, Watchman is a Facebook open source library constrained to provide
    maximum performance at their scale. One of its limitations is that it doesn’t listen for changes in symlinked locations and it’s unclear whether this feature is going to land
    anytime soon - not a priority.
    
    

    View Slide

60. Facebook scale
    At Facebook scale, lots of milliseconds that seem irrelevant to us matter a lot. It’s great that we get performance improvements for free. However, at the same time, I
    believe one should be able to opt-in for a slightly slower but more feature-rich solution.
    
    

    View Slide

61. Meet Haul
    That’s why, at Callstack, we made Haul. Haul is a drop-in replacement for existing packager. It builds on top of open tools, like Webpack. At its core, there’s Express
    server with pluggable middlewares. It’s a hackable dev server with predictable workflow. Most of the code snippets you have seen so far in this talk can be found in the
    repo.
    

    View Slide

62. Select platform
    Here’s how it feels.

    

    You first select a platform you want to prepare bundle for. If you don’t pass it as an option, Haul will ask you for it instead of exiting.
    

    View Slide

63. Then, wait for it
    At the same time, it will print how to call it next time to avoid interactive prompt from appearing.
    
    Then, you just wait for it to finish bundling…
    

    View Slide

64. Ready to go
    And you are ready to go!
    
    We designed it carefully around best developer experience possible taking as much inspiration as possible from community leaders such as create-react-app.
    

    View Slide

65. Reasons?
    There were a couple of reasons we made Haul.
    
    Two of them were already mentioned today, so to recap:
    
    (first) First one was to have a support for symlinks, something we could die for
    
    (second) Second was to embrace potential of plug-ability and make something, that can be extended in an easy way
    
    

    View Slide

66. Webpack is awesome!
    The last one was Webpack - it’s just awesome!
    
    What I like about it is its extensive, well documented interface with a broad ecosystem. It opens lots of exciting possibilities, like using new extensions for your files, using
    CSS, new loaders or even Typescript.
    
    Also, if you are already using Webpack on web, you could share the config and maximise the percentage of common business logic in your codebase.
    

    View Slide

67. What’s the future?
    There are lots of missing features we are working on. Currently, our top priority is to increase its adoption so that we get more bug reports and can move forward!
    
    If the idea of Haul sounds interesting to you and you are likely to use it in the future, please give it a go and let us know your feedback!
    

    View Slide

68. Make Haul a platform
    As I mentioned, I am really looking into making Haul a platform for other bundlers to run. There are faster alternatives to Webpack, like Fusebox.
    
    I believe that one should be able to use whatever works for him to bundle the app and still, get first class developer experience that Haul gives him.
    

    View Slide

69. Thank you
    Thank you!
    

    View Slide

70. Q & A
    Do we still have time for some questions?
    
    

    View Slide