Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Big app design for Node.js

Big app design for Node.js

Designing the code of a large application in production and scaling up to thousands of users while doing it all in Node.js is a challenge that no developer should face alone. I will discuss how we have developed a highly decoupled, plugin-based architecture and a decentralized infrastructure for Cloud9 IDE, along with the technologies we've developed and the difficulties we faced in order to build the largest Node.js application that exists in production.

Sergi Mansilla

November 10, 2012
Tweet

More Decks by Sergi Mansilla

Other Decks in Technology

Transcript

  1. BIG
    application
    design
    Saturday, November 10, 12

    View full-size slide

  2. @sergimansilla
    Saturday, November 10, 12

    View full-size slide

  3. http://c9.io
    Saturday, November 10, 12

    View full-size slide

  4. Smart code completion
    Saturday, November 10, 12

    View full-size slide

  5. Static analysis
    Saturday, November 10, 12

    View full-size slide

  6. Free linux VM
    Saturday, November 10, 12

    View full-size slide

  7. Bring your own machine
    Saturday, November 10, 12

    View full-size slide

  8. Real terminal
    Saturday, November 10, 12

    View full-size slide

  9. Hundreds of thousands
    of lines of JavaScript
    Saturday, November 10, 12

    View full-size slide

  10. dynamic,
    weakly-typed
    language
    Saturday, November 10, 12

    View full-size slide

  11. one single thread
    Saturday, November 10, 12

    View full-size slide

  12. Remote
    VM
    Cloud9 datacenter
    IDE server
    Remote
    VM
    Remote
    VM
    Openshift
    Saturday, November 10, 12

    View full-size slide

  13. Saturday, November 10, 12

    View full-size slide

  14. Saturday, November 10, 12

    View full-size slide

  15. Modularization
    Saturday, November 10, 12

    View full-size slide

  16. Black box coding
    Saturday, November 10, 12

    View full-size slide

  17. Saturday, November 10, 12

    View full-size slide

  18. Good at
    abstracting away
    require
    Saturday, November 10, 12

    View full-size slide

  19. Not that good at
    app-level abstraction
    require
    Saturday, November 10, 12

    View full-size slide

  20. duplicated modules
    maps to folder names
    Relies on FS
    require
    hard to configure module
    Saturday, November 10, 12

    View full-size slide

  21. Dependency error handling
    coding time compile time run time
    KA-BOOM!
    - Server crash
    - Unhappy customers
    - Developer gets fired
    Saturday, November 10, 12

    View full-size slide

  22. How to fix it?
    Static dependency list
    Named services
    Easy configuration
    Resolve at startup
    No FS required
    Pass an object
    Saturday, November 10, 12

    View full-size slide

  23. Architect
    github.com/c9/architect
    Saturday, November 10, 12

    View full-size slide

  24. Everything is a plugin
    Plugins can consume plugins
    An application is just a set of
    plugins
    Architect
    Saturday, November 10, 12

    View full-size slide

  25. Dependency model
    require
    Architect
    runtime
    compile time
    Saturday, November 10, 12

    View full-size slide

  26. Dependency error handling
    Architect
    coding time compile time run time
    - Happy customers
    - Developer keeps job
    - Fails before release
    Saturday, November 10, 12

    View full-size slide

  27. github.com/sergi/nodejsconfit2012
    Saturday, November 10, 12

    View full-size slide

  28. Saturday, November 10, 12

    View full-size slide

  29. Declare entity ‘cook’ with behavior
    Use ‘cook’ to do some cooking
    Saturday, November 10, 12

    View full-size slide

  30. Cook
    Cooking
    action
    Dependency model
    Saturday, November 10, 12

    View full-size slide

  31. package.json
    Builds dependency tree
    executing code
    without
    Saturday, November 10, 12

    View full-size slide

  32. {
    {
    cook/package.json
    do-­‐cooking/package.json
    Saturday, November 10, 12

    View full-size slide

  33. Next
    Extract the code
    Wrap in Architect plugin code
    Make two plugins
    Saturday, November 10, 12

    View full-size slide

  34. Function
    signature
    Call when
    done
    Saturday, November 10, 12

    View full-size slide

  35. Architect plugin code
    Module.exports
    Options - we’ll talk about it later
    Imports - everything you ‘consume’
    Register - call when done
    Saturday, November 10, 12

    View full-size slide

  36. Saturday, November 10, 12

    View full-size slide

  37. Saturday, November 10, 12

    View full-size slide

  38. Easy to test
    Mock dependencies
    Saturday, November 10, 12

    View full-size slide

  39. ‘cook’ should be called 8 times
    Saturday, November 10, 12

    View full-size slide

  40. No black magic
    Specify dependency model
    Feed architect a config file
    Call ‘createApp’
    Saturday, November 10, 12

    View full-size slide

  41. Saturday, November 10, 12

    View full-size slide

  42. Saturday, November 10, 12

    View full-size slide

  43. Configuration
    Per-plugin options
    No global options object
    Specify in config file
    Saturday, November 10, 12

    View full-size slide

  44. Saturday, November 10, 12

    View full-size slide

  45. Saturday, November 10, 12

    View full-size slide

  46. Options
    Passed in at startup
    Options can be dependencies
    Fail if options aren’t present
    Saturday, November 10, 12

    View full-size slide

  47. Saturday, November 10, 12

    View full-size slide

  48. Saturday, November 10, 12

    View full-size slide

  49. Your app as
    chunks of functionality
    rather than sets of classes
    Saturday, November 10, 12

    View full-size slide

  50. chunks of functionality
    Implicit type constraints
    Keep implementation private
    Swap features instead of interfaces
    Saturday, November 10, 12

    View full-size slide

  51. How does Cloud9 use it?
    Open source version
    Local version (OS + sync)
    Hosted version
    Normal
    FTP
    SSH
    Saturday, November 10, 12

    View full-size slide

  52. Saturday, November 10, 12

    View full-size slide

  53. Swap feature per
    implementation
    On OSS: talk local filesystem
    On FTP: talk FTP library
    On SSH: talk via a SSH bridge
    Saturday, November 10, 12

    View full-size slide

  54. Architect is a DI
    framework
    Saturday, November 10, 12

    View full-size slide

  55. But a quite
    special one
    Saturday, November 10, 12

    View full-size slide

  56. IDE instance
    (FTP)
    IDE instance
    (SSH)
    IDE instance
    (Normal)
    IDE instance
    (Normal)
    Single node.js process
    Other code
    (dashboard etc.)
    Saturday, November 10, 12

    View full-size slide

  57. Architect can do
    Multiple instances of same plugin
    Run independently
    But in the same process
    Saturday, November 10, 12

    View full-size slide

  58. Centralized event bus
    Loose coupling between plugins
    No hard dependencies!
    Can also do inter-context communication
    Saturday, November 10, 12

    View full-size slide

  59. Event
    bus
    Plugin
    Other plugin
    Emit event
    React on event
    Saturday, November 10, 12

    View full-size slide

  60. Saturday, November 10, 12

    View full-size slide

  61. Saturday, November 10, 12

    View full-size slide

  62. And now scale
    Swap it with i.e. Redis PubSub
    Plugins will never notice
    Modular awesomeness!
    Saturday, November 10, 12

    View full-size slide

  63. Modularize in feature blocks
    Don’t over engineer
    Don’t create too small blocks
    They aren’t interfaces!
    Saturday, November 10, 12

    View full-size slide

  64. Avoid context switching
    Less code!
    Less errors!
    Less boilerplate!
    Saturday, November 10, 12

    View full-size slide

  65. Loose coupling
    Use (don’t abuse) an event bus
    Smaller dependency graph
    Saturday, November 10, 12

    View full-size slide

  66. github.com/c9/architect
    Saturday, November 10, 12

    View full-size slide

  67. Happy coding!
    Saturday, November 10, 12

    View full-size slide

  68. @sergimansilla
    github.com/sergi
    http://c9.io
    Saturday, November 10, 12

    View full-size slide