Big app design for Node.js

Designing the code of a large application in production and scaling up to thousands of users while doing it all in Node.js is a challenge that no developer should face alone. I will discuss how we have developed a highly decoupled, plugin-based architecture and a decentralized infrastructure for Cloud9 IDE, along with the technologies we've developed and the difficulties we faced in order to build the largest Node.js application that exists in production.

Transcript

1. BIG
   application
   design
   Saturday, November 10, 12
   

   View Slide

2. @sergimansilla
   Saturday, November 10, 12
   

   View Slide

3. http://c9.io
   Saturday, November 10, 12
   

   View Slide

4. Smart code completion
   Saturday, November 10, 12
   

   View Slide

5. Static analysis
   Saturday, November 10, 12
   

   View Slide

6. Free linux VM
   Saturday, November 10, 12
   

   View Slide

7. Bring your own machine
   Saturday, November 10, 12
   

   View Slide

8. Real terminal
   Saturday, November 10, 12
   

   View Slide

9. Hundreds of thousands
   of lines of JavaScript
   Saturday, November 10, 12
   

   View Slide

10. dynamic,
    weakly-typed
    language
    Saturday, November 10, 12
    

    View Slide

11. one single thread
    Saturday, November 10, 12
    

    View Slide

12. Remote
    VM
    Cloud9 datacenter
    IDE server
    Remote
    VM
    Remote
    VM
    Openshift
    Saturday, November 10, 12
    

    View Slide

13. Saturday, November 10, 12
    

    View Slide

14. Saturday, November 10, 12
    

    View Slide

15. Modularization
    Saturday, November 10, 12
    

    View Slide

16. Black box coding
    Saturday, November 10, 12
    

    View Slide

17. Saturday, November 10, 12
    

    View Slide

18. Good at
    abstracting away
    require
    Saturday, November 10, 12
    

    View Slide

19. Not that good at
    app-level abstraction
    require
    Saturday, November 10, 12
    

    View Slide

20. duplicated modules
    maps to folder names
    Relies on FS
    require
    hard to configure module
    Saturday, November 10, 12
    

    View Slide

21. Dependency error handling
    coding time compile time run time
    KA-BOOM!
    - Server crash
    - Unhappy customers
    - Developer gets fired
    Saturday, November 10, 12
    

    View Slide

22. How to fix it?
    Static dependency list
    Named services
    Easy configuration
    Resolve at startup
    No FS required
    Pass an object
    Saturday, November 10, 12
    

    View Slide

23. Architect
    github.com/c9/architect
    Saturday, November 10, 12
    

    View Slide

24. Everything is a plugin
    Plugins can consume plugins
    An application is just a set of
    plugins
    Architect
    Saturday, November 10, 12
    

    View Slide

25. Dependency model
    require
    Architect
    runtime
    compile time
    Saturday, November 10, 12
    

    View Slide

26. Dependency error handling
    Architect
    coding time compile time run time
    - Happy customers
    - Developer keeps job
    - Fails before release
    Saturday, November 10, 12
    

    View Slide

27. github.com/sergi/nodejsconfit2012
    Saturday, November 10, 12
    

    View Slide

28. Saturday, November 10, 12
    

    View Slide

29. Declare entity ‘cook’ with behavior
    Use ‘cook’ to do some cooking
    Saturday, November 10, 12
    

    View Slide

30. Cook
    Cooking
    action
    Dependency model
    Saturday, November 10, 12
    

    View Slide

31. package.json
    Builds dependency tree
    executing code
    without
    Saturday, November 10, 12
    

    View Slide

32. {
    {
    cook/package.json
    do-­‐cooking/package.json
    Saturday, November 10, 12
    

    View Slide

33. Next
    Extract the code
    Wrap in Architect plugin code
    Make two plugins
    Saturday, November 10, 12
    

    View Slide

34. Function
    signature
    Call when
    done
    Saturday, November 10, 12
    

    View Slide

35. Architect plugin code
    Module.exports
    Options - we’ll talk about it later
    Imports - everything you ‘consume’
    Register - call when done
    Saturday, November 10, 12
    

    View Slide

36. Saturday, November 10, 12
    

    View Slide

37. Saturday, November 10, 12
    

    View Slide

38. Easy to test
    Mock dependencies
    Saturday, November 10, 12
    

    View Slide

39. ‘cook’ should be called 8 times
    Saturday, November 10, 12
    

    View Slide

40. No black magic
    Specify dependency model
    Feed architect a config file
    Call ‘createApp’
    Saturday, November 10, 12
    

    View Slide

41. Saturday, November 10, 12
    

    View Slide

42. Saturday, November 10, 12
    

    View Slide

43. Configuration
    Per-plugin options
    No global options object
    Specify in config file
    Saturday, November 10, 12
    

    View Slide

44. Saturday, November 10, 12
    

    View Slide

45. Saturday, November 10, 12
    

    View Slide

46. Options
    Passed in at startup
    Options can be dependencies
    Fail if options aren’t present
    Saturday, November 10, 12
    

    View Slide

47. Saturday, November 10, 12
    

    View Slide

48. Saturday, November 10, 12
    

    View Slide

49. Your app as
    chunks of functionality
    rather than sets of classes
    Saturday, November 10, 12
    

    View Slide

50. chunks of functionality
    Implicit type constraints
    Keep implementation private
    Swap features instead of interfaces
    Saturday, November 10, 12
    

    View Slide

51. How does Cloud9 use it?
    Open source version
    Local version (OS + sync)
    Hosted version
    Normal
    FTP
    SSH
    Saturday, November 10, 12
    

    View Slide

52. Saturday, November 10, 12
    

    View Slide

53. Swap feature per
    implementation
    On OSS: talk local filesystem
    On FTP: talk FTP library
    On SSH: talk via a SSH bridge
    Saturday, November 10, 12
    

    View Slide

54. Architect is a DI
    framework
    Saturday, November 10, 12
    

    View Slide

55. But a quite
    special one
    Saturday, November 10, 12
    

    View Slide

56. IDE instance
    (FTP)
    IDE instance
    (SSH)
    IDE instance
    (Normal)
    IDE instance
    (Normal)
    Single node.js process
    Other code
    (dashboard etc.)
    Saturday, November 10, 12
    

    View Slide

57. Architect can do
    Multiple instances of same plugin
    Run independently
    But in the same process
    Saturday, November 10, 12
    

    View Slide

58. Centralized event bus
    Loose coupling between plugins
    No hard dependencies!
    Can also do inter-context communication
    Saturday, November 10, 12
    

    View Slide

59. Event
    bus
    Plugin
    Other plugin
    Emit event
    React on event
    Saturday, November 10, 12
    

    View Slide

60. Saturday, November 10, 12
    

    View Slide

61. Saturday, November 10, 12
    

    View Slide

62. And now scale
    Swap it with i.e. Redis PubSub
    Plugins will never notice
    Modular awesomeness!
    Saturday, November 10, 12
    

    View Slide

63. Modularize in feature blocks
    Don’t over engineer
    Don’t create too small blocks
    They aren’t interfaces!
    Saturday, November 10, 12
    

    View Slide

64. Avoid context switching
    Less code!
    Less errors!
    Less boilerplate!
    Saturday, November 10, 12
    

    View Slide

65. Loose coupling
    Use (don’t abuse) an event bus
    Smaller dependency graph
    Saturday, November 10, 12
    

    View Slide

66. github.com/c9/architect
    Saturday, November 10, 12
    

    View Slide

67. Happy coding!
    Saturday, November 10, 12
    

    View Slide

68. @sergimansilla
    github.com/sergi
    http://c9.io
    Saturday, November 10, 12
    

    View Slide