Collaborative Editing and Scaling Web Based Editor

Tech talk on how to create collaborative editors and scale them to millions of users.

Transcript

1. Collaborative Editing Abhishek Gahlot ( HackerRank )
2. About Me • Same Guy who spoke in the Morning • Currently Software Engineer at HackerRank • Worked on Collaborative Editing • Email : me@abhishek.it
3. Why Collaboration? • Modern Web Application • Remote and Distributed Work • Evolution of Web
4. Collaboration Use Cases • Online Interviews ( HackerRank CodePair ) • Expert Sessions • Document Editing • Feedbacks • Online Learning
5. Eventual Consistency Convergence • Every edit on different clients should converge • Balancing servers in order to maintain sticky sessions • Avoid conflicts or divergence
6. Achieving Convergence • Operational Transformation • Differential Synchronization • CRDT’s ( Conflict-free Replicated Data Type )
7. Operational Transform Let T be Transform function such that T(x, y) = (x’, y’) x -> insert(1, ‘a’), delete(3) y -> delete(2) Transform function makes sure document converges
8. Operational Transform Example
9. Operational Transform Properties • Easy to apply operation • Difficult to compute transformation • Too complex and Many corner cases
10. Differential Synchronization • Diff - O(n*n) • Match • Patch ( Apply operation )
11. Differential Synchronization Example
12. Conflict-free Replicated Data Type
13. CRDT Properties 1 + 2 = 2 + 1 (Commutative) 1 + (2+3) = (1+2) + 3 (Associative) 1 + 1 = 2 ( Not Idempotent )
14. CRDT Properties A U B = B U A (Commutative) A U (B U C) = (A U B) U C (Associative) A U A= A ( Idempotent )
15. CRDT Types • CmRDT ( Not Idempotent ) • CvRDT ( Associative, Commutative, Idempotent )
16. Building • Frontend Implementation • Backend Implementation • Databases
17. Frontend Implementation
18. Web Based Editor • Ace Editor, CodeMirror, Monaco, Draftsjs • Extracting tokens ( Read, Update, Delete, Insert ) • Applying Tokens
19. Full Duplex Channel • WebSocket • Sticky sessions • Failure support • Javascript API
20. Other Client Side Requirements • Unique Identifier for Editor or Webpage • Local Storage • Collaborative Algorithm Library
21. Backend Implementation
22. Sticky Sessions • Request ending on same server and same process • Unique document Identifier • Ability to support failure or Node shutdown
23. Maintain Sticky Sessions • Use HAProxy ( Default websocket support ) • Redirects requests depending on Algorithm used • Algorithms like : IP, Session, URL fragment based
24. Multiple Servers
25. Multiple Servers • HAProxy acts a load balancer • Algorithms by HAProxy helps in Node Failure • High Availability
26. Multiple Processes Image From : http://insights.mindstix.com/building-a-node-js-infrastructure-that-offers-scalability-fault-resilience-and-zero-downtime/
27. Multiple Processes • HAProxy redirects to master process of NodeJS • Customised node module handles id with master process • Master Process redirect websocket connection to child process
28. Multiple Databases • Sharding • Replication • Connection Pooling from Application
29. Maintaining Connections • Make sure upgrade request is done by server • Use proper unique ID for document for Web Socket
30. In Production
31. Architecture
32. RethinkDB • Realtime Database • Scalable and Distributed • Doesn’t provide atomic level changes
33. Memory Issues • Garbage Collection of Node.JS • Huge JSON and Fast manipulation • Increase Heap Size of Node.JS • Atomic Updates
34. Network Issues • Flaky Internet • WebSockets Disconnection • High Latency
35. Divergence • Read from Database • Reset socket peers • Update clients
36. Scrite (In Progress) • http://github.com/scrite • scrite.io • me@abhishek.it
37. Questions? Can’t ask here? 
 
 Ping me at (me@abhishek.it)
38. Thanks!!