Realtime Web Applications

In this presentation, we will talk about the approach used by the JusBrasil website to keep all browser instances updated. The JusBrasil Live Update Framework (JusLive) send realtime updates to open browser windows informing the browser which HTML element changed, instead of re-rendering the entire page. JusLive is not just a server push approach (like comet and reverse ajax), instead, it is a complete component-based approach for page rendering. Its stack is composed by Python, JavaScript and HTML components that run on top of a Tornado webserver. Finally, we will present the first version of JusLive, which leveraged Ajax + Redis and we will explain why we ended up switching to a websockets + ZeroMQ approach for message broadcasting

Transcript

1. Realtime Web Applications Osvaldo Matos Júnior
2. Traditional web page
4. Dynamic web page
5. Dynamic web page
6. Realtime web page
7. Realtime web page
8. Realtime web page Chat News Feed Notifications Comments Vote up/down Text edition … many things to update
10. JusBrasil Challenge
11. Push vs Pull
14. JusBrasil Live Architecture
15. JusLive 1.0
16. Inspired on Quora Webnode2 and LiveNode http://www.quora.com/Shreyes-Seshasai/Posts/Tech-Talk-webnode2-and-LiveNode
17. Component-Based www www www
18. Component-Based
19. Component-Based
20. Component Types Complex Simple
21. Topic Follow Button Before Click After Click
22. Python Java Script HTML
23. Python class TopicFollowButton(Component): __alias__ = 'topic.followbutton' def __init__(self, tid, topic=None): self.tid = tid self.following = False self.topic = topic or load_from_database(self.tid) def follow(self, tid): self.following = True def unfollow(self, tid): self.following = False def __html__(self): return render('components/topic_followbutton.html', topic=self.topic, following=self.following, id=self.id)
24. Python class Component(object): def __init__(self, **kwargs): self.__dict__.update(kwargs) @property def name(self): return self.__class__.__name__ @property def id(self): return '__%s' % rand_id(ID_SIZE) def __html__(self): return NotImplementedError() def __js__(self): return 'new %s("%s", "", %s, "%s")' % (self.name, self.id, self.js_init, self.__alias__)
25. Controller @app.route('/topic/<tid>/<slug>') def topic_view(tid, slug): topic = get_topic(tid) components = { 'topic_follow_button' = TopicFollowButton(topic) } return render('topic/view.html', components=components)
26. Page rendering # templates/topic/view.html ... <div class="sidebar">{{components['topic_follow_button']}}</div> ... <script>loadComponents([new TopicFollowButton("__6YekPR9", "", {"tid": 10000036}, "topic.followbutton")]);</script>
27. HTML # templates/componentes/topic_followbutton.html {% if following %} <button id="{{id}}_unfollow" class="btn following"> <span class="icon spr-icons"></span> <span class="label"></span> </button> {% else %} <button id="{{id}}_follow" class="btn primary"> Seguir Tópico </button> {% endif %}
28. Load Components # templates/topic/view.html ... <div class="sidebar">{{components['topic_follow_button']}}</div> ... <script>loadComponents([new TopicFollowButton("__6YekPR9", "", {"tid": 10000036}, "topic.followbutton")]);</script>
29. Java Script TopicFollowButton = Component.extend({ onload: function() { this.$('#@follow').click( bind( this, function(){ this.serverCall('follow') .kwargs({tid : this.tid }) .target( this.$('#@follow')) .send(); return false; })); this.$('#@unfollow').click( bind( this, function(){ this.serverCall('unfollow') .kwargs({tid : this.tid}) .target( this.$('#@unfollow')) .send(); return false; })); } });
30. The @ prefix Each component has a unique ID. The @ sign is replaced by the component's ID: $('#@follow') -> $('#__xyz_follow')
31. serverCall() /ajax? callback=jQuery18206321731214411557_1399129 850063&json=%7B%22args%22%3A%5B%5D% 2C%22kwargs%22%3A%7B%7D% 7D&__vcon_json=%22topic.followbutton% 22&__vcon_method=follow&js_init=%7B% 22tid%22%3A542484%2C%22
32. @app.route('/ajax') def ajax(): module_alias = params.get('__vcon_json', None) method = params.get('__vcon_method') js_init = json.loads(params.get('js_init', "{}")) _json = json.loads(params.get('json', '{"args":[],"kwargs":{}}')) args = _json['args'] kwargs = _json['kwargs'] module = COMPONENTS[module_alias](**js_init) func = getattr(module, method, None) result = func(*args, **kwargs) data = {'js': result.__js__(), 'html': result.__html__()} return json.dumps(data)
33. Server-side Rendering
34. Register and Update register: register the events that the client is interested to. update: wait for updates (long polling).
35. Register and Update /register www www /updates
36. Register and Update
37. Ajax Cross-Domain JusBrasil uses subdomains for profile pages. http://stf.jusbrasil.com.br/ http://www.jusbrasil.com.br/ajax http://www.jusbrasil.com.br/live/update JSONP with subdomain: http://stf.jusbrasil.com.br/ajax http://live.jusbrasil.com.br/update Limit of browser connections per host http://xyz.live.jusbrasil.com.br/update
38. Redis
41. Event Table event_ID window_IDs event_1 window_1, window_2, window_3 event_2 window_2, window_5 event_3 window_1, window_5 event_n ...
42. Event Table event_ID window_IDs event_1 window_1, window_2, window_3 event_2 window_2, window_5 event_3 window_1, window_5 event_n ...
43. Window Table window_ID events window_1 event1_update window_2 event1_update window_3 event1_update window_n ...
44. Action Update pop(window) push(event) gunicorn tornado
45. HAProxy Load Balancer webapp Redis Broker tornado tornado tornado tornado Webnode Machines JusBrasil Live Architecture
46. checks for window updates tornado.ioloop.PeriodicCallback() 200ms
47. Heavy Network Traffic
48. JusLive 2.0
49. Goal: Remove Redis Polling 200ms polling sucks!
51. context = zmq.Context() subscribe = context.socket(zmq.SUB) message = subscribe.recv() # block ZeroMQ
52. ZMQStream sub = zmq_context.socket(zmq.SUB) ... stream = ZMQStream(sub) stream.on_recv(handle_message)
53. socket.bind(1001) socket.connect(1001) socket.connect(1001) listen:1001 socket.connect(1001)
54. new event Publisher Subscribers
55. MANY TO MANY
56. ZMQ Forwarder OUTGOING
57. HAProxy Load Balancer webapp ZMQ Broker tornado tornado tornado tornado Webnode Machines JusBrasil Live Architecture
58. Live Broker (Ring?) Live Broker Live Broker Live Broker Live Broker Live Broker
59. Live Broker (Ring?) Live Broker Live Broker Live Broker Live Broker Live Broker
60. Live Broker (Fully Connected) Live Broker Live Broker Live Broker Live Broker Live Broker
61. Live Broker: Broadcast Live Broker Live Broker Live Broker Live Broker Live Broker webapp tornado tornado tornado tornado tornado tornado
62. Tornado notifies all connected clients
63. event_ID window_IDs event_1 window_1, window_2, window_3 event_2 window_2, window_5 event_3 window_1, window_5 event_n ... window_ID events window_1 event1_update window_2 event1_update window_3 event1_update window_n ...
64. STATELESS!
65. WEBSOCKET!
67. Disadvantages
68. var _components = [new Feedback("__p7qstXI", "", {}, "feedback"),new MessagesAlerts("__dPIyzM0", "", {"render_type": "new", "pid": 3700432, "unseen" "last_sender": 3702921}, "profile.messages_alerts"),new CasesAlerts("__1ieMhRN", "", {"pid": 3700432, "unseen": 0}, "profile.cases_alerts"),new NotificationsAlerts("__UkjlYLy", "", {"unseen_notifications": 5, "pid": 3700432}, "profile.notifications_alerts"),new Newsletter("__f29y3Jo", "", {"called": false "newsletter"),new CompleteProfileAlert("__KeCvcGR", "", {"pid": 3700432, "unseen": true, "recent_updated": false}, "profile.complete_profile"),new AutocompleteMainSearch("__mrU2M5S", "", {"search": null}, "autocomplete_mainsearch"),new DocumentVote("__M32ozic", "", {"artefato": 3, "render_type "document_feed", "document_type": "noticias", "docid": 118270826}, "document.vote"),new WhoVotedDocument("__lSvXcYJ", "", {"doc_id": 118270826, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__IpblN2W", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__RXJnGiA", "", {"artefato": 4, "render_type": "document_feed", "document_type": "politica", "docid": 118242504}, "document.vote"),new WhoVotedDocument("__yLikWX5 "", {"doc_id": 118242504, "artifact": 4}, "who_voted"),new FeedProfileInteractions("__3awaVGX", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__cKsif3u", "", {"artefato": 3, "render_type": "document_feed", "document_type": "noticias", "docid": 118245933}, "document.vote"),new WhoVotedDocument("__vJHcxEm", "", {"doc_id": 118245933, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__eNY6HJu", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__VXXnzuh", "", {"artefato": 4, "render_type": "document_feed", "document_type": "politica", "docid": 118228731}, "document.vote"),new WhoVotedDocument("__YYlNK7s", "", {"doc_id": 118228731, "artifact": 4}, "who_voted"),new FeedProfileInteractions ("__6WEt2w1", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__eN2mIQ6", "", {"artefato": 3, "render_type": "document_feed", "document_type": "noticias", "docid": 118221888}, "document.vote"),new WhoVotedDocument("__w8CMmob", "", {"doc_id": 118221888, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__9mXMTdF", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__zruF4Fm", "", {"artefato": "render_type": "document_feed", "document_type": "noticias", "docid": 118253351}, "document.vote"),new WhoVotedDocument("__yNK4nd6", "", {"doc_id 118253351, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__qv2mGGs", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote ("__pHlx9IP", "", {"artefato": 3, "render_type": "document_feed", "document_type": "noticias", "docid": 118268126}, "document.vote"),new WhoVotedDocument("__2tkeaWe", "", {"doc_id": 118268126, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__SjT24P9", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__cM7nlim", "", {"artefato": 2, "render_type": "document_feed", "document_type": "artigo", "docid": 118241033}, "document.vote"),new WhoVotedDocument("__Zo99beH", "", {"doc_id": 118241033, "artifact": 2}, "who_voted"),new FeedProfileInteractions ("__LxqNB1a", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__8DaBbSU", "", {"artefato": 4, "render_type": "document_feed", "document_type": "politica", "docid": 118236134}, "document.vote"),new WhoVotedDocument("__DAUsbUr", "", {"doc_id": 118236134, "artifact": 4}, "who_voted"),new FeedProfileInteractions("__Vx4g7Rh", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__vszEzko", "", {"artefato": 3 "render_type": "document_feed", "document_type": "noticias", "docid": 118262774}, "document.vote"),new WhoVotedDocument("__3AkNtHt", "", {"doc_id" 118262774, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__q6TJcUM", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote ("__HTIJpBY", "", {"artefato": 3, "render_type": "document_feed", "document_type": "noticias", "docid": 118210064}, "document.vote"),new WhoVotedDocument("__tQnAK0d", "", {"doc_id": 118210064, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__IizEP09", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__Gex2Y7E", "", {"artefato": 3, "render_type": "document_feed", "document_type": "noticias", "docid": 118252543}, "document.vote"),new WhoVotedDocument("__m9mbXOY", "", {"doc_id": 118252543, "artifact": 3}, "who_voted"),new FeedProfileInteractions ("__IbWOXEw", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__hPl0Bu9", "", {"artefato": 3, "render_type": "document_feed", "document_type": "noticias", "docid": 118262349}, "document.vote"),new WhoVotedDocument("__7Zz8F2C", "", {"doc_id": 118262349, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__2owbSIO", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__ciMtOE9", "", {"artefato": 4 "render_type": "document_feed", "document_type": "politica", "docid": 118260885}, "document.vote"),new WhoVotedDocument("__71M5cYi", "", {"doc_id" 118260885, "artifact": 4}, "who_voted"),new FeedProfileInteractions("__mY0xuV5", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote ("__G5OjPnO", "", {"artefato": 3, "render_type": "document_feed", "document_type": "noticias", "docid": 118265182}, "document.vote"),new WhoVotedDocument("__Bm4e5G0", "", {"doc_id": 118265182, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__IPUaGGl", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__as8WaHF", "", {"artefato": 3, "render_type": "document_feed", "document_type": "noticias", "docid": 118262490}, "document.vote"),new WhoVotedDocument("__AFRYceq", "", {"doc_id": 118262490, "artifact": 3}, "who_voted"),new FeedProfileInteractions ("__RKLbRqE", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__hvPj4sb", "", {"artefato": 3, "render_type": "document_feed", "document_type": "noticias", "docid": 118255151}, "document.vote"),new WhoVotedDocument("__ZyJDky1", "", {"doc_id": 118255151, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__q0waQQB", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote("__G6osoSF", "", {"artefato": "render_type": "document_feed", "document_type": "noticias", "docid": 118224685}, "document.vote"),new WhoVotedDocument("__kOCEs7f", "", {"doc_id" 118224685, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__Jg9VAEn", "", {"profile_ids": []}, "perfil.feed_interactions"),new DocumentVote ("__sOYJzl8", "", {"artefato": 3, "render_type": "document_feed", "document_type": "noticias", "docid": 118257726}, "document.vote"),new WhoVotedDocument("__0D5A0F7", "", {"doc_id": 118257726, "artifact": 3}, "who_voted"),new FeedProfileInteractions("__w9omLuQ", "", {"profile_ids": []} "perfil.feed_interactions"),new DocumentVote("__ekLFRyy", "", {"artefato": 3, "render_type": "document_feed", "document_type": "noticias", "docid": 118257838}, "document.vote"),new WhoVotedDocument("__dDXbWLF", "", {"doc_id": 118257838, "artifact": 3}, "who_voted"),new FeedProfileInteractions ("__zKvCf77", "", {"profile_ids": []}, "perfil.feed_interactions"),new MoreQuentes("__aKOylBg", "", {"last_doc": {"artefato": "NOTICIAS", "idDocumento":
69. Not recommended for single user applications
70. Authorization github.com/jusbrasil/pycan
71. Thanks! That's all for today @tupydabahia github.com/tupy