The Evented Web

Given at the Denver Open Source Users Group (@dosug) Nov. 1, 2011

Transcript

1. The Evented Web webhooks and websockets the dynamic duo Mike Brevoort @ mbrevoort Denver Open Source User Group November 1, 2011 1
2. • The evented What? Why? • Webhooks • Websockets • Asynchronous I/O and node.js • Putting it all together 2
3. Semi-related points in time 3
4. 1990 HTML 1993 Common Gateway Interface (CGI) 1997 Java servlets 1999 XMLHTTP ActiveX control 2004 Gmail, “web 2.0” 2005 AJAX coined CometD/Bayeux 2006 2010 (Dec 2009) Websockets Draft 4
5. Conditioned... ...into aone sided conversation wait ...to http://sites.google.com/site/davidnchung/StakeElephant.jpg 5
6. When we should have been doing this! http://www.tecacentre.net/funpics/funpics/flying_elephant.jpg 6
7. The traditional web Call Me... I won’t call you. 7
8. boiling yet? boiling yet? boiling yet? boiling yet? boiling yet? boiling yet? boiling yet? boiling yet? boiling yet? boiling yet? boiling yet? http://www.charlesculp.com/isleroyale07/BoilingWater_IMG_3487_800_160.jpg 8
9. anonymous web the disconnected 9
10. it’s all about events http://www.flickr.com/photos/cizake/4164756091/ 10
11. I will call them Webhooks. And they will be good Jeff Lindsay @progrium 2006 11
12. what is a Webhook? callbacks over HTTP ... an HTTP POST that occurs when something happens 12
13. Server Server 13
14. scalability performance throughput realtime instant data 14
15. scalability performance throughput realtime instant data 14
17. I got it! 15
18. When something happens, I’ll just call you! I got it! 15
19. When something happens, I’ll just call you! I got it! So... relax 15
20. it’s simple 16
21. it’s simple “even a baby can do it” 16
22. configurable event handling user to infinity and beyond! 17
23. credit: Jeff Lindsay @progrium 18
24. receive data in real time 19
25. receive data in real time Superpoke /remove 19
26. receive data in real time Facebook User removes app Superpoke /remove 19
27. receive data in real time Facebook User removes app HTTP POST signed request w/ user id Superpoke /remove 19
28. build script it! resolve issue change 20
29. receive data, do something, pass it on $ cat apple.txt | wc | mail -s "The count" joe@me.com http://www.flickr.com/photos/russmorris/2713018257 21
30. Yo’ App Yo’ Warehouse 22
31. Customer Pays Yo’ App Yo’ Warehouse 22
32. Customer Pays Yo’ App Yo’ Warehouse event New Order HTTP POST 22
33. Customer Pays Yo’ App Yo’ Warehouse event New Order HTTP POST 22
34. Customer Pays Yo’ App Yo’ Warehouse event New Order HTTP POST New Order HTTP PO ST 22
35. Customer Pays Yo’ App Yo’ Warehouse event New Order HTTP POST New Order HTTP PO ST Come and get it HTTP POST 22
36. process data, give something in return like a typical API call but the contract is dictated by the caller (not the callee) 23
37. process data, give something in return like a typical API call but the contract is dictated by the caller (not the callee) when something happens, I’m going to send you Y 23
38. process data, give something in return like a typical API call but the contract is dictated by the caller (not the callee) when something happens, I’m going to send you Y OK 23
39. process data, give something in return like a typical API call but the contract is dictated by the caller (not the callee) when something happens, I’m going to send you Y just tell me where 23
40. process data, give something in return like a typical API call but the contract is dictated by the caller (not the callee) when something happens, I’m going to send you Y just tell me where OK 23
41. process data, give something in return like a typical API call but the contract is dictated by the caller (not the callee) when something happens, I’m going to send you Y just tell me where Respond back with X OR call me back HERE 23
42. process data, give something in return like a typical API call but the contract is dictated by the caller (not the callee) when something happens, I’m going to send you Y just tell me where Respond back with X OR call me back HERE OK 23
43. process data, give something in return like a typical API call but the contract is dictated by the caller (not the callee) when something happens, I’m going to send you Y just tell me where Respond back with X OR call me back HERE OK 23
45. Webhook Standards? • Yes, it’s called HTTP • OK, but there are a few related conventions/standards including RESTFul webhooks, pubsubhubbub, etc 25
46. Webhook Standards? • Yes, it’s called HTTP • OK, but there are a few related conventions/standards including RESTFul webhooks, pubsubhubbub, etc “even a baby can do it” 25
47. http://code.google.com/p/pubsubhubbub/ 26
48. http://code.google.com/p/pubsubhubbub/ 27
49. http://code.google.com/p/pubsubhubbub/ 28
50. http://code.google.com/p/pubsubhubbub/ 29
51. http://code.google.com/p/pubsubhubbub/ 30
52. http://code.google.com/p/pubsubhubbub/ 31
53. http://code.google.com/p/pubsubhubbub/ 32
54. Webhooks • powerful glue • plumbing of the evented web • empowering 33
55. Server Client Websockets 34
56. Server Websockets web Browser (the hard part) 34
57. Flash sockets Comet Silverlight Duplex Websockets Server web Browser 35
58. Websockets enables native bidirectional communication between a server and a web browser 36
59. W3C API and IETF Protocol Shares ports with HTTP/S (80, 443) Cross-Origin Works across firewalls, proxies and routers (theoretically) Websockets 37
60. Comet long held HTTP requests umbrella term for multiple techniques aka “Ajax Push”, “Reverse Ajax”, “HTTP Streaming” compatible 38
61. Flash Sockets raw TCP socket requires Flash runtime proxy and router woes can implement websocket protocol 39
62. Silverlight Duplex socket or polling requires Silverlight sockets over ports 4502-4534 can implement websocket protocol 40
63. Why not comet? • an inefficient hack? • 1/2 duplex • HTTP Overhead • latency http://websocket.org/quantum.html 41
64. Polling http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.ajax.devguide.help/docs/PureAjax_pubsub_clients.html Comet 42
65. Polling http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.ajax.devguide.help/docs/PureAjax_pubsub_clients.html Comet 42
66. http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.ajax.devguide.help/docs/PureAjax_pubsub_clients.html Long Polling Comet 43
67. Keep-Alive Streaming http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.ajax.devguide.help/docs/PureAjax_pubsub_clients.html Comet 44
68. a protocol http://tools.ietf.org/html/draft-ietf-hybi-thewebsocketprotocol-03 Websockets 45
69. an API http://dev.w3.org/html5/websockets/ var socket = new WebSocket("ws://host/socket/"); socket.onopen = function() { ! console.log("viva la socket!"); } socket.onmessage = function(msg) { ! console.log(msg);! } Websockets 46
70. Browser Support http://caniuse.com Websockets 47
71. Security Flaw • secure Websockets (wss://) not affected • not specifically Websockets, rather affects all sockets over HTTP including Flash and Java • IETF study of 47,338 HTTP proxies tested found 0.37% and 0.017% were vulnerable to the two attack types • Browser vendors want to bulletproof Websocket wire protocol to handle even buggy implementations of intercepting proxies • Websocket protocol R6 introduced 2/6/2011, actively being reviewed. More info: http://tools.ietf.org/wg/hybi/ http://security.sys-con.com/node/1642956 Websockets 48
72. Security Flaw • secure Websockets (wss://) not affected • not specifically Websockets, rather affects all sockets over HTTP including Flash and Java • IETF study of 47,338 HTTP proxies tested found 0.37% and 0.017% were vulnerable to the two attack types • Browser vendors want to bulletproof Websocket wire protocol to handle even buggy implementations of intercepting proxies • Websocket protocol R6 introduced 2/6/2011, actively being reviewed. More info: http://tools.ietf.org/wg/hybi/ http://security.sys-con.com/node/1642956 Websockets Resolved 48
73. What about the server? Websockets 49
74. Traditional Web Servers just won’t do 50
75. Traditional Web Servers just won’t do 50
76. Why? All these webhooks and websockets mean orders of magnitude more concurrent, long lived connections Thread per connection is not scalable So much time is wasted waiting... on I/O 51
77. Typical I/O Latency L1: 3 cycles L2: 14 cycles RAM: 250 cycles DISK: 41,000,000 cycles NETWORK: 240,000,000 cycles http://nodejs.org/jsconf.pdf L1 L2 RAM Disk Network 0 60,000,000 120,000,000 180,000,000 240,000,000 300,000,000 52
78. var result = db.query("select * from T"); // wait for result // use result console.log( result[0].C ); What’s wrong with this? waiting idle blocking for 240,000,000 cycles 53
79. Alternatively // the async way db.query("select * from T", function(result) { console.log( result[0].C ); }); 54
80. Asynchronous IO • non-blocking • event loop • callbacks 55
81. Apache vs Nginx http://blog.webfaction.com/a-little-holiday-present asynchronous vs synchronous I/O event loop vs thread per connection 56
82. Apache vs Nginx http://blog.webfaction.com/a-little-holiday-present asynchronous vs synchronous I/O event loop vs thread per connection 57
83. Some of the contenders • Twisted (python) • EventMachine (Ruby) • Node.js (javascript) • etc. (non-blocking, asyncronous I/O, event loop) 58
84. provide an easy way to build scalable network programs. • server-side Javascript • client-side development likeness • Javascript designed specifically to be used with an event loop • non-blocking libraries • FAST, underpinned by V8 • community momentum 59
85. a different way of thinking var callback = function(response) { // do something with response }; obj.getSomething(data, callback); 60
86. http://www.flickr.com/photos/way2go/5536458691/ But, don’t I have to wait until more browsers support Websockets? Ummm... 61
88. Don’t be ridiculous. We can use fallbacks! 62
89. Don’t be ridiculous. We can use fallbacks! With Socket.io we can eat IE6’s lunch! 62
90. Don’t be ridiculous. We can use fallbacks! With Socket.io we can eat IE6’s lunch! and IE7 62
91. Don’t be ridiculous. We can use fallbacks! With Socket.io we can eat IE6’s lunch! and IE7 and IE8 62
92. Don’t be ridiculous. We can use fallbacks! With Socket.io we can eat IE6’s lunch! and IE7 and IE8 and IE9 62
93. Don’t be ridiculous. We can use fallbacks! With Socket.io we can eat IE6’s lunch! and IE7 and IE8 and IE9 *sigh* I’m getting too old for this crap 62
94. Socket.io Socket.IO aims to make realtime apps possible in every browser and mobile device, blurring the differences between the different transport mechanisms 63
95. Websocket Flash Socket AJAX Polling & Multipart Forever Frame JSONP Polling Socket.io 64
96. IE 5.5+ Chrome 4+ Firefox 3+ Safari 3+ iOS Safari Android Webkit WebOs Webkit Opera 10.6 Socket.io 65
97. Demo 66
98. Demo webhooks + websockets /twillio /socket.io 67
99. Demo webhooks + websockets 1. Call /twillio /socket.io 67
100. Demo webhooks + websockets 1. Call 2. New Call GET (webhook) /twillio /socket.io 67
101. Demo webhooks + websockets 1. Call 2. New Call GET (webhook) 3. Status (websocket) /twillio /socket.io 67
102. Demo webhooks + websockets 1. Call 2. New Call GET (webhook) 3. Status (websocket) 4. Respond Say + Record /twillio /socket.io 67
103. Demo webhooks + websockets 1. Call 2. New Call GET (webhook) 3. Status (websocket) 4. Respond Say + Record 5. Say /twillio /socket.io 67
104. Demo webhooks + websockets 1. Call 2. New Call GET (webhook) 3. Status (websocket) 4. Respond Say + Record 5. Say 6. Record /twillio /socket.io 67
105. Demo webhooks + websockets 1. Call 2. New Call GET (webhook) 3. Status (websocket) 4. Respond Say + Record 5. Say 6. Record 7. New Recording POST (webhook) /twillio /socket.io 67
106. Demo webhooks + websockets 1. Call 2. New Call GET (webhook) 3. Status (websocket) 4. Respond Say + Record 5. Say 6. Record 8. Recording (websocket) 7. New Recording POST (webhook) /twillio /socket.io 67
107. Demo webhooks + websockets 1. Call 2. New Call GET (webhook) 3. Status (websocket) 4. Respond Say + Record 5. Say 6. Record 8. Recording (websocket) 7. New Recording POST (webhook) /twillio /socket.io 9. <audio/> GET Recording 67
108. https://github.com/mbrevoort/ Evented-Web-Demo 68
109. Beyond http://www.photostrend.com/wp-content/uploads/2011/09/Abstract-Stream-Light-hd.jpg Realtime = Web 3.0 69
110. Thank You (you’re free to fly now) 70