SocketStream NYCCamp 2015

The slides from the presentation about SocketStream at NYCCamp 2015

Transcript

1. NYC Camp, 17th July 2015
2. What is SocketStream?
3. SocketStream is a framework for realtime web apps
4. SocketStream • Created by Owen Barnes back in 2011 • Built on Node.js • Geared towards building Single Page Applications • Uses WebSockets to transmit data between the client and the server
5. SocketStream • Used to build apps as opposed to sites • Current apps in the wild include chat apps, games, dashboards, line of business applications, financial trading platforms
6. The origins of SocketStream
7. I want to tell you about this guy
8. Peter Lubbers Senior Program Manager, Scalable Developer Programs at Google Formerly at Kaazing Author of “Pro HTML5 Programming” by APress …and he owns that numberplate!
9. If there is a Godfather of WebSockets, it is Peter
11. Some colleagues at New Bamboo attend Peter’s training event at Skills Matter in London, UK
12. The next day, at the company hack- day they create a WebSockets API that powers a web traffic tool
13. They think it’s cool, work on it more over the next couple of months, and turn it into a company
15. 2010 http://www.cloudera.com/content/dam/cloudera/ logos/customers/ homepage_customer_logo_banner/aol- advertising-logo.png
16. I join a small team building Ruby on Rails apps for the UK Sales department
17. We were building Rails Apps with REST APIs that powered apps with AJAX interfaces
18. My boss (Owen Barnes) attends Peter’s training event at Skills Matter in London, UK (same event, 1year on)
19. He comes back excited about WebSockets, and sees a chance to rethink how web apps work
20. – Owen Barnes “What would happen if we sent all of the data your app needs over a WebSocket?”
21. He suggests a project to the UK Tech boss, it’s approved.
22. We build a new account management platform called AdSpeed
23. AOL AdSpeed
24. He extracts a web framework from it, calls it SocketStream
25. We show it to other people in AOL, they get excited
26. He demos SocketStream at the HN London Meetup June 2011
27. It goes to #1 on Hacker News
28. #1 Trending repo on Github, with 1k Stars in just over a week
29. Developers even say that AOL “looks cool”
31. AOL let Owen work on SocketStream for 1000 hours free!!!* *then $9.99 per month following
32. I’ve been very fortunate to see the creation of both Pusher and SocketStream
33. Peter’s training events were the catalyst for both. Without him, they might not have existed
34. So thank you Peter
35. How does SocketStream work?
36. Let’s explore by playing with a demo app
37. You can try this out as we go along
38. Step 1 - Go to socketstream.com and follow the instructions under “Getting Started”
39. RPC
40. RPC • This is a request/response API over WebSockets • We use it to send data from the client to the server, and have the server send back a response • Useful for handling data sent from the web app, that needs to be validated or verified as having been received
41. On the client
42. On the server
43. RPC • From the client, demo.sendMessage calls the sendMessage function in the server/rpc/demo.js file. • This convention allows you to structure the RPC API that you want in line with your files, so they are easy to create, find and modify. • It is one of the “opinions” of the framework
44. Pub/Sub
45. Pub/Sub • A publish/subscribe API over WebSockets • The server does the publish/subscribe actions, and the client listens for events • Commonly used for public and private communications (.e.g chat rooms, activity feeds, announcements, events in games)
46. On the server
47. On the client
48. Pub/Sub • That example broadcasts to all browsers, but Pub/ Sub also supports sending to: • Private channels • A specific user • Even a single browser tab
49. Pub/Sub • SocketStream supports Pub/Sub through 2 methods - internally within the Node.js process with EventEmitter (simple to start with, but means you lose channel subscriptions when the process is restarted • For that reason, SocketStream has a Pub/Sub plugin that uses Redis to persist the channels and subscriptions • There is even a 3rd-party Pub/Sub plugin using MongoDB to persist the data
50. Pub/Sub - app patterns • If you have a need for authenticated channels, implement an authentication method via RPC, and create a subscription with a channel name unique for that user, if the authentication succeeds • Always use Redis for Pub/Sub persistence when in staging/production modes • It’s a good idea to design the app so that channel subscriptions are kept at a minimum.
51. Client code organisation .js .css .html
53. Code Stores the client-side Javascript files CSS Stores the Cascading Style Sheets Static Stores any assets that need to be served statically (images, favicon.ico, fonts)
54. Templates Stores the client-side HTML templates that are rendered on the client Views Stores the HTML page that is sent to the browser from the server
55. This is an opinionated approach to organising client- side files, but with reasons
56. HTML Templates are loaded by the app based on their location in the templates folder
59. Next, the client-side JS is loaded as CommonJS modules, via a modified version of Browserify
60. Henrik (a contributor) has been working on the bundler, making it pluggable, so it can support using tools like WebPack and JSPM
62. We use the same style of module loading that Node.js uses, and it feels really nice
63. Also, SocketStream ships with a powerful asset pipeline for Single Page Apps
64. Defining a Single Page Client does a number of things
66. Firstly, it transpiles files app.coffee app.js layout.styl layout.css admin.jade admin.html
67. We support the following transpilers HTML CSS JS Jade EJS CoffeeKup RActive Handlebars Toffee JSX Stylus Less Scss Sass CoffeeScript IcedCoffeeScript TypeScript LiveScript GorillaScript
68. We then concatenate those files in the order defined for the single page client
69. After that, we minify those files so that they are the smallest possible size
70. We do this so that the Single Page App consists of just 1 JS file, 1 CSS file, 1 HTML file, the favicon, and the WebSocket connection (excluding images/other static assets)
71. We can even specify custom paths for the assets, so that they load from a CDN
72. How does the web page load those files?
74. The asset bundler will find the <SocketStream> tag, and insert the JS and CSS files in there
75. Oh, and we have live- reload built-in too
76. Don’t just optimise for the user experience, optimise for the developer experience too
77. There are more features… • Use either Engine.io or SockJS for the WebSocket transport
 • Load an Express app with SocketStream, and share Cookie information
 • Create your own WebSocket responder protocols (for ultra-efficient communication - e.g. game data) • Use Web Workers for doing intensive processing in non-blocking threads in the browser
78. There are more features… • Reconnect WebSockets that die (due to train tunnels and general lack of Internet connectivity) • There are some neat 3rd-party plugins for integrating with Backbone and Angular applications
 • We also have a tool for testing your web apps with Cucumber and Selenium, called SS-cucumber
79. if I try to run through them then all we’re going to miss Lunch… See socketstream.com for more details on those features
80. The challenges of building a web framework
81. Ninja coding
82. It’s not uncommon to build an app or a library without tests
83. …and some developers are that good that they do it and everything just works
84. I have met 2 people who I would call “Ninja coders”, and Owen is one of them
85. but what happens when the Ninja coder isn’t around anymore?
86. or, how do you maintain a web framework that doesn’t have unit tests?
87. this was the challenge when I became the Lead Developer for SocketStream
88. Do 2 things: 1 - Get a code coverage tool 2 - Read the code 3 - Write tests
89. Code coverage tools • Istanbul, from Krishnan Anantheswaran at Yahoo! • https://github.com/gotwarlost/istanbul • Let’s you check what code is covered by tests • Gives you colourful reports that highlight the weak areas of your codebase
90. Read the code • Joel Spolsky once said that people find reading code hard • The source code was written in CoffeeScript • I converted it by hand to JavaScript, and ported the code comments over
91. Write the tests • Myself, Roman Minkin, and Henrik Vendelbo have been chipping away at writing tests for the framework for some time • The framework’s code coverage is now at 76% • We still have some way to go, but we’re getting there
92. You are not superman (no matter how much you want to be)
93. Open Source software can become a full-time occupation
94. If you can make OSS a a full-time paid job, fantastic
95. If not, you spend what free time you have working on it
96. This has been my biggest challenge to date with working on SocketStream
97. You have to delegate responsibility
98. Thankfully, these people have helped a massive amount with the project
99. Roman Minkin Henrik Vendelbo Robert Hall Joshua Wylie Cullick
100. Rewrites are dangerous
101. In the early years of SocketStream, it was rewritten a couple of times
102. Some users became wary of the rewrites http://is.gd/qOToHC
103. And when work on 0.4 became stagnant, it lead people to question whether the framework was dead
104. When I became the new lead developer, I originally wanted to maintain both the current 0.3 branch, and then launch 0.4 by June 2014
105. It didn’t happen. In the end I chose to focus on 0.3 and cancel the 0.4 branch
106. It was not the easiest thing to do, but it made things better in the long run
107. Lesson Learned: Rewrites can cause duplicate work, split focus, and scare users
108. What lies ahead for SocketStream?
109. Scaling • At the moment, scaling requires a load balancer with session affinity enabled to make sure that long-polling handshakes go to the right servers • We want to explore an approach that doesn’t depend on session affinity
111. Test Coverage • As mentioned earlier, we want to get to as close to 100% as is reasonable (currently at 76%) • Once we’re there, we can have confidence in refactoring whole swathes of the codebase.
112. Removing sync calls • There are some synchronous calls in the codebase (mainly around file reading/app generation) • We want to remove all of them
113. Making the codebase modular • We’ve started to breakout SocketStream’s functionality into modules • This has the benefit of making the codebase easier to look through, test, and refactor • App generation was the first part of the code to break out into a module
114. github.com/socketstream/ss-generator
115. Hybrid app support • Single Page Apps are not good for SEO • React has become a very popular tool • We’re looking to improve support for building hybrid apps (server-side rendering + client-side rendering) - which currently exists with React and Flux
116. Looking for a new sponsor • At the very start of the project, AOL allowed Owen to work on SocketStream full time • But it meant he longer ran the Ad Ops team, who were merged into another team • The change along with company re-orgs ended up stifling momentum on the project
117. Looking for a new sponsor • Last year, I became a consultant, and managed to sponsor work on SocketStream through Anephenix (my company) • But ultimately, I am 1 person, and there is only so much I can support by myself • We’re looking for a sponsor
118. Apps built with SocketStream
119. Hollow hollowdocumentary.com
120. Vmux vmux.co
121. Dashku dashku.com
122. I want to talk a little about Dashku
123. After I left AOL in 2012, I joined a startup called Uberlife
124. It wasn’t going well - I wasn’t happy there
125. One Saturday afternoon
126. It was the right thing for them to do
127. But it also felt like a death sentence on my career
128. I felt like I had to prove that I wasn’t a bad developer I had to build something
129. So I built Dashku, using SocketStream
130. I spend every day for 2 months working on this app, and then I post it to Hacker News
131. Goes to #1 on HN in 44 minutes
132. Then I discover the joys of not persisting the ulimit config on Ubuntu 10.04
133. Chris Matthieu (then at Bechtel) calls and says he wants to use it
134. I also meet with a Dashboard company in London, and try to sell the app to them (without success)
135. After 4 months of not working my bank balance is running low
136. I find some Danish Kroner lying around and cash it in
137. I sell my iPad to pay the rent
138. I get to within £50 of my overdraft limit
139. I finally get a job in London, decide to focus hard on it, and open source Dashku
140. What happens next • I get contacted by a Indian Health startup, who want me to work for them to extend Dashku for their products • I get a free Ninja Block from the guys at Ninja Blocks, who ask me to work for them • Dashku gets ripped off by tazaboard.com • Another IoT company asks to use it for their product
141. One Wednesday morning, I get an email…
142. Subject: Dashku @ Facebook
143. A developer tells me that their playing with it at work
144. He then asks if I’d like to come visit California and consider working for them
145. It was an exceptional opportunity, but my family needed me, so I had to pass on it
146. But it helped me put that episode in my life to rest
147. Thank You Maxime Beauchemin
148. If it wasn’t for SocketStream, I would not have built Dashku as fast as I did
149. Thank you Owen
150. About me Paul Jensen @paulbjensen Founder ANEPHENIX
151. I’m writing a Book
152. Available at: manning.com/jensen 50% off code: mljensen
153. Thank you