Marrying Front with Backend

Developer Week 2013

Transcript

1. Marrying front with back end @BastianHofmann
2. Let's talk about... In the next hour I want to talk a bit bout
3. Application architecture application architecture and with that I mean how to structure your code
4. Code and component re-use how you can re-use code and components
5. Rapid Development so that you can develop new features quickly without breaking the rest of your application or accumulation lots of technical debt
6. Handling large code-bases especially if your code base becomes larger and larger
7. Frameworks to the rescue Normally if faced with these problems people say: don't reinvent the wheel, let's look what's already out there, let's use some framework (or some libraries), or if you have a bad case of the not-invented-here syndrom in your company, you are going to develop your own
8. ? languages? one/their own product? agency? long lived, short lived projects? large codebases? frameworks?
9. Most Web Frameworks are incomplete But I want to be a bit opinionated here: ...
10. they oftentimes either care about your backend on the server and serve html pages or apis
11. or they care about your frontend in the client
12. webserver HTML browser JS so the normal workflow of a web application is that some server code serves some html (and maybe some apis) to the browser, in the browser there is some (or a lot of) javascript that manipulates the DOM
13. webserver HTML browser JS Ajax and either links to other pages served by the server or does ajax requests back to the server. this concept is mostly true for either traditional apps as well as single page applications
14. webserver HTML browser JS Ajax but the code in both worlds, server and client, is seldom connected
15. duplication de duplication ode duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplicatio code duplicat code duplic code dup code du code cod co co which leads to lot of code duplication in templates, validation logic, models and lots of boiler plate code for communication between server and client
16. So? so, how can we change this? before i answer this question
17. A few words about me ...
18. I work at ResearchGate, the social network for scientists and researchers
19. ResearchGate gives science back to the people who make it happen. We help researchers build reputation and accelerate scientific progress. On their terms. ‟ the goal is to give...
20. over 2.9 million users
21. here some impressions of the page
22. you may have read about us in the news recently
23. http://gigaom.com/2013/06/05/heres-how-bill-gatess- researchgate-investment-might-change-the-world-for-the- better http://venturevillage.eu/researchgate
24. have this, and also work on some cool stuff
25. we are hiring
26. in addition to this i also speak frequently on conferences throughout the world
27. work and live in berlin
28. Questions? Ask by the way, if you have any questions throughout this talk, if you don't understand something, just raise your hand and ask. probably my fault anyways since i spoke to quickly or my accent was too bad
29. http://speakerdeck.com/u/bastianhofmann the slides will be available on speakerdeck
30. if we take a small peek into a possible future of web application development
31. Mojito http://developer.yahoo.com/cocktails/mojito/ there are frameworks already being worked on like mojito from yahoo
32. Meteor http://www.meteor.com or meteor, that try to bridge the gap between server and client. but they are still very alpha, good for prototyping, writing small apps, trying things out, but personally I wouldn't create a big million lines of code web application with 40+ developers yet
33. What about existing applications? and regardless of that, most of you have existing applications
34. Legacy Code with lots of legacy code
35. Incremental Refactoring the way to make the architecture of your existing application better and connect your backend with your frontend code is incremental refactoring, since you hopefully can't afford to sit down, stop development for 6 months and rewrite everything
36. many roads of course like always there are many ways to go about that, which totally depends on you existing code
37. http://backstage.soundcloud.com/2012/06/building-the- next-soundcloud/ some examples: soundcloud
38. http://twitter.github.io/flight/ twitter, check out flight.js
39. or what i'm going to talk about for the reminder of the talk, what we did at RG
40. http://www.youtube.com/watch?v=fqULJBBEVQE what's interesting about all these approaches and with all these totally different applications and totally different use cases and totally different technologies, that the underlying concepts people come up with are strikingly similar. basically it's thinking about your app in small components, that are as independent as possible. this may even make it directly into browsers: if you haven't heard about it yet, check out the proposed web components standard.
41. status quo so RG: when we started about over a year ago, the status quo was:
42. webserver loadbalancer pgsql memcached mongodb services classic web application, with some soa elements
43. webserver web application was written in php
44. self-written framework, Large, old codebase, some architecture and code smells, but also some nicely done things
45. MVC mvc-like
46. complicated routing
47. PHP templates templating done in with php scripts on the server
48. lazy DB queries in view thanks to doctrine1 with db queries executed in the view
49. on the frontend yui3, but every page was built with custom modules, not much reuse of existing modules but very basic ones (also some legacy pages with yui2, prototype.js and scriptaculous)
50. Duplication and only some Code re- use overall ...
51. We can do better so we sat down and thought ...
52. Components let's rethink the way we are building pages or our whole application and separate it into small components
53. so looking at a page (that's what our profile looked like over a year ago), you can identify lots of small components nested into each other on a page
54. Self contained our goal was that each component is ...
55. Can be addressed and rendered separately and ...
56. Server JS Browser JSON HTML HTML so it should have it's own url, and for seo reasons can just be included in a page rendered to a browser by the server or can be fetched separately or nested within other components by the apps javascript, where we only wanted to transport the data to the client and render it into html there
57. JS is part of the component the javascript to bind on dom events and manipulate the components should be part of the component
58. Share code between server and client and we wanted to share as much code between server and client as possible (considering our heterogeneous architectur, js client, php server)
59. Templates, Validation, Entities,... e.g. ..
60. It needs to be fast of course ... concerning performance as well as development speed and productivity
61. We called them Widgets
62. PHP Controller Mustache Template JavaScript view class Widget Providing data Handling browser events Displaying data so to sum it up an widget contains
63. Why no models in the frontend? http://backbonejs.org/#Model compared to flight.js, backbone we decided against having dedicated models in the frontend where views could subscribe on because of legacy reasons. so the data is coupled to the component, and if needed events synchronize data between components, a bit overhead, a bit duplication of data, but made development for us much faster, also we have a lot of different entities, so for us taking this route worked out very well. "
64. so if we go back to our page
65. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu we see, that it's actually kind of a tree structure, for the sake of this presentation i simplified it slightly, actually our profile consists of over 200 components
66. Remember: self contained remember all these 200 components are supposed to be self contained
67. Do not fetch data directly so something you shouldn't do is fetching this data directly on you server when composing a page or a subpage
68. Sssssssllllooooowww because that would be really slow
69. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu Account Account Account Account Account Publication1 Publication2 Publication3 if we take our simplified example, a lot of components need the account of the user that's displayed, while that could be solved with in memory caching, a list of publications each need the publication entity, if we display 20 publications, that would mean 20 database queries, doing it in one query would be much faster though, and you have lot's of stuff like this
70. Require stuff so instead of fetching stuff directly, it's way better to require stuff
71. http://www.infoq.com/presentations/Evolution-of-Code- Design-at-Facebook/ this concept is actually not that new, one small company who is doing this very much is for example facebook, who also did a very good talk about this a few years back
72. so how does it work? around all your components that just post data requirements you need an instance called a preparer
73. Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces Connector Implementations
74. Widget Widget Widget Widget Preparer Fetch Requirements Resolver Resolver Services Connector Interfaces Connector Implementations the preparer iterates over all components and fetches their requirments
75. Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces Connector Implementations Batch requirements and pass them to resolvers it batches them together as intelligently as possible, passes the requirements to resolvers
76. Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces Connector Implementations Call Services as effective as possible (Multi-GET,...) which call the services/storages/etc as effective as possible, or just execute some service class methods
77. Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces Connector Implementations Attach fetched data to Requirements and pass them back to the preparer the fetched data is attached to the requirements, passed back to the preparer
78. Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces Connector Implementations Distribute fetched data to the widgets that required it who in turn distributes the fetched data to the components
79. so in code a component could specify its requirements like this
80. EntityRequirement in our implementation we separate between
81. Service Requirement and .. which is just a technical reason
82. Request Cache nice thing is you can cache this stuff intelligently in memory of your current request
83. Multi-GET or use stuff like multi-get
84. Futures or futures / promises
85. Required / Optional requirements can be required or optional, meaning either it's ok if a call fails or not (important for error handling). if it's not ok we just deactivate the whole component
86. Widget Requirement but it does not need to be only data, for example including another widget is also a requirement for us. that's the way we actually specify the component trees seen earlier
87. RequestData Requirement or getting data out of the request scope can be a requirement
88. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu so in our example
89. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu in the first iteration the preparer asks the profile widget for its requirement that can be data as well as including all the subwidgets
90. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu next the preparer asks all the new widgets it got to know in the last iteration for their requirements
91. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu and so on
92. Data dependencies within a widget but oftentimes we have .. so it gets a bit more complex
93. => Callbacks we can use ... for that
94. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu CALLBACK CALLBACK CALLBACK so in the first iteration the profile widget returns its requirements as well as a callback function that should be executed once all the requirements have been fullfilled
95. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu CALLBACK CALLBACK CALLBACK after the first iteration this callback is then executed while collecting all the requirements of the new subwidgets. the callback can then return further requirments as well as also further callbacks
96. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu CALLBACK CALLBACK CALLBACK and so on, this iterative process is done as long as there are components or callbacks available that return new requirements
97. in code... you can imagine if you are not careful this can become some kind of a callback hell
98. PHP 5.5 Generators https://wiki.php.net/rfc/generators that's why i'm very excited about generators in php 5.5.
99. public function collect() { yield array( new EntityRequirement( 'account', Account::class, array('accountId' => $this->requestContext->getAccountId()) ), ); yield array( new ServiceRequirement( 'scienceDisciplines', AccountService::class, 'getScienceDisciplines', array('account' => $this->account) ) ); } because then you can write it like this:
100. Data dependencies between Widgets also oftentimes you may have... if only for performance reasons,
101. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu e.g. for a list of publications that you got from a solr search, you may not want to go to the database again to fetch each publication in each list item, but since a list item is supposed to be renderable separately as well, it needs this logic in there
102. Prefills what you can do then is prefilling this data from the parent widget
103. in code
104. that means if the subwidget has a requirment like this, the preparer would directly put the prefilled data in there instead of passing it on to the resolvers
105. Whoa, this looks awfully complicated to debug you're right, it takes a bit of time to get used to program this way, but everyone who joined our company in the last few months since we are doing it says after a week or so, that they actually can't imagine working any other way again (i'll talk more about the benefits later)
106. but still it's complex, so a good thing is taking a bit of effort to make it as transparent as possible to see what happens in your application (which is a very good thing anyways). we have a debug toolbar that shows all the component on the page
107. and you can also drill down into all the iterations the preparer did
108. and how long each step took DEMO
109. Rendering so we have now all the data fetched together on our server lets render it now
110. HTML i said earlier we want our widget tree to be renderable on the server side as html
111. JSON and just put out as json and then be rendered on the client side by our javascript
112. Templates and of course we want to reuse our templates
113. Mustache } http://mustache.github.com/ so we decided to use mustache templates, we are actually using twitters mustache implementation called hogan
114. that's what a template may look like
115. Helper Methods there is also a php implementation of mustache, but it's not the fastest and one thing is, that mustache oftentimes relies on small helper functions
116. •nl2br •truncate •pluralize •wordwrap •highlight •... like ... if you now use a php implementation on the server and a js implementation on the client, we have to develop these functions twice ... not a good idea
117. http://pecl.php.net/package/v8js V8js that why we actually execute javascript through a php extension that makes the v8 js library available on the server. so we are executing the same code on the server and on the client to render the same templates. suprise: it's acutally really really fast
118. JavaScript so we handled templates and rendering, the backend controllers that provide the data, on the javascript side in the client each component can have a view object to bind on DOM events
119. WidgetViews
120. here is an example
121. Loading widgets from JavaScript of course since every widget has its own url its very easy to load other widgets from the server and then put them somewhere in the dom
122. templates and javascript is all loaded automatically
123. Benefits
124. Enables developers to only focus on their components
125. Easy re-using of components
126. Easier Refactoring
127. Error Handling also error handling can be really simplified
128. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu
129. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu EXCEPTION if an exception occurs in one of our components while fulfilling a requirement that is not optional
130. Profile Publications Publication Publication Publication LeftColumn Image Instiution Menu we just deactivate this component and the rest of the page is still functional
131. Testing also this approach simplifies testing, and with that i mean A/B testing
132. we are doing this excessively, and have over 90 or so experiments with sometimes over 20 variants running. the way we can do this quickly and easily is by just switching components for the different variants
133. Bandit Algorithm http://untyped.com/untyping/2011/02/11/stop-ab-testing- and-make-out-like-a-bandit/ tip to read up on testing:
134. Caching of Components also you can cache the components easily in a varnish
135. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu <esi:include src="..." /> because every component has it's own url you can just render out a esi placeholder instead of the widget to tell varnish to fetch it separately and provided it has caching headers, get it out of the cache
136. Load components asynchronously the same way you can also load components of the widget tree asynchronously
137. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu <div id="placeholder"></div> <script>loadWidget('/aboutMe', function(w) { w.render({ replace : '#placeholder' }); })</script> so instead of rendering the widget you render a placeholder dom element and a script tag that loads the widget with an ajax request and then renders it on the client side
138. BigPipe https://www.facebook.com/note.php? note_id=389414033919 even more sophisticated: something facebook calls bigpipe (we call it nozzle)
139. if you look at your widget tree you can mostly identify larger parts, which are widgets itself
140. Profile Menu Header LeftColumn RightColumn like this, so what you can do to dramatically increase the perceived load time is prioritizing the rendering
141. so our http request looks like this, first you compute and render the important parts of the page, like the top menu and the profile header as well as the rest of the layout, for the left column and right column which are expensive to compute you just render placeholders and then flush the content to the client so that the browser already renders this
142. still in the same http request you render out the javascript needed to make the already rendered components work, so people can use the menu for example
143. still in the same http request you then compute the data for the left column and render out some javascript that takes this data and renders it into the components template client side and then replaces the placeholder with the rendered template
144. still in the same request you then can do this with the right column -> flush content as early as possible, don't wait for the whole site to be computed
145. pushState and if you are at that, when you switch pages, you can also just load the differences between and use pushState to change the url (if supported) to make your app faster
146. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Instiution Menu
147. Profile Questions Question Question Question AboutMe LeftColumn Image Instiution Menu
148. Conclusion?
149. Think about your architecture
150. Refactor and make it better continuously
151. Frontend and backend are part of the same application
152. Don't rewrite your whole codebase in one go
153. http://twitter.com/BastianHofmann http://profiles.google.com/bashofmann http://lanyrd.com/people/BastianHofmann http://speakerdeck.com/u/bastianhofmann https://github.com/bashofmann https://www.researchgate.net/profile/Bastian_Hofmann/ mail@bastianhofmann.de thanks, you can contact me on any of these platforms or via mail.