Marrying front with back end

Marrying front with back end

from PHPBenelux 2014

8e82eb7e128a14a16d642ae55227339b?s=128

Bastian Hofmann

January 25, 2014
Tweet

Transcript

  1. Marrying front with back end @BastianHofmann

  2. Let's talk about... In the next hour I want to

    talk a bit about
  3. Application architecture application architecture and with that I mean how

    to structure your code
  4. Code and component re-use how you can re-use code and

    components
  5. Rapid Development so that you can develop new features quickly

    without breaking the rest of your application or accumulation lots of technical debt
  6. Handling large code-bases especially if your code base becomes larger

    and larger, developed by big, growing teams
  7. Frameworks to the rescue Normally if faced with these problems

    people say: don't reinvent the wheel, let's look what's already out there, let's use some framework (or some libraries), or if you have a bad case of the not-invented-here syndrom in your company, you are going to develop your own
  8. ? languages? one/their own product? agency? long lived, short lived

    projects? large codebases? frameworks?
  9. But I want to be a bit opinionated here: ...

  10. Most Web Frameworks are incomplete

  11. they oftentimes either care about your backend on the server

    and serve html pages or apis
  12. or they care about your frontend in the client

  13. webserver HTML browser JS so the normal flow of a

    web application is that some server code serves some html (and maybe some apis) to the browser, in the browser there is some (or a lot of) javascript that manipulates the DOM
  14. webserver HTML browser JS Ajax and either links to other

    pages served by the server or does ajax requests back to the server. this concept is mostly true for either traditional apps as well as single page applications
  15. webserver HTML browser JS Ajax but the code in both

    worlds, server and client, is seldom connected
  16. de duplication ode duplication code duplication code duplication code duplication

    code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplicatio code duplicat code duplic code dup code du code cod co co which leads to lot of code duplication in templates, validation logic, models and lots of boiler plate code for communication between server and client, marshalling data
  17. So? so, how can we change this? before i answer

    this question
  18. A few words about me ...

  19. I work at ResearchGate, the social network for scientists and

    researchers
  20. ResearchGate gives science back to the people who make it

    happen. We help researchers build reputation and accelerate scientific progress. On their terms. ‟ the goal is to give...
  21. over 3 million users

  22. here some impressions of the page

  23. you may have read about us in the news recently

  24. have this, and also work on some cool stuff

  25. have this, and also work on some cool stuff

  26. have this, and also work on some cool stuff

  27. we are hiring

  28. Questions? Ask by the way, if you have any questions

    throughout this talk, if you don't understand something, just raise your hand and ask.
  29. http://speakerdeck.com/u/bastianhofmann the slides will be available on speakerdeck

  30. if we take a small peek into a possible future

    of web application development
  31. Meteor http://www.meteor.com or meteor, that try to bridge the gap

    between server and client. but they are still very alpha, good for prototyping, writing small apps, trying things out, but personally I wouldn't create a big million lines of code web application with 40+ developers yet
  32. What about existing applications? and regardless of that, most of

    you have existing applications
  33. Legacy Code with lots of legacy code

  34. Incremental Refactoring the way to make the architecture of your

    existing application better and connect your backend with your frontend code is incremental refactoring, since you hopefully can't afford to sit down, stop development for 6 months and rewrite everything
  35. many roads of course like always there are many ways

    to go about that, which totally depends on you existing code
  36. or what i'm going to talk about for the reminder

    of the talk, what we did at RG
  37. status quo so RG: when we started about over a

    year ago, the status quo was:
  38. webserver loadbalancer pgsql memcached mongodb services classic web application, with

    some soa elements
  39. webserver web application was written in php

  40. self-written framework, Large, old codebase, some architecture and code smells,

    but also some nicely done things
  41. MVC mvc-like, one page, one controller, one template

  42. complicated routing

  43. PHP templates templating done in with php scripts on the

    server
  44. on the frontend yui3, but every page was built with

    custom modules, not much reuse of existing modules but very basic ones (also some legacy pages with yui2, prototype.js and scriptaculous)
  45. Duplication and only some Code re- use overall ...

  46. We can do better so we sat down and thought

    ...
  47. Components let's rethink the way we are building pages or

    our whole application and separate it into small components
  48. so looking at a page (that's what our profile looked

    like over a year ago), you can identify lots of small components nested into each other on a page
  49. Self contained our goal was that each component is ...

  50. Can be addressed and rendered separately and ...

  51. Server JS Browser JSON HTML HTML so it should have

    it's own url, and for seo reasons can just be included in a page rendered to a browser by the server or can be fetched separately or nested within other components by the apps javascript, where we only wanted to transport the data to the client and render it into html there
  52. None
  53. JS is part of the component the javascript to bind

    on dom events and manipulate the components should be part of the component
  54. CSS can be part of the component

  55. Share code between server and client and we wanted to

    share as much code between server and client as possible (considering our heterogeneous architectur, js client, php server)
  56. Templates, Validation, Entities,... e.g. ..

  57. It needs to be fast of course ... concerning performance

    as well as development speed and productivity
  58. We called them Widgets

  59. PHP Controller Mustache Template JavaScript view class Widget Providing data

    Handling browser events Displaying data CSS so to sum it up an widget contains
  60. so if we go back to our page

  61. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Institution Menu

    we see, that it's actually kind of a tree structure, for the sake of this presentation i simplified it slightly, actually our profile consists of over 200 components
  62. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Server

    Request Response Institution i said earlier, that all these components have their own URL. So they can be requested and rendered by the client seperately. For better user experience on the first load or SEO reasons these components can also be bundled together and rendered in a single page load on the backend.
  63. LeftColumn Image Menu Server Request Response if we want to

    fetch the left column separately it looks like this
  64. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Institution Menu

    so you have this tree
  65. Widget Requirement this tree structure is achieved through so called

    widget requirements, why they are called requirements
  66. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution

    that means the profile widget requires: publications, aboutMe, leftColumn and institution
  67. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution

    and Publications requires several publicationItem widgets and leftColumn the Image and the Menu widget
  68. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution

  69. Remember: self contained remember all these 200 components are supposed

    to be self contained
  70. Do not fetch data directly so something you shouldn't do

    is fetching this data directly on you server when composing a page or a subpage
  71. Sssssssllllooooowww because that would be really slow

  72. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Account

    Account Account Account Account Publication1 Publication2 Publication3 Institution if we take our simplified example, a lot of components need the account of the user that's displayed, while that could be solved with in memory caching, a list of publications each need the publication entity, if we display 20 publications, that would mean 20 database queries, doing it in one query would be much faster though, and you have lot's of stuff like this
  73. Require stuff so instead of fetching stuff directly, it's way

    better to require stuff (that's also why building up the widget tree was done with widget "requirements"
  74. http://www.infoq.com/presentations/Evolution-of-Code- Design-at-Facebook/ this concept is actually not that new, one

    small company who is doing this very much is for example facebook, who also did a very good talk about this a few years back
  75. so how does it work? around all your components that

    just state data (or widget) requirements you need an instance called a preparer
  76. Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces

    Connector Implementations
  77. Widget Widget Widget Widget Preparer Fetch Requirements Resolver Resolver Services

    Connector Interfaces Connector Implementations the preparer iterates over all components and fetches their requirments
  78. Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces

    Connector Implementations Batch requirements and pass them to resolvers it batches them together as intelligently as possible, passes the requirements to resolvers
  79. Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces

    Connector Implementations Call Services as effective as possible (Multi-GET,...) which call the services/storages/etc as effective as possible, or just execute some service class methods
  80. Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces

    Connector Implementations Attach fetched data to Requirements and pass them back to the preparer the fetched data is attached to the requirements, passed back to the preparer
  81. Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces

    Connector Implementations Distribute fetched data to the widgets that required it who in turn distributes the fetched data to the components
  82. class Widget { /** @var Request */ public $request; public

    $account; public $scienceDisciplines; public function collect() { return new RequirementCollection([ new EntityRequirement( 'account', Account::class, ['id' => $this->request->get('id')] ), ], function() { return new RequirementCollection([ new ServiceRequirement( 'scienceDisciplines', AccountService::class, 'getScienceDisciplines', ['account' => $this->account] ) ]); }); } } in code... you can imagine if you are not careful this can become some kind of a callback hell
  83. Request Cache nice thing is you can cache this stuff

    intelligently in memory of your current request
  84. Multi-GET or use stuff like multi-get

  85. Futures or futures / promises

  86. class Widget { /** @var Request */ public $request; public

    $account; public $scienceDisciplines; public function collect() { return new RequirementCollection([ new EntityRequirement( 'account', Account::class, ['id' => $this->request->get('id')] ), ], function() { return new RequirementCollection([ new ServiceRequirement( 'scienceDisciplines', AccountService::class, 'getScienceDisciplines', ['account' => $this->account] ) ]); }); } } as you are seeing here
  87. Data dependencies within a widget but oftentimes we have ..

    so it gets a bit more complex
  88. => Callbacks we can use ... for that

  89. public function collect() { return new RequirementCollection([ new EntityRequirement( 'account',

    Account::class, ['id' => $this->request->get('id')] ), ], function() { return new RequirementCollection([ new ServiceRequirement( 'scienceDisciplines', AccountService::class, 'getScienceDisciplines', ['account' => $this->account] ) ]); }); } in code... you can imagine if you are not careful this can become some kind of a callback hell
  90. PHP 5.5 Generators http://php.net/manual/de/language.generators.overview.php that's why i'm very excited about

    generators in php 5.5.
  91. <?php function someGenerator() { yield 'start of the generator'; for

    ($i = 1; $i <= 4; $i++) { // Note that $i is preserved between yields. if ($i % 2 === 0) { yield $i; } } if (rand(1, 2) === 1) { // this closes the generator return; } yield anotherFunction(); // this is also executed } function anotherFunction() { return 'result from another function'; } foreach (someGenerator() as $value) { echo "$value\n"; } start  of  the  generator 2 4 result  from  another  function start  of  the  generator 2 4
  92. public function collect() { yield [ new EntityRequirement( 'account', Account::class,

    ['id' => $this->request->get('id')] ), ]; yield [ new ServiceRequirement( 'scienceDisciplines', AccountService::class, 'getScienceDisciplines', ['account' => $this->account] ) ]; } because then you can write it like this:
  93. PHP 5.4 Traits http://us1.php.net/trait that's why i'm very excited about

    generators in php 5.5.
  94. yield [ serviceRequirement( 'job', JobService::getCall()->getById(248) ) ];

  95. trait ServiceRequirementFactory { /** @return $this */ public static function

    getCall() { return new ServiceRequirementFactoryProxy( ! ! ! static::class ! ! ! ); } }
  96. class ServiceRequirementFactoryProxy { private $className; public function __construct($className) { $this->className

    = $className; } public function __call($name, $arguments) { return [ 'serviceClass' => $this->className, 'methodName' => $name, 'arguments' => $arguments ]; } }
  97. Data dependencies between Widgets also oftentimes you may have... if

    only for performance reasons,
  98. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution

    e.g. for a list of publications that you got from a solr search, you may not want to go to the database again to fetch each publication in each list item, but since a list item is supposed to be renderable separately as well, it needs this logic in there
  99. Prefills what you can do then is prefilling this data

    from the parent widget
  100. class PublicationKeywordSearch { public $publications; public $publicationListItems = []; public

    function collect() { yield [ serviceRequirement( 'publications', PublicationService::getCall()->getByKeywords( ['virology', 'cancer'], 10, 0 ) ) ]; foreach ($this->publications as $publication) { yield new WidgetRequirement( 'publicationListItems', PublicationItem::CLASS, [ ! ! ! ! ! ! ! ! 'publicationId' => $publication->getId(), ! ! ! ! ! ! ! ! 'publication' => $publication ! ! ! ! ! ! ! ] ); } } } in code
  101. class PublicationItem { public $publicationId; public $publication; public function collect()

    { yield new RequestDataRequirement('publicationId'); yield [ new EntityRequirement( 'publication', Publication::class, ['id' => $this->publicationId] ) ]; } } that means if the subwidget has a requirment like this, the preparer would directly put the prefilled data in there instead of passing it on to the resolvers
  102. Whoa, this looks awfully complicated to debug you're right, it

    takes a bit of time to get used to program this way, but everyone who joined our company in the last few months since we are doing it says after a week or so, that they actually can't imagine working any other way again (i'll talk more about the benefits later)
  103. but still it's complex, so a good thing is taking

    a bit of effort to make it as transparent as possible to see what happens in your application (which is a very good thing anyways). we have a debug toolbar that shows all the component on the page
  104. and you can also drill down into all the iterations

    the preparer did
  105. None
  106. None
  107. None
  108. So that was the backend

  109. we now have the date for all the components on

    our page
  110. Rendering so we have now all the data fetched together

    on our server lets render it now
  111. HTML i said earlier we want our widget tree to

    be renderable on the server side as html
  112. JSON and just put out as json and then be

    rendered on the client side by our javascript
  113. Templates and of course we want to reuse our templates

  114. Mustache } http://mustache.github.com/ so we decided to use mustache templates,

    we are actually using twitters mustache implementation called hogan
  115. <div id="{{widgetId}}"> <h3> Publications <a href="javascript:" class="js-showAll"> ({{count}}) </a> </h3>

    <ul> {{#publicationItems}} <li>{{{.}}}</li> {{/publicationItems}} </ul> {{#showLegalFooter}} {{{legalFooter}}} {{/showLegalFooter}} </div> that's what a template may look like
  116. Helper Methods there is also a php implementation of mustache,

    but it's not the fastest and one thing is, that mustache oftentimes relies on small helper functions
  117. •nl2br •truncate •pluralize •wordwrap •highlight •... like ... if you

    now use a php implementation on the server and a js implementation on the client, we have to develop these functions twice ... not a good idea
  118. http://pecl.php.net/package/v8js V8js that why we actually execute javascript through a

    php extension that makes the v8 js library available on the server. so we are executing the same code on the server and on the client to render the same templates. suprise: it's acutally really really fast
  119. JavaScript so we handled templates and rendering, the backend controllers

    that provide the data, on the javascript side in the client each component can have a view object to bind on DOM events
  120. WidgetViews

  121. <div id="{{widgetId}}"> <h3> Publications <a href="javascript:" class="js-showAll"> ({{count}}) </a> </h3>

    <ul> {{#publicationItems}} <li>{{{.}}}</li> {{/publicationItems}} </ul> {{#showLegalFooter}} {{{legalFooter}}} {{/showLegalFooter}} </div> remember this template
  122. YUI.add('views.publicationList', function (Y) { Y.views.publicationList = Y.Base.create(Y.WidgetView, { events :

    { '.js-showALl' : { click : 'showALl' } }, showAll : function () { var node = this.get('container'); node.addClass('loading'); Y.loadWidget(‚/publications/all', { keywords : this.data.keywords }, function (widget) { widget.render({ replace : node }); }); } }); }); here is an example
  123. if you think about this a bit more, every component

    having it's own template, javascript and maybe css
  124. http://twitter.github.io/flight/ twitter, check out flight.js

  125. http://facebook.github.io/react/ by the way: if you’ll have a look at

    react: it is strikingly similar
  126. http://www.youtube.com/watch?v=fqULJBBEVQE .. this is exactly what web components is about.

    they have their own html, css, js and are even more sandboxed through a shadow dom to limit interactions between different components. an example for webcomponents are the browser controls for videos or forms by the way.
  127. http://www.polymer-project.org/

  128. <polymer-element name="tk-element" attributes="owner color"> <template> <span>This is <strong>{{owner}}</strong>'s tk-element.</span> <span

    id="el" style="color: {{color}}">Not ready...</span> </template> <script> Polymer('tk-element', { owner: "Bastian", color: "red", ready: function() { this.$.el.innerText = this.owner + " is ready!"; } }); </script> </polymer-element>
  129. <!DOCTYPE html> <html> <head> <script src="bower_components/platform/platform.js"> </script> <link rel="import" href="elements/tk-element.html">

    </head> <body> <tk-element color="blue"></tk-element> </body> </html>
  130. Benefits

  131. Enables developers to only focus on their components

  132. Rapid prototyping

  133. Easier Refactoring even if the implementation of one component is

    very ugly and crude, at least it's encapsulated (hopefully tested) and you can either leave it or revisit this component later on
  134. Error Handling also error handling can be really simplified and

    end-user friendly
  135. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution

    so going back to our widget tree
  136. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu EXCEPTION

    Institution if an exception occurs in one of our components while fulfilling a requirement that is not optional
  137. Profile Publications Publication Publication Publication LeftColumn Image Menu Institution we

    just deactivate this component and the rest of the page is still functional
  138. Experiments (A/B Testing) also this approach simplifies testing, and with

    that i mean A/B testing
  139. we are doing this excessively, and have over 90 or

    so experiments with sometimes over 20 variants running. the way we can do this quickly and easily is by just switching components for the different variants
  140. Caching of Components also you can cache the components easily

    in a varnish
  141. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu <esi:include

    src="..." /> Institution because every component has it's own url you can just render out a esi placeholder instead of the widget to tell varnish to fetch it separately and provided it has caching headers, get it out of the cache
  142. Easy re-using of components

  143. None
  144. Load components asynchronously the same way you can also load

    components of the widget tree asynchronously
  145. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu <div

    id="placeholder"></div> <script>loadWidget('/aboutMe', function(w) { w.render({ replace : '#placeholder' }); })</script> Institution so instead of rendering the widget you render a placeholder dom element and a script tag that loads the widget with an ajax request and then renders it on the client side
  146. BigPipe https://www.facebook.com/note.php? note_id=389414033919 even more sophisticated: something facebook calls bigpipe

    (we call it nozzle)
  147. if you look at your widget tree you can mostly

    identify larger parts, which are widgets itself
  148. Profile Menu Header LeftColumn RightColumn like this, so what you

    can do to dramatically increase the perceived load time is prioritizing the rendering
  149. so our http request looks like this, first you compute

    and render the important parts of the page, like the top menu and the profile header as well as the rest of the layout, for the left column and right column which are expensive to compute you just render placeholders and then flush the content to the client so that the browser already renders this
  150. still in the same http request you render out the

    javascript needed to make the already rendered components work, so people can use the menu for example
  151. still in the same http request you then compute the

    data for the left column and render out some javascript that takes this data and renders it into the components template client side and then replaces the placeholder with the rendered template
  152. still in the same request you then can do this

    with the right column -> flush content as early as possible, don't wait for the whole site to be computed
  153. pushState and if you are at that, when you switch

    pages, you can also just load the differences between and use pushState to change the url (if supported) to make your app faster
  154. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution

  155. Profile Questions Question Question Question AboutMe LeftColumn Image Menu Institution

  156. Conclusion?

  157. Think about your architecture

  158. Refactor and make it better continuously

  159. Frontend and backend are part of the same application

  160. Don't rewrite your whole codebase in one go

  161. https://joind.in/10279

  162. http://twitter.com/BastianHofmann http://lanyrd.com/people/BastianHofmann http://speakerdeck.com/u/bastianhofmann mail@bastianhofmann.de thanks, you can contact me on

    any of these platforms or via mail.
  163. Start of an opensource implementation https://github.com/P2EE