Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Marrying front with back end

Marrying front with back end

Presentation from RG Developer Day

Bastian Hofmann

November 30, 2013
Tweet

More Decks by Bastian Hofmann

Other Decks in Programming

Transcript

  1. Marrying front
    with back end
    @BastianHofmann

    View Slide

  2. Let's talk about...
    In the next hour I want to talk a bit about

    View Slide

  3. Application
    architecture
    application architecture and with that I mean how to structure your code

    View Slide

  4. Code and
    component
    re-use
    how you can re-use code and components

    View Slide

  5. Rapid
    Development
    so that you can develop new features quickly without breaking the rest of your application or
    accumulation lots of technical debt

    View Slide

  6. Handling large
    code-bases
    especially if your code base becomes larger and larger, developed by big, growing teams

    View Slide

  7. Frameworks to the
    rescue
    Normally if faced with these problems people say: don't reinvent the wheel, let's look what's
    already out there, let's use some framework (or some libraries), or if you have a bad case of
    the not-invented-here syndrom in your company, you are going to develop your own

    View Slide

  8. ?
    languages? one/their own product? agency? long lived, short lived projects? large codebases?
    frameworks?

    View Slide

  9. Most Web Frameworks
    are incomplete
    But I want to be a bit opinionated here: ...

    View Slide

  10. they oftentimes either care about your backend on the server and serve html pages or apis

    View Slide

  11. or they care about your frontend in the client

    View Slide

  12. webserver
    HTML
    browser
    JS
    so the normal flow of a web application is that some server code serves some html (and
    maybe some apis) to the browser, in the browser there is some (or a lot of) javascript that
    manipulates the DOM

    View Slide

  13. webserver
    HTML
    browser
    JS
    Ajax
    and either links to other pages served by the server or does ajax requests back to the server.
    this concept is mostly true for either traditional apps as well as single page applications

    View Slide

  14. webserver
    HTML
    browser
    JS
    Ajax
    but the code in both worlds, server and client, is seldom connected

    View Slide

  15. duplication
    de duplication
    ode duplication
    code duplication
    code duplication
    code duplication
    code duplication
    code duplication
    code duplication
    code duplication
    code duplication
    code duplication
    code duplication
    code duplication
    code duplicatio
    code duplicat
    code duplic
    code dup
    code du
    code
    cod
    co
    co
    which leads to lot of code duplication in templates, validation logic, models and lots of boiler
    plate code for communication between server and client, marshalling data

    View Slide

  16. So?
    so, how can we change this? before i answer this question

    View Slide

  17. A few words
    about us
    ...

    View Slide

  18. I work at ResearchGate, the social network for scientists and researchers

    View Slide

  19. ResearchGate gives
    science back to the
    people who make it
    happen.
    We help researchers
    build reputation and
    accelerate scientific
    progress.
    On their terms.

    the goal is to give...

    View Slide

  20. over 3 million users

    View Slide

  21. here some impressions of the page

    View Slide

  22. you may have read about us in the news recently

    View Slide

  23. have this, and also work on some cool stuff

    View Slide

  24. we are hiring

    View Slide

  25. Questions? Ask
    by the way, if you have any questions throughout this talk, if you don't understand
    something, just raise your hand and ask.

    View Slide

  26. http://speakerdeck.com/u/bastianhofmann
    the slides will be available on speakerdeck

    View Slide

  27. View Slide

  28. if we take a small peek into a possible future of web application development

    View Slide

  29. Mojito
    http://developer.yahoo.com/cocktails/mojito/
    there are frameworks already being worked on like mojito from yahoo

    View Slide

  30. Meteor
    http://www.meteor.com
    or meteor, that try to bridge the gap between server and client. but they are still very alpha,
    good for prototyping, writing small apps, trying things out, but personally I wouldn't create a
    big million lines of code web application with 40+ developers yet

    View Slide

  31. What about existing
    applications?
    and regardless of that, most of you have existing applications

    View Slide

  32. Legacy Code
    with lots of legacy code

    View Slide

  33. Incremental
    Refactoring
    the way to make the architecture of your existing application better and connect your
    backend with your frontend code is incremental refactoring, since you hopefully can't afford
    to sit down, stop development for 6 months and rewrite everything

    View Slide

  34. many roads
    of course like always there are many ways to go about that, which totally depends on you
    existing code

    View Slide

  35. http://backstage.soundcloud.com/2012/06/building-the-
    next-soundcloud/
    some examples: soundcloud

    View Slide

  36. http://twitter.github.io/flight/
    twitter, check out flight.js

    View Slide

  37. http://www.youtube.com/watch?v=fqULJBBEVQE
    what's interesting about all these approaches and with all these totally different applications
    and totally different use cases and totally different technologies, that the underlying concepts
    people come up with are strikingly similar. basically it's thinking about your app in small
    components, that are as independent as possible. this may even make it directly into
    browsers: if you haven't heard about it yet, check out the proposed web components
    standard.

    View Slide

  38. or what i'm going to talk about for the reminder of the talk, what we did at RG

    View Slide

  39. status quo
    so RG: when we started about over a year ago, the status quo was:

    View Slide

  40. webserver
    loadbalancer
    pgsql memcached mongodb
    services
    classic web application, with some soa elements

    View Slide

  41. webserver
    web application was written in php

    View Slide

  42. self-written framework, Large, old codebase, some architecture and code smells, but also
    some nicely done things

    View Slide

  43. MVC
    mvc-like, one page, one controller, one template

    View Slide

  44. complicated routing

    View Slide

  45. PHP templates
    templating done in with php scripts on the server

    View Slide

  46. lazy DB queries in
    view
    thanks to doctrine1 with db queries executed in the view

    View Slide

  47. on the frontend yui3, but every page was built with custom modules, not much reuse of
    existing modules but very basic ones (also some legacy pages with yui2, prototype.js and
    scriptaculous)

    View Slide

  48. Duplication and
    only some Code re-
    use
    overall ...

    View Slide

  49. We can do
    better
    so we sat down and thought ...

    View Slide

  50. Components
    let's rethink the way we are building pages or our whole application and separate it into small
    components

    View Slide

  51. so looking at a page (that's what our profile looked like over a year ago), you can identify lots
    of small components nested into each other on a page

    View Slide

  52. Self contained
    our goal was that each component is ...

    View Slide

  53. Can be addressed and
    rendered separately
    and ...

    View Slide

  54. Server
    JS
    Browser
    JSON
    HTML
    HTML
    so it should have it's own url, and for seo reasons can just be included in a page rendered to
    a browser by the server or can be fetched separately or nested within other components by
    the apps javascript, where we only wanted to transport the data to the client and render it
    into html there

    View Slide

  55. View Slide

  56. JS is part of the
    component
    the javascript to bind on dom events and manipulate the components should be part of the
    component

    View Slide

  57. Share code
    between server
    and client
    and we wanted to share as much code between server and client as possible (considering our
    heterogeneous architectur, js client, php server)

    View Slide

  58. Templates,
    Validation,
    Entities,...
    e.g. ..

    View Slide

  59. It needs to be fast
    of course ...
    concerning performance as well as development speed and productivity

    View Slide

  60. We called them
    Widgets

    View Slide

  61. PHP Controller Mustache
    Template
    JavaScript
    view class
    Widget
    Providing data
    Handling browser events
    Displaying data
    so to sum it up an widget contains

    View Slide

  62. so if we go back to our page

    View Slide

  63. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu
    we see, that it's actually kind of a tree structure, for the sake of this presentation i simplified
    it slightly, actually our profile consists of over 200 components

    View Slide

  64. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu
    Server
    Request
    Response
    i said earlier, that all these components have their own URL. So they can be requested and
    rendered by the client seperately. For better user experience on the first load or SEO reasons
    these components can also be bundled together and rendered in a single page load on the
    backend.

    View Slide

  65. Widget
    Requirement
    this tree structure is achieved through so called widget requirements, why they are called
    requirements: more to this later

    View Slide

  66. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu
    that means the profile widget requires: publications, aboutMe, leftColumn and institution

    View Slide

  67. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu
    and Publications requires several publicationItem widgets and leftColumn the Image and the
    Menu widget

    View Slide

  68. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu

    View Slide

  69. Remember:
    self contained
    remember all these 200 components are supposed to be self contained

    View Slide

  70. Do not fetch data
    directly
    so something you shouldn't do is fetching this data directly on you server when composing a
    page or a subpage

    View Slide

  71. Sssssssllllooooowww
    because that would be really slow

    View Slide

  72. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu
    Account
    Account
    Account
    Account
    Account
    Publication1
    Publication2
    Publication3
    if we take our simplified example, a lot of components need the account of the user that's
    displayed, while that could be solved with in memory caching, a list of publications each need
    the publication entity, if we display 20 publications, that would mean 20 database queries,
    doing it in one query would be much faster though, and you have lot's of stuff like this

    View Slide

  73. Require stuff
    so instead of fetching stuff directly, it's way better to require stuff (that's also why building
    up the widget tree was done with widget "requirements"

    View Slide

  74. http://www.infoq.com/presentations/Evolution-of-Code-
    Design-at-Facebook/
    this concept is actually not that new, one small company who is doing this very much is for
    example facebook, who also did a very good talk about this a few years back

    View Slide

  75. so how does it work? around all your components that just state data (or widget)
    requirements you need an instance called a preparer

    View Slide

  76. Widget Widget Widget Widget
    Preparer
    Resolver Resolver
    Services
    Connector Interfaces
    Connector Implementations

    View Slide

  77. Widget Widget Widget Widget
    Preparer
    Fetch Requirements
    Resolver Resolver
    Services
    Connector Interfaces
    Connector Implementations
    the preparer iterates over all components and fetches their requirments

    View Slide

  78. Widget Widget Widget Widget
    Preparer
    Resolver Resolver
    Services
    Connector Interfaces
    Connector Implementations
    Batch
    requirements
    and pass
    them to
    resolvers
    it batches them together as intelligently as possible, passes the requirements to resolvers

    View Slide

  79. Widget Widget Widget Widget
    Preparer
    Resolver Resolver
    Services
    Connector Interfaces
    Connector Implementations
    Call Services as
    effective as possible
    (Multi-GET,...)
    which call the services/storages/etc as effective as possible, or just execute some service
    class methods

    View Slide

  80. Widget Widget Widget Widget
    Preparer
    Resolver Resolver
    Services
    Connector Interfaces
    Connector Implementations
    Attach fetched
    data to
    Requirements and
    pass them back to
    the preparer
    the fetched data is attached to the requirements, passed back to the preparer

    View Slide

  81. Widget Widget Widget Widget
    Preparer
    Resolver Resolver
    Services
    Connector Interfaces
    Connector Implementations
    Distribute fetched
    data to the widgets
    that required it
    who in turn distributes the fetched data to the components

    View Slide

  82. so in code a component could specify its requirements like this

    View Slide

  83. Request Cache
    nice thing is you can cache this stuff intelligently in memory of your current request

    View Slide

  84. Multi-GET
    or use stuff like multi-get

    View Slide

  85. Futures
    or futures / promises

    View Slide

  86. Required / Optional
    requirements can be required or optional, meaning either it's ok if a call fails or not
    (important for error handling). if it's not ok we just deactivate the whole component

    View Slide

  87. Data
    dependencies
    within a widget
    but oftentimes we have .. so it gets a bit more complex

    View Slide

  88. => Callbacks
    we can use ... for that

    View Slide

  89. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu
    CALLBACK
    CALLBACK
    CALLBACK
    so in the first iteration the profile widget returns its requirements as well as a callback
    function that should be executed once all the requirements have been fullfilled

    View Slide

  90. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu
    CALLBACK
    CALLBACK
    CALLBACK
    after the first iteration this callback is then executed while collecting all the requirements of
    the new subwidgets. the callback can then return further requirments as well as also further
    callbacks

    View Slide

  91. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu
    CALLBACK
    CALLBACK
    CALLBACK
    and so on, this iterative process is done as long as there are components or callbacks
    available that return new requirements

    View Slide

  92. in code...
    you can imagine if you are not careful this can become some kind of a callback hell

    View Slide

  93. PHP 5.5 Generators
    http://php.net/manual/de/language.generators.overview.php
    that's why i'm very excited about generators in php 5.5.

    View Slide

  94. public function collect() {
    yield array(
    new EntityRequirement(
    'account',
    Account::class,
    array('accountId' => $this->requestContext->getAccountId())
    ),
    );
    yield array(
    new ServiceRequirement(
    'scienceDisciplines',
    AccountService::class,
    'getScienceDisciplines',
    array('account' => $this->account)
    )
    );
    }
    because then you can write it like this:

    View Slide

  95. Data
    dependencies
    between Widgets
    also oftentimes you may have...
    if only for performance reasons,

    View Slide

  96. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu
    e.g. for a list of publications that you got from a solr search, you may not want to go to the
    database again to fetch each publication in each list item, but since a list item is supposed to
    be renderable separately as well, it needs this logic in there

    View Slide

  97. Prefills
    what you can do then is prefilling this data from the parent widget

    View Slide

  98. in code

    View Slide

  99. that means if the subwidget has a requirment like this, the preparer would directly put the
    prefilled data in there instead of passing it on to the resolvers

    View Slide

  100. Whoa, this looks
    awfully
    complicated to
    debug
    you're right, it takes a bit of time to get used to program this way, but everyone who joined
    our company in the last few months since we are doing it says after a week or so, that they
    actually can't imagine working any other way again (i'll talk more about the benefits later)

    View Slide

  101. but still it's complex, so a good thing is taking a bit of effort to make it as transparent as
    possible to see what happens in your application (which is a very good thing anyways).
    we have a debug toolbar that shows all the component on the page

    View Slide

  102. and you can also drill down into all the iterations the preparer did

    View Slide

  103. View Slide

  104. View Slide

  105. View Slide

  106. So that was the
    backend

    View Slide

  107. we now have the date for all the components on our page

    View Slide

  108. Rendering
    so we have now all the data fetched together on our server lets render it now

    View Slide

  109. HTML
    i said earlier we want our widget tree to be renderable on the server side as html

    View Slide

  110. JSON
    and just put out as json and then be rendered on the client side by our javascript

    View Slide

  111. Templates
    and of course we want to reuse our templates

    View Slide

  112. Mustache
    }
    http://mustache.github.com/
    so we decided to use mustache templates, we are actually using twitters mustache
    implementation called hogan

    View Slide

  113. that's what a template may look like

    View Slide

  114. Helper Methods
    there is also a php implementation of mustache, but it's not the fastest and one thing is, that
    mustache oftentimes relies on small helper functions

    View Slide

  115. •nl2br
    •truncate
    •pluralize
    •wordwrap
    •highlight
    •...
    like ...
    if you now use a php implementation on the server and a js implementation on the client, we
    have to develop these functions twice ... not a good idea

    View Slide

  116. http://pecl.php.net/package/v8js
    V8js
    that why we actually execute javascript through a php extension that makes the v8 js library
    available on the server. so we are executing the same code on the server and on the client to
    render the same templates.
    suprise: it's acutally really really fast

    View Slide

  117. JavaScript
    so we handled templates and rendering, the backend controllers that provide the data, on the
    javascript side in the client each component can have a view object to bind on DOM events

    View Slide

  118. WidgetViews

    View Slide

  119. here is an example

    View Slide

  120. if you think about this a bit more, every component having it's own template, javascript and
    maybe css

    View Slide

  121. http://www.youtube.com/watch?v=fqULJBBEVQE
    .. this is exactly what web components is about. they have their own html, css, js and are
    even more sandboxed through a shadow dom to limit interactions between different
    components. an example for webcomponents are the browser controls for videos or forms by
    the way.

    View Slide

  122. Benefits

    View Slide

  123. Enables developers
    to only focus on
    their components

    View Slide

  124. Easier Refactoring
    even if the implementation of one component is very ugly and crude, at least it's
    encapsulated (hopefully tested) and you can either leave it or revisit this component later on

    View Slide

  125. Error Handling
    also error handling can be really simplified and end-user friendly

    View Slide

  126. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu
    so going back to our widget tree

    View Slide

  127. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu
    EXCEPTION
    if an exception occurs in one of our components while fulfilling a requirement that is not
    optional

    View Slide

  128. Profile Publications Publication
    Publication
    Publication
    LeftColumn Image
    Instiution
    Menu
    we just deactivate this component and the rest of the page is still functional

    View Slide

  129. Testing
    also this approach simplifies testing, and with that i mean A/B testing

    View Slide

  130. we are doing this excessively, and have over 90 or so experiments with sometimes over 20
    variants running.
    the way we can do this quickly and easily is by just switching components for the different
    variants

    View Slide

  131. Bandit Algorithm
    http://untyped.com/untyping/2011/02/11/stop-ab-testing-
    and-make-out-like-a-bandit/
    tip to read up on testing:

    View Slide

  132. Caching of
    Components
    also you can cache the components easily in a varnish

    View Slide

  133. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu

    because every component has it's own url you can just render out a esi placeholder instead of
    the widget to tell varnish to fetch it separately and provided it has caching headers, get it out
    of the cache

    View Slide

  134. Easy re-using of
    components

    View Slide

  135. View Slide

  136. Load components
    asynchronously
    the same way you can also load components of the widget tree asynchronously

    View Slide

  137. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu

    loadWidget('/aboutMe', function(w) {<br/>w.render({ replace : '#placeholder' });<br/>})
    so instead of rendering the widget you render a placeholder dom element and a script tag
    that loads the widget with an ajax request and then renders it on the client side

    View Slide

  138. BigPipe
    https://www.facebook.com/note.php?
    note_id=389414033919
    even more sophisticated: something facebook calls bigpipe (we call it nozzle)

    View Slide

  139. if you look at your widget tree you can mostly identify larger parts, which are widgets itself

    View Slide

  140. Profile Menu
    Header
    LeftColumn
    RightColumn
    like this, so what you can do to dramatically increase the perceived load time is prioritizing
    the rendering

    View Slide

  141. so our http request looks like this, first you compute and render the important parts of the
    page, like the top menu and the profile header as well as the rest of the layout, for the left
    column and right column which are expensive to compute you just render placeholders and
    then flush the content to the client so that the browser already renders this

    View Slide

  142. still in the same http request you render out the javascript needed to make the already
    rendered components work, so people can use the menu for example

    View Slide

  143. still in the same http request you then compute the data for the left column and render out
    some javascript that takes this data and renders it into the components template client side
    and then replaces the placeholder with the rendered template

    View Slide

  144. still in the same request you then can do this with the right column
    -> flush content as early as possible, don't wait for the whole site to be computed

    View Slide

  145. pushState
    and if you are at that, when you switch pages, you can also just load the differences between
    and use pushState to change the url (if supported) to make your app faster

    View Slide

  146. Profile Publications Publication
    Publication
    Publication
    AboutMe
    LeftColumn Image
    Instiution
    Menu

    View Slide

  147. Profile Questions Question
    Question
    Question
    AboutMe
    LeftColumn Image
    Instiution
    Menu

    View Slide

  148. Conclusion?

    View Slide

  149. Think about your
    architecture

    View Slide

  150. Refactor and make
    it better
    continuously

    View Slide

  151. Frontend and
    backend are part
    of the same
    application

    View Slide

  152. Don't rewrite your
    whole codebase in
    one go

    View Slide

  153. http://twitter.com/BastianHofmann
    http://lanyrd.com/people/BastianHofmann
    http://speakerdeck.com/u/bastianhofmann
    [email protected]
    thanks, you can contact me on any of these platforms or via mail.

    View Slide