Marrying Front with Backend

Marrying front
with back end
@BastianHofmann

View Slide

Let's talk about...
In the next hour I want to talk a bit bout

View Slide

Application
architecture
application architecture and with that I mean how to structure your code

View Slide

Code and
component
re-use
how you can re-use code and components

View Slide

Rapid
Development
so that you can develop new features quickly without breaking the rest of your application or
accumulation lots of technical debt

View Slide

Handling large
code-bases
especially if your code base becomes larger and larger

View Slide

Frameworks to the
rescue
Normally if faced with these problems people say: don't reinvent the wheel, let's look what's
already out there, let's use some framework (or some libraries), or if you have a bad case of
the not-invented-here syndrom in your company, you are going to develop your own

View Slide

?
languages? one/their own product? agency? long lived, short lived projects? large codebases?
frameworks?

View Slide

Most Web Frameworks
are incomplete
But I want to be a bit opinionated here: ...

View Slide

they oftentimes either care about your backend on the server and serve html pages or apis

View Slide

or they care about your frontend in the client

View Slide

webserver
HTML
browser
JS
so the normal workﬂow of a web application is that some server code serves some html (and
maybe some apis) to the browser, in the browser there is some (or a lot of) javascript that
manipulates the DOM

View Slide

webserver
HTML
browser
JS
Ajax
and either links to other pages served by the server or does ajax requests back to the server.
this concept is mostly true for either traditional apps as well as single page applications

View Slide

webserver
HTML
browser
JS
Ajax
but the code in both worlds, server and client, is seldom connected

View Slide

duplication
de duplication
ode duplication
code duplication
code duplication
code duplication
code duplication
code duplication
code duplication
code duplication
code duplication
code duplication
code duplication
code duplication
code duplicatio
code duplicat
code duplic
code dup
code du
code
cod
co
co
which leads to lot of code duplication in templates, validation logic, models and lots of boiler
plate code for communication between server and client

View Slide

So?
so, how can we change this? before i answer this question

View Slide

A few words
about me
...

View Slide

I work at ResearchGate, the social network for scientists and researchers

View Slide

ResearchGate gives
science back to the
people who make it
happen.
We help researchers
build reputation and
accelerate scientific
progress.
On their terms.
‟
the goal is to give...

View Slide

over 2.9 million users

View Slide

here some impressions of the page

View Slide

you may have read about us in the news recently

View Slide

http://gigaom.com/2013/06/05/heres-how-bill-gatess-
researchgate-investment-might-change-the-world-for-the-
better
http://venturevillage.eu/researchgate

View Slide

have this, and also work on some cool stuff

View Slide

we are hiring

View Slide

in addition to this i also speak frequently on conferences throughout the world

View Slide

work and live in berlin

View Slide

Questions? Ask
by the way, if you have any questions throughout this talk, if you don't understand
something, just raise your hand and ask. probably my fault anyways since i spoke to quickly
or my accent was too bad

View Slide

http://speakerdeck.com/u/bastianhofmann
the slides will be available on speakerdeck

View Slide

if we take a small peek into a possible future of web application development

View Slide

Mojito
http://developer.yahoo.com/cocktails/mojito/
there are frameworks already being worked on like mojito from yahoo

View Slide

Meteor
http://www.meteor.com
or meteor, that try to bridge the gap between server and client. but they are still very alpha,
good for prototyping, writing small apps, trying things out, but personally I wouldn't create a
big million lines of code web application with 40+ developers yet

View Slide

What about existing
applications?
and regardless of that, most of you have existing applications

View Slide

Legacy Code
with lots of legacy code

View Slide

Incremental
Refactoring
the way to make the architecture of your existing application better and connect your
backend with your frontend code is incremental refactoring, since you hopefully can't afford
to sit down, stop development for 6 months and rewrite everything

View Slide

many roads
of course like always there are many ways to go about that, which totally depends on you
existing code

View Slide

http://backstage.soundcloud.com/2012/06/building-the-
next-soundcloud/
some examples: soundcloud

View Slide

http://twitter.github.io/ﬂight/
twitter, check out ﬂight.js

View Slide

or what i'm going to talk about for the reminder of the talk, what we did at RG

View Slide

http://www.youtube.com/watch?v=fqULJBBEVQE
what's interesting about all these approaches and with all these totally different applications
and totally different use cases and totally different technologies, that the underlying concepts
people come up with are strikingly similar. basically it's thinking about your app in small
components, that are as independent as possible. this may even make it directly into
browsers: if you haven't heard about it yet, check out the proposed web components
standard.

View Slide

status quo
so RG: when we started about over a year ago, the status quo was:

View Slide

webserver
loadbalancer
pgsql memcached mongodb
services
classic web application, with some soa elements

View Slide

webserver
web application was written in php

View Slide

self-written framework, Large, old codebase, some architecture and code smells, but also
some nicely done things

View Slide

MVC
mvc-like

View Slide

complicated routing

View Slide

PHP templates
templating done in with php scripts on the server

View Slide

lazy DB queries in
view
thanks to doctrine1 with db queries executed in the view

View Slide

on the frontend yui3, but every page was built with custom modules, not much reuse of
existing modules but very basic ones (also some legacy pages with yui2, prototype.js and
scriptaculous)

View Slide

Duplication and
only some Code re-
use
overall ...

View Slide

We can do
better
so we sat down and thought ...

View Slide

Components
let's rethink the way we are building pages or our whole application and separate it into small
components

View Slide

so looking at a page (that's what our proﬁle looked like over a year ago), you can identify lots
of small components nested into each other on a page

View Slide

Self contained
our goal was that each component is ...

View Slide

Can be addressed and
rendered separately
and ...

View Slide

Server
JS
Browser
JSON
HTML
HTML
so it should have it's own url, and for seo reasons can just be included in a page rendered to
a browser by the server or can be fetched separately or nested within other components by
the apps javascript, where we only wanted to transport the data to the client and render it
into html there

View Slide

JS is part of the
component
the javascript to bind on dom events and manipulate the components should be part of the
component

View Slide

Share code
between server
and client
and we wanted to share as much code between server and client as possible (considering our
heterogeneous architectur, js client, php server)

View Slide

Templates,
Validation,
Entities,...
e.g. ..

View Slide

It needs to be fast
of course ...
concerning performance as well as development speed and productivity

View Slide

We called them
Widgets

View Slide

PHP Controller Mustache
Template
JavaScript
view class
Widget
Providing data
Handling browser events
Displaying data
so to sum it up an widget contains

View Slide

Why no models in
the frontend?
http://backbonejs.org/#Model
compared to ﬂight.js, backbone we decided against having dedicated models in the frontend
where views could subscribe on because of legacy reasons. so the data is coupled to the
component, and if needed events synchronize data between components, a bit overhead, a
bit duplication of data, but made development for us much faster, also we have a lot of
different entities, so for us taking this route worked out very well.
"

View Slide

so if we go back to our page

View Slide

Profile Publications Publication
Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu
we see, that it's actually kind of a tree structure, for the sake of this presentation i simplified
it slightly, actually our profile consists of over 200 components

View Slide

Remember:
self contained
remember all these 200 components are supposed to be self contained

View Slide

Do not fetch data
directly
so something you shouldn't do is fetching this data directly on you server when composing a
page or a subpage

View Slide

Sssssssllllooooowww
because that would be really slow

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu
Account
Account
Account
Account
Account
Publication1
Publication2
Publication3
if we take our simpliﬁed example, a lot of components need the account of the user that's
displayed, while that could be solved with in memory caching, a list of publications each need
the publication entity, if we display 20 publications, that would mean 20 database queries,
doing it in one query would be much faster though, and you have lot's of stuff like this

View Slide

Require stuff
so instead of fetching stuff directly, it's way better to require stuff

View Slide

http://www.infoq.com/presentations/Evolution-of-Code-
Design-at-Facebook/
this concept is actually not that new, one small company who is doing this very much is for
example facebook, who also did a very good talk about this a few years back

View Slide

so how does it work? around all your components that just post data requirements you need
an instance called a preparer

View Slide

Widget Widget Widget Widget
Preparer
Resolver Resolver
Services
Connector Interfaces
Connector Implementations

View Slide

Preparer
Fetch Requirements
Resolver Resolver
Services
the preparer iterates over all components and fetches their requirments

View Slide

Preparer
Resolver Resolver
Services
Batch
requirements
and pass
them to
resolvers
it batches them together as intelligently as possible, passes the requirements to resolvers

View Slide

Preparer
Resolver Resolver
Services
Call Services as
effective as possible
(Multi-GET,...)
which call the services/storages/etc as effective as possible, or just execute some service
class methods

View Slide

Preparer
Resolver Resolver
Services
Attach fetched
data to
Requirements and
pass them back to
the preparer
the fetched data is attached to the requirements, passed back to the preparer

View Slide

Preparer
Resolver Resolver
Services
Distribute fetched
data to the widgets
that required it
who in turn distributes the fetched data to the components

View Slide

so in code a component could specify its requirements like this

View Slide

EntityRequirement
in our implementation we separate between

View Slide

Service
Requirement
and ..
which is just a technical reason

View Slide

Request Cache
nice thing is you can cache this stuff intelligently in memory of your current request

View Slide

Multi-GET
or use stuff like multi-get

View Slide

Futures
or futures / promises

View Slide

Required / Optional
requirements can be required or optional, meaning either it's ok if a call fails or not
(important for error handling). if it's not ok we just deactivate the whole component

View Slide

Widget
Requirement
but it does not need to be only data, for example including another widget is also a
requirement for us. that's the way we actually specify the component trees seen earlier

View Slide

RequestData
Requirement
or getting data out of the request scope can be a requirement

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu
so in our example

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu
in the ﬁrst iteration the preparer asks the proﬁle widget for its requirement that can be data
as well as including all the subwidgets

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu
next the preparer asks all the new widgets it got to know in the last iteration for their
requirements

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu
and so on

View Slide

Data
dependencies
within a widget
but oftentimes we have .. so it gets a bit more complex

View Slide

=> Callbacks
we can use ... for that

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu
CALLBACK
CALLBACK
CALLBACK
so in the first iteration the profile widget returns its requirements as well as a callback
function that should be executed once all the requirements have been fullfilled

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu
CALLBACK
CALLBACK
CALLBACK
after the ﬁrst iteration this callback is then executed while collecting all the requirements of
the new subwidgets. the callback can then return further requirments as well as also further
callbacks

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu
CALLBACK
CALLBACK
CALLBACK
and so on, this iterative process is done as long as there are components or callbacks
available that return new requirements

View Slide

in code...
you can imagine if you are not careful this can become some kind of a callback hell

View Slide

PHP 5.5 Generators
https://wiki.php.net/rfc/generators
that's why i'm very excited about generators in php 5.5.

View Slide

public function collect() {
yield array(
new EntityRequirement(
'account',
Account::class,
array('accountId' => $this->requestContext->getAccountId())
),
);
yield array(
new ServiceRequirement(
'scienceDisciplines',
AccountService::class,
'getScienceDisciplines',
array('account' => $this->account)
)
);
}
because then you can write it like this:

View Slide

Data
dependencies
between Widgets
also oftentimes you may have...
if only for performance reasons,

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu
e.g. for a list of publications that you got from a solr search, you may not want to go to the
database again to fetch each publication in each list item, but since a list item is supposed to
be renderable separately as well, it needs this logic in there

View Slide

Prefills
what you can do then is preﬁlling this data from the parent widget

View Slide

in code

View Slide

that means if the subwidget has a requirment like this, the preparer would directly put the
preﬁlled data in there instead of passing it on to the resolvers

View Slide

Whoa, this looks
awfully
complicated to
debug
you're right, it takes a bit of time to get used to program this way, but everyone who joined
our company in the last few months since we are doing it says after a week or so, that they
actually can't imagine working any other way again (i'll talk more about the beneﬁts later)

View Slide

but still it's complex, so a good thing is taking a bit of effort to make it as transparent as
possible to see what happens in your application (which is a very good thing anyways).
we have a debug toolbar that shows all the component on the page

View Slide

and you can also drill down into all the iterations the preparer did

View Slide

and how long each step took
DEMO

View Slide

Rendering
so we have now all the data fetched together on our server lets render it now

View Slide

HTML
i said earlier we want our widget tree to be renderable on the server side as html

View Slide

JSON
and just put out as json and then be rendered on the client side by our javascript

View Slide

Templates
and of course we want to reuse our templates

View Slide

Mustache
}
http://mustache.github.com/
so we decided to use mustache templates, we are actually using twitters mustache
implementation called hogan

View Slide

that's what a template may look like

View Slide

Helper Methods
there is also a php implementation of mustache, but it's not the fastest and one thing is, that
mustache oftentimes relies on small helper functions

View Slide

•nl2br
•truncate
•pluralize
•wordwrap
•highlight
•...
like ...
if you now use a php implementation on the server and a js implementation on the client, we
have to develop these functions twice ... not a good idea

View Slide

http://pecl.php.net/package/v8js
V8js
that why we actually execute javascript through a php extension that makes the v8 js library
available on the server. so we are executing the same code on the server and on the client to
render the same templates.
suprise: it's acutally really really fast

View Slide

JavaScript
so we handled templates and rendering, the backend controllers that provide the data, on the
javascript side in the client each component can have a view object to bind on DOM events

View Slide

WidgetViews

View Slide

here is an example

View Slide

Loading widgets
from JavaScript
of course since every widget has its own url its very easy to load other widgets from the
server and then put them somewhere in the dom

View Slide

templates and javascript is all loaded automatically

View Slide

Benefits

View Slide

Enables developers
to only focus on
their components

View Slide

Easy re-using of
components

View Slide

Easier Refactoring

View Slide

Error Handling
also error handling can be really simpliﬁed

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu
EXCEPTION
if an exception occurs in one of our components while fulﬁlling a requirement that is not
optional

View Slide

Publication
Publication
LeftColumn Image
Instiution
Menu
we just deactivate this component and the rest of the page is still functional

View Slide

Testing
also this approach simpliﬁes testing, and with that i mean A/B testing

View Slide

we are doing this excessively, and have over 90 or so experiments with sometimes over 20
variants running.
the way we can do this quickly and easily is by just switching components for the different
variants

View Slide

Bandit Algorithm
http://untyped.com/untyping/2011/02/11/stop-ab-testing-
and-make-out-like-a-bandit/
tip to read up on testing:

View Slide

Caching of
Components
also you can cache the components easily in a varnish

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu

because every component has it's own url you can just render out a esi placeholder instead of
the widget to tell varnish to fetch it separately and provided it has caching headers, get it out
of the cache

View Slide

Load components
asynchronously
the same way you can also load components of the widget tree asynchronously

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu

loadWidget('/aboutMe', function(w) {<br/>w.render({ replace : '#placeholder' });<br/>})
so instead of rendering the widget you render a placeholder dom element and a script tag
that loads the widget with an ajax request and then renders it on the client side

View Slide

BigPipe
https://www.facebook.com/note.php?
note_id=389414033919
even more sophisticated: something facebook calls bigpipe (we call it nozzle)

View Slide

if you look at your widget tree you can mostly identify larger parts, which are widgets itself

View Slide

Proﬁle Menu
Header
LeftColumn
RightColumn
like this, so what you can do to dramatically increase the perceived load time is prioritizing
the rendering

View Slide

so our http request looks like this, first you compute and render the important parts of the
page, like the top menu and the profile header as well as the rest of the layout, for the left
column and right column which are expensive to compute you just render placeholders and
then flush the content to the client so that the browser already renders this

View Slide

still in the same http request you render out the javascript needed to make the already
rendered components work, so people can use the menu for example

View Slide

still in the same http request you then compute the data for the left column and render out
some javascript that takes this data and renders it into the components template client side
and then replaces the placeholder with the rendered template

View Slide

still in the same request you then can do this with the right column
-> ﬂush content as early as possible, don't wait for the whole site to be computed

View Slide

pushState
and if you are at that, when you switch pages, you can also just load the differences between
and use pushState to change the url (if supported) to make your app faster

View Slide

Publication
Publication
AboutMe
LeftColumn Image
Instiution
Menu

View Slide

Proﬁle Questions Question
Question
Question
AboutMe
LeftColumn Image
Instiution
Menu

View Slide

Conclusion?

View Slide

Think about your
architecture

View Slide

Refactor and make
it better
continuously

View Slide

Frontend and
backend are part
of the same
application

View Slide

Don't rewrite your
whole codebase in
one go

View Slide

http://twitter.com/BastianHofmann
http://profiles.google.com/bashofmann
http://lanyrd.com/people/BastianHofmann
http://speakerdeck.com/u/bastianhofmann
https://github.com/bashofmann
https://www.researchgate.net/profile/Bastian_Hofmann/
[email protected]
thanks, you can contact me on any of these platforms or via mail.

View Slide

Marrying Front with Backend

Marrying Front with Backend

Bastian Hofmann

More Decks by Bastian Hofmann

Other Decks in Programming

Featured

Transcript