Marrying front with back end

Slide 1

Slide 1 text

Marrying front with back end @BastianHofmann

Slide 2

Slide 2 text

Let's talk about... In the next hour I want to talk a bit about

Slide 3

Slide 3 text

Application architecture application architecture and with that I mean how to structure your code

Slide 4

Slide 4 text

Code and component re-use how you can re-use code and components

Slide 5

Slide 5 text

Rapid Development so that you can develop new features quickly without breaking the rest of your application or accumulation lots of technical debt

Slide 6

Slide 6 text

Handling large code-bases especially if your code base becomes larger and larger, developed by big, growing teams

Slide 7

Slide 7 text

Frameworks to the rescue Normally if faced with these problems people say: don't reinvent the wheel, let's look what's already out there, let's use some framework (or some libraries), or if you have a bad case of the not-invented-here syndrom in your company, you are going to develop your own

Slide 8

Slide 8 text

? languages? one/their own product? agency? long lived, short lived projects? large codebases? frameworks?

Slide 9

Slide 9 text

But I want to be a bit opinionated here: ...

Slide 10

Slide 10 text

Most Web Frameworks are incomplete

Slide 11

Slide 11 text

they oftentimes either care about your backend on the server and serve html pages or apis

Slide 12

Slide 12 text

or they care about your frontend in the client

Slide 13

Slide 13 text

webserver HTML browser JS so the normal ﬂow of a web application is that some server code serves some html (and maybe some apis) to the browser, in the browser there is some (or a lot of) javascript that manipulates the DOM

Slide 14

Slide 14 text

webserver HTML browser JS Ajax and either links to other pages served by the server or does ajax requests back to the server. this concept is mostly true for either traditional apps as well as single page applications

Slide 15

Slide 15 text

webserver HTML browser JS Ajax but the code in both worlds, server and client, is seldom connected

Slide 16

Slide 16 text

de duplication ode duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplication code duplicatio code duplicat code duplic code dup code du code cod co co which leads to lot of code duplication in templates, validation logic, models and lots of boiler plate code for communication between server and client, marshalling data

Slide 17

Slide 17 text

So? so, how can we change this? before i answer this question

Slide 18

Slide 18 text

A few words about me ...

Slide 19

Slide 19 text

I work at ResearchGate, the social network for scientists and researchers

Slide 20

Slide 20 text

over 3 million users

Slide 21

Slide 21 text

here some impressions of the page

Slide 22

Slide 22 text

you may have read about us in the news recently

Slide 23

Slide 23 text

have this, and also work on some cool stuff

Slide 24

Slide 24 text

have this, and also work on some cool stuff

Slide 25

Slide 25 text

have this, and also work on some cool stuff

Slide 26

Slide 26 text

we are hiring

Slide 27

Slide 27 text

Questions? Ask by the way, if you have any questions throughout this talk, if you don't understand something, just raise your hand and ask.

Slide 28

Slide 28 text

http://speakerdeck.com/u/bastianhofmann the slides will be available on speakerdeck

Slide 29

Slide 29 text

if we take a small peek into a possible future of web application development

Slide 30

Slide 30 text

Meteor http://www.meteor.com or meteor, that try to bridge the gap between server and client. but they are still very alpha, good for prototyping, writing small apps, trying things out, but personally I wouldn't create a big million lines of code web application with 40+ developers yet

Slide 31

Slide 31 text

What about existing applications? and regardless of that, most of you have existing applications

Slide 32

Slide 32 text

Legacy Code with lots of legacy code

Slide 33

Slide 33 text

Incremental Refactoring the way to make the architecture of your existing application better and connect your backend with your frontend code is incremental refactoring, since you hopefully can't afford to sit down, stop development for 6 months and rewrite everything

Slide 34

Slide 34 text

many roads of course like always there are many ways to go about that, which totally depends on you existing code

Slide 35

Slide 35 text

or what i'm going to talk about for the reminder of the talk, what we did at RG

Slide 36

Slide 36 text

status quo so RG: when we started about over a year ago, the status quo was:

Slide 37

Slide 37 text

webserver loadbalancer pgsql memcached mongodb services classic web application, with some soa elements

Slide 38

Slide 38 text

webserver web application was written in php

Slide 39

Slide 39 text

self-written framework, Large, old codebase, some architecture and code smells, but also some nicely done things

Slide 40

Slide 40 text

MVC mvc-like, one page, one controller, one template

Slide 41

Slide 41 text

PHP templates templating done in with php scripts on the server

Slide 42

Slide 42 text

Duplication and only some Code re- use overall ...

Slide 43

Slide 43 text

We can do better so we sat down and thought ...

Slide 44

Slide 44 text

Components let's rethink the way we are building pages or our whole application and separate it into small components

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

Self contained our goal was that each component is ...

Slide 54

Slide 54 text

Can be addressed and rendered separately and ...

Slide 55

Slide 55 text

Server JS Browser JSON HTML HTML so it should have it's own url, and for seo reasons can just be included in a page rendered to a browser by the server or can be fetched separately or nested within other components by the apps javascript, where we only wanted to transport the data to the client and render it into html there

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

JS is part of the component the javascript to bind on dom events and manipulate the components should be part of the component

Slide 58

Slide 58 text

CSS can be part of the component

Slide 59

Slide 59 text

Share code between server and client and we wanted to share as much code between server and client as possible (considering our heterogeneous architectur, js client, php server)

Slide 60

Slide 60 text

Templates, Validation, Entities,... e.g. ..

Slide 61

Slide 61 text

It needs to be fast of course ... concerning performance as well as development speed and productivity

Slide 62

Slide 62 text

PHP Controller Mustache Template JavaScript view class Widget Providing data Handling browser events Displaying data CSS so to sum it up an widget contains

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

Profile Publications Publication Publication Publication AboutMe LeftColumn Image Institution Menu we see, that it's actually kind of a tree structure, for the sake of this presentation i simplified it slightly, actually our profile consists of over 200 components

Slide 65

Slide 65 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Menu Server Request Response Institution i said earlier, that all these components have their own URL. So they can be requested and rendered by the client seperately. For better user experience on the ﬁrst load or SEO reasons these components can also be bundled together and rendered in a single page load on the backend.

Slide 66

Slide 66 text

LeftColumn Image Menu Server Request Response if we want to fetch the left column separately it looks like this

Slide 67

Slide 67 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Institution Menu so you have this tree

Slide 68

Slide 68 text

Widget Requirement this tree structure is achieved through so called widget requirements, why they are called requirements

Slide 69

Slide 69 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution that means the proﬁle widget requires: publications, aboutMe, leftColumn and institution

Slide 70

Slide 70 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution and Publications requires several publicationItem widgets and leftColumn the Image and the Menu widget

Slide 71

Slide 71 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution

Slide 72

Slide 72 text

Remember: self contained remember all these 200 components are supposed to be self contained

Slide 73

Slide 73 text

Do not fetch data directly so something you shouldn't do is fetching this data directly on you server when composing a page or a subpage

Slide 74

Slide 74 text

Sssssssllllooooowww because that would be really slow

Slide 75

Slide 75 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Menu Account Account Account Account Account Publication1 Publication2 Publication3 Institution if we take our simpliﬁed example, a lot of components need the account of the user that's displayed, while that could be solved with in memory caching, a list of publications each need the publication entity, if we display 20 publications, that would mean 20 database queries, doing it in one query would be much faster though, and you have lot's of stuff like this

Slide 76

Slide 76 text

Require stuff so instead of fetching stuff directly, it's way better to require stuff (that's also why building up the widget tree was done with widget "requirements"

Slide 77

Slide 77 text

http://www.infoq.com/presentations/Evolution-of-Code- Design-at-Facebook/ this concept is actually not that new, one small company who is doing this very much is for example facebook, who also did a very good talk about this a few years back

Slide 78

Slide 78 text

so how does it work? around all your components that just state data (or widget) requirements you need an instance called a preparer

Slide 79

Slide 79 text

Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces Connector Implementations

Slide 80

Slide 80 text

Widget Widget Widget Widget Preparer Fetch Requirements Resolver Resolver Services Connector Interfaces Connector Implementations the preparer iterates over all components and fetches their requirments

Slide 81

Slide 81 text

Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces Connector Implementations Batch requirements and pass them to resolvers it batches them together as intelligently as possible, passes the requirements to resolvers

Slide 82

Slide 82 text

Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces Connector Implementations Call Services as effective as possible (Multi-GET,...) which call the services/storages/etc as effective as possible, or just execute some service class methods

Slide 83

Slide 83 text

Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces Connector Implementations Attach fetched data to Requirements and pass them back to the preparer the fetched data is attached to the requirements, passed back to the preparer

Slide 84

Slide 84 text

Widget Widget Widget Widget Preparer Resolver Resolver Services Connector Interfaces Connector Implementations Distribute fetched data to the widgets that required it who in turn distributes the fetched data to the components

Slide 85

Slide 85 text

public function collect() { return [ new EntityRequirement( 'account', Account::class, ['id' => $this->request->get('id')] ), ]; } because then you can write it like this:

Slide 86

Slide 86 text

Request Cache nice thing is you can cache this stuff intelligently in memory of your current request

Slide 87

Slide 87 text

Multi-GET or use stuff like multi-get

Slide 88

Slide 88 text

Futures or futures / promises

Slide 89

Slide 89 text

Data dependencies within a widget but oftentimes we have .. so it gets a bit more complex

Slide 90

Slide 90 text

PHP 5.5 Generators http://php.net/manual/de/language.generators.overview.php that's why i'm very excited about generators in php 5.5.

Slide 91

Slide 91 text

public function collect() { yield [ new EntityRequirement( 'account', Account::class, ['id' => $this->request->get('id')] ), ]; yield [ new ServiceRequirement( 'scienceDisciplines', AccountService::class, 'getScienceDisciplines', ['account' => $this->account] ) ]; } because then you can write it like this:

Slide 92

Slide 92 text

PHP 5.4 Traits http://us1.php.net/trait that's why i'm very excited about generators in php 5.5.

Slide 93

Slide 93 text

yield [ serviceRequirement( 'job', JobService::getCall()->getById(248) ) ];

Slide 94

Slide 94 text

trait ServiceRequirementFactory { /** @return $this */ public static function getCall() { return new ServiceRequirementFactoryProxy( ! ! ! static::class ! ! ! ); } }

Slide 95

Slide 95 text

class ServiceRequirementFactoryProxy { private $className; public function __construct($className) { $this->className = $className; } public function __call($name, $arguments) { return [ 'serviceClass' => $this->className, 'methodName' => $name, 'arguments' => $arguments ]; } }

Slide 96

Slide 96 text

Data dependencies between Widgets also oftentimes you may have... if only for performance reasons,

Slide 97

Slide 97 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution e.g. for a list of publications that you got from a solr search, you may not want to go to the database again to fetch each publication in each list item, but since a list item is supposed to be renderable separately as well, it needs this logic in there

Slide 98

Slide 98 text

Prefills what you can do then is preﬁlling this data from the parent widget

Slide 99

Slide 99 text

class PublicationKeywordSearch { public $publications; public $publicationListItems = []; public function collect() { yield [ serviceRequirement( 'publications', PublicationService::getCall()->getByKeywords( ['virology', 'cancer'], 10, 0 ) ) ]; foreach ($this->publications as $publication) { yield new WidgetRequirement( 'publicationListItems', PublicationItem::CLASS, [ ! ! ! ! ! ! ! ! 'publicationId' => $publication->getId(), ! ! ! ! ! ! ! ! 'publication' => $publication ! ! ! ! ! ! ! ] ); } } } in code

Slide 100

Slide 100 text

class PublicationItem { public $publicationId; public $publication; public function collect() { yield new RequestDataRequirement('publicationId'); yield [ new EntityRequirement( 'publication', Publication::class, ['id' => $this->publicationId] ) ]; } } that means if the subwidget has a requirment like this, the preparer would directly put the preﬁlled data in there instead of passing it on to the resolvers

Slide 101

Slide 101 text

Whoa, this looks awfully complicated to debug you're right, it takes a bit of time to get used to program this way, but everyone who joined our company in the last few months since we are doing it says after a week or so, that they actually can't imagine working any other way again (i'll talk more about the beneﬁts later)

Slide 102

Slide 102 text

but still it's complex, so a good thing is taking a bit of effort to make it as transparent as possible to see what happens in your application (which is a very good thing anyways). we have a debug toolbar that shows all the component on the page

Slide 103

Slide 103 text

and you can also drill down into all the iterations the preparer did

Slide 104

Slide 104 text

No content

Slide 105

Slide 105 text

No content

Slide 106

Slide 106 text

No content

Slide 107

Slide 107 text

So that was the backend

Slide 108

Slide 108 text

so we have all the data for all tese small components

Slide 109

Slide 109 text

Rendering so we have now all the data fetched together on our server lets render it now

Slide 110

Slide 110 text

HTML i said earlier we want our widget tree to be renderable on the server side as html

Slide 111

Slide 111 text

JSON and just put out as json and then be rendered on the client side by our javascript

Slide 112

Slide 112 text

Templates and of course we want to reuse our templates

Slide 113

Slide 113 text

Mustache } http://mustache.github.com/ so we decided to use mustache templates, we are actually using twitters mustache implementation called hogan

Slide 114

Slide 114 text

Publications ({{count}})

{{{.}}}

{{#showLegalFooter}} {{{legalFooter}}} {{/showLegalFooter}}

that's what a template may look like

Slide 115

Slide 115 text

Helper Methods there is also a php implementation of mustache, but it's not the fastest and one thing is, that mustache oftentimes relies on small helper functions

Slide 116

Slide 116 text

•nl2br •truncate •pluralize •wordwrap •highlight •... like ... if you now use a php implementation on the server and a js implementation on the client, we have to develop these functions twice ... not a good idea

Slide 117

Slide 117 text

http://pecl.php.net/package/v8js V8js that why we actually execute javascript through a php extension that makes the v8 js library available on the server. so we are executing the same code on the server and on the client to render the same templates. suprise: it's acutally really really fast

Slide 118

Slide 118 text

JavaScript so we handled templates and rendering, the backend controllers that provide the data, on the javascript side in the client each component can have a view object to bind on DOM events

Slide 119

Slide 119 text

WidgetViews

Slide 120

Slide 120 text

Publications ({{count}})

{{{.}}}

{{#showLegalFooter}} {{{legalFooter}}} {{/showLegalFooter}}

remember this template

Slide 121

Slide 121 text

YUI.add('views.publicationList', function (Y) { Y.views.publicationList = Y.Base.create(Y.WidgetView, { events : { '.js-showALl' : { click : 'showALl' } }, showAll : function () { var node = this.get('container'); node.addClass('loading'); Y.loadWidget(‚/publications/all', { keywords : this.data.keywords }, function (widget) { widget.render({ replace : node }); }); } }); }); here is an example

Slide 122

Slide 122 text

http://twitter.github.io/ﬂight/ twitter, check out ﬂight.js

Slide 123

Slide 123 text

http://facebook.github.io/react/ by the way: if you’ll have a look at react: it is strikingly similar

Slide 124

Slide 124 text

http://www.youtube.com/watch?v=fqULJBBEVQE .. this is exactly what web components is about. they have their own html, css, js and are even more sandboxed through a shadow dom to limit interactions between different components. an example for webcomponents are the browser controls for videos or forms by the way.

Slide 125

Slide 125 text

http://www.polymer-project.org/

Slide 126

Slide 126 text

Benefits

Slide 127

Slide 127 text

Enables developers to only focus on their components

Slide 128

Slide 128 text

Rapid prototyping

Slide 129

Slide 129 text

Easier Refactoring even if the implementation of one component is very ugly and crude, at least it's encapsulated (hopefully tested) and you can either leave it or revisit this component later on

Slide 130

Slide 130 text

Error Handling also error handling can be really simpliﬁed and end-user friendly

Slide 131

Slide 131 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution so going back to our widget tree

Slide 132

Slide 132 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Menu EXCEPTION Institution if an exception occurs in one of our components while fulﬁlling a requirement that is not optional

Slide 133

Slide 133 text

Proﬁle Publications Publication Publication Publication LeftColumn Image Menu Institution we just deactivate this component and the rest of the page is still functional

Slide 134

Slide 134 text

Experiments (A/B Testing) also this approach simpliﬁes testing, and with that i mean A/B testing

Slide 135

Slide 135 text

we are doing this excessively, and have over 90 or so experiments with sometimes over 20 variants running. the way we can do this quickly and easily is by just switching components for the different variants

Slide 136

Slide 136 text

Caching of Components also you can cache the components easily in a varnish

Slide 137

Slide 137 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution because every component has it's own url you can just render out a esi placeholder instead of the widget to tell varnish to fetch it separately and provided it has caching headers, get it out of the cache

Slide 138

Slide 138 text

Easy re-using of components

Slide 139

Slide 139 text

No content

Slide 140

Slide 140 text

Load components asynchronously the same way you can also load components of the widget tree asynchronously

Slide 141

Slide 141 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Menu

loadWidget('/aboutMe', function(w) { w.render({ replace : '#placeholder' }); }) Institution so instead of rendering the widget you render a placeholder dom element and a script tag that loads the widget with an ajax request and then renders it on the client side

Slide 142

Slide 142 text

BigPipe https://www.facebook.com/note.php? note_id=389414033919 even more sophisticated: something facebook calls bigpipe (we call it nozzle)

Slide 143

Slide 143 text

if you look at your widget tree you can mostly identify larger parts, which are widgets itself

Slide 144

Slide 144 text

Proﬁle Menu Header LeftColumn RightColumn like this, so what you can do to dramatically increase the perceived load time is prioritizing the rendering

Slide 145

Slide 145 text

so our http request looks like this, first you compute and render the important parts of the page, like the top menu and the profile header as well as the rest of the layout, for the left column and right column which are expensive to compute you just render placeholders and then flush the content to the client so that the browser already renders this

Slide 146

Slide 146 text

still in the same http request you render out the javascript needed to make the already rendered components work, so people can use the menu for example

Slide 147

Slide 147 text

still in the same http request you then compute the data for the left column and render out some javascript that takes this data and renders it into the components template client side and then replaces the placeholder with the rendered template

Slide 148

Slide 148 text

still in the same request you then can do this with the right column -> ﬂush content as early as possible, don't wait for the whole site to be computed

Slide 149

Slide 149 text

pushState and if you are at that, when you switch pages, you can also just load the differences between and use pushState to change the url (if supported) to make your app faster

Slide 150

Slide 150 text

Proﬁle Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution

Slide 151

Slide 151 text

Proﬁle Questions Question Question Question AboutMe LeftColumn Image Menu Institution

Slide 152

Slide 152 text

Conclusion?

Slide 153

Slide 153 text

Think about your architecture

Slide 154

Slide 154 text

Refactor and make it better continuously

Slide 155

Slide 155 text

Frontend and backend are part of the same application

Slide 156

Slide 156 text

Don't rewrite your whole codebase in one go

Slide 157

Slide 157 text

http://twitter.com/BastianHofmann http://lanyrd.com/people/BastianHofmann http://speakerdeck.com/u/bastianhofmann [email protected] thanks, you can contact me on any of these platforms or via mail.