Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ben Shaw: An introduction to Helio

Ben Shaw: An introduction to Helio

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Ben Shaw:
An Introduction to Helio
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2013 - Saturday, 07 Sep 2013 - Track 1
http://nz.pycon.org/

**Audience level**

Intermediate

**Description**

At Yellow, we love building web-applications that leverage the best parts of client and server side languages, so we created the open-source Helio. Its modular design lets us compose Python, Javascript, HTML and CSS in harmony, allowing each to perform in the area it excels. This talk is about Helio's creation, and how it integrates with your favourite Python web framework(s) in your next project.

**Abstract**

Background

When building a dynamic “AJAX” website with Python, there are many Javascript based MVC libraries to provide assistance, however, they move much of the templating and logic into the HTML and Javascript, leaving the Python backend as little more than a JSON or XML generator. Helio was created to move the web content generation tasks back to the Python web framework, while still allowing dynamic pages that are interactive and can update portions of themselves without a complete refresh. Helio lets the Python web framework assume its traditional role of data retrieval and template rendering, while Javascript is left to marshall browser events. This talk details the features of Helio, gives an overview into how it works, and outlines how it can be integrated with a Python web framework.

Components

Helio is built on the concept of self-contained, reusable and extendable Components, that encapsulate a Python view controller, and (optionally) Javascript, CSS and HTML files.

View Controller
A view controller class, written in Python, responds to events received from the browser, fetches data, and renders HTML which the browser retrieves. It is also responsible for defining what Javascript and CSS should apply to the component. View controllers can inherit from other controllers, and can overwrite as much or as little of its parents functionality as necessary. For example, a child view controller could choose to just implement its own CSS, while retaining the actions provided by the parent’s Javascript. The view controller can persist its data via the web framework’s session storage.

Javascript
A component’s Javascript file is where Javascript events (e.g. onclick) for the component are attached in the browser. The Javascript file is dynamically retrieved and its events attached when the first instance of the component is loaded by the browser. Any Javascript library, like jQuery, can be integrated.

CSS
The component’s CSS file is dynamically retrieved when the first instance of the component is loaded by the browser, and then unloaded when no instances remain. Any CSS framework or library (such as Foundation, Twitter Bootstrap or various grid libraries) can be integrated.

Included Components
Helio includes a number of components that can be used as the base for custom components and extended as necessary. Some example are sortable tables with pagination and master detail views.

View Controller Tree

An instantiated component is stored server side in a view controller tree. Each component instance has a unique path identifying its location in the tree, based on its parent’s location and an identifier assigned by the parent. For example, the component at path page.document.0 is child 0 of page.document, which in turn is child document of page (the root component). The path allows any component in the tree to be addressed, both server side and in the browser’s DOM.

Rendered View Tree
The markup for the view controller tree that is displayed by the browser can be generated and requested in full or in part, depending on how much of the page the browser needs to refresh. For example, on the initial load, the browser will request the HTML for page, which would render the tree from page (the root component) down; this includes page.document and page.sidebar. If the browser only needs to refresh the sidebar, it can request just page.sidebar, which will render the view controller tree from page.sidebar down.

The path for each component is encapsulated in its HTML as its id attribute.

Server Side Events
Events can be sent between components on the server side via the NotificationCentre; events can either be sent directly to a component via its address, or components can choose to subscribe to events of a certain type (an implementation of the observer pattern).

Javascript Events
A view controller can generate an event, in Python, that is returned to the browser for Javascript to interpret; likewise the browser can generate an event in Javascript for the python view controller to interpret. An example of this might be a controller that supports pagination. The browser will generate and send a change-page event to the controller at tree-address page.results-view (including a payload indicating what page is to be selected). The controller will change the pagination, generate new HTML, then return a load event to the browser, letting the browser know to update that component with the new HTML.

Development and use at Yellow

Helio was conceived as the base of Axle, Yellow’s new order, asset and content management system. Axle is built upon Django, which allows the use of Django’s URL routing, templating, middleware and other popular niceties, supplemented by the dynamic interaction that Helio provides. The nature of Helio components meant each Axle developer could work on their own small part of the system without worrying that they would affect someone else’s development, and yet know that their component would work correctly when integrated with the rest. Component inheritance in Helio also meant that as base components were enhanced, the components inheriting from them were also improved, for example, after adding table sorting to the base table view, all table components were then sortable. The development of Helio was not without challenges, particularly in tracking down errors in Javascript events and classes as they are dynamically loaded. Extensive use of automated unit and regression testing has been used to make sure that Helio performs as designed.

Web Framework Integration

Helio is customisable and supports multiple Python web frameworks. With simple configuration changes it can be integrated with Django, Flask and Pyramid, and will probably work with any web framework provided that framework can render templates, load static files and store session data.

Summary

Helio allows the Python framework to perform the data manipulation and template rendering that it is designed for, and Javascript is only used to send events to update/retrieve data, not for templating or data manipulation. However, if a developer chooses, they can add extra Javascript libraries that they are familiar with if they want more templating or data manipulation in the browser.

**YouTube**

https://www.youtube.com/watch?v=w_fFONTSzig

New Zealand Python User Group

September 07, 2013
Tweet

More Decks by New Zealand Python User Group

Other Decks in Programming

Transcript

  1. menu.html <ul>        <li>        

           <a  href="#"  data-­‐component="blue">Blue</a>        </li>        <li>                <a  href="#"  data-­‐component="red">Red</a>        </li>        <li>                <a  href="#"  data-­‐component="adder">Adder</a>        </li>        <li>                <a  href="#"  data-­‐component="multiplier">Multiplier</a>        </li> </ul>
  2. Menu Controller from  helio.controller.base  import  BaseViewController from  helio.helio_exceptions  import  ControllerImportError

    class  MenuViewController(BaseViewController):        component_name  =  'menu'        has_js  =  True        has_css  =  True        def  handle_notification(self,  notification_name,  data,  request,  **kwargs):                try:                        new_component_id  =  'colouredbackground.'  +  data['component']                        self.view_state.insert_new_controller('page.document',  new_component_id)                except  ControllerImportError:                        self.view_state.insert_new_controller('page.document',  data['component'])                self.nc.queue_load('page.document') Controller  =  MenuViewController
  3. Menu Controller from  helio.controller.base  import  BaseViewController from  helio.helio_exceptions  import  ControllerImportError

    class  MenuViewController(BaseViewController):        component_name  =  'menu'        has_js  =  True        has_css  =  True        def  handle_notification(self,  notification_name,  data,  request,  **kwargs):                try:                        new_component_id  =  'colouredbackground.'  +  data['component']                        self.view_state.insert_new_controller('page.document',  new_component_id)                except  ControllerImportError:                        self.view_state.insert_new_controller('page.document',  data['component'])                self.nc.queue_load('page.document') Controller  =  MenuViewController
  4. menu.css .menu  {        padding-­‐left:  4px; } .menu

     a{        color:  white; } .menu  >  ul  >  li  {        line-­‐height:  150%; }
  5. menu.js registerClass('menu',  null,  function(){        return  Controller.extend({  

                 attach:  function(){                        var  self  =  this;                        this.$container.find('a').click(function(ev){                                ev.preventDefault();                                postNotification( self.controllerPath, 'menu-­‐change', {'component':  $(this).data('component')}    );                                return  false;                        });                }        }); });
  6. menu.js registerClass('menu',  null,  function(){        return  Controller.extend({  

                 attach:  function(){                        var  self  =  this;                        this.$container.find('a').click(function(ev){                                ev.preventDefault();                                postNotification( self.controllerPath, 'menu-­‐change', {'component':  $(this).data('component')}    );                                return  false;                        });                }        }); });
  7. menu.js registerClass('menu',  null,  function(){        return  Controller.extend({  

                 attach:  function(){                        var  self  =  this;                        this.$container.find('a').click(function(ev){                                ev.preventDefault();                                postNotification( self.controllerPath, 'menu-­‐change', {'component':  $(this).data('component')}    );                                return  false;                        });                }        }); });
  8. PageViewController class  PageViewController(BaseViewController):        component_name  =  'page'  

         has_css  =  True        def  post_attach(self):                self.set_child('header',  init_controller('header'))                self.set_child('menu',  init_controller('menu'))                self.set_child('document',  init_controller('blue'))                self.set_child('footer',  init_controller('footer'))
  9. <div  id="{{  header.path  }}"  class="header">{{  header  }}</div> <div>    

       <div  id="{{  menu.path  }}"  class="menu">{{  menu  }}</div>        <div  id="{{  document.path  }}"  class="document">{{  document  }} </div> </div> <div  id="{{  footer.path  }}"  class="footer">{{  footer  }}</div> page.html
  10. <div  id="{{  header.path  }}"  class="header">{{  header  }}</div> <div>    

       <div  id="{{  menu.path  }}"  class="menu">{{  menu  }}</div>        <div  id="{{  document.path  }}"  class="document">{{  document  }} </div> </div> <div  id="{{  footer.path  }}"  class="footer">{{  footer  }}</div> page.html
  11. <div  id="{{  header.path  }}"  class="header">{{  header  }}</div> <div>    

       <div  id="{{  menu.path  }}"  class="menu">{{  menu  }}</div>        <div  id="{{  document.path  }}"  class="document">{{  document  }} </div> </div> <div  id="{{  footer.path  }}"  class="footer">{{  footer  }}</div> page.html
  12. ViewState Tree page page . header page . menu page

    . document page . footer page . document . page_one
  13. ViewState Tree page page . header page . menu page

    . document page . footer page header blue menu footer
  14. ViewState Tree page page . header page . menu page

    . document page . footer page header blue menu footer
  15. ViewState Tree page page . header page . menu page

    . document page . footer page header red menu footer
  16. Notification to Server JavaScript Controller Helio Notification Centre User Click

    Python Controller Event page.menu menu-change red menu-change red Event
  17. Menu Controller from  helio.controller.base  import  BaseViewController from  helio.helio_exceptions  import  ControllerImportError

    class  MenuViewController(BaseViewController):        component_name  =  'menu'        has_js  =  True        has_css  =  True        def  handle_notification(self,  notification_name,  data,  request,  **kwargs):                try:                        new_component_id  =  'colouredbackground.'  +  data['component']                        self.view_state.insert_new_controller('page.document',  new_component_id)                except  ControllerImportError:                        self.view_state.insert_new_controller('page.document',  data['component'])                self.nc.queue_load('page.document') Controller  =  MenuViewController
  18. Menu Controller from  helio.controller.base  import  BaseViewController from  helio.helio_exceptions  import  ControllerImportError

    class  MenuViewController(BaseViewController):        component_name  =  'menu'        has_js  =  True        has_css  =  True        def  handle_notification(self,  notification_name,  data,  request,  **kwargs):                try:                        new_component_id  =  'colouredbackground.'  +  data['component']                        self.view_state.insert_new_controller('page.document',  new_component_id)                except  ControllerImportError:                        self.view_state.insert_new_controller('page.document',  data['component'])                self.nc.queue_load('page.document') Controller  =  MenuViewController menu- change {‘component’: ‘red’} Insert here
  19. ViewState Tree page page . header page . menu page

    . document page . footer page header red menu footer
  20. from  helio.controller.base  import  BaseViewController from  helio.helio_exceptions  import  ControllerImportError class  MenuViewController(BaseViewController):

           component_name  =  'menu'        has_js  =  True        has_css  =  True        def  handle_notification(self,  notification_name,  data,  request,  **kwargs):                try:                        new_component_id  =  'colouredbackground.'  +  data['component']                        self.view_state.insert_new_controller('page.document',  new_component_id)                except  ControllerImportError:                        self.view_state.insert_new_controller('page.document',  data['component'])                self.nc.queue_load('page.document') Controller  =  MenuViewController Menu Controller menu- change {‘component’: ‘red’} Insert Queues a controller refresh
  21. Notification to Client JavaScript Controller Helio Notification Centre Python Controller

    Event page.document load <div class=“red… page.document load <div class=“red… Event Insert into DOM at controller path
  22. Server Side Notifications Controller Instance Helio Notification Centre Controller Instance

    Controller Instance Subscribe to notification A Subscribe to notification A
  23. heliomath Controller class  MathController(BaseViewController):        def  handle_notification(…):  

                 if  self.form.is_valid():                        self.result  =  self._do_operation(number1,  number2)                        self.nc.post_notification('result',  data=self.result)
  24. heliomath Controller class  MathController(BaseViewController):        def  handle_notification(…):  

                 if  self.form.is_valid():                        self.result  =  self._do_operation(number1,  number2)                        self.nc.post_notification('result',  data=self.result)
  25. Footer Controller class  FooterViewController(BaseViewController):        def  handle_notification(self,  notification_name,

     data,  …):                if  notification_name  ==  'result':                        if  self.result  !=  data:                                self.result  =  data                                self.queue_load()        def  context_setup(self):                self.context['result']  =  self.result
  26. heliomath Controller class  MathController(BaseViewController):        def  handle_notification(…):  

                 if  self.form.is_valid():                        self.result  =  self._do_operation(number1,  number2)                        self.nc.post_notification('result',  data=self.result)
  27. heliomath JavaScript postNotification(self.controllerPath,        'submit',      

       {                'form-­‐data':  $(this).serialize()        } );
  28. heliomath Controller class  NumericForm(forms.Form):        number1  =  forms.DecimalField(required=True)

           number2  =  forms.DecimalField(required=True) class  MathController(BaseViewController):        def  handle_notification(self,  notification_name,  data,  …):                if  notification_name  ==  'submit':                        qd  =  QueryDict(data['form-­‐data'])                        self.form  =  NumericForm(qd)                        if  self.form.is_valid():                                #  happy  path                        else:                                #  let  Django  render  the  form  with  errors
  29. heliomath Controller class  NumericForm(forms.Form):        number1  =  forms.DecimalField(required=True)

           number2  =  forms.DecimalField(required=True) class  MathController(BaseViewController):        def  handle_notification(self,  notification_name,  data,  …):                if  notification_name  ==  'submit':                        qd  =  QueryDict(data['form-­‐data'])                        self.form  =  NumericForm(qd)                        if  self.form.is_valid():                                #  happy  path                        else:                                #  let  Django  render  the  form  with  errors
  30. heliomath Controller class  NumericForm(forms.Form):        number1  =  forms.DecimalField(required=True)

           number2  =  forms.DecimalField(required=True) class  MathController(BaseViewController):        def  handle_notification(self,  notification_name,  data,  …):                if  notification_name  ==  'submit':                        qd  =  QueryDict(data['form-­‐data'])                        self.form  =  NumericForm(qd)                        if  self.form.is_valid():                                #  happy  path                        else:                                #  let  Django  render  the  form  with  errors
  31. Footer Controller class  FooterViewController(BaseViewController):        def  handle_notification(self,  notification_name,

     data,  …):                if  notification_name  ==  'result':                        if  self.result  !=  data:                                self.result  =  data                                self.queue_load()        def  context_setup(self):                self.context['result']  =  self.result
  32. helio.settings.TEMPLATE_RENDERER  =  'helio.heliodjango.renderers.render' STATICFILES_FINDERS  =  (        …

           'helio.heliodjango.finders.ComponentStaticFinder' ) TEMPLATE_LOADERS  =  (        …        'helio.heliodjango.finders.ComponentTemplateLoader' ) INSTALLED_APPS  =  (        …        'helio.heliodjango' ) urlpatterns  =  patterns('',        …        url('',  include('helio.heliodjango.urls')) )
  33. import  os import  cPickle  as  pickle import  base64 import  hmac

    import  hashlib import  random import  string import  datetime from  uuid  import  uuid4 from  collections  import  OrderedDict from  werkzeug.datastructures  import  CallbackDict from  flask.sessions  import  SessionInterface,  SessionMixin def  _generate_sid():        return  str(uuid4()) def  _calc_hmac(body,  secret):        return  base64.b64encode(hmac.new(secret,  body,  hashlib.sha1).digest()) class  ManagedSession(CallbackDict,  SessionMixin):        def  __init__(self,  initial=None,  sid=None,  new=False,  randval=None,  hmac_digest=None):                def  on_update(self):                        self.modified  =  True                CallbackDict.__init__(self,  initial,  on_update)                self.sid  =  sid                self.new  =  new                self.modified  =  False                self.randval  =  randval                self.hmac_digest  =  hmac_digest        def  sign(self,  secret):                if  not  self.hmac_digest:                        self.randval  =  ''.join(random.sample(string.lowercase+string.digits,  20))                        self.hmac_digest  =  _calc_hmac('%s:%s'  %  (self.sid,  self.randval),  secret) class  SessionManager(object):        def  new_session(self):                'Create  a  new  session'                raise  NotImplementedError        def  exists(self,  sid):                'Does  the  given  session-­‐id  exist?'                raise  NotImplementedError        def  remove(self,  sid):                'Remove  the  session'                raise  NotImplementedError        def  get(self,  sid,  digest):                'Retrieve  a  managed  session  by  session-­‐id,  checking  the  HMAC  digest'                raise  NotImplementedError        def  put(self,  session):                'Store  a  managed  session'                raise  NotImplementedError class  CachingSessionManager(SessionManager):        def  __init__(self,  parent,  num_to_store):                self.parent  =  parent                self.num_to_store  =  num_to_store                self._cache  =  OrderedDict()        def  _normalize(self):                print  "Session  cache  size:  %s"  %  len(self._cache)                if  len(self._cache)  >  self.num_to_store:                        while  len(self._cache)  >  (self.num_to_store  *  0.8):    #  flush  20%  of  the  cache                                self._cache.popitem(False)        def  new_session(self):                session  =  self.parent.new_session()                self._cache[session.sid]  =  session                self._normalize()                return  session        def  remove(self,  sid):                self.parent.remove(sid)                if  sid  in  self._cache:                        del  self._cache[sid]        def  exists(self,  sid):                if  sid  in  self._cache:                        return  True                return  self.parent.exists(sid)        def  get(self,  sid,  digest):                session  =  None                if  sid  in  self._cache:                        session  =  self._cache[sid]                        if  session.hmac_digest  !=  digest:                                session  =  None                        #  reset  order  in  OrderedDict                        del  self._cache[sid]                if  not  session:                        session  =  self.parent.get(sid,  digest)                self._cache[sid]  =  session                self._normalize()                return  session        def  put(self,  session):                self.parent.put(session)                if  session.sid  in  self._cache:                        del  self._cache[session.sid]                self._cache[session.sid]  =  session                self._normalize() lass  FileBackedSessionManager(SessionManager):        def  __init__(self,  path,  secret):                self.path  =  path                self.secret  =  secret                if  not  os.path.exists(self.path):                        os.makedirs(self.path)        def  exists(self,  sid):                fname  =  os.path.join(self.path,  sid)                return  os.path.exists(fname)        def  remove(self,  sid):                print  'Removing  session:  %s'  %  sid                fname  =  os.path.join(self.path,  sid)                if  os.path.exists(fname):                        os.unlink(fname)        def  new_session(self):                sid  =  _generate_sid()                fname  =  os.path.join(self.path,  sid)                while  os.path.exists(fname):                        sid  =  _generate_sid()                        fname  =  os.path.join(self.path,  sid)                #  touch  the  file                with  open(fname,  'w'):                        pass                print  "Created  new  session:  %s"  %  sid                return  ManagedSession(sid=sid)        def  get(self,  sid,  digest):                'Retrieve  a  managed  session  by  session-­‐id,  checking  the  HMAC  digest'                print  "Looking  for  session:  %s,  %s"  %  (sid,  digest)                fname  =  os.path.join(self.path,  sid)                data  =  None                hmac_digest  =  None                randval  =  None                if  os.path.exists(fname):                        try:                                with  open(fname)  as  f:                                        randval,  hmac_digest,  data  =  pickle.load(f)                        except:                                print  "Error  loading  session  file"                if  not  data:                        print  "Missing  data?"                        return  self.new_session()                #  This  assumes  the  file  is  correct,  if  you  really  want  to                #  make  sure  the  session  is  good  from  the  server  side,  you                #  can  re-­‐calculate  the  hmac                if  hmac_digest  !=  digest:                        print  "Invalid  HMAC  for  session"                        return  self.new_session()                return  ManagedSession(data,  sid=sid,  randval=randval,  hmac_digest=hmac_digest)        def  put(self,  session):                'Store  a  managed  session'                print  "Storing  session:  %s"  %  session.sid                if  not  session.hmac_digest:                        session.sign(self.secret)                fname  =  os.path.join(self.path,  session.sid)                with  open(fname,  'w')  as  f:                        pickle.dump((session.randval,  session.hmac_digest,  dict(session)),  f) class  ManagedSessionInterface(SessionInterface):        def  __init__(self,  manager,  skip_paths,  cookie_timedelta):                self.manager  =  manager                self.skip_paths  =  skip_paths                self.cookie_timedelta  =  cookie_timedelta        def  get_expiration_time(self,  app,  session):                if  session.permanent:                        return  app.permanent_session_lifetime                return  datetime.datetime.now()  +  self.cookie_timedelta        def  open_session(self,  app,  request):                cookie_val  =  request.cookies.get(app.session_cookie_name)                if  not  cookie_val  or  not  '!'  in  cookie_val:                        #  Don't  bother  creating  a  cookie  for  static  resources                        for  sp  in  self.skip_paths:                                if  request.path.startswith(sp):                                        return  None                        print  'Missing  cookie'                        return  self.manager.new_session()                sid,  digest  =  cookie_val.split('!',  1)                if  self.manager.exists(sid):                        return  self.manager.get(sid,  digest)                return  self.manager.new_session()        def  save_session(self,  app,  session,  response):                domain  =  self.get_cookie_domain(app)                if  not  session:                        self.manager.remove(session.sid)                        if  session.modified:                                response.delete_cookie(app.session_cookie_name,  domain=domain)                        return                if  not  session.modified:                        #  no  need  to  save  an  unaltered  session                        #  TODO:  put  logic  here  to  test  if  the  cookie  is  older  than  N  days,  if  so,  update  the  expiration  date                        return                self.manager.put(session)                session.modified  =  False                cookie_exp  =  self.get_expiration_time(app,  session)                response.set_cookie(app.session_cookie_name,                                                        '%s!%s'  %  (session.sid,  session.hmac_digest),                                                        expires=cookie_exp,  httponly=True,  domain=domain)
  34. Standard Interface Get ViewState Retrieve controller data (HTML, JavaScript and

    CSS) Receive notification from server Helio tells the framework how to find templates and static files You tell Helio what renderer to use
  35. Requests (Django Only) URL Time (s) Size /longtable/ 1.688596 1,599,182b

    /longtable/ Y9MyJ 1.781953 1,599,226b /longtable/ 8wQCz 1.843202 1,599,226b Average 1.771250 1,599,211b
  36. Requests (Django w/Helio) URL Time (s) Size / 0.006202 980b

    /get-view- state/ 0.004864 1b /longtable/ 8wQCz 1.843202 1,757,977b …longtable.js 0.002 417b detail-update 0.003659 138b detail-update 0.003222 138b detail-update 0.002332 138b Detail Average 0.003071 138b POSTs
  37. 0s 3.75s 7.5s 11.25s 15s 1 2 3 4 5

    6 7 Helio Django Only Cumulative Load Time
  38. 0MB 2.75MB 5.5MB 8.25MB 11MB 1 2 3 4 5

    6 7 Helio Django Only Cumulative Data Transfer