Offline rules (BDConf 2013)

In the last couple of years a deluge of new web technologies have appeared, allowing for the creation of ever richer and more immersive web applications. The FT is one of the pioneers in the use of newly minted HTML5 technologies to build web apps that are virtually indistinguishable from native apps. But using these technologies is far from easy. I'll cover some of the compelling reasons for choosing HTML5 and investing in the web platform, with a focus on offline technologies that enable web apps to run without a network connection. The examples will feature real-life code from our FT and the Economist applications, so you know that the techniques here are applicable to large, complex problems.

Transcript

1. Offline rules
   Taking the FT offline on the eb
   ndre Betts (@triblondon)
   FT Labs (@ftlabs)
   

   View Slide

2. 1880 1900 1920 1940 1960 1980 2000 2020
   e’ve been doing this for a hile.
   

   View Slide

3. orks offline
   Portable
   Long battery life
   Can be read in bright sunlight
   Cheap
   Ubiquity
   Bookmarking
   Sharing
   Nesprint is a great client
   Fast start up
   Clipping/saving
   Can be read in the dark
   Updates in real time
   Electronic delivery
   Search
   Personalisation
   Deep linking
   

   View Slide

4. orks offline
   Portable
   Long battery life
   Can be read in bright sunlight
   Cheap
   Ubiquity
   Bookmarking
   Sharing
   ‘Traditional’ eb
   Fast start up
   Clipping/saving
   Can be read in the dark
   Updates in real time
   Electronic delivery
   Search
   Personalisation
   Deep linking
   

   View Slide

5. orks offline
   Portable
   Long battery life
   Can be read in bright sunlight
   Cheap
   Ubiquity
   Bookmarking
   Sharing
   pps
   Fast start up
   Clipping/saving
   Can be read in the dark
   Updates in real time
   Electronic delivery
   Search
   Personalisation
   Deep linking
   

   View Slide

6. e need to care about
   supporting existing
   features
   as much as getting ne ones.
   

   View Slide

7. HTML5 is not a ‘mobile thing’.
   It’s not an alternative to native apps.
   It’s a ay of making better ebsites.
   

   View Slide

8. Lose the constraints
   •  Tablet
   •  Phone
   •  Desktop
   •  Laptop
   •  TV
   •  Games console
   •  In car screen
   •  In flight screen
   •  Billboard
   •  Kiosk
   •  e-ink reader
   

   View Slide

9. Screenshot  of  the  app  
   

   View Slide

10. Offline vs dependency management
    Offline vs responsive images
    Offline vs media queries
    Offline vs analytics
    Offline vs login state
    

    View Slide

11. Offline makes
    everything harder.
    

    View Slide

12. Client side storage options
    Meet the family from hell.
    

    View Slide

13. ppCache
    Teen
    localStorage
    Dad
    IndexedDB
    Mum
    Cookies
    Grandad
    Files PI
    Imaginary
    friend
    

    View Slide

14. View Slide

15. appCache
    indexedDB
    

    View Slide

16. Speed: read/rite cycles
    242  
    296  
    175  
    259  
    6873  
    29897  
    1 10 100 1000 10000 100000
    IDB Large
    IDB Small
    ebSQL Large
    ebSQL Small
    localStor Large
    localStor Small
    Operations per second (more is better)
    h7p://jsperf.com/indexeddb-­‐vs-­‐localstorage/13  
    

    View Slide

17. Bootstrapping
    Start small and enhance
    

    View Slide

18. CONTENT  
    adver  
    Jsing  
    adverJsing   adverJsing  
    twaddle  
    crap  
    nonsense  
    tomfoolery  
    tripe  
    baloney  
    garbage  
    poppycock  
    trash  
    rubbish  
    filler  
    irrelevance  
    faff  
    hidden  
    unseen  
    ferreted  
    concealed  
    invisible  
    buried  
    latent  
    sequestered  
    shrouded  
    unrevealed  
    withheld  
    

    View Slide

19. hy Bootstrap?
    •  Keep content clean
    •  Start ‘enhanced experience’ on any URL
    – Need front-end router / front controller
    •  Handle errors
    •  Run only hat e need
    •  Mollify attention seeking app cache
    

    View Slide

20. Main content
    Basic CSS / fonts
    Boot and error handling code
    JS modules
    CSS modules
    HTML fragments
    dditional content
    Initial pageload
    appCache/Netork
    Modules and dependencies
    Local storage / appCache / netork
    uthentication tokens
    Cookies
    Extra content and prefetching
    ebSQL / IndexedDB / netork
    uthentication token
    

    View Slide

21. Taming the appCache
    Grump grump grump grump grump...
    

    View Slide

22. CACHE MANIFEST
    # 14120120420
    /
    /favicon.ico
    /lib/fonts/ofsnmd__.ttf
    /lib/imgs/sprite.png
    FALLBACK:
    /issues /fb/issues
    /api /fb/api
    NETWORK:
    *
    Manifest version
    Explicit caching
    llo stuff not in appcache
    Offline controllers
    

    View Slide

23. ny page ith manifest attribute ill
    get cached, like it or not.
    ny cached page ill be used in
    preference to fetching a fresh copy,
    even hen online.
    

    View Slide

24.  
     
    

    View Slide

25. Only cache one ‘master’ URL:
    ...
    
    

    View Slide

26. Internet Explorer ill not populate
    an appCache unless the manifest is
    cacheable.
    Firefox on’t store any resource if
    it’s served ith no-store, even if
    explicitly listed in the manifest.
    

    View Slide

27. Cache-control: max-age=1, private;
    Prevent proxy caching
    

    View Slide

28. hen the manifest is updated,
    every entry in it is redonloaded
    

    View Slide

29. CACHE MANIFEST
    # 14120120420
    /js/mymodule1.js?v=1.15
    /js/mymodule2.js?v=1.18
    /js/mymodule3.js?v=1.1
    /js/mymodule4.js?v=1.18
    /js/mymodule5.js?v=1.19
    FALLBACK:
    /issues /fb/issues?v=2.34
    /api /fb/api?v=2.34
    NETWORK:
    *
    Manifest version
    Far future cached
    resources
    

    View Slide

30. ppCache
    HTTP
    Cache
    Server
    Updated manifest
    Donload
    just one
    

    View Slide

31. On slo netorks a request must
    time out before the fallback is used.
    

    View Slide

32. View Slide

33. Ideas for fixing it:
    •  Separate the cache and offline router
    •  More granular atomicity
    •  Read/rite/load/purge PI in JavaScript
    •  Spec the quota behaviour
    

    View Slide

34. •  Redefine appcache syntax
    •  Still mostly declarative
    •  Sub-manifests: multiple atomic
    groups
    •  JS PI to bring transparency
    •  eb orker option for
    advanced usage
    •  Pattern matching fallbacks
    •  Remove the ‘magic’, like auto-
    adding masters
    •  Ditch appcache, define ne PI
    •  lmost entirely imperative
    •  Individual atomic cache groups
    accessed via PIs
    •  Install a controller to intercept
    and route requests
    •  “Bring your on unicorn”
    EVOLUTION
    REVOLUTION
    

    View Slide

35. Mozilla proposed manifest
    {
    "expiration": 300,
    "cache": ["index.html", "index.js", "index.css",
    "show_bug_handler.html", "forum.html"],
    "urlmap": [
    {
    url: "show_bug?id=*",
    page: "show_bug_handler.html"
    },
    ],
    "network-controller": "httpworker.js”
    }
    

    View Slide

36. Google proposed controller
    navigator.controller.register("/*", "/navctrl.js")
    URL space to
    control
    Path to controller
    script
    

    View Slide

37. this.caches.set(”app", new Cache([
    "/resources/js/bundle.js?v=2.3",
    "/resources/css/bundle.css?v=4",
    "/resources/html/templatepack.js?v=1.1",
    "/fallbacks/main.html"
    ]));
    this.caches.set("edition-20130301", new Cache([
    "/issues/20130301/uk/article1.html",
    "/issues/20130301/world/article3.html",
    "/issues/20130301/media/images/chavez.jpg",
    "http://cdn.example.com/audio/readaloud.mp3"
    ]));
    pplication
    (high priority)
    Content
    (loer priority)
    navcntl.js – setting up caches
    

    View Slide

38. this.onrequest = function(e) {
    var resource = this.caches.get("edition-20130301").match(e.url);
    if (resource) {
    e.preventDefault();
    e.respondWith(resource);
    return;
    }
    }
    lays serve from cache
    (if available)
    navcntl.js – handling requests
    

    View Slide

39. this.onrequest = function(e) {
    if (this.onLine) return;
    var resource = this.caches.get("edition-20130301").match(e.url);
    if (resource) {
    e.preventDefault();
    e.respondWith(resource);
    return;
    }
    }
    Or, only if offline
    (if you prefer)
    navcntl.js – handling requests
    

    View Slide

40. this.onrequest = function(e) {
    if (this.onLine) return;
    var resource = this.caches.get("edition-20130301").match(e.url);
    if (resource) {
    e.preventDefault();
    e.respondWith(resource)
    } else {
    e.preventDefault();
    var fallback = this.caches.get("app")
    .match('/fallbacks/main.html');
    e.respondWith(fallback);
    }
    }
    navcntl.js – handling requests
    Choose to return a
    different resource
    

    View Slide

41. hich should in?
    •  Both?
    •  NavigationController offers rich and highly
    granular control to advanced developers
    •  Moz advanced appCache provides the
    declarative option for simpler use cases
    •  NC could be the ‘netork-controller’ of the
    Moz manifest.
    

    View Slide

42. •  NetorkController
    •  ResourceController
    •  ProxyController
    •  FetchController
    •  NavigationController
    •  CacheController
    

    View Slide

43. Meanhile...
    

    View Slide

44. Meanhile... appCache is good for:
    YES:
    •  Fonts
    •  Splash image
    •  pp icon
    •  Entry page
    •  Fallback bootstrap
    NO:
    •  CSS
    •  HTML
    •  JavaScript
    Local
    Storage
    

    View Slide

45. Storage optimisation
    ll storage technologies are limited by tiny
    quotas. e need to learn to live ith less.
    

    View Slide

46. ebSQL () /
    IndexedDB (i)
    ppcache localStorage Cookies
    iOS Safari (6.1) : 5MB / 50MB 25MB 5MB
    4KB total
    Up to 50
    per domain
    MacOS Safari (5.2) : 5MB / Unlimited Unlimited 5MB
    ndroid broser (2.3) ? Unlimited 5MB
    Chrome (26) Unlimited 260MB (ish) 5MB
    IE (10) i: 500MB 50MB 10MB
    Opera (12) : 5MB / Unlimited 50MB / Unlimited 5MB / Unlimited
    Firefox (20) i: 50MB / Unlimited Unlimited 10MB
    Storage limits
    Two  figures  separated  by  /  indicate  limit  with  and  without  user  consent  prompt,  which  appears  automaJcally  when  you  try  to  
    store  more  than  the  lower  limit.    The  user  may  not  consent  to  more  than  the  upper  limit.  
    

    View Slide

47. UTF-16 in JavaScript
    •  Provides great compatibility but halves
    precious quota!
    •  Can make compromises on compatibility
    to get more storage efficiency
    00   48   00   45   00   4C   00   4C   00   4F   00   20   00   57   00   4F   00   52   00   4C   00   44  
    H   E   L   L   O   W   O   R   L   D  
    

    View Slide

48. View Slide

49. Squeezing your bits
    Text   S   i   m   p   l   e  
    Decimal   83   105   109   112   108   101  
    As  binary   01010011   01101001   01101101   01110000   01101100   01100101  
    Shi[ed   01010011  01101001   01101101  01110000   01101100  01100101  
    As  hex   53  69   6D  70   6C  65  
    UTF-­‐16   卩   浰   汥  
    

    View Slide

50. SCII packed into UTF-16!
    Simple = 卩浰汥
    6 chars 3 chars
    

    View Slide

51. View Slide

52. SCII to UTF-16
    function  compress(s)  {  
     var  i,  l,  out='';  
     if  (s.length  %  2  !==  0)  s  +=  '  ';  
     for  (i  =  0,  l  =  s.length;  i  <  l;  i+=2)  {  
       out  +=  String.fromCharCode((s.charCodeAt(i)<<8)  +  
                 s.charCodeAt(i+1));  
     }  
     return  out;  
    }  
    

    View Slide

53. Too simple?
    

    View Slide

54. Bit-shifting base-64 data?
    •  Could go further if e kno e’re only using
    base 64.
    •  64 characters requires only six bits
    •  Potential 63% compression ratio
    •  6 divides into 48
    •  8 base-64 chars per 3 UTF-16 chars
    

    View Slide

55. Base64 hiding in UTF-16
    Original   A   B   C   D   o   p   q   9  
    b64  index   0   1   2   3   40   41   42   61  
    As  binary   000000   000001   000010   000011   101000   101001   101010   111101  
    Shi[ed   00000000  00010000   10000011  10100010   10011010  10111101  
    As  hex   00  10   83  A2   9A  BD  
    UTF-­‐16   ☐   莢   丢  
    

    View Slide

56. UTF-8 hiding inside UTF-16
    UTF-­‐8   R   o   y   ’   s  
    Decimal   82   111   121   8217   115  
    As  binary   01010010   01101111   01111001   11100010  10000000  10011001   01110011  
    Shi[ed   01010010  01101111   01111001  11100010   10000000  10011001   01110011  
    As  hex   52  6F   79  E2   80  99   73  00  
    UTF-­‐16   副   秢   肙   猀  
    

    View Slide

57. Connection state
    Taking online and offline events
    at face value
    

    View Slide

58. Q: If the value of
    indo.navigator.onLine is true, hat
    does that mean?
    : The device might be online.
    

    View Slide

59. Donloading!
    Not sure.
    Still trying...
    Bah.
    e give up.
    

    View Slide

60. Open source releases
    e did it already. Use our code.
    

    View Slide

61. View Slide

62. FT Columnflo
    orld's most advanced column layout tool
    Forked by The Telegraph and Baidu
    bit.ly/Mmhd1z
    

    View Slide

63. FT Scroller
    Momentum scrolling support ith ultra ide
    compatibility including Internet Explorer 10
    bit.ly/SJcJR4
    

    View Slide

64. FT Fastclick
    Virtually instant responsiveness for touch UIs
    bit.ly/Ndvbmo
    

    View Slide

65. A new kind of one-day conference on
    advanced web technologies for
    developers and browser vendors.
    September 2013, New York City
    edgeconf.com
    

    View Slide

66. Thanks
    andre@labs.ft.com
    @triblondon, @FTLabs
    

    View Slide