Storage in the broser ndre Betts London JS, 16 pril 2012

;

elcome back to the eb The FT eb app provides a touch optimised user experience ithout native code.

hy e need local storage •  Reduce aiting times •  Launch hile offline •  Maintain state

vailable technologies •  HTML5 application cache •  localStorage and sessionStorage •  ebSQL and IndexedDB •  Cookies

Cookies Tiny size limit Sent in every request to the server Idiots in Brussels really hate them So retro your dad’s probably heard of them Universally supported (almost)

IndexedDB / ebSQL ebSQL died the moment it as born So slo it’s sometimes faster to query the netork (yes really) Only accessible from JavaScript Indexed/queryable data store in the broser. Marvelous. Reasonably ide support if you use both

localStorage Synchronous. JS execution blocks on IO Limited to lookup by key Only accessible from JS Simple persistent key/value store Really fast

HTML5 application cache tomic, nested, impossible to version effectively, buggy JS PI, hard to invalidate, etc etc. (ho long do e have?) Caches HTTP responses, so can serve first page request, before JS PIs are available. Easy to cache binary files

No one technology solves all the problems e need to use them all

•  Load an article page hile offline •  Not in broser cache, not in app cache •  pp cache serves fallback

•  Fallback page acts as client side front controller •  Load main JS, CSS, and HTML from localStorage •  Check for errors, trigger startup. Splash  screen   Basic  CSS   Error  msgs  /  support  info   Boot  code  

Splash  screen   Basic  CSS   Error  msgs  /  support  info   Boot  code   Main  app  code   Main  app  CSS   App  HTML  templates   •  Check cookie for user authentication •  Remove redundant content only needed for boot •  Examine URL, read requested content from ebSQL •  Render page

Storing content images •  Can’t use app cache, because it’s aful. •  Other storage options don’t offer a URL •  Base-64 encode image data •  Store in localStorage or ebSQL •  Insert into DOM as data: URIs •  30% bloat, but also avoids netork operator recompression

Scaling don ll storage technologies are limited by tiny quotas. e need to learn to live ith less.

ebSQL () / IndexedDB (i) ppcache localStorage Cookies iOS (5.1) : 5MB / 50MB 10MB 5MB 4KB total Up to 50 per domain ndroid (4) ? Unlimited 5MB Safari (5.2) : 5MB / Unlimited Unlimited 5MB Chrome (18) : Unlimited 5MB 5MB IE (10) i: 500MB 50MB 10MB Opera (11) : 5MB / Unlimited 50MB / Unlimited 5MB / Unlimited Firefox (11) i: 50MB / Unlimited Unlimited 10MB Storage limits Two  figures  separated  by  /  indicate  limit  with  and  without  user  consent  prompt,  which  appears  automaBcally  when  you  try  to   store  more  than  the  lower  limit.    The  user  may  not  consent  to  more  than  the  upper  limit.  

Getting creative ith limited storage

Let’s talk compression •  JavaScript internally uses UTF-16 for text encoding •  Great idea for processing: fast string operations, full support for Unicode BMP •  Terrible idea for storage of English text or base-64 encoded images.

To bytes per character •  Provides great compatibility but halves precious quota! •  Can make compromises on compatibility to get more storage efficiency 00   48   00   45   00   4C   00   4C   00   4F   00   20   00   57   00   4F   00   52   00   4C   00   44   H   E   L   L   O   W   O   R   L   D  

Convert to UTF-8? •  Obvious strategy •  Use Byterray in JavaScript to store UTF-8 data •  Convert back to string pretending UTF-8 chars are UTF-16 ones (ssssh) •  Just under 50% compression ratio •  Good: Efficient storage •  Bad: Slo to convert, lots of code

UTF-8 hiding inside UTF-16 UTF-­‐8   R   o   y   ’   s   Decimal   82   111   121   8217   115   As  binary   01010010   01101111   01111001   11100010  10000000  10011001   01110011   ShiSed   01010010  01101111   01111001  11100010   10000000  10011001   01110011   As  hex   52  6F   79  E2   80  99   73  00   UTF-­‐16   副   秢   肙   猀  

eird. Not your father’s compression algorithm, that’s for sure.

Use SCII? •  Replace all non-SCII chars ith HTML entities server side •  Read char codes in pairs, construct ne string ith one char bitshifted on top of the other. •  Good: Fast, almost as efficient as UTF-8, code is tiny.

SCII in UTF-16 UTF-­‐8   S   i   m   p   l   e   Decimal   83   105   109   112   108   101   As  binary   01010011   01101001   01101101   01110000   01101100   01100101   ShiSed   01010011  01101001   01101101  01110000   01101100  01100101   As  hex   53  69   6D  70   6C  65   UTF-­‐16   卩   浰   汥  

SCII to UTF-16 function  compress(s)  {    var  i,  l,  out='';    if  (s.length  %  2  !==  0)  s  +=  '  ';    for  (i  =  0,  l  =  s.length;  i  <  l;  i+=2)  {      out  +=  String.fromCharCode((s.charCodeAt(i)*256)  +                s.charCodeAt(i+1));    }    return  out;   }  

Bit-shifting base-64 data? •  Could go further if e kno e’re only using base 64. •  64 characters requires only six bits •  Potential 63% compression ratio •  6 divides into 48 •  8 base-64 chars per 3 UTF-16 chars

Base64 in UTF-16 Original   A   B   C   D   o   p   q   9   b64  index   0   1   2   3   40   41   42   61   As  binary   000000   000001   000010   000011   101000   101001   101010   111101   ShiSed   00000000  00010000   10000011  10100010   10011010  10111101   As  hex   00  10   83  A2   9A  BD   UTF-­‐16   ☐   莢   丢  

Base64 to UTF-16 function  compress(s)  {    var  i,  l,  out='',  map={A:0,  B:1,  C:2,  ...  "+":62,  "/":63};    var  bits  =  16,  chr  =  0,  rem  =  0;    for  (i  =  0,  l  =  s.length;  i  <  l;  i++)  {      if  (bits  >  6)  {        bits  -­‐=  6;  chr  +=  map[i]  <<  bits;        }  else  {        rem  =  6  -­‐  bits;        chr  +=  map[i]  >>  rem;        out  +=  String.fromCharCode(chr);        chr  =  (map[i]  %  rem)  <<  (16-­‐rem);        bits  =  16  –  rem;      }    }    return  out;   }  

Summary •  Client side storage is essential part of ne generation eb apps •  Currently hampered by fragmented implementation and tiny quota limits •  ith effort, can be used to great effect

Thanks andre@labs.ft.com @triblondon Do you ant to build this stuff? Join in. [email protected]