Strategies for Working with Offline Data

When your app needs to operate without a network connection, you will need to consider your options for offline or cached data. This session will explore methods for storing data locally and techniques for synchronizing with the backend when your app is online again.

Transcript

1. Andrew Duncan, Co-founder, SwarmOnline @andrewmduncan andrew@swarmonline.com Strategies for Working with Offline Data
3. Agenda • Application Cache vs Offline Storage • Data storage options • Connecting offline storage to the server • Considerations - Managing Conflicts - Deltas - When to Sync - Sync Frameworks - Security/Privacy - Hybrid App Containers - Performance
4. Application Cache (AppCache) • Application logic • Browser caches files • Specified by developer • Advantages - App works without connectivity - Increased load times - Reduced server load • Cache Manifest • Hybrid App Containers - package resources, e.g. static images, large documents etc...
5. - Eric Bidelman Offline storage is about capturing specific data generated by the user, or resources the user has expressed interest in.
6. Reasons for going o ine
7. Make the app work o ine
8. Improve performance
9. Where can we store our Data?
10. LocalStorage Cookies WebSQL IndexedDB SessionStorage
11. Memory • Reads data from local memory • Contents are not retained between sessions • For reading only • Useful to help you translate your data into the correct format in your model.
12. Memory Proxy Ext.define('MyApp.store.Location',  {        extend:  'Ext.data.Store'          autoLoad:  true,        model:  'Location',        proxy:  {                type:  'memory',                reader:  {                        type:  'json'                }        } });
13. SessionStorage Proxy • Extends WebStorage Proxy • SessionStorage property on Window object - accessible from window.SessionStorage • String Key/Value Pairs • Lifetime - data deleted when browsing session ends • Scope - document origin - per-window basis • Size limit - system memory
14. Session Proxy Ext.define('MyApp.store.Location',  {        extend:  'Ext.data.Store'          autoLoad:  true,        model:  'Location',        proxy:  {                type:  'sessionstorage',                id:  'LocationsProxy',                reader:  {                        type:  'json'                }        } });
15. LocalStorage • LocalStorage property on Window object • Lifetime - permanent - deleted by the app - deleted by the user • Scope - document origin - protocol, hostname & port must be the same - browser vendor • Size Limit - 5MB
16. LocalStorage Proxy Ext.define('MyApp.store.Location',  {        extend:  'Ext.data.Store'          autoLoad:  true,        model:  'Location',        proxy:  {                type:  'localstorage',                id:  'LocationsProxy',                reader:  {                        type:  'json'                }        } });
17. WebSQL Database • Proxy added in Sencha Touch 2.1 • Size - defaults to 5MB - developer can request more space from user • Lifetime - permanent - deleted by the app - deleted by the user • Scope - document origin - protocol, hostname & port must be the same - browser vendor • Rollbacks & Data Integrity
18. SQL Proxy Ext.define('MyApp.store.Location',  {        extend:  'Ext.data.Store'          autoLoad:  true,        model:  'Location',        proxy:  {                type:  'sql',                database:  'Assets',                table:  'AssetLocations',                reader:  {                        type:  'json'                }        } });
19. IndexedDB • No SQL • Key-value mapping • Better performance through indexes • Limited Support
20. Enabling O ine Working
21. Central Storage Local Storage Mobile Application Web Server SYNC
22. Multiple Proxies may help • Data can be changed in two places - In the backend e.g. by external systems - Requires a connection via a Server Proxy - On the client - Requires a Client Proxy • Benefits are more than just offline working - Perceived performance boost as syncing is local and server syncing can be done in the background - Alleviates slow network connectivity or slow backend • We need a way of knowing/merging the data between the proxies
23. Ext.ux.O ineSyncStore https://github.com/SwarmOnline/Ext.ux.OfflineSyncStore
24. Overview • Sencha Touch plugin • Extends Ext.data.Store • Two proxies - LocalProxy - ServerProxy • Persists a local copy • Keeps a record of added, updated and deleted records • Syncs these changes with your server • Enables you to control when syncing takes place
25. OfflineSyncStore Ext.data.Store LocalProxy Model Ext.data.Model Client Side Storage ServerProxy Backend Systems ModifiedRecordsCollection
26. Ext.define('Example.store.Person',  {        extend:  'Ext.ux.OfflineSyncStore',        config:  {                model:  'Example.model.Person',                localProxy:  {                        type:  'localstorage',                        id:  'offline-­‐sync-­‐store'                },                serverProxy:  {                        type:  'ajax',                        api:  {                                read:  '../../server/select.php',                                create:  '../../server/save.php'                        },                        reader:  {                                type:  'json',                                rootProperty:  'rows'                        },                        writer:  {                                allowSingle:  false                        }                }        } });
27. Conflicts between the Proxies • Dirty and phantom records are no longer known • We keep our own record of these - to avoid overwriting data between the proxies and a server load
28. ServerProxy is synced New model added to store in ‘offline’ mode
29. Managing Conflicts
30. Conflicts with the Backend • Timestamp all data • Keep a version number • Possible Solutions - Last write wins - trust the local time to be truthful - Privilege • Application dependent, no right or wrong answer
31. When Should I Sync?
32. When Should I Sync? • Try all the time • Ajax request will fail if no connectivity. • Stack up failures and resend • Time driven • Event/Action driven • Monitoring Connectivity • Phonegap • navigator.connection.type • Sencha Touch • Ext.device.Connection • HTML5 • navigator.onLine • definition of ‘online’ varies • local network or internet • Solution is detect a heartbeat Poor Connectivity As soon as connectivity is regained? Good Connectivity All the time?
33. Data Loss During the Sync • Problem - New data can be lost if the store is currently ‘syncing’ • Cause - Slower & older devices - large datasets • Possible Solution - Pause user interaction while sync is in progress. - Framework Override to stop the date clear happening so early.
34. Cross Platform Considerations
35. Deltas
36. Deltas • Improves performance / Reduces Bandwidth • Require your app to have knowledge of when its data was last updated • Server only sends newly updated between the current time and the last sync • Data is merged with store • Sencha Touch handles this well for us already
37. Other tools that might help?
38. Is the hardwork done for us? • RhoConnect • WebSqlSync - https://github.com/orbitaloop/WebSqlSync/ • PouchDB
39. Storage, Privacy & Security Considerations
40. HTML5 Storage Security • Do not store session identifiers - Cookies can use the httpOnly flag - SessionStorage might be ok for • Encryption is an option - Stanford Javascript Crypto Library - http://crypto.stanford.edu/sjcl/ - AES - Override the proxy
41. Hybrid App Containers • RhoMobile - Rhom Local database - SQLite - Database encryption • PhoneGap - Use SQLLite Plugin - SQLCipher - Open Source - 256-bit encryption - http://brodyspark.blogspot.co.uk/ - Don’t store the key - derive from users password • Sencha Space
42. Remote Wiping Data • Use a mobile device management suite - AirWatch - Soti MobiControl • Sencha Space
44. Hackathon
45. Take the Survey! • Session Survey - Available on the SenchaCon mobile app - http://app.senchacon.com • Be Social! - @SenchaCon - #SenchaCon - @andrewmduncan
46. Questions?