Immer & überall: Offlinefähige Progressive Web Apps – am Beispiel Angular

Immer & überall: Offlinefähige Progressive Web Apps – am Beispiel Angular

“Keine Internetverbindung” – ein Satz, den wir alle aus genutzten Apps kennen, sei es Desktopanwendung oder Smartphone App. Oftmals sind die Clients auf dem jeweiligen Zielgerät installiert, bedienen sich aber einer externen Datenquelle, bspw. in Form einer Web API. Bei einer offline-fähigen Implementierung sind nicht nur Lese-, sondern vor allem auch Schreibzugriffe wichtig. Sprich, die Daten müssen in beide Richtungen synchronisiert werden können, sobald die Internetverbindung wiederhergestellt wurde. Gerade in der heutigen Zeit, in der Menschen immer mobiler werden, in Autos, Zügen und Flugzeugen unterwegs sind und arbeiten, benötigt man oftmals offline Zugriff auf die Anwendungsdaten. Wie man diese echt offline-fähigen Single-Page Applications/Progressive Web Apps realisiert, zeigen Thomas Hilzendegen und Manuel Rauber von Thinktecture in diesem Ganztagesworkshop. Allgemein werden Themen wie Synchronisationsstrategien von Anwendungsdaten und großen Binärdaten, Konfliktmanagement oder Datenbankarchitektur sowohl für neue als auch bestehende Anwendungen diskutiert. Am Beispiel einer Angular-basierten App mit einem exemplarischen Backend in .NET Core werden Client und Server zur Umsetzung der besprochenen Konzepte und Patterns implementiert.



Manuel Rauber

October 10, 2019


  1. Offlinefähige Progressive Web Apps Immer & überall Thomas Hilzendegen @hilzendegent

    Consultant Manuel Rauber @ManuelRauber Consultant
  2. Thomas Hilzendegen Speakers Manuel Rauber @hilzendegent @manuelrauber

    Microsoft MVP
  3. Timetable Time Doing 09:00 - 10:30 Part I 10:30 -

    11:00 ☕ 11:00 - 12:30 Part II 12:30 - 13:30 13:30 - 15:00 Part III 15:00 - 15:30 ☕ 15:30 - 17:00 Part IV
  4. • Progressive Web App • Offline availability • Offline synchronization

    • Change tracking • Security considerations • Pitfalls Agenda
  5. Intro

  6. Offline… but why?

  7. • Just because the phone has a connection,
 e.g. Edge,

    it does not mean we are online • Depends on the use case which connection
 quality is required to determine, if the 
 app is reliable online • Connection quality could be measured
 by the time an exclusive request takes • Duration < 150 ms: online • Duration !>=150 ms: bad connection What does “offline” mean?
  8. • A lot of reasons to be offline • Traveling,

    Train, Flights (bad or no signal) • Server is not available • Routing problems • Roaming • Costs Motivation
  9. Demo Application

  10. • A little application to manage boardgames • Angular, .NET

    Core, MS SQL Server • Brownfield application, started as a pure online application • As typical for any good demo: no security • Available on Azure: • GitHub: Demo Application “Thinktecture Boardist”
  11. Demo Application Database Scheme

  12. Live Demo

  13. Now let’s go offline!

  14. Does it work in flight mode? NOPE!

  15. Progressive Web App

  16. • Motivation: Get rid of app stores! • Web App

    = App App • Support native features like: Push notifications, offline availability, installable • Backwards-compatibility: runs in non-PWA browsers, with a reduced feature set Progressive Web App
  17. PWA Technology Overview Progressive Web Apps HTML5, JavaScript, CSS3 Service

    Worker API Web App Manifest HTTPS Fetch API Web Notifications Web Workers Push API
  18. PWA Platform Support 40 44 11.1 17 4.1 11.3

  19. Progressive Web App 
 is not a technology, 

    a collection of features an application must/should support
  20. PWA Collection of Features Responsive Linkable Discoverable Installable App-Like Connectivity

    Independent Fresh Safe Re-engageable Progressive
  21. • A JavaScript running in its own thread • No

    access to the DOM • Communicates via postMessage-API • Acts as a controller/proxy/interceptor • Can perform background tasks • Has to be installed before usage • Runs whenever the browser is running PWA ServiceWorker
  22. PWA ServiceWorker Website Internet Cache Service Worker

  23. PWA ServiceWorker Lifecycle Parse Installing Error Activated Idle Terminated fetch/message

  24. PWA Connectivity Independent Website Internet Cache storage Server Remote storage

  25. • Cache only • Network only • Cache falling back

    to network • Cache & network race • Network falling back to cache • Cache then network • Generic fallback • ServiceWorker-side templating PWA Caching Strategies
  26. PWA Cache Then Network Website Internet Cache storage Server Remote

    storage ServiceWorker HTTP 1. Lookup 2. fetch 1. Lookup
  27. • “One-size-fits-all” ServiceWorker implementation • Instrumented via ngsw-config.json • Restricted

    feature set (e.g. no communication between app and SW) • Initial caching of static content • Caching external content • Dynamic data caching • Push notifications PWA Angular ServiceWorker
  28. Live Demo

  29. Does it work offline now? STILL NOPE!

  30. Real offline availability

  31. • ServiceWorker is only able to take data offline which

    has been requested by the application • If all the URLs are known beforehand, the ServiceWorker could cache them all • Data which was not requested, is not available offline (no real offline synchronisation) • But what about … the Background Sync API? What’s the problem?
  32. • Name is misleading • Does not offer any data

    synchronisation possibilities • Just raises an event periodically or after a delay (optionally based on the network type) • It’s totally up to the developer what to do in the onsync event • Since the sync is done in the ServiceWorker, the page can be closed, the sync will be fulfilled anyway • It does not help anything with syncing your actual data! Background Sync API
  33. Background Sync API

  34. • Cookies (not meant for large data or binaries) •

    Web Storage like session storage or local storage (not meant for large data or binaries) • IndexedDB • Cache Storage (based on request/response) Storage capabilities
  35. User can clear anything at any time by “Remove temporary

    internet files” or “Clear website data”
  36. • Key-value database of the browser • Stores data permanently

    • ServiceWorker and Web App share access to the same IndexedDB • Possibility of scenarios, where the ServiceWorker (or Web App) stores synchronised data in the IndexedDB and the Web App reads the data • Like everything based on the origin IndexedDB
  37. IndexedDB Availability

  38. • The standard API of IndexedDB is inconvenient to use

    (lots of callback) • Dexie.js is a minimalistic wrapper for IndexedDB • Operations are promise-based • Near native performance (even for bulk inserts) • Open Source @ GitHub: IndexedDB API
  39. Offline Sync

  40. • Offline Sync means to download all data available to

    client into a persistent offline storage, without the user having to explicitly request the data • Depending on the scenario, client can do CRUD on the offline data • Data will be synced back to the server, whenever a connection is possible Offline Sync Basics
  41. • Online/offline recognition • Conflict management • Binary data •

    Local changes • Deleting data • Update interval (incoming new data) • Error handling • Primary key generation Offline Sync Challenges
  42. • Having Edge may not mean that we are online

    • Being online could depend on several factors • Connection quality (Edge, 3G, 4G, 5G) • Connection speed (latency to your backend) • Reachability of backend systems (database, 3rd party system, storages) Offline Sync Challenges - Online/offline recognition
  43. • Client needs to be online for write operations •

    Locks the data, so no other client can overwrite it • Data stays locked, until the client either saves or discards changes • Last One (Write) Wins • Visual conflict management (diffing like in Git, SVN, etc.) Offline Sync Challenges - Conflict management
  44. • Downloading large binary data may not be possible on

    all platforms • A browser does not have a big enough storage for storing the data • Native apps (Cordova, Electron) could be a solution to directly access the platform’s file system • Uploading large binary data • Server needs to support chunk based upload with connection interruptions • Client needs permanent access to the binary file being uploaded, which could be hard in a browser-only scenario Offline Sync Challenges - Binary data
  45. • User opens an edit form and makes changes to

    data • Incoming sync from server would change the data the user is editing • Depending on your use-case, decide what to do • Inform the user about the data change? • Show a visual diff? • Override the local changed data? • “Do nothing” and override the server data on next sync Offline Sync Challenges - Local changes
  46. • Mark “deleted” data with a flag only • Use

    a trigger to keep the deleted ID in a separate table • Client needs some info about “deleted” data • Simple list of IDs “deleted” since the last sync Offline Sync Challenges - Deleting data
  47. • Decide when you need to sync which data •

    Not all data needs to be synced in the same interval • An additional real time connection (WebSocket) could be established • The server can send a signal when some data needs to be synchronised Offline Sync Challenges - Update interval
  48. • Show it, don’t hide them! • The user needs

    to know that something has occurred which shouldn’t • If you can’t sync to the client anymore, you may need to re-sync everything • Depending on your use case, you may want to write all data in one transaction on the server to not have partial data written into the database Offline Sync Challenges - Error handling
  49. Primary key generation

  50. • Inserted offline data needs to be available in the

    client immediately (e.g. the users reloads the application) • Behaves like “real synced data”, even if it was not written to server yet • Client is able to route to offline inserted data • Therefor it needs some kind of primary key/id to route to • Primary key can be generated on server-side and on client-side Primary key generation - challenge
  51. • Server is responsible for generating the primary key •

    Client will generate a temporary offline primary key • After syncing, the client needs to replace the temporary offline key with the server generated primary key • Allows to use sequential GUIDs on the server side (MSSQL) Primary key generation - server-side
  52. • Client is responsible for generating the primary key (GUIDs

    only) • Server inserts the client generated key into the database as the primary key • Server additionally needs to have a clustered key • Otherwise inserting a client non-sequential GUID will lead to a reclustering of the database, every time a client syncs new data Primary key generation - client-side
  53. Change Tracking

  54. • Possibilities to determine the client’s current state • Use

    highest rowversion within result • Determine by result set or additional query • Result of @@DBTS • Returns the last used rowversion of the database • Result of MIN_ACTIVE_ROWVERSION() • Returns the lowest (active) rowversion of the database Change Tracking - MS SQL Server’s ROWVERSION
  55. • If you do a million updates a second, every

    second, the timestamp will wrap around in about 585000 years • Could be used for concurrency checks as well (see Conflict Management) • Before writing data into a row, you could check if it still as the client’s saved row version • If yes: data was not modified, saving is easily possible • If no: data was modified, decide per use case what to do! Change Tracking - MS SQL Server’s ROWVERSION
  56. Change Tracking - Highest ROWVERSION within result Transaction #1 ID

    ROWVERSION 1 0x01 2 0x02 … … 41 0x41 Transaction #2 ID ROWVERSION 1 0x01 2 0x02 … … 41 0x41 42 0x42 43 0x43 SELECT MAX(ROWVERSION) FROM TABLE Client’s state Last known rowversion 41 43 NOT RELIABLE!
  57. • A later started transaction but committed first results in

    a queryable rowversion higher than the pending ones • Data will be lost (never seen by the client’s delta request) Change Tracking - Highest ROWVERSION within result
  58. Change Tracking - Query @@DBTS Transaction #1 ID ROWVERSION 1

    0x01 2 0x02 … … 41 0x41 Transaction #2 ID ROWVERSION 1 0x01 2 0x02 … … 41 0x41 42 0x42 43 0x43 SELECT @@DBTS Client’s state Last known rowversion 41 42 43 NOT RELIABLE!
  59. • Value is database global • Running transaction updates the

    value instantly (before commit or rollback) • Data will be lost (never seen by the client’s delta request) Change Tracking - Query @@DBTS
  60. Change Tracking - Query MIN_ACTIVE_ROWVERSION() Transaction #1 ID ROWVERSION 1

    0x01 2 0x02 … … 41 0x41 Transaction #2 ID ROWVERSION 1 0x01 2 0x02 … … 41 0x41 42 0x42 43 0x43 SELECT MIN_ACTIVE_ROWVERSION() Client’s state Last known rowversion 41 42 44 RELIABLE!
  61. • Value is database global • Running transaction updates the

    value instantly (before commit or rollback) • Data will be safe • Needs to be compared with greater or equal • May query data multiple times Change Tracking - Query MIN_ACTIVE_ROWVERSION()
  62. Backend Preparation

  63. • All syncable entities need to have rowversion column •

    rowversion is updated by MS SQL Server automatically whenever the row is changed (created & updated) • For deleted entities • Either set a IsDeleted flag to true (never delete any rows physically) • Or save the deleted IDs of the entities somewhere else (by trigger) Backend Preparation - MS SQL Server
  64. Backend Preparation - MS SQL Server public class Syncable :

    ISyncable { public Guid Id { get; set; } public byte[] RowVersion { get; set; } } public abstract class SyncableEntityTypeConfiguration<T> : IEntityTypeConfiguration<T> where T : class, ISyncable { public virtual void Configure(EntityTypeBuilder<T> builder) { builder.HasKey(p !=> p.Id); builder.Property(p !=> p.RowVersion).IsRowVersion(); } }
  65. Backend Preparation - MS SQL Server public async Task<SyncDto<TResult!>> SyncAsync<TSource,

    TResult>(string timestamp) where TSource : Syncable where TResult : SyncableDto { var rowVersion = Convert.FromBase64String(timestamp !?? string.Empty); var baseQuery = _context.Set<TSource>() .Where(p !=> (ulong)(object)p.RowVersion !>= (ulong)(object)rowVersion); var changed = await _mapper.ProjectTo<TResult>(baseQuery.WithoutDeleted()).ToListAsync(); var deleted = await baseQuery.Where(p !=> p.IsDeleted).Select(p !=> p.Id).ToListAsync(); return new SyncDto<TResult>() { Timestamp = await _context.GetMinActiveRowVersionAsync(), Changed = changed, Deleted = deleted }; }
  66. Backend Preparation - MS SQL Server public class Context :

    DbContext { private DbQuery<DbQueryValue> DbQueryValue { get; set; } public async Task<byte[]> GetMinActiveRowVersionAsync() { return await DbQueryValue .FromSql("SELECT MIN_ACTIVE_ROWVERSION() AS Value") .Select(p !=> p.Value) .FirstAsync(); } }
  67. • Use equivalents of rowversion and triggers • Manual implement

    mechanism in business logic (error-prone!) • Update tracking column manually by incrementing a database global number (during one transaction!) • Will be very hard für multi-row updates/inserts • Manual implement mechanism in triggers (if available) Backend Preparation - Other Database Systems
  68. Frontend Preparation

  69. • Choose storage area for data (e.g. IndexedDB) • Write

    all the code • Periodic data synchronization • Binary synchronization when data changes • Tracking of timestamps Frontend Preparation
  70. • Implementation of the observable pattern in JavaScript • In

    our case used for observing a stream of synced data • Syncing is done in the background in periodic intervals • RxJS is helping to build a pipeline of actions for the sync process RxJS
  71. Security

  72. Typical token-based security Security - API level Browser Identity Provider

    Web API
  73. • User is only able to see data based on

    his security level • Permissions • Roles • Policies • What happens, if the user permission change is leading to different data visible to the user? • What about data, which he does not see due to rights, but is connected to other data? • Remove frontend data after logout Security - Data Level
  74. Pitfalls

  75. • De-normalize the relational data (document style) • Results in

    multiple rowversion for one entry (use most recent one) • Multiplies the data (more traffic) • Consistent data for one entry • Keep the relations up to the frontend • Explicit rowversion for each entry • Partial consistency (related data may not be synced yet) Relational Data
  76. • Keep some relational data and de-normalize some of it

    • De-normalize many-to-many relations • Needs trigger or business logic to change main entry’s rowversion when relation changes • Partial consistency (related data may not be synced yet) Relational Data
  77. • Native Cordova app out-of-the box possible • Native features

    could be used • Store restrictions apply • What about Electron? Native Packaging??
  78. • PWA helps taking offline the application, but not the

    data • Online != Edge is available • Offline data may be “temporary” (“Remove temporary internet files”) • Use MIN_ACTIVE_ROWVERSION() and “greater or equal than” operator • RxJS helps building a sync engine, but a lot knowledge is needed • Think about data level security (permissions, roles, etc.) • Think about conflict management • Depends heavily on your use case Summary
  79. And… now? • Slides: • Repository: