Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Immer & überall: Offlinefähige Progressive Web Apps – am Beispiel Angular

Immer & überall: Offlinefähige Progressive Web Apps – am Beispiel Angular

“Keine Internetverbindung” – ein Satz, den wir alle aus genutzten Apps kennen, sei es Desktopanwendung oder Smartphone App. Oftmals sind die Clients auf dem jeweiligen Zielgerät installiert, bedienen sich aber einer externen Datenquelle, bspw. in Form einer Web API. Bei einer offline-fähigen Implementierung sind nicht nur Lese-, sondern vor allem auch Schreibzugriffe wichtig. Sprich, die Daten müssen in beide Richtungen synchronisiert werden können, sobald die Internetverbindung wiederhergestellt wurde. Gerade in der heutigen Zeit, in der Menschen immer mobiler werden, in Autos, Zügen und Flugzeugen unterwegs sind und arbeiten, benötigt man oftmals offline Zugriff auf die Anwendungsdaten. Wie man diese echt offline-fähigen Single-Page Applications/Progressive Web Apps realisiert, zeigen Thomas Hilzendegen und Manuel Rauber von Thinktecture in diesem Ganztagesworkshop. Allgemein werden Themen wie Synchronisationsstrategien von Anwendungsdaten und großen Binärdaten, Konfliktmanagement oder Datenbankarchitektur sowohl für neue als auch bestehende Anwendungen diskutiert. Am Beispiel einer Angular-basierten App mit einem exemplarischen Backend in .NET Core werden Client und Server zur Umsetzung der besprochenen Konzepte und Patterns implementiert.

GitHub: https://github.com/thinktecture/thinktecture-boardist

Manuel Rauber

October 10, 2019
Tweet

More Decks by Manuel Rauber

Other Decks in Programming

Transcript

  1. Offlinefähige Progressive Web Apps
    Immer & überall
    Thomas Hilzendegen
    @hilzendegent
    Consultant
    Manuel Rauber
    @ManuelRauber
    Consultant

    View full-size slide

  2. Thomas Hilzendegen
    Speakers
    Manuel Rauber
    [email protected]
    @hilzendegent
    [email protected]
    @manuelrauber
    https://manuel-rauber.com
    Microsoft MVP

    View full-size slide

  3. Timetable
    Time Doing
    09:00 - 10:30 Part I
    10:30 - 11:00 ☕
    11:00 - 12:30 Part II
    12:30 - 13:30
    13:30 - 15:00 Part III
    15:00 - 15:30 ☕
    15:30 - 17:00 Part IV

    View full-size slide

  4. • Progressive Web App
    • Offline availability
    • Offline synchronization
    • Change tracking
    • Security considerations
    • Pitfalls
    Agenda

    View full-size slide

  5. Offline… but why?

    View full-size slide

  6. • Just because the phone has a connection,

    e.g. Edge, it does not mean we are online
    • Depends on the use case which connection

    quality is required to determine, if the 

    app is reliable online
    • Connection quality could be measured

    by the time an exclusive request takes
    • Duration < 150 ms: online
    • Duration !>=150 ms: bad connection
    What does “offline” mean?

    View full-size slide

  7. • A lot of reasons to be offline
    • Traveling, Train, Flights (bad or no signal)
    • Server is not available
    • Routing problems
    • Roaming
    • Costs
    Motivation

    View full-size slide

  8. Demo Application

    View full-size slide

  9. • A little application to manage boardgames
    • Angular, .NET Core, MS SQL Server
    • Brownfield application, started as a pure online application
    • As typical for any good demo: no security
    • Available on Azure: https://tt-boardist.azurewebsites.net
    • GitHub: https://github.com/thinktecture/thinktecture-boardist
    Demo Application “Thinktecture Boardist”

    View full-size slide

  10. Demo Application Database Scheme

    View full-size slide

  11. Now let’s go offline!

    View full-size slide

  12. Does it work in flight mode?
    NOPE!

    View full-size slide

  13. Progressive Web App

    View full-size slide

  14. • Motivation: Get rid of app stores!
    • Web App = App App
    • Support native features like: Push notifications,
    offline availability, installable
    • Backwards-compatibility: runs in non-PWA
    browsers, with a reduced feature set
    Progressive Web App

    View full-size slide

  15. PWA Technology Overview
    Progressive Web Apps
    HTML5, JavaScript, CSS3
    Service Worker API Web App Manifest HTTPS
    Fetch API
    Web
    Notifications
    Web
    Workers
    Push API

    View full-size slide

  16. PWA Platform Support
    40 44 11.1 17 4.1 11.3

    View full-size slide

  17. Progressive Web App 

    is not a technology, 

    but a collection of features an
    application must/should support

    View full-size slide

  18. PWA Collection of Features
    Responsive Linkable Discoverable Installable App-Like
    Connectivity

    Independent
    Fresh Safe Re-engageable Progressive

    View full-size slide

  19. • A JavaScript running in its own thread
    • No access to the DOM
    • Communicates via postMessage-API
    • Acts as a controller/proxy/interceptor
    • Can perform background tasks
    • Has to be installed before usage
    • Runs whenever the browser is running
    PWA ServiceWorker

    View full-size slide

  20. PWA ServiceWorker
    Website Internet
    Cache Service
    Worker

    View full-size slide

  21. PWA ServiceWorker Lifecycle
    Parse Installing
    Error
    Activated Idle Terminated
    fetch/message

    View full-size slide

  22. PWA Connectivity Independent
    Website Internet
    Cache storage
    Server
    Remote storage
    ServiceWorker

    View full-size slide

  23. • Cache only
    • Network only
    • Cache falling back to network
    • Cache & network race
    • Network falling back to cache
    • Cache then network
    • Generic fallback
    • ServiceWorker-side templating
    PWA Caching Strategies

    View full-size slide

  24. PWA Cache Then Network
    Website Internet
    Cache storage
    Server
    Remote storage
    ServiceWorker
    HTTP
    1. Lookup
    2. fetch
    1. Lookup

    View full-size slide

  25. • “One-size-fits-all” ServiceWorker implementation
    • Instrumented via ngsw-config.json
    • Restricted feature set (e.g. no communication between app and SW)
    • Initial caching of static content
    • Caching external content
    • Dynamic data caching
    • Push notifications
    PWA Angular ServiceWorker

    View full-size slide

  26. Does it work offline now?
    STILL NOPE!

    View full-size slide

  27. Real offline availability

    View full-size slide

  28. • ServiceWorker is only able to take data offline which has been requested by
    the application
    • If all the URLs are known beforehand, the ServiceWorker could cache them
    all
    • Data which was not requested, is not available offline (no real offline
    synchronisation)
    • But what about … the Background Sync API?
    What’s the problem?

    View full-size slide

  29. • Name is misleading
    • Does not offer any data synchronisation possibilities
    • Just raises an event periodically or after a delay (optionally based on the
    network type)
    • It’s totally up to the developer what to do in the onsync event
    • Since the sync is done in the ServiceWorker, the page can be closed, the
    sync will be fulfilled anyway
    • It does not help anything with syncing your actual data!
    Background Sync API

    View full-size slide

  30. Background Sync API

    View full-size slide

  31. • Cookies (not meant for large data or binaries)
    • Web Storage like session storage or local storage (not meant for large data
    or binaries)
    • IndexedDB
    • Cache Storage (based on request/response)
    Storage capabilities

    View full-size slide

  32. User can clear anything
    at any time by
    “Remove temporary internet files”
    or “Clear website data”

    View full-size slide

  33. • Key-value database of the browser
    • Stores data permanently
    • ServiceWorker and Web App share access to the same IndexedDB
    • Possibility of scenarios, where the ServiceWorker (or Web App) stores
    synchronised data in the IndexedDB and the Web App reads the data
    • Like everything based on the origin
    IndexedDB

    View full-size slide

  34. IndexedDB Availability

    View full-size slide

  35. • The standard API of IndexedDB is inconvenient to use (lots of callback)
    • Dexie.js is a minimalistic wrapper for IndexedDB
    • Operations are promise-based
    • Near native performance (even for bulk inserts)
    • Open Source @ GitHub: https://github.com/dfahlander/Dexie.js
    IndexedDB API

    View full-size slide

  36. Offline Sync

    View full-size slide

  37. • Offline Sync means to download all data available to client into a persistent
    offline storage, without the user having to explicitly request the data
    • Depending on the scenario, client can do CRUD on the offline data
    • Data will be synced back to the server, whenever a connection is possible
    Offline Sync Basics

    View full-size slide

  38. • Online/offline recognition
    • Conflict management
    • Binary data
    • Local changes
    • Deleting data
    • Update interval (incoming new data)
    • Error handling
    • Primary key generation
    Offline Sync Challenges

    View full-size slide

  39. • Having Edge may not mean that we are online
    • Being online could depend on several factors
    • Connection quality (Edge, 3G, 4G, 5G)
    • Connection speed (latency to your backend)
    • Reachability of backend systems (database, 3rd party system, storages)
    Offline Sync Challenges - Online/offline recognition

    View full-size slide

  40. • Client needs to be online for write operations
    • Locks the data, so no other client can overwrite it
    • Data stays locked, until the client either saves or discards changes
    • Last One (Write) Wins
    • Visual conflict management (diffing like in Git, SVN, etc.)
    Offline Sync Challenges - Conflict management

    View full-size slide

  41. • Downloading large binary data may not be possible on all platforms
    • A browser does not have a big enough storage for storing the data
    • Native apps (Cordova, Electron) could be a solution to directly access the
    platform’s file system
    • Uploading large binary data
    • Server needs to support chunk based upload with connection interruptions
    • Client needs permanent access to the binary file being uploaded, which
    could be hard in a browser-only scenario
    Offline Sync Challenges - Binary data

    View full-size slide

  42. • User opens an edit form and makes changes to data
    • Incoming sync from server would change the data the user is editing
    • Depending on your use-case, decide what to do
    • Inform the user about the data change?
    • Show a visual diff?
    • Override the local changed data?
    • “Do nothing” and override the server data on next sync
    Offline Sync Challenges - Local changes

    View full-size slide

  43. • Mark “deleted” data with a flag only
    • Use a trigger to keep the deleted ID in a separate table
    • Client needs some info about “deleted” data
    • Simple list of IDs “deleted” since the last sync
    Offline Sync Challenges - Deleting data

    View full-size slide

  44. • Decide when you need to sync which data
    • Not all data needs to be synced in the same interval
    • An additional real time connection (WebSocket) could be established
    • The server can send a signal when some data needs to be synchronised
    Offline Sync Challenges - Update interval

    View full-size slide

  45. • Show it, don’t hide them!
    • The user needs to know that something has occurred which shouldn’t
    • If you can’t sync to the client anymore, you may need to re-sync everything
    • Depending on your use case, you may want to write all data in one
    transaction on the server to not have partial data written into the database
    Offline Sync Challenges - Error handling

    View full-size slide

  46. Primary key generation

    View full-size slide

  47. • Inserted offline data needs to be available in the client immediately (e.g. the
    users reloads the application)
    • Behaves like “real synced data”, even if it was not written to server yet
    • Client is able to route to offline inserted data
    • Therefor it needs some kind of primary key/id to route to
    • Primary key can be generated on server-side and on client-side
    Primary key generation - challenge

    View full-size slide

  48. • Server is responsible for generating the primary key
    • Client will generate a temporary offline primary key
    • After syncing, the client needs to replace the temporary offline key with the
    server generated primary key
    • Allows to use sequential GUIDs on the server side (MSSQL)
    Primary key generation - server-side

    View full-size slide

  49. • Client is responsible for generating the primary key (GUIDs only)
    • Server inserts the client generated key into the database as the primary key
    • Server additionally needs to have a clustered key
    • Otherwise inserting a client non-sequential GUID will lead to a
    reclustering of the database, every time a client syncs new data
    Primary key generation - client-side

    View full-size slide

  50. Change Tracking

    View full-size slide

  51. • Possibilities to determine the client’s current state
    • Use highest rowversion within result
    • Determine by result set or additional query
    • Result of @@DBTS
    • Returns the last used rowversion of the database
    • Result of MIN_ACTIVE_ROWVERSION()
    • Returns the lowest (active) rowversion of the database
    Change Tracking - MS SQL Server’s ROWVERSION

    View full-size slide

  52. • If you do a million updates a second, every second, the timestamp will wrap
    around in about 585000 years
    • Could be used for concurrency checks as well (see Conflict Management)
    • Before writing data into a row, you could check if it still as the client’s
    saved row version
    • If yes: data was not modified, saving is easily possible
    • If no: data was modified, decide per use case what to do!
    Change Tracking - MS SQL Server’s ROWVERSION

    View full-size slide

  53. Change Tracking - Highest ROWVERSION within result
    Transaction #1
    ID ROWVERSION
    1 0x01
    2 0x02
    … …
    41 0x41
    Transaction #2
    ID ROWVERSION
    1 0x01
    2 0x02
    … …
    41 0x41
    42 0x42 43 0x43
    SELECT MAX(ROWVERSION) FROM TABLE
    Client’s state
    Last known rowversion 41
    43
    NOT RELIABLE!

    View full-size slide

  54. • A later started transaction but committed first results in a queryable
    rowversion higher than the pending ones
    • Data will be lost (never seen by the client’s delta request)
    Change Tracking - Highest ROWVERSION within result

    View full-size slide

  55. Change Tracking - Query @@DBTS
    Transaction #1
    ID ROWVERSION
    1 0x01
    2 0x02
    … …
    41 0x41
    Transaction #2
    ID ROWVERSION
    1 0x01
    2 0x02
    … …
    41 0x41
    42 0x42 43 0x43
    SELECT @@DBTS
    Client’s state
    Last known rowversion 41
    42
    43
    NOT RELIABLE!

    View full-size slide

  56. • Value is database global
    • Running transaction updates the value instantly (before commit or rollback)
    • Data will be lost (never seen by the client’s delta request)
    Change Tracking - Query @@DBTS

    View full-size slide

  57. Change Tracking - Query MIN_ACTIVE_ROWVERSION()
    Transaction #1
    ID ROWVERSION
    1 0x01
    2 0x02
    … …
    41 0x41
    Transaction #2
    ID ROWVERSION
    1 0x01
    2 0x02
    … …
    41 0x41
    42 0x42 43 0x43
    SELECT MIN_ACTIVE_ROWVERSION()
    Client’s state
    Last known rowversion 41
    42
    44
    RELIABLE!

    View full-size slide

  58. • Value is database global
    • Running transaction updates the value instantly (before commit or rollback)
    • Data will be safe
    • Needs to be compared with greater or equal
    • May query data multiple times
    Change Tracking - Query MIN_ACTIVE_ROWVERSION()

    View full-size slide

  59. Backend Preparation

    View full-size slide

  60. • All syncable entities need to have rowversion column
    • rowversion is updated by MS SQL Server automatically whenever the row
    is changed (created & updated)
    • For deleted entities
    • Either set a IsDeleted flag to true (never delete any rows physically)
    • Or save the deleted IDs of the entities somewhere else (by trigger)
    Backend Preparation - MS SQL Server

    View full-size slide

  61. Backend Preparation - MS SQL Server
    public class Syncable : ISyncable
    {
    public Guid Id { get; set; }
    public byte[] RowVersion { get; set; }
    }
    public abstract class SyncableEntityTypeConfiguration : IEntityTypeConfiguration
    where T : class, ISyncable
    {
    public virtual void Configure(EntityTypeBuilder builder)
    {
    builder.HasKey(p !=> p.Id);
    builder.Property(p !=> p.RowVersion).IsRowVersion();
    }
    }

    View full-size slide

  62. Backend Preparation - MS SQL Server
    public async Task> SyncAsync(string timestamp)
    where TSource : Syncable
    where TResult : SyncableDto
    {
    var rowVersion = Convert.FromBase64String(timestamp !?? string.Empty);
    var baseQuery = _context.Set()
    .Where(p !=> (ulong)(object)p.RowVersion !>= (ulong)(object)rowVersion);
    var changed = await _mapper.ProjectTo(baseQuery.WithoutDeleted()).ToListAsync();
    var deleted = await baseQuery.Where(p !=> p.IsDeleted).Select(p !=> p.Id).ToListAsync();
    return new SyncDto()
    {
    Timestamp = await _context.GetMinActiveRowVersionAsync(),
    Changed = changed,
    Deleted = deleted
    };
    }

    View full-size slide

  63. Backend Preparation - MS SQL Server
    public class Context : DbContext {
    private DbQuery DbQueryValue { get; set; }
    public async Task GetMinActiveRowVersionAsync()
    {
    return await DbQueryValue
    .FromSql("SELECT MIN_ACTIVE_ROWVERSION() AS Value")
    .Select(p !=> p.Value)
    .FirstAsync();
    }
    }

    View full-size slide

  64. • Use equivalents of rowversion and triggers
    • Manual implement mechanism in business logic (error-prone!)
    • Update tracking column manually by incrementing a database global
    number (during one transaction!)
    • Will be very hard für multi-row updates/inserts
    • Manual implement mechanism in triggers (if available)
    Backend Preparation - Other Database Systems

    View full-size slide

  65. Frontend Preparation

    View full-size slide

  66. • Choose storage area for data (e.g. IndexedDB)
    • Write all the code
    • Periodic data synchronization
    • Binary synchronization when data changes
    • Tracking of timestamps
    Frontend Preparation

    View full-size slide

  67. • Implementation of the observable pattern in JavaScript
    • In our case used for observing a stream of synced data
    • Syncing is done in the background in periodic intervals
    • RxJS is helping to build a pipeline of actions for the sync process
    RxJS

    View full-size slide

  68. Typical token-based security
    Security - API level
    Browser
    Identity
    Provider
    Web API

    View full-size slide

  69. • User is only able to see data based on his security level
    • Permissions
    • Roles
    • Policies
    • What happens, if the user permission change is leading to different data
    visible to the user?
    • What about data, which he does not see due to rights, but is connected to
    other data?
    • Remove frontend data after logout
    Security - Data Level

    View full-size slide

  70. • De-normalize the relational data (document style)
    • Results in multiple rowversion for one entry (use most recent one)
    • Multiplies the data (more traffic)
    • Consistent data for one entry
    • Keep the relations up to the frontend
    • Explicit rowversion for each entry
    • Partial consistency (related data may not be synced yet)
    Relational Data

    View full-size slide

  71. • Keep some relational data and de-normalize some of it
    • De-normalize many-to-many relations
    • Needs trigger or business logic to change main entry’s rowversion when
    relation changes
    • Partial consistency (related data may not be synced yet)
    Relational Data

    View full-size slide

  72. • Native Cordova app out-of-the box possible
    • Native features could be used
    • Store restrictions apply
    • What about Electron?
    Native Packaging??

    View full-size slide

  73. • PWA helps taking offline the application, but not the data
    • Online != Edge is available
    • Offline data may be “temporary” (“Remove temporary internet files”)
    • Use MIN_ACTIVE_ROWVERSION() and “greater or equal than” operator
    • RxJS helps building a sync engine, but a lot knowledge is needed
    • Think about data level security (permissions, roles, etc.)
    • Think about conflict management
    • Depends heavily on your use case
    Summary

    View full-size slide

  74. And… now?
    • Slides: https://speakerdeck.com/manuelrauber
    • Repository: https://github.com/thinktecture/thinktecture-boardist

    View full-size slide