Offline storage & automatic data synchronization for your web and mobile applications

More and more web and mobile developers are using local storage to ensure good response times and that applications continue to work in the absence of a network connection. But how do you sync local storage with the backend? How do you resolve conflicts? In this talk, I present to you an open source framework (Amplify DataStore) that allows you to develop modern web and mobile applications that answer these questions with a minimum of a line of code.

Transcript

1. OFFLINE FIRST.
2. OFFLINE FIRST Core features should function with or without an internet connection. WHAT DOES IT MEAN? #01
3. OFFLINE FIRST Data is written to and queried locally from the end user’s device and periodically uploaded and replicated in the database. WHAT DOES IT MEAN? #02
4. OFFLINE FIRST Provide end users with a consistent UX when internet connectivity is slow or non-existent. WHAT DOES IT MEAN? #03
5. REAL TIME.
6. REAL TIME An application that functions within a time frame that the user senses as immediate or very close to it. WHAT DOES IT MEAN #01
7. REAL TIME Cohesive system where data is in sync across all connected devices. WHAT DOES IT MEAN #02
8. THE FUTURE OF REAL-TIME | OFFLINE | DATA
9. #01 Sébastien Stormacq INTRODUCTION. @sebsto /sebsto
10. CLIENT TECHNOLOGIES AWS AMPLIFY AWS APPSYNC
11. DEMO WITH .
14. WHY REAL TIME & OFFLINE APPLICATIONS
15. WHY MODERN APPLICATION DEVELOPMENT
16. WHY SLOW or NONEXISTENT 
 NETWORKS
17. WHY USER EXPERIENCE
18. CHALLENGES REAL TIME & OFFLINE APPLICATION
19. CACHING.
20. CACHE !== DATABASE
21. REAL TIME.
22. CONFLICT DETECTION.
23. CONFLICT RESOLUTION.
24. MULTIPLE PLATFORMS.
25. SCALABILITY.
26. SOLUTIONS REAL TIME & OFFLINE APPLICATION
27. CACHING
28. CACHING mutation create($item: InputType) { createItem(input: $item) { id name status } } database cache Writes
29. Cache hydration query listItems { items { id name location } } database
30. Querying query listItems { items(filter: { eq: { location: “NYC” } }) { id name location } } cache
32. MENTAL MODEL struct Episode: Identifiable, Hashable, Codable { let id: String let date: String let title: String let duration: String let description: String? } let newEpisode = Episode(id: UUID().uuidString, date: Date(), title: “New Episode", duration: “00:12:14”, description: "Description")
33. AMPLIFY DATASTORE Multi-platform on-device persistent storage engine that automatically synchronizes data between mobile/web apps and the cloud using GraphQL.
34. Storage Engine Storage Adapter CRUD OBSERVES WRITES READS WRITES OBSERVES Sync Engine QUERIES/MUTATIONS SUBSCRIPTIONS ARCHITECTURE DataStore API Developer
35. DATA MODELING WITH GRAPHQL Developer AUTHORING BUILD codegen typescript, java, kotlin, swift type EpisodeData { id: ID! date: String! title: String! duration: String! description: String } struct Episode { let id: String let date: String let title: String let duration: String let description: String? }
36. DATASTORE Fluent interface Mutations / writes let episode = new Episode(date: "14 nov 2022", title: "New Episode", duration: "New Duration") try await Amplify.DataStore.save(episode)
37. DATASTORE Fluent interface Queries / reads Amplify.DataStore .query(for: Episode.self)
38. DATASTORE Fluent interface - with predicates Amplify.DataStore .query( for: Episode.self, where: e.podcastEpisodesId == podcast.id)
39. DATASTORE Mutations - with predicates try await Amplify.DataStore.delete(Episode.self, withId: episode.id)
40. Storage Engine Storage Adapter CRUD OBSERVES WRITES READS ARCHITECTURE DataStore API Developer Model converted to / from the storage adapter of choice Object and action persistence with specific storage engine (SQL Lite, IndexDB, etc..)
41. Storage Engine Storage Adapter CRUD OBSERVES WRITES READS OBSERVES WRITES / READS Sync Engine QUERIES/MUTATIONS SUBSCRIPTIONS ARCHITECTURE DataStore API Developer Model converted to GraphQL statement Response converted to Model and written to storage engine.
42. REAL TIME
43. LOCAL STORE + SUBSCRIPTIONS Fluent interface - observing data models let e = EpisodeData.keys Amplify.DataStore .observeQuery( for: EpisodeData.self, where: e.podcastDataEpisodesId == podcast.id)
44. SINGLE REHYDRATE QUERY
45. SINGLE REHYDRATE QUERY let e = EpisodeData.keys Amplify.DataStore .observeQuery( for: EpisodeData.self, where: e.podcastDataEpisodesId == podcast.id)
46. CONFLICT DETECTION AND 
 RESOLUTION
47. Provide a way to merge concurrent modi fi cations, always, in any order. CONFLICT DETECTION & RESOLUTION
48. CONFLICT DETECTION & RESOLUTION Objects tagged with metadata
49. person = { name: “Jennifer”, age: 32 } person = { name: “Jennifer”, age: 32, title: “Developer”, } person = { name: “Jennifer”, age: 32, title: “Engineer” } original item client a client b CONFLICT RESOLUTION - AUTO MERGE
50. person = { name: “Jennifer”, age: 32, title: “Developer” } person = { name: “Jennifer”, age: 32, title: “Engineer” } client a client b CONFLICT RESOLUTION - AUTO MERGE person = { name: “Jennifer”, age: 32, title: “Developer” } new item
51. person = { name: “Jennifer”, age: 32 } person = { name: “Jennifer”, age: 32, location: “Nebraska”, } person = { name: “Jennifer”, age: 32, title: “Engineer” } original item client a client b CONFLICT RESOLUTION - AUTO MERGE
52. person = { name: “Jennifer”, age: 32, location: “Nebraska” } person = { name: “Jennifer”, age: 32, title: “Engineer” } client a client b CONFLICT RESOLUTION - AUTO MERGE person = { name: “Jennifer”, age: 32, location: “Nebraska”, title: “Engineer” } new item
53. person = { name: “Jennifer”, age: 32 hobbies: [“climbing"] } person = { name: “Jennifer”, age: 32, hobbies: [“climbing”, “reading”] } person = { name: “Jennifer”, age: 32, hobbies: [“climbing”, “karaoke”] } original item client a client b CONFLICT RESOLUTION - AUTO MERGE Lists
54. person = { name: “Jennifer”, age: 32, hobbies: [“climbing”, “reading”] } person = { name: “Jennifer”, age: 32, hobbies: [“climbing”,“karaoke”] } client a client b CONFLICT RESOLUTION - AUTO MERGE Lists person = { name: “Jennifer”, age: 32, hobbies: [“climbing”, “karaoke”, “reading”] } new item
55. AMPLIFY DATASTORE CONFLICT DETECTION & RESOLUTION • Monotonic counters • Controlled by the backend (AppSync) • GraphQL types with versions used for merge or reject decisions • Base table & change table with soft deletes & TTL
56. MONOTONIC COUNTER person = { id: 100, age: 30, _version: 1 } person = { id: 100, age: 31, _version: 2 } person = { id: 100, age: 32, _version: 3 } id age name _version 100 32 Amy 3 101 28 Charles 9 102 47 Melissa 8 Base Table
57. ds_pk ds_sk age name _version _lastChangedAt Person-9386:2020-01-23 13:16:08.127:9386:1 32 Amy 1 1579785368127 Person-9386:2020-01-23 13:16:21.785:9386:2 33 Amy 2 1579785381785 Person-9386:2020-01-23 13:16:37.859:9386:3 34 Amy 3 1579785397859 Change Table typename + uuid + date timestamp + uuid + version CONFLICT RESOLUTION
58. AUTO MERGE CONFLICT RESOLUTION
59. person = { age: 34, age: 35, location: “Seattle” _version: 4 } User A User B person = { age: 32, _version: 1 } person = { age: 33, _version: 2 } person = { age: 33, _version: 2 } person = { age: 32, _version: 1 } person = { age: 35, _version: 3 } OFFLINE ONLINE person = { age: 34, location: “Seattle” _version: 3 }
60. User A User B person = { age: 32, _version: 1 } person = { age: 33, _version: 2 } person = { age: 33, _version: 2 } person = { age: 32, _version: 1 } person = { age: 35, _version: 3 } OFFLINE ONLINE person = { age: 34, location: “Seattle” _version: 3 } person = { age: 34, location: “Seattle” _version: 4 } person = { age: 35, location: “Seattle” _version: 4 }
61. AUTO MERGE SYNC ENABLED GRAPHQL RESOLVERS Base table + change table + GraphQL types with versions used for merge or reject decisions
62. AWS APPSYNC CONFLICT RESOLUTION STRATEGIES Automerge AWS Lambda Optimistic concurrency
63. SYNCHRONIZATION
64. id age name _version 100 32 Amy 8 101 28 Charles 9 DELETE ID 102 SYNCHRONIZATION - BASIC IMPLEMENTATION 102 47 Melissa 3
65. DELTA SYNC SYNCHRONIZATION
66. ds_pk ds_sk age name _version _lastChangedAt Person-9387:2020-01-23 13:16:08.127:9387:7 22 James 7 1579785368127 Person-9388:2020-01-23 13:16:21.785:9388:3 41 Amy 3 1579785381785 Change Table SYNCHRONIZATION - DELTA SYNC syncPerson(lastSync: 13:16:22.127) Person-9389:2020-01-23 13:16:37.859:9389:1 34 Jennifer 1 1579785397859 Person-9389:2020-01-23 13:16:37.859:9389:1 34 Jennifer 1 1579785397859
67. TIME TRAVEL SYNCHRONIZATION
68. id age name _version _deleted _ttl 100 32 Amy 8 101 28 Charles 9 102 47 Melissa 3 TRUE 1582384003 SYNCHRONIZATION ds_pk ds_sk age _deleted _ttl Person-9386:2020-01-23 13:16:08.127:9386:1 32 1582384003 Person-9386:2020-01-23 13:16:21.785:9386:2 33 1582384003 Person-9386:2020-01-23 13:16:37.859:9386:3 47 TRUE 1582384003 Change Table Base Table
69. DB OPTIMIZATION SYNCHRONIZATION
70. TIME TO LIVE ds_pk ds_sk age _deleted _ttl Person-9386:2020-01-23 13:16:08.127:9386:1 32 1582384003 Person-9386:2020-01-23 13:16:21.785:9386:2 33 1582384003 Person-9386:2020-01-23 13:16:37.859:9386:3 47 TRUE 1582384003 1582384003 1582384003 1582384003
71. PLATFORMS
72. PLATFORMS
73. https://docs.amplify.aws/
74. https://github.com/sebsto
75. THANK YOU! /sebAWS @sebsto /sebsto /sebsto
76. https://eventbox.dev/survey/ADK80QX ANONYMOUS SURVEY $25 AWS VOUCHERS