Upgrade to Pro — share decks privately, control downloads, hide ads and more …

two-way sync

Dima
December 19, 2015

two-way sync

#cocoaheadskyiv

Dima

December 19, 2015
Tweet

More Decks by Dima

Other Decks in Programming

Transcript

  1. Why? • Client-server application • Full offline support • “All

    changes have to be available online automatically” - client • Simultaneous editing from multiple clients: iOS, Android, Web What needs to be done
  2. remote local t { id: 12341, username: "john.doe", description: “hire

    me" } * { id: 12341, username: "john.doe", description: "Android" } to find conflicts
  3. remote local t { id: 12341, username: "john.doe", description: “hire

    me" } * { id: 12341, username: "john.doe", description: "Android" } to find conflicts
  4. { id: 12341, username: "john.doe", description: “hire me" } *

    push remote local t to find unsaved changes
  5. { id: 12341, username: "john.doe", description: “hire me" } *

    ? push remote local t to find unsaved changes
  6. changes tracking • Timestamp (lastModified: 17.12.2015T19:20:22.133145Z) • Version (revision: 245)

    • Status (dirty: true, deleted: false) • Combined aka Change Data Capture
  7. { id: 12341, username: "john.doe", description: "iOS dev" } data

    structure { id: 12341, username: "john.doe", description: "iOS dev", revision: 213, modified: 12.12.2016, deleted: false, dirty: false } trackable plain
  8. changelog local t pull changes tracking { id: 12341, revision:

    213, modified: 12.12.2016, deleted: false dirty: true } *
  9. changelog local t changes tracking { id: 12341, revision: 215,

    modified: 12.12.2016, deleted: false } { id: 12341, revision: 213, modified: 12.12.2016, deleted: false dirty: true } *
  10. data authority • API is the only data authority •

    Client doesn't change revision nor timestamp, only via API • Timestamp management on client is hard • Client’s changelog can maintain its own revision, independent from the API’s one
  11. db versioning • Depends on changes tracking implementation • Tied

    with changelog • Allows to identify changes deltas • Acts as an offset revision = 1023
  12. changelog • Ordered log of data change records • Store

    records such as update or delete for specific object • Can track only fact of the particular event or full change representation (affected keys, old values, new values)
  13. db version + changelog • Database.version == changelog.HEAD.version • Database.version

    == most recent data.version • Any data change increments database version • New change.version == Database.version + 1
  14. changelog API { id: 24, change: update, entity: "Note", objectID:

    442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=1
  15. changelog API { id: 24, change: update, entity: "Note", objectID:

    442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=1 POST /notes/
  16. POST /notes/ changelog API { id: 24, change: update, entity:

    "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2
  17. { id: 443, revision: 2, content: “First Note" } Note

    changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2
  18. changelog API { id: 24, change: update, entity: "Note", objectID:

    442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note
  19. changelog API { id: 24, change: update, entity: "Note", objectID:

    442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 25, change: update, entity: "Note", objectID: 443 } { id: 443, revision: 2, content: “First Note" } Note
  20. changelog API { id: 24, change: update, entity: "Note", objectID:

    442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 }
  21. changelog API { id: 24, change: update, entity: "Note", objectID:

    442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } PUT /notes/442/
  22. changelog API { id: 24, change: update, entity: "Note", objectID:

    442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } PUT /notes/442/
  23. changelog API { id: 24, change: update, entity: "Note", objectID:

    442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } PUT /notes/442/
  24. changelog API { id: 24, change: update, entity: "Note", objectID:

    442 } db + changelog data Note revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } { id: 442, revision: 3, content: "Cocoaheads" }
  25. { id: 24, change: update, entity: "Note", objectID: 442 }

    changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } { id: 442, revision: 3, content: "Cocoaheads" } Note
  26. { id: 24, change: update, entity: "Note", objectID: 442 }

    changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } { id: 26, change: update, entity: "Note", objectID: 442 } { id: 442, revision: 3, content: "Cocoaheads" } Note
  27. { id: 24, change: update, entity: "Note", objectID: 442 }

    changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 442, revision: 3, content: "Cocoaheads" } Note { id: 26, change: update, entity: "Note", objectID: 442 } { id: 25, change: update, entity: "Note", objectID: 443 }
  28. { id: 24, change: update, entity: "Note", objectID: 442 }

    changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 442, revision: 3, content: "Cocoaheads" } Note DELETE /notes/442/ { id: 26, change: update, entity: "Note", objectID: 442 } { id: 25, change: update, entity: "Note", objectID: 443 }
  29. { id: 442, revision: 3, content: "Cocoaheads" } Note {

    id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } DELETE /notes/442/ { id: 26, change: update, entity: "Note", objectID: 442 }
  30. { id: 442, revision: 4, content: "Cocoaheads", deleted: true }

    Note { id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } { id: 26, change: update, entity: "Note", objectID: 442 }
  31. { id: 442, revision: 4, content: "Cocoaheads", deleted: true }

    Note { id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } { id: 26, change: update, entity: "Note", objectID: 442 }
  32. { id: 27, change: delete, entity: "Note", objectID: 442 }

    { id: 442, revision: 4, content: "Cocoaheads", deleted: true } Note { id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } { id: 26, change: update, entity: "Note", objectID: 442 }
  33. simple changelog, version per row, no changed data • Easy

    to implement • Easy to maintain • Doesn’t contain actual data deltas • No way to rollback invalid changes • Impossible to do field-by-field conflicts resolution pros cons
  34. API • Client’s database version is required for any API

    data mutation API or deltas fetch • Data mutation from the unsynced client is rejected • Push is request-per-change via REST
  35. API GET /sync-chunk/?revision=1331&limit=100 { "max_revision": 1334, "result": { "Note": {

    "updates": [ { "id": 812, "content": "Cocoaheads Kiev", "revision": 1332 }, { "id": 442, "content": "Android Dev", "revision": 1333 } ], "deletes": [ { "id": 113, "revision": 1334 } ] } } }
  36. API with update and delete of the same object, single

    page { "max_revision": 1335, "result": { "Note": { "updates": [ { "id": 443, "content": "Android Dev", "revision": 1333 } ], "deletes": [ { "id": 442, "revision": 1335 } ] } } }
  37. API with update and delete of the same object, two

    pages { "max_revision": 1334, "result": { "Note": { "updates": [ { "id": 442, "content": "Android Dev", "revision": 1333 }, { "id": 443, "content": "CocoaHeads", "revision": 1334 } ] } } } { "max_revision": 1335, "result": { "Note": { "deletes": [ { "id": 443, "revision": 1335 } ] } } } page #0 page #1
  38. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision":

    1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body
  39. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision":

    1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”]
  40. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision":

    1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true
  41. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision":

    1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true
  42. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision":

    1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true STATUS CODE: 409 (Conflict)
  43. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision":

    1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true STATUS CODE: 409 (Conflict) false { "id": 442, "content": "Android Dev", "revision": 1334 } Revision: 1334 headers body
  44. resolving conflicts • Client resolves any merge conflict • Merge

    changed data pushed to the API • Client-based resolution allows to use UI • Simplifies backend code, client becomes more complex • Graceful resolving requires additional effort • per-field changes tracking • usage of combined changes tracking on client done on client
  45. resolving conflicts in dummy way pseudocode class OverwriteConflictResolvingStrategy { let

    object: Object let changelog: Changelog let update: ChangelogEvent? let delete: ChangelogEvent? func resolve(changes: [String: AnyObject]) { if let update = update { changelog.removeChange(update) } if let delete = delete { changelog.removeChange(delete) } } }
  46. Chunk Store Sync Scheduler Pull Operation Push Operation Sync Chunk

    Fetch Merge Chunk Merge Change Push Changelog
  47. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store
  48. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store
  49. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store
  50. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store revision: 431
  51. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store revision: 481
  52. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store revision: 481
  53. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store revision: 501
  54. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store revision: 501
  55. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store revision: 501
  56. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store revision: 501
  57. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store revision: 501
  58. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store
  59. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog pull Chunk Store
  60. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog push Chunk Store
  61. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog push Chunk Store
  62. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog push Chunk Store revision: 431
  63. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog push Chunk Store
  64. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge

    Chunk Merge Change Push Changelog push Chunk Store
  65. merge pseudocode from Merge Operation class ChunkStore { var chunks:

    [SyncChunk] { return [] } } class Merge { enum EntityType { case User, Note, Notebook } let chunkStore = ChunkStore() func executeMerge() { let orderedTypes: [EntityType] = [.User, .Notebook, .Note] for type in orderedTypes { for chunk in chunkStore.chunks { mergeChunkUpdates(chunk, ofType: type) mergeChunkDeletes(chunk, ofType: type) } } } }
  66. expect unexpected termination • Design you code with restoration /

    preservation in mind • Preserve execution state after every sync operation (entity merged, change pushed, etc) • Delete preserved data once done if needed
  67. changes push • Push changes in a serial queue •

    Push and Pull are mutually exclusive and can not be performed concurrently • Handle validation errors to prevent pushing queue hang (e.g. mark change as invalid in case of double validation error, etc)
  68. SyncScheduler • Respect reachability and connection errors: make a delay

    before retry • Don’t use timer with small delay for check. It drains your battery • On Changelog update notify SyncScheduler about local state is being dirty (Observer, NotificationCenter, etc)
  69. changes tracking: prefer Version over Timestamp • Integers easier to

    compare, store and use • Utilize microseconds to prevent data lost (yyyy.MM.dd’T’HH:mm:ss.SSSSSS’Z’) • NSDateFormatter doesn’t parse microseconds • Prefer NSDateComponents over NSDate
  70. changes tracking • One place for all • Harder to

    implement • Requires filtering of unnecessary data. Leads to hell. • Simple implementation • Requires explicit tracking invocation • Tracks only necessary data auto manual
  71. changelog as a separate database • Looks like a more

    robust solution • Allows you to imp. Changelog as a separate module • Doesn’t give a lot of benefit • Requires complex synchronization of Changeling’s context and Database’s context. (not available for iOS 7) • DB saving error may leave you up with inconsistent state: changelog doesn’t reflect database state.
  72. deletes • Don’t delete objects immediately, use deleted flag instead

    • Purge deleted records on: • Delete change push • After running “full sync”
  73. relational database sync • Apply updates from the top of

    your relationship tree • Beware of unexpected cascade deletion during deletes deltas applying • Try to omit many-to-many relationships
  74. performance tuning • Move sync to the background queue /

    context • Use separate contexts: multiple general-purpose contexts, single sync context • Use data prefetching, batch saving
  75. where to go • Add per-field sync metadata • Store

    changed values alongside with the change record • Client mutate objects sync data for more precise merge