Tweet
Tweet

Slide 1

Slide 1 text

two-way sync Simplest implementation #cocoaheadskyiv Dima Zen @ 17.12.2015

Slide 2

Slide 2 text

Why? • Client-server application • Full offline support • “All changes have to be available online automatically” - client • Simultaneous editing from multiple clients: iOS, Android, Web What needs to be done

Slide 3

Slide 3 text

Whats available? • Backend written on Django (python) • REST API • google.com, stackoverflow.com

Slide 4

Slide 4 text

What do we need for sync?

Slide 5

Slide 5 text

sync changes
 tracking conflicts API architecture background changelog db versioning experience data authority

Slide 6

Slide 6 text

changes tracking why?

Slide 7

Slide 7 text

remote local t to find conflicts

Slide 8

Slide 8 text

remote pull local t to find conflicts

Slide 9

Slide 9 text

remote local t { id: 12341, username: "john.doe", description: "iOS dev" } to find conflicts

Slide 10

Slide 10 text

remote local t { id: 12341, username: "john.doe", description: “hire me" } * pull to find conflicts

Slide 11

Slide 11 text

remote local t { id: 12341, username: "john.doe", description: “hire me" } * { id: 12341, username: "john.doe", description: "Android" } to find conflicts

Slide 12

Slide 12 text

remote local t { id: 12341, username: "john.doe", description: “hire me" } * { id: 12341, username: "john.doe", description: "Android" } to find conflicts

Slide 13

Slide 13 text

{ id: 12341, username: "john.doe", description: "iOS dev" } remote local t to find unsaved changes

Slide 14

Slide 14 text

{ id: 12341, username: "john.doe", description: “hire me" } * push remote local t to find unsaved changes

Slide 15

Slide 15 text

{ id: 12341, username: "john.doe", description: “hire me" } * ? push remote local t to find unsaved changes

Slide 16

Slide 16 text

changes tracking • Timestamp (lastModified: 17.12.2015T19:20:22.133145Z) • Version (revision: 245) • Status (dirty: true, deleted: false) • Combined aka Change Data Capture

Slide 17

Slide 17 text

{ id: 12341, username: "john.doe", description: "iOS dev" } data structure { id: 12341, username: "john.doe", description: "iOS dev", revision: 213, modified: 12.12.2016, deleted: false, dirty: false } trackable plain

Slide 18

Slide 18 text

changelog local t changes tracking

Slide 19

Slide 19 text

changelog local t { id: 12341, revision: 213, modified: 12.12.2016, deleted: false } changes tracking

Slide 20

Slide 20 text

changelog local t pull changes tracking { id: 12341, revision: 213, modified: 12.12.2016, deleted: false dirty: true } *

Slide 21

Slide 21 text

changelog local t changes tracking { id: 12341, revision: 215, modified: 12.12.2016, deleted: false } { id: 12341, revision: 213, modified: 12.12.2016, deleted: false dirty: true } *

Slide 22

Slide 22 text

sync changes
 tracking conflicts API architecture background changelog db versioning experience data authority

Slide 23

Slide 23 text

data authority • API is the only data authority • Client doesn't change revision nor timestamp, only via API • Timestamp management on client is hard • Client’s changelog can maintain its own revision, independent from the API’s one

Slide 24

Slide 24 text

db versioning

Slide 25

Slide 25 text

remote local without db version

Slide 26

Slide 26 text

remote pull local without db version

Slide 27

Slide 27 text

remote local without db version ? ? ? ? ? ?

Slide 28

Slide 28 text

remote local with db version

Slide 29

Slide 29 text

remote local with db version pull revision=215

Slide 30

Slide 30 text

remote local with db version {no changes}

Slide 31

Slide 31 text

db versioning • Depends on changes tracking implementation • Tied with changelog • Allows to identify changes deltas • Acts as an offset revision = 1023

Slide 32

Slide 32 text

changelog

Slide 33

Slide 33 text

changelog • Ordered log of data change records • Store records such as update or delete for specific object • Can track only fact of the particular event or full change representation (affected keys, old values, new values)

Slide 34

Slide 34 text

db version + changelog • Database.version == changelog.HEAD.version • Database.version == most recent data.version • Any data change increments database version • New change.version == Database.version + 1

Slide 35

Slide 35 text

changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=1

Slide 36

Slide 36 text

changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=1 POST /notes/

Slide 37

Slide 37 text

POST /notes/ changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2

Slide 38

Slide 38 text

{ id: 443, revision: 2, content: “First Note" } Note changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2

Slide 39

Slide 39 text

changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note

Slide 40

Slide 40 text

changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 25, change: update, entity: "Note", objectID: 443 } { id: 443, revision: 2, content: “First Note" } Note

Slide 41

Slide 41 text

changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 }

Slide 42

Slide 42 text

changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } PUT /notes/442/

Slide 43

Slide 43 text

changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } PUT /notes/442/

Slide 44

Slide 44 text

changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } PUT /notes/442/

Slide 45

Slide 45 text

changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog data Note revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } { id: 442, revision: 3, content: "Cocoaheads" }

Slide 46

Slide 46 text

{ id: 24, change: update, entity: "Note", objectID: 442 } changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } { id: 442, revision: 3, content: "Cocoaheads" } Note

Slide 47

Slide 47 text

{ id: 24, change: update, entity: "Note", objectID: 442 } changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } { id: 26, change: update, entity: "Note", objectID: 442 } { id: 442, revision: 3, content: "Cocoaheads" } Note

Slide 48

Slide 48 text

{ id: 24, change: update, entity: "Note", objectID: 442 } changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 442, revision: 3, content: "Cocoaheads" } Note { id: 26, change: update, entity: "Note", objectID: 442 } { id: 25, change: update, entity: "Note", objectID: 443 }

Slide 49

Slide 49 text

{ id: 24, change: update, entity: "Note", objectID: 442 } changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 442, revision: 3, content: "Cocoaheads" } Note DELETE /notes/442/ { id: 26, change: update, entity: "Note", objectID: 442 } { id: 25, change: update, entity: "Note", objectID: 443 }

Slide 50

Slide 50 text

{ id: 442, revision: 3, content: "Cocoaheads" } Note { id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } DELETE /notes/442/ { id: 26, change: update, entity: "Note", objectID: 442 }

Slide 51

Slide 51 text

{ id: 442, revision: 4, content: "Cocoaheads", deleted: true } Note { id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } { id: 26, change: update, entity: "Note", objectID: 442 }

Slide 52

Slide 52 text

{ id: 442, revision: 4, content: "Cocoaheads", deleted: true } Note { id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } { id: 26, change: update, entity: "Note", objectID: 442 }

Slide 53

Slide 53 text

{ id: 27, change: delete, entity: "Note", objectID: 442 } { id: 442, revision: 4, content: "Cocoaheads", deleted: true } Note { id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } { id: 26, change: update, entity: "Note", objectID: 442 }

Slide 54

Slide 54 text

simple changelog, version per row, no changed data • Easy to implement • Easy to maintain • Doesn’t contain actual data deltas • No way to rollback invalid changes • Impossible to do field-by-field conflicts resolution pros cons

Slide 55

Slide 55 text

API • Client’s database version is required for any API data mutation API or deltas fetch • Data mutation from the unsynced client is rejected • Push is request-per-change via REST

Slide 56

Slide 56 text

API GET /sync-chunk/?revision=1331&limit=100 { "max_revision": 1334, "result": { "Note": { "updates": [ { "id": 812, "content": "Cocoaheads Kiev", "revision": 1332 }, { "id": 442, "content": "Android Dev", "revision": 1333 } ], "deletes": [ { "id": 113, "revision": 1334 } ] } } }

Slide 57

Slide 57 text

API with update and delete of the same object, single page { "max_revision": 1335, "result": { "Note": { "updates": [ { "id": 443, "content": "Android Dev", "revision": 1333 } ], "deletes": [ { "id": 442, "revision": 1335 } ] } } }

Slide 58

Slide 58 text

API with update and delete of the same object, two pages { "max_revision": 1334, "result": { "Note": { "updates": [ { "id": 442, "content": "Android Dev", "revision": 1333 }, { "id": 443, "content": "CocoaHeads", "revision": 1334 } ] } } } { "max_revision": 1335, "result": { "Note": { "deletes": [ { "id": 443, "revision": 1335 } ] } } } page #0 page #1

Slide 59

Slide 59 text

API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body

Slide 60

Slide 60 text

API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”]

Slide 61

Slide 61 text

API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true

Slide 62

Slide 62 text

API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true

Slide 63

Slide 63 text

API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true STATUS CODE: 409 (Conflict)

Slide 64

Slide 64 text

API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true STATUS CODE: 409 (Conflict) false { "id": 442, "content": "Android Dev", "revision": 1334 } Revision: 1334 headers body

Slide 65

Slide 65 text

sync changes
 tracking conflicts API architecture background changelog db versioning experience data authority

Slide 66

Slide 66 text

resolving conflicts • Client resolves any merge conflict • Merge changed data pushed to the API • Client-based resolution allows to use UI • Simplifies backend code, client becomes more complex • Graceful resolving requires additional effort • per-field changes tracking • usage of combined changes tracking on client done on client

Slide 67

Slide 67 text

resolving conflicts in dummy way pseudocode class OverwriteConflictResolvingStrategy { let object: Object let changelog: Changelog let update: ChangelogEvent? let delete: ChangelogEvent? func resolve(changes: [String: AnyObject]) { if let update = update { changelog.removeChange(update) } if let delete = delete { changelog.removeChange(delete) } } }

Slide 68

Slide 68 text

sync changes
 tracking conflicts API architecture background changelog db versioning experience data authority

Slide 69

Slide 69 text

architecture

Slide 70

Slide 70 text

Chunk Store Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog

Slide 71

Slide 71 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store

Slide 72

Slide 72 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store

Slide 73

Slide 73 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store

Slide 74

Slide 74 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 431

Slide 75

Slide 75 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 481

Slide 76

Slide 76 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 481

Slide 77

Slide 77 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 501

Slide 78

Slide 78 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 501

Slide 79

Slide 79 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 501

Slide 80

Slide 80 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 501

Slide 81

Slide 81 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 501

Slide 82

Slide 82 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store

Slide 83

Slide 83 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store

Slide 84

Slide 84 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog push Chunk Store

Slide 85

Slide 85 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog push Chunk Store

Slide 86

Slide 86 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog push Chunk Store revision: 431

Slide 87

Slide 87 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog push Chunk Store

Slide 88

Slide 88 text

Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog push Chunk Store

Slide 89

Slide 89 text

merge pseudocode from Merge Operation class ChunkStore { var chunks: [SyncChunk] { return [] } } class Merge { enum EntityType { case User, Note, Notebook } let chunkStore = ChunkStore() func executeMerge() { let orderedTypes: [EntityType] = [.User, .Notebook, .Note] for type in orderedTypes { for chunk in chunkStore.chunks { mergeChunkUpdates(chunk, ofType: type) mergeChunkDeletes(chunk, ofType: type) } } } }

Slide 90

Slide 90 text

experience

Slide 91

Slide 91 text

expect unexpected termination • Design you code with restoration / preservation in mind • Preserve execution state after every sync operation (entity merged, change pushed, etc) • Delete preserved data once done if needed

Slide 92

Slide 92 text

changes push • Push changes in a serial queue • Push and Pull are mutually exclusive and can not be performed concurrently • Handle validation errors to prevent pushing queue hang (e.g. mark change as invalid in case of double validation error, etc)

Slide 93

Slide 93 text

SyncScheduler • Respect reachability and connection errors: make a delay before retry • Don’t use timer with small delay for check. It drains your battery • On Changelog update notify SyncScheduler about local state is being dirty (Observer, NotificationCenter, etc)

Slide 94

Slide 94 text

changes tracking: prefer Version over Timestamp • Integers easier to compare, store and use • Utilize microseconds to prevent data lost (yyyy.MM.dd’T’HH:mm:ss.SSSSSS’Z’) • NSDateFormatter doesn’t parse microseconds • Prefer NSDateComponents over NSDate

Slide 95

Slide 95 text

changes tracking • One place for all • Harder to implement • Requires filtering of unnecessary data. Leads to hell. • Simple implementation • Requires explicit tracking invocation • Tracks only necessary data auto manual

Slide 96

Slide 96 text

changelog as a separate database • Looks like a more robust solution • Allows you to imp. Changelog as a separate module • Doesn’t give a lot of benefit • Requires complex synchronization of Changeling’s context and Database’s context. (not available for iOS 7) • DB saving error may leave you up with inconsistent state: changelog doesn’t reflect database state.

Slide 97

Slide 97 text

deletes • Don’t delete objects immediately, use deleted flag instead • Purge deleted records on: • Delete change push • After running “full sync”

Slide 98

Slide 98 text

relational database sync • Apply updates from the top of your relationship tree • Beware of unexpected cascade deletion during deletes deltas applying • Try to omit many-to-many relationships

Slide 99

Slide 99 text

performance tuning • Move sync to the background queue / context • Use separate contexts: multiple general-purpose contexts, single sync context • Use data prefetching, batch saving

Slide 100

Slide 100 text

where to go • Add per-field sync metadata • Store changed values alongside with the change record • Client mutate objects sync data for more precise merge

Slide 101

Slide 101 text

Links • https://www.objc.io/issues/10-syncing-data/data- synchronization/ • https://dev.evernote.com/media/pdf/edam-sync.pdf • https://en.wikipedia.org/wiki/Change_data_capture • https://technet.microsoft.com/en-us/library/ bb933994(v=sql.105).aspx

Slide 102

Slide 102 text

Wants more? • https://msdn.microsoft.com/en-US/library/bb677158.aspx • http://floriankugler.com/2013/04/29/concurrent-core- data-stack-performance-shootout/ • https://en.wikipedia.org/wiki/Operational_transformation

Slide 103

Slide 103 text

questions?

Slide 104

Slide 104 text

No content

Slide 105

Slide 105 text

thanks :)