HOLY SYNC: a sane approach to offline-first cross-platform data syncing

Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Eugenio Marletti @workingkills

Slide 3

Slide 3 text

2014 fastest growing app category* *according to Google HEALTH & FITNESS

Slide 4

Slide 4 text

PERIOD TRACKERS

Slide 5

Slide 5 text

“An app that doesn’t fart rainbows.”

Slide 6

Slide 6 text

DATA SYNCING & SHARING

Slide 7

Slide 7 text

* cross-platform solution * no reinventing the wheel (possibly) * do as little as possible (not our core business) * open source and actively developed (avoid lock-in)

Slide 8

Slide 8 text

LOCAL SERVICE

Slide 9

Slide 9 text

SINGLE SERVER, ALWAYS CONNECTED

Slide 10

Slide 10 text

MULTIPLE SERVERS

Slide 11

Slide 11 text

DATA SYNCING BETWEEN DEVICES (AND PLATFORMS)

Slide 12

Slide 12 text

NOT ALWAYS CONNECTED

Slide 13

Slide 13 text

DATA SHARING BETWEEN USERS

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

in case you didn’t get the joke the first time

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

LITE embedded NoSQL database that speaks the CouchDB protocol (and is designed for mobile)

Slide 18

Slide 18 text

embedded NoSQL database that speaks the CouchDB protocol

Slide 19

Slide 19 text

* no schema > data consistency is now your responsibility * de-normalized data > until “it hurts” * views > forget queries, embrace map/reduce embedded NoSQL database

Slide 20

Slide 20 text

DOCUMENTS

Slide 21

Slide 21 text

{ "_id": "uniqueID", "_rev": "1-abc", "key": "value" }

Slide 22

Slide 22 text

{ "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id": "uniqueID", "_rev": "2-efg", "key": ":trollface:" } { "_id": "uniqueID", "_rev": "1-abc", "key": "value" }

Slide 23

Slide 23 text

CONFLICTS

Slide 24

Slide 24 text

“CONFLICTS are not an error condition, they are the result of your infrastructure allowing the same dataset to be modified across disconnected systems. The introduction of such conflicts in such a topology is the expected behavior and their programmatic resolution is a core piece of application logic.”

Slide 25

Slide 25 text

{ "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id": "uniqueID", "_rev": "2-efg", "key": ":trollface:" } { "_id": "uniqueID", "_rev": "1-abc", "key": "value" }

Slide 26

Slide 26 text

{ "_id": "uniqueID", "_rev": "3-ghi", "_deleted": true } { "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id": "uniqueID", "_rev": "2-efg", "key": ":trollface:" }

Slide 27

Slide 27 text

{ "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id": "uniqueID", "_rev": "2-efg "key": ":trollface: } { "_id": "uniqueID", "_rev": "3-ghi", "_deleted": true }

Slide 28

Slide 28 text

REPLICATION

Slide 29

Slide 29 text

PULL REPLICATION PUSH

Slide 30

Slide 30 text

* distributed system * based on versioning * conflicts are solved by appending a revision * data is pulled & pushed * wait a second…

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

QUERIES

Slide 34

Slide 34 text

Slide 35

Slide 35 text

* get document by id * “query” a view * iterate through all documents

Slide 36

Slide 36 text

VIEWS

Slide 37

Slide 37 text

View phoneView = database.getView("phones"); phoneView.setMap(new Mapper() { @Override public void map(Map document, Emitter emitter) { List phones = (List)document.get("phones"); for (String phone : phones) { emitter.emit(phone, document.get("name")); } } }, "2");

Slide 38

Slide 38 text

Slide 39

Slide 39 text

* null * false, true (in that order) * numbers, in numeric order (duh) * strings, case-insensitive (all symbols sort before) * arrays (item-by-item) * maps/dictionaries (don’t use - except empty!)

Slide 40

Slide 40 text

[ "2015-01-03", "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03", "blue" ] [ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ]

Slide 41

Slide 41 text

[ "2015-01-03", "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03", "blue" ] [ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ]

Slide 42

Slide 42 text

[ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-03", "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03", "blue" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ] [ "2015-01-03" ] startKey [ "2015-01-03", { } ] endKey

Slide 43

Slide 43 text

BACKEND TROUBLE

Slide 44

Slide 44 text

DATABASE PROXY

Slide 45

Slide 45 text

TIPS AND TRICKS

Slide 46

Slide 46 text

don’t be scared of metadata

Slide 47

Slide 47 text

"$created_at" "$updated_at"

Slide 48

Slide 48 text

ids are awesome

Slide 49

Slide 49 text

type user_averages

Slide 50

Slide 50 text

type|day pill|2015-01-01

Slide 51

Slide 51 text

type|day|value tag|2015-01-01|!

Slide 52

Slide 52 text

type|value tag_list_item|stress

Slide 53

Slide 53 text

be wary of deletion

Slide 54

Slide 54 text

{ "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }

Slide 55

Slide 55 text

{ "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" } { "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" } { "_id": "pill|…", "_rev": "2-efg", "_deleted": true, "$updated_at": "2" }

Slide 56

Slide 56 text

{ "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" } { "_id": "pill|…", "_rev": "2-efg "_deleted": true "$updated_at": "2 } { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }

Slide 57

Slide 57 text

{ "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" } { "_id": "pill|…", "_rev": "2-efg", "$removed": true, "$updated_at": "2" } { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }

Slide 58

Slide 58 text

Android gotchas

Slide 59

Slide 59 text

* models are iOS only :( * “to trust [your data] is good, not to trust is better” * find the right time to start replications * when saving, objects get converted to json * when reading, json is parsed as Map * models

Slide 60

Slide 60 text

BIT.LY/HOLY-SYNC @workingkills