Upgrade to Pro — share decks privately, control downloads, hide ads and more …

HOLY SYNC: a sane approach to offline-first cross-platform data syncing

HOLY SYNC: a sane approach to offline-first cross-platform data syncing

Today users expect to have their data available on every device, and to be able to view and edit it at any time without an Internet connection.
Given how common the need is, anyone who tried to approach this problem might have found out that it’s (un)surprisingly complex, and that there are very few available tools!
In this talk I want to show how we solved it at Clue, without going crazy, on both Android and iOS.

In particular:

* Who needs this, and who doesn’t.
* Couchbase Lite: an open-source, cross-platform, embedded NoSQL database.
* Transitioning from a standard SQL data model to a NoSQL one, with deep implications on how that affects every level of the stack.
* How embracing a “Git-like approach” to conflict management can be tremendously effective.
* Keeping performance smooth on Android.

Video at Droidcon Italy 2015:
https://www.youtube.com/watch?v=Yp1h3cd8dsg&lc

Keynote presentation (animated!):
https://drive.google.com/file/d/0B-g3x7VcJPIqLVlKX3doTkY4cmM/view?usp=sharing

Wallpaper "deal with git" (2880×1800):
https://drive.google.com/file/d/0B-g3x7VcJPIqbnFkRUlibUp5dlE/view?usp=sharing

Eugenio Marletti

April 10, 2015
Tweet

More Decks by Eugenio Marletti

Other Decks in Programming

Transcript

  1. * cross-platform solution * no reinventing the wheel (possibly) *

    do as little as possible (not our core business) * open source and actively developed (avoid lock-in)
  2. * no schema > data consistency is now your responsibility

    * de-normalized data > until “it hurts” * views > forget queries, embrace map/reduce embedded NoSQL database
  3. { "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id":

    "uniqueID", "_rev": "2-efg", "key": ":trollface:" } { "_id": "uniqueID", "_rev": "1-abc", "key": "value" }
  4. “CONFLICTS are not an error condition, they are the result

    of your infrastructure allowing the same dataset to be modified across disconnected systems. The introduction of such conflicts in such a topology is the expected behavior and their programmatic resolution is a core piece of application logic.”
  5. { "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id":

    "uniqueID", "_rev": "2-efg", "key": ":trollface:" } { "_id": "uniqueID", "_rev": "1-abc", "key": "value" }
  6. { "_id": "uniqueID", "_rev": "3-ghi", "_deleted": true } { "_id":

    "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id": "uniqueID", "_rev": "2-efg", "key": ":trollface:" }
  7. { "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id":

    "uniqueID", "_rev": "2-efg "key": ":trollface: } { "_id": "uniqueID", "_rev": "3-ghi", "_deleted": true }
  8. * distributed system * based on versioning * conflicts are

    solved by appending a revision * data is pulled & pushed * wait a second…
  9. NO

  10. * get document by id * “query” a view *

    iterate through all documents
  11. View phoneView = database.getView("phones"); phoneView.setMap(new Mapper() { @Override public void

    map(Map<String, Object> document, Emitter emitter) { List<String> phones = (List)document.get("phones"); for (String phone : phones) { emitter.emit(phone, document.get("name")); } } }, "2");
  12. KEY VALUE "1" | "Genny" "123" | "Genny" "123" |

    "Maria" "2" | "Maria" "456" | "Genny" "666" | "Satan" "789" | "Genny"
  13. * null * false, true (in that order) * numbers,

    in numeric order (duh) * strings, case-insensitive (all symbols sort before) * arrays (item-by-item) * maps/dictionaries (don’t use - except empty!)
  14. [ "2015-01-03", "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03",

    "blue" ] [ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ]
  15. [ "2015-01-03", "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03",

    "blue" ] [ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ]
  16. [ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-03",

    "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03", "blue" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ] [ "2015-01-03" ] startKey [ "2015-01-03", { } ] endKey
  17. { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }

    { "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" } { "_id": "pill|…", "_rev": "2-efg", "_deleted": true, "$updated_at": "2" }
  18. { "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" }

    { "_id": "pill|…", "_rev": "2-efg "_deleted": true "$updated_at": "2 } { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }
  19. { "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" }

    { "_id": "pill|…", "_rev": "2-efg", "$removed": true, "$updated_at": "2" } { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }
  20. * models are iOS only :( * “to trust [your

    data] is good, not to trust is better” * find the right time to start replications * when saving, objects get converted to json * when reading, json is parsed as Map<String,Object> * models