HOLY SYNC: a sane approach to offline-first cross-platform data syncing

HOLY SYNC: a sane approach to offline-first cross-platform data syncing

Today users expect to have their data available on every device, and to be able to view and edit it at any time without an Internet connection.
Given how common the need is, anyone who tried to approach this problem might have found out that it’s (un)surprisingly complex, and that there are very few available tools!
In this talk I want to show how we solved it at Clue, without going crazy, on both Android and iOS.

In particular:

* Who needs this, and who doesn’t.
* Couchbase Lite: an open-source, cross-platform, embedded NoSQL database.
* Transitioning from a standard SQL data model to a NoSQL one, with deep implications on how that affects every level of the stack.
* How embracing a “Git-like approach” to conflict management can be tremendously effective.
* Keeping performance smooth on Android.

Video at Droidcon Italy 2015:
https://www.youtube.com/watch?v=Yp1h3cd8dsg&lc

Keynote presentation (animated!):
https://drive.google.com/file/d/0B-g3x7VcJPIqLVlKX3doTkY4cmM/view?usp=sharing

Wallpaper "deal with git" (2880×1800):
https://drive.google.com/file/d/0B-g3x7VcJPIqbnFkRUlibUp5dlE/view?usp=sharing

Fde10bcf0813b2162545477be4e7470b?s=128

Eugenio Marletti

April 10, 2015
Tweet

Transcript

  1. None
  2. Eugenio Marletti @workingkills

  3. 2014 fastest growing app category* *according to Google HEALTH &

    FITNESS
  4. PERIOD TRACKERS

  5. “An app that doesn’t fart rainbows.”

  6. DATA SYNCING & SHARING

  7. * cross-platform solution * no reinventing the wheel (possibly) *

    do as little as possible (not our core business) * open source and actively developed (avoid lock-in)
  8. LOCAL SERVICE

  9. SINGLE SERVER, ALWAYS CONNECTED

  10. MULTIPLE SERVERS

  11. DATA SYNCING BETWEEN DEVICES (AND PLATFORMS)

  12. NOT ALWAYS CONNECTED

  13. DATA SHARING BETWEEN USERS

  14. None
  15. in case you didn’t get the joke the first time

  16. None
  17. LITE embedded NoSQL database that speaks the CouchDB protocol (and

    is designed for mobile)
  18. embedded NoSQL database that speaks the CouchDB protocol

  19. * no schema > data consistency is now your responsibility

    * de-normalized data > until “it hurts” * views > forget queries, embrace map/reduce embedded NoSQL database
  20. DOCUMENTS

  21. { "_id": "uniqueID", "_rev": "1-abc", "key": "value" }

  22. { "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id":

    "uniqueID", "_rev": "2-efg", "key": ":trollface:" } { "_id": "uniqueID", "_rev": "1-abc", "key": "value" }
  23. CONFLICTS

  24. “CONFLICTS are not an error condition, they are the result

    of your infrastructure allowing the same dataset to be modified across disconnected systems. The introduction of such conflicts in such a topology is the expected behavior and their programmatic resolution is a core piece of application logic.”
  25. { "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id":

    "uniqueID", "_rev": "2-efg", "key": ":trollface:" } { "_id": "uniqueID", "_rev": "1-abc", "key": "value" }
  26. { "_id": "uniqueID", "_rev": "3-ghi", "_deleted": true } { "_id":

    "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id": "uniqueID", "_rev": "2-efg", "key": ":trollface:" }
  27. { "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id":

    "uniqueID", "_rev": "2-efg "key": ":trollface: } { "_id": "uniqueID", "_rev": "3-ghi", "_deleted": true }
  28. REPLICATION

  29. PULL REPLICATION PUSH

  30. * distributed system * based on versioning * conflicts are

    solved by appending a revision * data is pulled & pushed * wait a second…
  31. None
  32. None
  33. QUERIES

  34. NO

  35. * get document by id * “query” a view *

    iterate through all documents
  36. VIEWS

  37. View phoneView = database.getView("phones"); phoneView.setMap(new Mapper() { @Override public void

    map(Map<String, Object> document, Emitter emitter) { List<String> phones = (List)document.get("phones"); for (String phone : phones) { emitter.emit(phone, document.get("name")); } } }, "2");
  38. KEY VALUE "1" | "Genny" "123" | "Genny" "123" |

    "Maria" "2" | "Maria" "456" | "Genny" "666" | "Satan" "789" | "Genny"
  39. * null * false, true (in that order) * numbers,

    in numeric order (duh) * strings, case-insensitive (all symbols sort before) * arrays (item-by-item) * maps/dictionaries (don’t use - except empty!)
  40. [ "2015-01-03", "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03",

    "blue" ] [ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ]
  41. [ "2015-01-03", "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03",

    "blue" ] [ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ]
  42. [ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-03",

    "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03", "blue" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ] [ "2015-01-03" ] startKey [ "2015-01-03", { } ] endKey
  43. BACKEND TROUBLE

  44. DATABASE PROXY

  45. TIPS AND TRICKS

  46. don’t be scared of metadata

  47. "$created_at" "$updated_at"

  48. ids are awesome

  49. type user_averages

  50. type|day pill|2015-01-01

  51. type|day|value tag|2015-01-01|!

  52. type|value tag_list_item|stress

  53. be wary of deletion

  54. { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }

  55. { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }

    { "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" } { "_id": "pill|…", "_rev": "2-efg", "_deleted": true, "$updated_at": "2" }
  56. { "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" }

    { "_id": "pill|…", "_rev": "2-efg "_deleted": true "$updated_at": "2 } { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }
  57. { "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" }

    { "_id": "pill|…", "_rev": "2-efg", "$removed": true, "$updated_at": "2" } { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }
  58. Android gotchas

  59. * models are iOS only :( * “to trust [your

    data] is good, not to trust is better” * find the right time to start replications * when saving, objects get converted to json * when reading, json is parsed as Map<String,Object> * models
  60. BIT.LY/HOLY-SYNC @workingkills