HOLY SYNC: a sane approach to offline-first cross-platform data syncing

Today users expect to have their data available on every device, and to be able to view and edit it at any time without an Internet connection.
Given how common the need is, anyone who tried to approach this problem might have found out that it’s (un)surprisingly complex, and that there are very few available tools!
In this talk I want to show how we solved it at Clue, without going crazy, on both Android and iOS.

In particular:

* Who needs this, and who doesn’t.
* Couchbase Lite: an open-source, cross-platform, embedded NoSQL database.
* Transitioning from a standard SQL data model to a NoSQL one, with deep implications on how that affects every level of the stack.
* How embracing a “Git-like approach” to conflict management can be tremendously effective.
* Keeping performance smooth on Android.

Video at Droidcon Italy 2015:
https://www.youtube.com/watch?v=Yp1h3cd8dsg&lc

Keynote presentation (animated!):
https://drive.google.com/file/d/0B-g3x7VcJPIqLVlKX3doTkY4cmM/view?usp=sharing

Wallpaper "deal with git" (2880×1800):
https://drive.google.com/file/d/0B-g3x7VcJPIqbnFkRUlibUp5dlE/view?usp=sharing

Transcript

1. View Slide

2. Eugenio
   Marletti
   @workingkills
   

   View Slide

3. 2014 fastest growing app category*
   *according to Google
   HEALTH & FITNESS
   

   View Slide

4. PERIOD TRACKERS
   

   View Slide

5. “An app that doesn’t fart rainbows.”
   

   View Slide

6. DATA SYNCING & SHARING
   

   View Slide

7. * cross-platform solution
   * no reinventing the wheel (possibly)
   * do as little as possible (not our core business)
   * open source and actively developed (avoid lock-in)
   

   View Slide

8. LOCAL
   SERVICE
   

   View Slide

9. SINGLE SERVER,
   ALWAYS CONNECTED
   

   View Slide

10. MULTIPLE
    SERVERS
    

    View Slide

11. DATA SYNCING
    BETWEEN DEVICES
    (AND PLATFORMS)
    

    View Slide

12. NOT
    ALWAYS
    CONNECTED
    

    View Slide

13. DATA SHARING
    BETWEEN USERS
    

    View Slide

14. View Slide

15. in case you didn’t get the joke the first time
    

    View Slide

16. View Slide

17. LITE
    embedded NoSQL database
    that speaks the CouchDB protocol
    (and is designed for mobile)
    

    View Slide

18. embedded NoSQL database
    that speaks the CouchDB protocol
    

    View Slide

19. * no schema > data consistency is now your responsibility
    * de-normalized data > until “it hurts”
    * views > forget queries, embrace map/reduce
    embedded NoSQL database
    

    View Slide

20. DOCUMENTS
    

    View Slide

21. {
    "_id": "uniqueID",
    "_rev": "1-abc",
    "key": "value"
    }
    

    View Slide

22. {
    "_id": "uniqueID",
    "_rev": "2-cde",
    "key": "anotherValue"
    }
    {
    "_id": "uniqueID",
    "_rev": "2-efg",
    "key": ":trollface:"
    }
    {
    "_id": "uniqueID",
    "_rev": "1-abc",
    "key": "value"
    }
    

    View Slide

23. CONFLICTS
    

    View Slide

24. “CONFLICTS are not an error condition, they
    are the result of your infrastructure allowing
    the same dataset to be modified across
    disconnected systems. The introduction of
    such conflicts in such a topology is the
    expected behavior and their programmatic
    resolution is a core piece of application logic.”
    

    View Slide

25. {
    "_id": "uniqueID",
    "_rev": "2-cde",
    "key": "anotherValue"
    }
    {
    "_id": "uniqueID",
    "_rev": "2-efg",
    "key": ":trollface:"
    }
    {
    "_id": "uniqueID",
    "_rev": "1-abc",
    "key": "value"
    }
    

    View Slide

26. {
    "_id": "uniqueID",
    "_rev": "3-ghi",
    "_deleted": true
    }
    {
    "_id": "uniqueID",
    "_rev": "2-cde",
    "key": "anotherValue"
    }
    {
    "_id": "uniqueID",
    "_rev": "2-efg",
    "key": ":trollface:"
    }
    

    View Slide

27. {
    "_id": "uniqueID",
    "_rev": "2-cde",
    "key": "anotherValue"
    }
    {
    "_id": "uniqueID",
    "_rev": "2-efg
    "key": ":trollface:
    }
    {
    "_id": "uniqueID",
    "_rev": "3-ghi",
    "_deleted": true
    }
    

    View Slide

28. REPLICATION
    

    View Slide

29. PULL
    REPLICATION
    PUSH
    

    View Slide

30. * distributed system
    * based on versioning
    * conflicts are solved by appending a revision
    * data is pulled & pushed
    * wait a second…
    

    View Slide

31. View Slide

32. View Slide

33. QUERIES
    

    View Slide

34. NO
    

    View Slide

35. * get document by id
    * “query” a view
    * iterate through all documents
    

    View Slide

36. VIEWS
    

    View Slide

37. View phoneView = database.getView("phones");
    phoneView.setMap(new Mapper() {
    @Override
    public void map(Map document,
    Emitter emitter) {
    List phones = (List)document.get("phones");
    for (String phone : phones) {
    emitter.emit(phone, document.get("name"));
    }
    }
    }, "2");
    

    View Slide

38. KEY VALUE
    "1" | "Genny"
    "123" | "Genny"
    "123" | "Maria"
    "2" | "Maria"
    "456" | "Genny"
    "666" | "Satan"
    "789" | "Genny"
    

    View Slide

39. * null
    * false, true (in that order)
    * numbers, in numeric order (duh)
    * strings, case-insensitive (all symbols sort before)
    * arrays (item-by-item)
    * maps/dictionaries (don’t use - except empty!)
    

    View Slide

40. [ "2015-01-03", "blue" ]
    [ "2015-01-03", "red" ]
    [ "2015-01-03", "blue" ]
    [ "2015-01-04", "blue" ]
    [ "2015-01-05", "red" ]
    [ "2015-01-01", "blue" ]
    [ "2015-01-02", "red" ]
    

    View Slide

41. [ "2015-01-03", "blue" ]
    [ "2015-01-03", "red" ]
    [ "2015-01-03", "blue" ]
    [ "2015-01-04", "blue" ]
    [ "2015-01-05", "red" ]
    [ "2015-01-01", "blue" ]
    [ "2015-01-02", "red" ]
    

    View Slide

42. [ "2015-01-04", "blue" ]
    [ "2015-01-05", "red" ]
    [ "2015-01-03", "blue" ]
    [ "2015-01-03", "red" ]
    [ "2015-01-03", "blue" ]
    [ "2015-01-01", "blue" ]
    [ "2015-01-02", "red" ]
    [ "2015-01-03" ]
    startKey
    [ "2015-01-03", { } ]
    endKey
    

    View Slide

43. BACKEND TROUBLE
    

    View Slide

44. DATABASE
    PROXY
    

    View Slide

45. TIPS AND TRICKS
    

    View Slide

46. don’t be scared of metadata
    

    View Slide

47. "$created_at"
    "$updated_at"
    

    View Slide

48. ids are awesome
    

    View Slide

49. type
    user_averages
    

    View Slide

50. type|day
    pill|2015-01-01
    

    View Slide

51. type|day|value
    tag|2015-01-01|!
    

    View Slide

52. type|value
    tag_list_item|stress
    

    View Slide

53. be wary of deletion
    

    View Slide

54. {
    "_id": "pill|…",
    "_rev": "1-abc",
    "value": "missed",
    "$updated_at": "0"
    }
    

    View Slide

55. {
    "_id": "pill|…",
    "_rev": "1-abc",
    "value": "missed",
    "$updated_at": "0"
    }
    {
    "_id": "pill|…",
    "_rev": "2-cde",
    "value": "taken",
    "$updated_at": "1"
    }
    {
    "_id": "pill|…",
    "_rev": "2-efg",
    "_deleted": true,
    "$updated_at": "2"
    }
    

    View Slide

56. {
    "_id": "pill|…",
    "_rev": "2-cde",
    "value": "taken",
    "$updated_at": "1"
    }
    {
    "_id": "pill|…",
    "_rev": "2-efg
    "_deleted": true
    "$updated_at": "2
    }
    {
    "_id": "pill|…",
    "_rev": "1-abc",
    "value": "missed",
    "$updated_at": "0"
    }
    

    View Slide

57. {
    "_id": "pill|…",
    "_rev": "2-cde",
    "value": "taken",
    "$updated_at": "1"
    }
    {
    "_id": "pill|…",
    "_rev": "2-efg",
    "$removed": true,
    "$updated_at": "2"
    }
    {
    "_id": "pill|…",
    "_rev": "1-abc",
    "value": "missed",
    "$updated_at": "0"
    }
    

    View Slide

58. Android gotchas
    

    View Slide

59. * models are iOS only :(
    * “to trust [your data] is good, not to trust is better”
    * find the right time to start replications
    * when saving, objects get converted to json
    * when reading, json is parsed as Map
    * models
    

    View Slide

60. BIT.LY/HOLY-SYNC
    @workingkills
    

    View Slide