ADDC 2019 - Gopal Sharma - Building great offline-first apps

This talk will go through what an “offline-first” app is, why you should build offline-first apps, what some of the challenges of building them are, the problems (which appear simple on the surface, but are subtly complex) my team has faced building them, and how we solved those problems. We’ll look at the problem from a UX perspective first, and talk about some of the things to think about when designing offline-first apps. I will then look at the problem through a developer’s lens and talk about the architecture and techniques we used. We’ll finally briefly get into specific tools for iOS and Android that helped us get everything working.

Recordings & more: https://addconf.com/

Transcript

1. Building Great Offline-First Apps 26 June 2019 Gopal Sharma @gopalkri https://bohr.in
2. What Is an “Offline-First” App? • Work without internet connection • “Catch up” to a server later
3. Example: Notes
5. Do You Need to Be Offline-First? • Ubiquitous networks • Being offline-first is non-trivial • You may not need to be offline-first • Unless required, don’t do it!
6. Why Build Offline-First Apps?
7. Networks • Networks are (still) slow, flaky or expensive • International roaming • Rural areas • Networks are not as pervasive as we think • Airplanes • Basements • Elevators • Hospitals
9. 14,000 KM from BLR to SFO 47ms At speed of light 240ms Verizon: India ↔ North America 360ms Me: India ↔ North America 0.05ms Flash storage latency https://enterprise.verizon.com/terms/latency/ https://www.snia.org/sites/default/education/tutorials/2010/spring/solid/LeviNorman_Latency_The_Heartbeat_SSD.pdf
10. “You should never wait for your phone, your phone should wait for you” -Me
11. User Experience • > 100 ms latency ➡ waiting on your phone * • People really dislike slow apps *https://www.youtube.com/watch?v=vOvQCPLkPt4
12. Challenges
13. Challenges • Conflicts • Inconsistencies • Errors • Performance
14. Conflicts Device A Device B 09:00 Offline Offline 09:02 Note 1 Device B 09:03 Note 1 Device A 09:04 Online 09:05 Online 09:06 ??? ??? Note 1 Server 09:01 Note 1 Note 1 Note 1 Device A ??? ???
15. Options? • Pick a version, discard other • Duplicate, keep both • Combine both
16. Pick A Version • Data loss ⚠ • Even if that’s okay, which version? • First? • Which one was first? • Client timestamps can’t be trusted • Server timestamps are better, but still not 100% accurate
17. Duplicate • No data loss • Possibly annoying
18. Combine • How? • No generic answer • Options • diff3 • CRDTs http://www.cis.upenn.edu/~bcpierce/papers/diff3-short.pdf https://pages.lip6.fr/Marc.Shapiro/papers/RR-7687.pdf
19. It Gets Worse…
20. Example: Calendar Device B Merged Device A
21. Inconsistencies • Records are conflict free • Data across records is inconsistent
22. Example: Timing • Each block of time is a “Timer” record • Start time • End time • Project id • Can’t be doing two things at once • Records may not be in state of conflict • But, two records may be inconsistent • 9-10AM Project 1, 9:30 to 10:30 AM Project 2
23. Building Consensus • Multiple devices • Server • Canonical source of truth? (~ SVN) • Distributed truth? (~ git)
24. CloudKit Model http://www.vldb.org/pvldb/vol11/p540-shraer.pdf
25. Server Device A Device B Note - id - changeToken - text - etc. Note id: 1 ct: 1 text: ABC Note id: 1 ct: 1 text: ABC Note id: 1 ct: 1 text: ABC Time: t0
26. Server Device A Device B Note id: 1 ct: 1 text: ABCD Note id: 1 ct: 1 text: ABC Note id: 1 ct: 1 text: ABCE Time: t1
27. Note id: 1 ct: 2 text: ABCD Note id: 1 ct: 1 text: ABCD Server Device A Device B Note id: 1 ct: 1 text: ABC Note id: 1 ct: 1 text: ABCE Time: t2 Note id: 1 ct: 1 text: ABCD Note id: 1 ct: 2 text: ABCD Note id: 1 ct: 2 text: ABCD
28. Note id: 1 ct: 2 text: ABCD Server Device A Device B Note id: 1 ct: 2 text: ABCD Note id: 1 ct: 1 text: ABCE Time: t3 Note id: 1 ct: 1 text: ABCE Note id: 1 ct: 2 text: ABCD Note id: 1 ct: 1 text: ABCE Note id: 1 ct: 1 text: ABC
29. Note id: 1 ct: 1 text: ABCE Note id: 1 ct: 2 text: ABCD Server Device A Device B Note id: 1 ct: 2 text: ABCD Note id: 1 ct: 2 text: ABCDE Time: t4 Note id: 1 ct: 2 text: ABCD Note id: 1 ct: 1 text: ABCE Note id: 1 ct: 1 text: ABC
30. Note id: 1 ct: 2 text: ABCDE Note id: 1 ct: 2 text: ABCD Server Device A Device B Note id: 1 ct: 2 text: ABCD Note id: 1 ct: 3 text: ABCDE Time: t5 Note id: 1 ct: 2 text: ABCDE Note id: 1 ct: 3 text: ABCDE Note id: 1 ct: 3 text: ABCDE
31. Note id: 1 ct: 2 text: ABCDE Note id: 1 ct: 2 text: ABCD Server Device A Device B Note id: 1 ct: 3 text: ABCDE Note id: 1 ct: 3 text: ABCDE Time: t6 Note id: 1 ct: 3 text: ABCDE getLatest?since=2 Note id: 1 ct: 3 text: ABCDE
32. Error Handling
33. Traditional Error Handling - Sync enum GarbageError: Error { case garbage(value: UInt32) } func maybeThrow() throws -> Int { let random = arc4random() if random > 10 { throw GarbageError.garbage(value: random) } return random } do { let value = try maybeThrow() print("Value is: \(value)") } catch { print("Error is: \(error)") } func maybeError() -> Int { let random = arc4random() if random > 10 { return -1 } return random } let value = maybeError() if value < 0 { print("Error!") } else { print("Value: \(value)") }
34. Traditional Error Handling - Async enum GarbageError: Error { case garbage(value: UInt32) } func maybeError(callback: (Result<UInt32, GarbageError>) -> Void) { let random = arc4random() if random > 10 { callback(.failure(.garbage(value: random))) } else { callback(.success(random)) } } maybeError { result in switch result { case .success(let value): print("\(value)") case .failure(let error): print("Error: \(error)") } }
35. Device A Device B ???
36. A Solution • Persist errors in database • Create observable that fires when an error is found • Make common UI to get user to resolve error • Opt screens into observing error observable and triggering UI when it happens • Remove error from DB when resolved • Also works for detecting inconsistencies
37. Performance
38. Performance • How does data move between devices? • Cannot send complete data set back & forth • Need to sync deltas • Forward sync • Start with first record, move forwards • Get new records since last known change token
39. Note id: 1 ct: 3 text: ABCDE Note id: 1 ct: 2 text: ABCD Note id: 1 ct: 2 text: ABCDE Server Device A Device B Note id: 1 ct: 3 text: ABCDE Note id: 1 ct: 3 text: ABCDE Note id: 1 ct: 3 text: ABCDE getLatest?since=2
40. Architecture
41. Primary DB ViewController Store Rx Observable<ViewData> Observable<Error> View 1 View 2 View 1 Data View 2 Data Read Path
42. Write Path Primary DB ViewController Store Model View 1 View 2 View 1 Event View 2 Event
43. Architecture - Sync Primary DB SyncManager Store ViewController Start sync APIClient Model Model
44. Libraries & Tools • iOS • GRDB • RxSwift • RxGRDB • Android • SQLDelight • RxJava/Kotlin • Architecture components
45. Summary • Building offline-first apps is hard • Offline-first apps can be great for your users • Superior user experience • Flaky internet connections • Challenges & solutions • Conflicts • Inconsistencies • Error handling • Performance
46. Thank You! Questions? @gopalkri https://bohr.in