Implementation of Mobile Offline Folders

Overview of the evolution of the architecture used by Dropbox to implement its Mobile Offline Folders feature, focusing on the challenges presented by the Android platform (e.g. background processing and storage limitations). Presented at DroidKaigi 2018 in Tokyo, Japan.

Transcript

1. Implementation of Mobile Offline Folders Rodrigo Gomez Avila rodrigo@dropbox.com
2. What are offline folders? First implementation Second try The Ask Q & A Agenda A little bit of history…
3. Credit where credit is due Even though I’m presenting here today, a lot of people contributed to implementing and shipping this feature. Aakash Josh Asmaa Bill
4. Evolution of the Dropbox app
5. A little bit of history… c. 2011 Dropbox Android App Browsing + ”Favoriting” of some files c. 2008 Dropbox Desktop App Synchronizes all contents to the web
6. What are Offline Files/Folders? (2016) The user can mark specific files or folders to be stored locally so they’re available while offline and kept up-to- date while online. We do not want to keep ”everything” in sync (like we do on desktop), because… well… space on mobile devices is limited. Plus there’s network and battery constraints.
7. The ask
8. The ask Hey, can you check what’s needed to ship Offline Folders? It was implemented by an intern last year, almost finished. Offline Files has been around for some time now. LMK! Sure, let me take a look! Doesn’t look too bad… Boss Me
9. How do Offline Files work? Easy peasy… for (String path : offlinePaths) { // File name, size, icon name, // path, hash… syncMetadata(); // File and additional assets // (Preview and thumbnails) syncContents(); }
10. Single Offline Files Initial Architecture File Store Offline Manager for every path in the queue sync metadata and download files Metadata Manager Download Manager Assets Cache* List<Path>
11. First Implementation
12. Ok! How do we implement offline folders? Some things to think about: 1) Retrieving and storing the metadata of an entire folder structure takes longer than the metadata of a single file. 2) Downloading all of those files is going to take time… A significant part of that will happen when our application is in the background. 3) User might run out of space on their device What if a folder contains 10, 000 files?
13. First Implementation Delta Metadata Syncing (Gandalf’ed) Instead of downloading the metadata for the entire folder, retrieve only what’s changed (a delta) since the last time we checked and store a cursor. Background Syncing Depending on the Android™ OS version, background work can be scheduled using JobScheduler, GcmNetworkManager or AlarmManager. For simplicity, we use the Evernote ™ Job Library to execute syncing in the background Protect the user It’s easy to use up all of the space on the device when downloading a large folder; let’s prevent syncing it if it would leave less than X megabytes of storage.
14. A couple months later… Excellent! Oh! BTW, design came up and said that we should have a global progress of files (number of files and megabytes). I think we’re ready! Fixed one or two things. Let’s QA the hell out of this thing! Boss Me Me
15. What global progress should look like…
16. Why is global progress so hard to get right? Some things to consider: 1) What happens when you offline a new folder in the middle of a download? Do you interrupt the download? Do you do both update metadata and download files in parallel? 2) What happens when the user unmarks a folder for offline sync in the middle of it syncing? There’s no way to show progress unless I have all the metadata for all folders and their files
17. “If I mark a folder as offline while another folder is downloading, nothing appears to happen” Order matters! If we’re downloading the files for a huge folder, then we won’t execute the next offline job until we’re done with everything in the queue. time to sync metadata <<< time to sync files
18. Use a Priority Queue! Metadata retrieval should always happen before files downloading, otherwise we have no way of showing overall progress. Let’s split our ”download job” into three parts: Sync Compute Download (Interruptable)
19. Let’s try again… File Store Offline Manager Metadata Manager Download Manager Assets Cache OfflineQueue ComputeJob MetdataSyncJob DownloadJob
20. “Downloading stops when I connect to WiFi” In Android, when you move between networks, the connection is interrupted for a brief moment. So an ongoing download will be cancelled
21. Let’s implement Retry! We already had a shiny priority queue! 1) Let’s just push the failed job back into the back of the queue and keep going 2) If we have more than N consecutive failures then we stop the entire download
22. “Crash-loop when trying to download a file larger than 2GBs” OK! So retry works... but... In Android <= Marshmallow, FileUtils.copyFile() fails for files larger than 2GB. You need to use IOUtils.copyLarge(). Good to know…
23. “After a couple days I cannot preview offline files" OK… so previews where there before, and now they’re gone! How! InternalStorage.getCacheDir() vs InternalStorage.getDir() Before we were OK, because our cache was relatively small. But when we put thousands of files there, well, Android was eager to reclaim that space!
24. We need 2 different stores for our assets! Assets for offline files should be stored in a persistent storage (InternalStore.getDir()) Assets for all other assets should be evictable by Android (InternalStore.getCacheDir(
25. Finally! Offline Manager Metadata Manager Download Manager OfflineQueue ComputeJob MetdataSyncJob DownloadJob File Store Assets Cache Cache Persistent Assets Store
26. Wrapping it up… Learnings • You want your architecture to be flexible enough to account for later changes. • Adding tests coverage. (adding job queue). • Changes need to be staged. (non-delta to delta). What we could have done differently? • Design lockdown • Don’t make assumptions about the underlying working of a system (AssetsCache)
27. ありがとう ございます rodrigo@dropbox.com
28. Q & A