Handle large dataset in Android

At the beginning of product development, we used SQLite and Content Provider in Data Layer. Time flies and those data scale up to more than thousand of records and it causes application performance go down. After a period of time struggling around, we find Snappy DB the Saviour.

Transcript

1. Handle large dataset in Android © CloudJay, 2014 1
2. About me • Tech guy • Android & Sails.js • Developer to co-founder @ CloudJay • Teamwork & Entrepreneurship © CloudJay, 2014 2
3. Our story © CloudJay, 2014 3
4. Our story CJayNet -- Shipping Container Network • Camera -> Rugged Android Smartphone • Paper work -> Cloud • 2G Network: unstable, untrustable • Offline: work smoothly when network down So we need to keep all data in client. © CloudJay, 2014 4
5. Architecture © CloudJay, 2014 5
6. Problem • Complex DB Schema with 4 main tables and 6 minor tables. • Thousands number of records • Performance © CloudJay, 2014 6
7. Snappy DB • Homepage: http:// www.snappydb.com/ • NoSQL: keys and collections • Based on LevelDB: data is sorted by key © CloudJay, 2014 7
8. Features • Read/write: primitive types, object, serializable DB db = DBFactory.open(context); MyPojo pojo = new MyPojo (); db.put(KEY, pojo); /*...*/ MyPojo myObject = db.getObject(KEY, MyPojo.class); © CloudJay, 2014 8
9. Features • Search: by prefix, by range, with offset and limit // Return keys starting with `PREFIX_KEY` String [] keys = db.findKeys(PREFIX_KEY); // return the fourth key starting with `PREFIX_KEY` (offset 3, limit 1) keys = db.findKeys(PREFIX_KEY, 3, 1); // keys = db.findKeysBetween(PREFIX_KEY + "01", PREFIX_KEY + "09"); • Count: count = db.countKeys(PREFIX_KEY); © CloudJay, 2014 9
10. Features • Iterator public boolean hasNext(); // Whether or not this is the last key. public String[] next(int max); // Get an array of next keys (maximum [max] keys). void close(); // Closes the iterator. Usage: // An iterator to all keys it = snappyDB.allKeysIterator(); /*...*/ it.close(); © CloudJay, 2014 10
11. New architecture • SnappyDB • EventBus 1 1 For more details about implementation: http://blog.joanzapata.com/robust-architecture-for-an- android-app/ © CloudJay, 2014 11
12. Benchmark • LevelDB: http://leveldb.googlecode.com/svn/ trunk/doc/benchmark.html • SnappyDB: © CloudJay, 2014 12
13. Benchmark // Sample model public class Container { long id; int localStep; int step; int uploadStatus; long operatorId; long depotId; long preStatus; long status; boolean retry; String modifiedAt; String containerId; String operatorCode; String depotCode; String checkInTime; String checkOutTime; List<GateImage> gateImages; List<AuditItem> auditItems; } © CloudJay, 2014 13
14. Benchmark Find a record in 10k records to update. Device: Casio Gz'One. • SQLite: 8s • SnappyDB: ~ 50 ms © CloudJay, 2014 14
15. Pros • Read and write data in main thread • No need to implement LoaderCallback, CursorLoader and CursorAdapter • Just basic implementation © CloudJay, 2014 15
16. Cons • Not yet support full text search • Only a single process (possibly multi- threaded) can access a particular database at a time. • Cause locked by thread exception or Serialization exception 2 2 It's fast enough to apply Command Design Pattern to execute one query at the time. © CloudJay, 2014 16
17. Apply Command Pattern © CloudJay, 2014 17
18. Apply Command Pattern // abstract command public abstract class Command implements Task<Command.Callback> { public interface Callback { void onSuccess(String url); void onFailure(Throwable e); } protected abstract void run() throws SnappydbException; @Override public void execute(Callback callback) { try { run(); callback.onSuccess(""); } catch (Throwable e) { e.printStackTrace(); callback.onFailure(e); } } } © CloudJay, 2014 18
19. Apply Command Pattern // Concrete command public class SearchCommand extends Command { Context context; String keyword; // Construction public SearchCommand(Context context, String keyword) { this.context = context; this.keyword = keyword; } @Override protected void run() { // Do stuff DataCenter dataCenter = DataCenter_.getInstance_(context); List<Container> containers = dataCenter.getListContainers(context, keyword, ""); // Post back data via EventBus EventBus.getDefault().post(new ContainerSearchedEvent(containers)); } } © CloudJay, 2014 19
20. Apply Command Pattern public class MyActivity extends Activity { /* .. */ void performSearch() { // Add command to queue String keyword = editText.getText().toString(); // Add to queue and start Invoker/service queue.add(new SearchCommand(this, keyword)); } /* .. */ } © CloudJay, 2014 20
21. Apply Command Pattern // Invoker public class QueryService extends IntentService implements Command.Callback { private void executeNext() { if (processing) return; // Only one task at a time. Command task = queue.peek(); if (task != null) { processing = true; task.execute(this); } else { stopSelf(); // No more tasks are present. Stop. } } @Override protected void onHandleIntent(Intent intent) { executeNext(); } @Override public void onSuccess(String url) { // dequeue and execute next item } @Override public void onFailure(Throwable e) { // handle failure and execute next item } } © CloudJay, 2014 21
22. More discussions about leveldb: • https://gist.github.com/cbess/5872144 • https://news.ycombinator.com/item? id=2526032 • https://www.quora.com/LevelDB • http://blog.newitfarmer.com/nosql/key-value- store-nosql/5553/repost-leveldb-fast-and- lightweight-keyvalue-database-from-the- authors-of-mapreduce-and-bigtable-2 © CloudJay, 2014 22
23. Q & A © CloudJay, 2014 23
24. Challenge When should we use SnappyDB? © CloudJay, 2014 24