Persistent Queues with Tape

The best apps don't make users wait. They dispatch tasks to the background. Offloading tasks to the background is a great way to simulate the feeling of responsiveness.

But processing background tasks can be tricky. What happens if the device runs out of battery? What if the network is down? Persisting to disk helps you handle such edge cases. Enter Tape, a collection of queue related classes.

This talk will primarily be about it’s core component, QueueFile — a lightning fast, transactional, persistent file-based FIFO queue. We'll dig into it's implementation and see how it guarantees both reliability and efficiency. And you'll learn to make the most of QueueFile with real world examples.

Transcript

1. Persistent Queues with Tape Prateek Srivastava
3. curl -X "POST" "https://api.segment.io/v1/track" \ -H "Content-Type: application/json" \ -u apiKey: \ -d "{\"event\":\"hello world\",\"userId\":\"prateek\"}"
5. Background Tasks
7. Networks ar flaky
8. Battery life is limited
9. RAM is volatile
11. Guarantees
12. Durability
13. Atomicity
14. Efficiency
15. –Murphy’s Law “Anything that can go wrong, will go wrong.”
16. Solutions
17. java.io.File https://developer.android.com/reference/java/io/File.html
18. AtomicFile https://developer.android.com/reference/android/util/AtomicFile.html https://developer.android.com/reference/android/support/v4/util/AtomicFile.html
19. if (backupFile.exists()) { file.delete(); backupFile.renameTo(file); }
20. SharedPreferences https://developer.android.com/reference/android/content/SharedPreferences.html
21. SQLite https://developer.android.com/reference/android/database/sqlite/package- summary.html https://developer.android.com/training/basics/data-storage/databases.html
22. Tape https://github.com/square/tape
23. ✔ fast ✔ durable ✔ atomic
24. File System
25. renaming is atomic
26. File temp = new File("file.tmp"); initialize(temp); File file = new File(“file.real"); f.renameTo(file);
27. fsync is durable
28. segment writes are atomic
29. RandomAccessFile
30. class RandomAccessFile { void seek(long pos); <T> read{<T>}(); void write{<T>}(T data); }
31. new RandomAccessFile(file, "rwd");
32. -RWD Open for reading and writing, as with "rw", and also require that every update to the file's content be written synchronously to the underlying storage device.
33. Element 3 Element 2 Element 1 File Header
35. @jake 16 16 1 4096 Last Element Position First Element Position Count File Size 8 bytes 8 bytes 4 bytes 63 bits
36. @jake 16 16 1 4096 Last Element Position First Element Position Count File Size 8 bytes 8 bytes 4 bytes 63 bits
37. Version 1 1 bit Version 0 1 bit @jake
38. @1.2.3
39. @1.2.3
40. Adding Elements
41. queueFile.add(data);
42. int newLastPosition = lastPosition + 4 + lastLength; 4096 1 32 32 Element 1 10
43. raf.seek(newLastPosition); raf.write(data.length); 4096 1 32 32 26 Element 1 10
44. raf.seek(newLastPosition + 4); raf.write(data); 4096 1 32 32 Element 1 10 Element 2 26
45. raf.seek(0); writeHeader(fileLength, elementCount + 1, firstPosition, newLastPosition); 4096 2 32 46 Element 1 10 Element 2 26
46. 4096 1 32 32 26 Element 1 10 4096 1 32 32 Element 1 10 Element 2 26 4096 1 32 32 Element 1 10
47. Removing Elements
48. queueFile.remove();
49. int newFirstPosition = firstPosition + 4 + firstLength; 4096 2 32 46 Element 1 10 Element 2 26
50. writeHeader(fileLength, elementCount - 1, newFirstPosition, lastPosition); 4096 1 46 46 Element 1 10 Element 2 26
51. raf.seek(oldFirstPosition); byte[] empty = new byte[firstLength + 4]; raf.write(empty); 4096 1 46 46 Element 2 26
52. 4096 1 46 46 Element 1 10 Element 2 26 4096 1 46 46 Element 2 26
53. API @master
54. class QueueFile { void add(byte[] data); byte[] peek(); void remove(); int size(); } @1.2.3
55. class QueueFile { void add(byte[] data); byte[] peek(); void remove(); int size(); } @master
56. class QueueFile { void add(byte[] data); byte[] peek(); void remove(); int size(); } @master
57. class QueueFile { void add(byte[] data); byte[] peek(); void remove(); int size(); } @master
58. Usage
59. // Read and remove the data at the // head of the queue. byte[] element = queueFile.remove(); // Do something with the data. process(element);
60. // Read and remove the data at the // head of the queue. byte[] element = queueFile.remove(); // Do something with the data. process(element);
61. // Read data at the head. byte[] element = queueFile.peek(); // Do something with the data. process(element); // Remove data at the head. queueFile.remove();
62. Example
63. byte[] message = queueFile.peek(); Connection connection = open(“/v1/track"); connection.getOutputStream().write(message); connection.close(); queueFile.remove();
64. Handling HTTP Errors
65. retry network errors
66. retry server errors (5xx)
67. ignore client errors (4xx)
68. try { connection.close(); } catch (IOException e) { return; // Network error, retry later. } int responseCode = connection.getResponseCode(); if (responseCode >= 500) { return; // Server error, retry later. } // Client error, e.g. invalid JSON. Don’t retry. if (responseCode >= 400) { log("server rejected message"); } queueFile.remove();
69. @master class QueueFile implements Iterable<byte[]> { void add(byte[] d, int offset, int count); boolean isEmpty(); void remove(int n); void clear(); }
70. class QueueFile implements Iterable<byte[]> { void add(byte[] d, int offset, int count); boolean isEmpty(); void remove(int n); void clear(); } @master
71. class QueueFile implements Iterable<byte[]> { void add(byte[] d, int offset, int count); boolean isEmpty(); void remove(int n); void clear(); } @master
72. class QueueFile implements Iterable<byte[]> { void add(byte[] d, int offset, int count); boolean isEmpty(); void remove(int n); void clear(); } @master
73. @master class QueueFile implements Iterable<byte[]> { void add(byte[] d, int offset, int count); boolean isEmpty(); void remove(int n); void clear(); }
74. Usage
75. Iterator<byte[]> iterator = queueFile.iterator(); while (iterator.hasNext()) { byte[] data = iterator.next(); process(data); iterator.remove(); }
76. // Read the data at the head of the queue. List<byte[]> data = new ArrayList<>(); for (byte[] d : queueFile) { data.add(d); } // Do something with the elements. process(data); // Remove elements from the queue. queueFile.remove(data.size());
77. Example
78. HttpURLConnection connection = open(“/v1/import"); int count = 0; int size = 0; for (byte[] data : queueFile) { if (size + data.length > MAX_LIMIT) { break; } write(data, connection); count++; size += data.length; } connection.close(); queueFile.remove(count);
79. ObjectQueue API @master
80. @master abstract class ObjectQueue<T> implements Iterable<T> { abstract void add(T entry); abstract T peek(); List<T> peek(int max); List<T> asList(); void remove(); abstract void remove(int n); void clear(); abstract int size(); }
81. @master abstract class ObjectQueue<T> implements Iterable<T> { abstract void add(T entry); abstract T peek(); List<T> peek(int max); List<T> asList(); void remove(); abstract void remove(int n); void clear(); abstract int size(); }
82. @master abstract class ObjectQueue<T> implements Iterable<T> { abstract void add(T entry); abstract T peek(); List<T> peek(int max); List<T> asList(); void remove(); abstract void remove(int n); void clear(); abstract int size(); }
83. @1.2.3 interface Converter<T> { T from(byte[] bytes); void toStream(T o, OutputStream os); }
84. ObjectQueue Converter QueueFile
85. ObjectQueue<Pojo> queue = ObjectQueue.createInMemory(); ObjectQueue<Pojo> queue = ObjectQueue.create(file, converter);
86. Usage
87. // Read data at the head. Pojo pojo = queue.peek(); // Do something with the data. process(pojo); // Remove data at the head. queue.remove();
88. // Read the data at the head of the queue. List<Pojo> data = new ArrayList<>(); for (Pojo pojo : queue) { data.add(pojo); } // Do something with the elements. process(data); // Remove elements from the queue. queue.remove(data.size());
89. Beyond Tape
90. GCM Network Manager https://developers.google.com/cloud-messaging/network-manager
91. Firebase JobDispatcher https://github.com/firebase/firebase-jobdispatcher-android
92. JobScheduler https://developer.android.com/reference/android/app/job/JobScheduler.html
93. Android Job http://evernote.github.io/android-job/
94. Priority JobQueue https://github.com/yigit/android-priority-jobqueue