Persistent Queues with Tape

The best apps don't make users wait. They dispatch tasks to the background. Offloading tasks to the background is a great way to simulate the feeling of responsiveness.

But processing background tasks can be tricky. What happens if the device runs out of battery? What if the network is down? Persisting to disk helps you handle such edge cases. Enter Tape, a collection of queue related classes.

This talk will primarily be about it’s core component, QueueFile — a lightning fast, transactional, persistent file-based FIFO queue. We'll dig into it's implementation and see how it guarantees both reliability and efficiency. And you'll learn to make the most of QueueFile with real world examples.

Transcript

1. Persistent Queues
   with Tape
   Prateek Srivastava
   

   View Slide

2. View Slide

3. curl -X "POST" "https://api.segment.io/v1/track" \
   -H "Content-Type: application/json" \
   -u apiKey: \
   -d "{\"event\":\"hello world\",\"userId\":\"prateek\"}"
   

   View Slide

4. View Slide

5. Background Tasks
   

   View Slide

6. View Slide

7. Networks ar flaky
   

   View Slide

8. Battery life is limited
   

   View Slide

9. RAM is volatile
   

   View Slide

10. View Slide

11. Guarantees
    

    View Slide

12. Durability
    

    View Slide

13. Atomicity
    

    View Slide

14. Efficiency
    

    View Slide

15. –Murphy’s Law
    “Anything that can go wrong, will go wrong.”
    

    View Slide

16. Solutions
    

    View Slide

17. java.io.File
    https://developer.android.com/reference/java/io/File.html
    

    View Slide

18. AtomicFile
    https://developer.android.com/reference/android/util/AtomicFile.html
    https://developer.android.com/reference/android/support/v4/util/AtomicFile.html
    

    View Slide

19. if (backupFile.exists()) {
    file.delete();
    backupFile.renameTo(file);
    }
    

    View Slide

20. SharedPreferences
    https://developer.android.com/reference/android/content/SharedPreferences.html
    

    View Slide

21. SQLite
    https://developer.android.com/reference/android/database/sqlite/package-
    summary.html
    https://developer.android.com/training/basics/data-storage/databases.html
    

    View Slide

22. Tape
    https://github.com/square/tape
    

    View Slide

23. ✔ fast
    ✔ durable
    ✔ atomic
    

    View Slide

24. File System
    

    View Slide

25. renaming is atomic
    

    View Slide

26. File temp = new File("file.tmp");
    initialize(temp);
    File file = new File(“file.real");
    f.renameTo(file);
    

    View Slide

27. fsync is durable
    

    View Slide

28. segment writes are atomic
    

    View Slide

29. RandomAccessFile
    

    View Slide

30. class RandomAccessFile {
    void seek(long pos);
    read{}();
    void write{}(T data);
    }
    

    View Slide

31. new RandomAccessFile(file, "rwd");
    

    View Slide

32. -RWD
    Open for reading and writing, as with "rw", and also require that
    every update to the file's content be written synchronously to the
    underlying storage device.
    

    View Slide

33. Element 3
    Element 2
    Element 1
    File Header
    

    View Slide

34. View Slide

35. @jake
    16
    16
    1
    4096
    Last Element Position
    First Element Position
    Count
    File Size
    8 bytes
    8 bytes
    4 bytes
    63 bits
    

    View Slide

36. @jake
    16
    16
    1
    4096
    Last Element Position
    First Element Position
    Count
    File Size
    8 bytes
    8 bytes
    4 bytes
    63 bits
    

    View Slide

37. Version
    1
    1 bit
    Version
    0
    1 bit
    @jake
    

    View Slide

38. @1.2.3
    

    View Slide

39. @1.2.3
    

    View Slide

40. Adding Elements
    

    View Slide

41. queueFile.add(data);
    

    View Slide

42. int newLastPosition =
    lastPosition + 4 + lastLength;
    4096 1 32 32 Element 1
    10
    

    View Slide

43. raf.seek(newLastPosition);
    raf.write(data.length);
    4096 1 32 32 26
    Element 1
    10
    

    View Slide

44. raf.seek(newLastPosition + 4);
    raf.write(data);
    4096 1 32 32 Element 1
    10 Element 2
    26
    

    View Slide

45. raf.seek(0);
    writeHeader(fileLength,
    elementCount + 1,
    firstPosition,
    newLastPosition);
    4096 2 32 46 Element 1
    10 Element 2
    26
    

    View Slide

46. 4096 1 32 32 26
    Element 1
    10
    4096 1 32 32 Element 1
    10 Element 2
    26
    4096 1 32 32 Element 1
    10
    

    View Slide

47. Removing Elements
    

    View Slide

48. queueFile.remove();
    

    View Slide

49. int newFirstPosition = firstPosition +
    4 + firstLength;
    4096 2 32 46 Element 1
    10 Element 2
    26
    

    View Slide

50. writeHeader(fileLength,
    elementCount - 1,
    newFirstPosition,
    lastPosition);
    4096 1 46 46 Element 1
    10 Element 2
    26
    

    View Slide

51. raf.seek(oldFirstPosition);
    byte[] empty = new byte[firstLength + 4];
    raf.write(empty);
    4096 1 46 46 Element 2
    26
    

    View Slide

52. 4096 1 46 46 Element 1
    10 Element 2
    26
    4096 1 46 46 Element 2
    26
    

    View Slide

53. API
    @master
    

    View Slide

54. class QueueFile {
    void add(byte[] data);
    byte[] peek();
    void remove();
    int size();
    }
    @1.2.3
    

    View Slide

55. class QueueFile {
    void add(byte[] data);
    byte[] peek();
    void remove();
    int size();
    }
    @master
    

    View Slide

56. class QueueFile {
    void add(byte[] data);
    byte[] peek();
    void remove();
    int size();
    }
    @master
    

    View Slide

57. class QueueFile {
    void add(byte[] data);
    byte[] peek();
    void remove();
    int size();
    }
    @master
    

    View Slide

58. Usage
    

    View Slide

59. // Read and remove the data at the
    // head of the queue.
    byte[] element = queueFile.remove();
    // Do something with the data.
    process(element);
    

    View Slide

60. // Read and remove the data at the
    // head of the queue.
    byte[] element = queueFile.remove();
    // Do something with the data.
    process(element);
    

    View Slide

61. // Read data at the head.
    byte[] element = queueFile.peek();
    // Do something with the data.
    process(element);
    // Remove data at the head.
    queueFile.remove();
    

    View Slide

62. Example
    

    View Slide

63. byte[] message = queueFile.peek();
    Connection connection = open(“/v1/track");
    connection.getOutputStream().write(message);
    connection.close();
    queueFile.remove();
    

    View Slide

64. Handling HTTP Errors
    

    View Slide

65. retry network errors
    

    View Slide

66. retry server errors (5xx)
    

    View Slide

67. ignore client errors (4xx)
    

    View Slide

68. try {
    connection.close();
    } catch (IOException e) {
    return; // Network error, retry later.
    }
    int responseCode = connection.getResponseCode();
    if (responseCode >= 500) {
    return; // Server error, retry later.
    }
    // Client error, e.g. invalid JSON. Don’t retry.
    if (responseCode >= 400) {
    log("server rejected message");
    }
    queueFile.remove();
    

    View Slide

69. @master
    class QueueFile implements Iterable {
    void add(byte[] d, int offset, int count);
    boolean isEmpty();
    void remove(int n);
    void clear();
    }
    

    View Slide

70. class QueueFile implements Iterable {
    void add(byte[] d, int offset, int count);
    boolean isEmpty();
    void remove(int n);
    void clear();
    }
    @master
    

    View Slide

71. class QueueFile implements Iterable {
    void add(byte[] d, int offset, int count);
    boolean isEmpty();
    void remove(int n);
    void clear();
    }
    @master
    

    View Slide

72. class QueueFile implements Iterable {
    void add(byte[] d, int offset, int count);
    boolean isEmpty();
    void remove(int n);
    void clear();
    }
    @master
    

    View Slide

73. @master
    class QueueFile implements Iterable {
    void add(byte[] d, int offset, int count);
    boolean isEmpty();
    void remove(int n);
    void clear();
    }
    

    View Slide

74. Usage
    

    View Slide

75. Iterator iterator = queueFile.iterator();
    while (iterator.hasNext()) {
    byte[] data = iterator.next();
    process(data);
    iterator.remove();
    }
    

    View Slide

76. // Read the data at the head of the queue.
    List data = new ArrayList<>();
    for (byte[] d : queueFile) {
    data.add(d);
    }
    // Do something with the elements.
    process(data);
    // Remove elements from the queue.
    queueFile.remove(data.size());
    

    View Slide

77. Example
    

    View Slide

78. HttpURLConnection connection = open(“/v1/import");
    int count = 0;
    int size = 0;
    for (byte[] data : queueFile) {
    if (size + data.length > MAX_LIMIT) {
    break;
    }
    write(data, connection);
    count++;
    size += data.length;
    }
    connection.close();
    queueFile.remove(count);
    

    View Slide

79. ObjectQueue API
    @master
    

    View Slide

80. @master
    abstract class ObjectQueue
    implements Iterable {
    abstract void add(T entry);
    abstract T peek();
    List peek(int max);
    List asList();
    void remove();
    abstract void remove(int n);
    void clear();
    abstract int size();
    }
    

    View Slide

81. @master
    abstract class ObjectQueue
    implements Iterable {
    abstract void add(T entry);
    abstract T peek();
    List peek(int max);
    List asList();
    void remove();
    abstract void remove(int n);
    void clear();
    abstract int size();
    }
    

    View Slide

82. @master
    abstract class ObjectQueue
    implements Iterable {
    abstract void add(T entry);
    abstract T peek();
    List peek(int max);
    List asList();
    void remove();
    abstract void remove(int n);
    void clear();
    abstract int size();
    }
    

    View Slide

83. @1.2.3
    interface Converter {
    T from(byte[] bytes);
    void toStream(T o, OutputStream os);
    }
    

    View Slide

84. ObjectQueue
    Converter
    QueueFile
    

    View Slide

85. ObjectQueue queue = ObjectQueue.createInMemory();
    ObjectQueue queue = ObjectQueue.create(file, converter);
    

    View Slide

86. Usage
    

    View Slide

87. // Read data at the head.
    Pojo pojo = queue.peek();
    // Do something with the data.
    process(pojo);
    // Remove data at the head.
    queue.remove();
    

    View Slide

88. // Read the data at the head of the queue.
    List data = new ArrayList<>();
    for (Pojo pojo : queue) {
    data.add(pojo);
    }
    // Do something with the elements.
    process(data);
    // Remove elements from the queue.
    queue.remove(data.size());
    

    View Slide

89. Beyond Tape
    

    View Slide

90. GCM Network Manager
    https://developers.google.com/cloud-messaging/network-manager
    

    View Slide

91. Firebase JobDispatcher
    https://github.com/firebase/firebase-jobdispatcher-android
    

    View Slide

92. JobScheduler
    https://developer.android.com/reference/android/app/job/JobScheduler.html
    

    View Slide

93. Android Job
    http://evernote.github.io/android-job/
    

    View Slide

94. Priority JobQueue
    https://github.com/yigit/android-priority-jobqueue
    

    View Slide