Tweet
Tweet

Slide 1

Slide 1 text

Persistent Queues

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

-Murphy’s Law “Anything that can go wrong, will go wrong.”

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

Networks are flaky

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Batteries are limited

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Memory is constrained

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

RAM is volatile

Slide 16

Slide 16 text

Tasks must be durable

Slide 17

Slide 17 text

Non-Volatile Storage

Slide 18

Slide 18 text

Files java.io.File AtomicFile

Slide 19

Slide 19 text

java.io.File lacks atomicity

Slide 20

Slide 20 text

AtomicFile burdens caller with recovery

Slide 21

Slide 21 text

if (backupFile.exists()) { file.delete(); backupFile.renameTo(file); }

Slide 22

Slide 22 text

SharedPreferences loads entire queue into memory swallows errors

Slide 23

Slide 23 text

SQLite complex

Slide 24

Slide 24 text

Tape https://github.com/square/tape

Slide 25

Slide 25 text

QueueFile https://www.parleys.com/tutorial/ 5148922b0364bc17fc56c97b/chapter15

Slide 26

Slide 26 text

atomic durable efficient

Slide 27

Slide 27 text

What do we start with?

Slide 28

Slide 28 text

File System renaming is atomic fsync is durable segment writes are atomic

Slide 29

Slide 29 text

class RandomAccessFile { void seek(long pos); read{}(); void write{}(T data); }

Slide 30

Slide 30 text

new RandomAccessFile(file, "rwd");

Slide 31

Slide 31 text

“rwd” Open for reading and writing, as with "rw", and also require that every update to the file's content be written synchronously to the underlying storage device.

Slide 32

Slide 32 text

Putting it all together

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

Adding

Slide 37

Slide 37 text

queueFile.add(data); *not to scale

Slide 38

Slide 38 text

int next = lastPosition + 4 + lastLength;

Slide 39

Slide 39 text

raf.seek(next); raf.write(data.length);

Slide 40

Slide 40 text

raf.seek(next + 4); raf.write(data);

Slide 41

Slide 41 text

raf.seek(0); writeHeader(fileLength, elementCount + 1, firstPosition, newLastPosition);

Slide 42

Slide 42 text

= =

Slide 43

Slide 43 text

Removing

Slide 44

Slide 44 text

queueFile.remove();

Slide 45

Slide 45 text

int newFirstPosition = firstPosition + 4 + firstLength;

Slide 46

Slide 46 text

writeHeader(fileLength, elementCount - 1, newFirstPosition, lastPosition);

Slide 47

Slide 47 text

raf.seek(firstPosition); byte[] empty = new byte[firstLength + 4]; raf.write(empty);

Slide 48

Slide 48 text

=

Slide 49

Slide 49 text

API

Slide 50

Slide 50 text

class QueueFile { void add(byte[] data); byte[] peek(); void remove(); void clear(); int size(); void forEach(ElementReader reader); } @1.2.3

Slide 51

Slide 51 text

interface ElementReader { void read(InputStream in, int length); }

Slide 52

Slide 52 text

// Read the data at the head of the queue. byte[] element = queueFile.peek(); // Process it. process(element); // Remove if from the queue. queueFile.remove();

Slide 53

Slide 53 text

example byte[] message = queueFile.peek(); if (message != null) { return; } HttpURLConnection connection = open("https://segment.com/import"); connection.getOutputStream().write(message);

Slide 54

Slide 54 text

Handling Retries

Slide 55

Slide 55 text

retry server and network errors

Slide 56

Slide 56 text

ignore client errors

Slide 57

Slide 57 text

try { connection.close(); } catch (IOException e) { // Network error, retry later. return; } int responseCode = connection.getResponseCode(); if (responseCode >= 500) { // Server error, retry later. return; } // Client error, e.g. invalid json. Don’t retry. if (responseCode >= 400) { log("server rejected message"); } queueFile.remove(); example

Slide 58

Slide 58 text

Idempotent Tasks

Slide 59

Slide 59 text

UUID.randomUUID().toString()

Slide 60

Slide 60 text

Batch

Slide 61

Slide 61 text

example if (queueFile.size() < 0) { return; } HttpURLConnection connection = open("https://segment.com/import"); OutputStream out = connection.getOutputStream(); queueFile.forEach((in, length) -> { Utils.copy(in, out); });

Slide 62

Slide 62 text

example for (int i = 0; i < n; i++) { queueFile.remove(); }

Slide 63

Slide 63 text

Enhanced Batch API

Slide 64

Slide 64 text

class QueueFile { void remove(int n); int forEach(ElementVisitor visitor); } @master

Slide 65

Slide 65 text

interface ElementVisitor { boolean read(InputStream in, int length); }

Slide 66

Slide 66 text

interface ElementVisitor { boolean read(InputStream in, int length); }

Slide 67

Slide 67 text

List elements = read(queueFile); process(elements); queueFile.remove(elements.size());

Slide 68

Slide 68 text

example HttpURLConnection connection = open("https://segment.com/import"); OutputStream out = connection.getOutputStream(); *int count = 0; // simplified queueFile.forEach((in, length) -> { if (count + length > MAX_LIMIT) { return; } count += Utils.copy(in, out); return; });

Slide 69

Slide 69 text

example HttpURLConnection connection = open("https://segment.com/import"); OutputStream out = connection.getOutputStream(); *int count = 0; // simplified queueFile.forEach((in, length) -> { if (count + length > MAX_LIMIT) { return false; } count += Utils.copy(in, out); return true; });

Slide 70

Slide 70 text

example HttpURLConnection connection = open("https://segment.com/import"); OutputStream out = connection.getOutputStream(); *int count = 0; // simplified queueFile.forEach((in, length) -> { if (count + length > MAX_LIMIT) { return false; } count += Utils.copy(in, out); return true; });

Slide 71

Slide 71 text

Queue API

Slide 72

Slide 72 text

@1.2.3 interface ObjectQueue { void add(T entry); T peek(); void remove(); int size(); void setListener(Listener listener); }

Slide 73

Slide 73 text

interface Listener { void onAdd(ObjectQueue queue, T entry); void onRemove(ObjectQueue queue); }

Slide 74

Slide 74 text

@master class FileObjectQueue { List peek(int n); List asList(); void remove(int n); }

Slide 75

Slide 75 text

@1.2.3 interface Converter { T from(byte[] bytes); void toStream(T o, OutputStream os); }

Slide 76

Slide 76 text

FileObjectQueue Converter QueueFile

Slide 77

Slide 77 text

Beyond Tape

Slide 78

Slide 78 text

backoff priority *dependencies *battery, network

Slide 79

Slide 79 text

Priority JobQueue https://github.com/path/android-priority-jobqueue

Slide 80

Slide 80 text

JobScheduler https://developer.android.com/reference/android/app/job/ JobScheduler.html

Slide 81

Slide 81 text

GCM Network Manager https://developers.google.com/cloud-messaging/network- manager

Slide 82

Slide 82 text

Android Job https://github.com/evernote/android-job

Slide 83

Slide 83 text

Cassette* https://github.com/segmentio/cassette

Slide 84

Slide 84 text

Questions? @f2prateek f2prateek.com