How To Cache and Load Data Without Even Trying

Slide 1

Slide 1 text

Store Data Loading Made Easy-ish Brian Plummer - NY Times

Slide 2

Slide 2 text

We ❤ Making Apps

Slide 3

Slide 3 text

We ❤ Open Source

Slide 4

Slide 4 text

Why? Open Source Makes Life Easier

Slide 5

Slide 5 text

Networking is easier: HTTPURLCONNECTION Volley Retroﬁt / Okhttp

Slide 6

Slide 6 text

Storage is Easier Shared Preferences Firebase Realm SqlBrite/SqlDelight

Slide 7

Slide 7 text

Parsing is Easier JSONObject Jackson Gson Moshi

Slide 8

Slide 8 text

Fetching, Persisting & Parsing data has become easy

Slide 9

Slide 9 text

What's NOT Easy? DATA LOADING Everyone does it differently

Slide 10

Slide 10 text

What is Data Loading? The Act of getting data from an external system to a user's screen

Slide 11

Slide 11 text

Loading is complicated Rotation Handling is a special snowﬂake

Slide 12

Slide 12 text

New York Times built Store to simplify data loading github.com/NYTimes/store

Slide 13

Slide 13 text

OUR GOALS

Slide 14

Slide 14 text

Data should survive conﬁguration changes agnostic of where it comes from or how it is needed

Slide 15

Slide 15 text

Activities and presenters should stop retaining MBs of data

Slide 16

Slide 16 text

Ofﬂine as conﬁguration Caching as standard, not an exception

Slide 17

Slide 17 text

API should be simple enough for an intern to use, yet robust enough for every data load.

Slide 18

Slide 18 text

How do we work with Data at NY Times?

Slide 19

Slide 19 text

80% case: Need data, don't care if fresh or cached

Slide 20

Slide 20 text

Need fresh data background updates & pull to refresh

Slide 21

Slide 21 text

Requests need to be async & reactive

Slide 22

Slide 22 text

Data is dependent on each other, Data Loading should be too

Slide 23

Slide 23 text

Performance is important New calls should hook into in ﬂight responses

Slide 24

Slide 24 text

Parsing should be done efﬁciently and minimally Parse once and cache the result for future calls

Slide 25

Slide 25 text

Let's Use Repository Pattern By creating reactive and persistent Data Stores

Slide 26

Slide 26 text

Repository Pattern Separate the logic that retrieves the data and maps it to the [view] model from the [view] logic that acts on the model. The repository mediates between the data layer and the [view] layer of the application.

Slide 27

Slide 27 text

Why Repository? Maximize the amount of code that can be tested with automation by isolating the data layer

Slide 28

Slide 28 text

Why Repository? Data source from many locations will be centrally managed with consistent access rules and logic

Slide 29

Slide 29 text

Our Implementation https://github.com/NYTimes/Store

Slide 30

Slide 30 text

What is a Store? A class that manages the fetching, parsing, and storage of a speciﬁc data model

Slide 31

Slide 31 text

Tell a Store: What to fetch

Slide 32

Slide 32 text

Tell a Store: What to fetch Where to cache

Slide 33

Slide 33 text

Tell a Store: What to fetch Where to cache How to parse

Slide 34

Slide 34 text

Tell a Store: What to fetch Where to cache How to parse The Store handles ﬂow

Slide 35

Slide 35 text

Stores are Observable Observable get( V key); Observable fetch(V key) Observable stream() void clear(V key)

Slide 36

Slide 36 text

Let's see how stores helped us achieve our goals by loading a currywurst

Slide 37

Slide 37 text

Get is for Handling our 80% Use Case public ﬁnal class CurrywurstActivity { Store currywurstStore; void onCreate() { store.get("ketchup") .subscribe(currywurst -> show(currywurst)); } }

Slide 38

Slide 38 text

On Conﬁguration Change You only need to Save/Restore your key to get your cached data public ﬁnal class CurrywurstActivity { void onRecreate(String topping) { store.get(topping) .subscribe(currywurst -> show(currywurst)); } } Please do not serialize a Currywurst It would be very messy

Slide 39

Slide 39 text

What we gain?: Fragments/Presenters don't need to be retained No TransactionTooLargeExceptions Only keys need to be Parcelable

Slide 40

Slide 40 text

Efﬁciency is Important: for(i=0;i<20;i++){ store.get(topping) .subscribe(currywurst -> getView() .setData(currywurst)); } Many concurrent Get requests will still only hit network once

Slide 41

Slide 41 text

Fetching New Data public class CurrywurstPresenter { Store store; void onPTR() { store.fetch("ketchup") .subscribe(currywurst -> getView().setData(currywurst)); } }

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

Stream listens for events public class CurrywurstBar { Store store; void showCurrywurstDone() { store.stream() .subscribe(currywurst -> showSnackBar(currywurst)); } }

Slide 44

Slide 44 text

How do We build a Store? By implementing interfaces

Slide 45

Slide 45 text

Interfaces Fetcher{ Observable fetch(Key key); } Persister{} Observable read(Key key); Observable write(Key key, Raw raw); } Parser extends Func1{ Parsed call(Raw raw); }

Slide 46

Slide 46 text

Fetcher deﬁnes how a Store will get new data Fetcher currywurstFetcher = new Fetcher() { public Observable fetch(String topping) { return currywurstApi.order(topping); };

Slide 47

Slide 47 text

Becoming Observable Fetcher currywurstFetcher = topping -> Observable.fromCallable(() -> client.fetch(topping));

Slide 48

Slide 48 text

Parsers help with Fetchers that don't return View Models

Slide 49

Slide 49 text

Some Parsers Transform Parser boxParser = new Parser<>() { public CurrywurstBox call(Currywurst currywurst) { return new CurrywurstBox(currywurst); } };

Slide 50

Slide 50 text

Others read Streams Parser parser = source -> { InputStreamReader reader = new InputStreamReader(source.inputStream()); return gson.fromJson(reader, Currywurst.class); }

Slide 51

Slide 51 text

MiddleWare is for the common JSON cases Parser parser = GsonParserFactory.createSourceParser(gson,Currywurst.class) Parser parser = GsonParserFactory.createReaderParser(gson,Currywurst.class) Parser parser = GsonParserFactory.createStringParser(gson,Currywurst.class) 'com.nytimes.android:middleware:CurrentVersion' 'com.nytimes.android:middleware:jackson:CurrentVersion' 'com.nytimes.android:middleware-moshi:CurrentVersion'

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

Adding Ofﬂine with Persisters

Slide 54

Slide 54 text

File System Record Persister FileSystemRecordPersister.create( ﬁleSystem,key -> "currywurst"+key, 1, TimeUnit.DAYS);

Slide 55

Slide 55 text

File System is KISS storage interface FileSystem { BufferedSource read(String var) throws FileNotFoundException; void write(String path, BufferedSource source) throws IOException; void delete(String path) throws IOException; } compile 'com.nytimes.android:ﬁlesystem:CurrentVersion'

Slide 56

Slide 56 text

Don't like our persisters? No Problem! You can implement your own

Slide 57

Slide 57 text

Persister interfaces Persister { Maybe read(Key key); Single write(Key key, Raw raw); } Clearable { void clear(Key key); } RecordProvider { RecordState getRecordState( Key key);

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

We have our components! Let's build and open a store

Slide 60

Slide 60 text

MultiParser List parsers=new ArrayList<>(); parsers.add(GsonParserFactory.createSourceParser(gson, Currywurst.class)); parsers.add(boxParser);

Slide 61

Slide 61 text

Store Builder List parsers=new ArrayList<>(); parsers.add(GsonParserFactory.createSourceParser(gson, Currywurst.class)); parsers.add(boxParser); StoreBuilder parsedWithKey()

Slide 62

Slide 62 text

Add Our Parser List parsers=new ArrayList<>(); parsers.add(GsonParserFactory.createSourceParser(gson, Currywurst.class)); parsers.add(boxParser); StoreBuilder parsedWithKey() .fetcher(topping -> currywurstSource.fetch(topping))

Slide 63

Slide 63 text

Now Our Persister List parsers=new ArrayList<>(); parsers.add(GsonParserFactory.createSourceParser(gson, Currywurst.class)); parsers.add(boxParser); StoreBuilder parsedWithKey() .fetcher(topping -> currywurstSource.fetch(topping)) .persister( FileSystemRecordPersister.create( ﬁleSystem, key -> "preﬁx"+key, 1, TimeUnit.DAYS))

Slide 64

Slide 64 text

And Our Parsers List parsers=new ArrayList<>(); parsers.add(GsonParserFactory.createSourceParser(gson, Currywurst.class)); parsers.add(boxParser); StoreBuilder parsedWithKey() .fetcher(topping -> currywurstSource.fetch(topping)) .persister( FileSystemRecordPersister.create( ﬁleSystem, key -> "preﬁx"+key, 1, TimeUnit.DAYS)) .parsers(parsers)

Slide 65

Slide 65 text

Time to Open a Store List parsers=new ArrayList<>(); parsers.add(GsonParserFactory.createSourceParser(gson, Currywurst.class)); parsers.add(boxParser); Store currywurstStore = StoreBuilder parsedWithKey() .fetcher(topping -> currywurstSource.fetch(topping)) .persister( FileSystemRecordPersister.create( ﬁleSystem, key -> "preﬁx"+key, 1, TimeUnit.DAYS)) .parsers(parsers) .open();

Slide 66

Slide 66 text

Conﬁguring caches

Slide 67

Slide 67 text

Conﬁguring Memory Policy StoreBuilder .parsedWithKey() .fetcher(topping -> currywurstSource.fetch(topping)) .memoryPolicy(MemoryPolicy .builder() .setMemorySize(10) .setExpireAfter(24) .setExpireAfterTimeUnit(HOURS) .build()) .open()

Slide 68

Slide 68 text

Refresh On Stale - BackFilling the Cache Store currywurstStore = StoreBuilder .parsedWithKey() .fetcher(topping -> currywurstSource.fetch(topping)) .parsers(parsers) .persister(persister) .refreshOnStale() .open();

Slide 69

Slide 69 text

Network Before Stale Policy Store currywurstStore = StoreBuilder .parsedWithKey() .fetcher(topping -> currywurstSource.fetch(topping)) .parsers(parsers) .persister(persister) .networkBeforeStale() .open();

Slide 70

Slide 70 text

Stores in the wild

Slide 71

Slide 71 text

Intern Project: Best Sellers List public interface Api { @GET Observable getBooks(@Path("category") String category);

Slide 72

Slide 72 text

Store provideBooks(FileSystem ﬁleSystem, Gson gson, Api api) { return StoreBuilder. parsedWithKey() .fetcher(category -> api.getBooks(category)) .persister(create(ﬁleSystem)) .parser(createSourceParser(gson, Books.class)) .open(); }

Slide 73

Slide 73 text

public class BookActivity{ ... onCreate(...){ bookStore .get(category) .subscribe(); }

Slide 74

Slide 74 text

How do books get updated? public class BackgroundUpdater{ ... bookStore .fetch(category) .subscribe(); }

Slide 75

Slide 75 text

Data Available when screen needs it UI calls get, background services call fresh

Slide 76

Slide 76 text

LiveData, no problem

Slide 77

Slide 77 text

public class WurstLiveData extends WurstLiveData { protected void onActive() { curryworstStore .get("ketchup") .subscribe(new SingleObserver() { ...... @Override public void onSuccess(Currywurst currywurst) { setValue(currywurst); } ...... });

Slide 78

Slide 78 text

LiveData getCurrywurst(String topping) { MutableLiveData data = new MutableLiveData<>(); curryworstStore .get(topping) .subscribe(new SingleObserver() { ...... @Override public void onSuccess(Currywurst currywurst) { data.setValue(currywurst); } ...... });

Slide 79

Slide 79 text

Dependent Calls

Slide 80

Slide 80 text

Map one Store get to another public Observable getSectionFront(String name) { return feedStore.get() .map(feed -> feed.getSectionMeta(name)) .ﬂatMap(meta -> sfStore.get(meta))); }

Slide 81

Slide 81 text

Relationships by overriding Get/Fetch

Slide 82

Slide 82 text

Video Store Returns single videos Playlist Store returns playlist How do we get a playlist with all videos?

Slide 83

Slide 83 text

Step 1: Create Video Store Store provideVideoStore(VideoFetcher fetcher, Gson gson) { return StoreBuilder.parsedWithKey() .fetcher(fetcher) .parser(createSourceParser(gson, Video.class)) .open(); }

Slide 84

Slide 84 text

Step 2: Create Playlist Store public class VideoPlaylistStore extends RealStore { ﬁnal Store videoStore; public VideoPlaylistStore(@NonNull PlaylistFetcher fetcher, @NonNull Store videoStore, @NonNull Gson gson) { super(fetcher, NoopPersister.create(), createSourceParser(gson, Playlist.class)); this.videoStore = videoStore; }

Slide 85

Slide 85 text

Step 3: Override store.get public class VideoPlaylistStore extends RealStore { ﬁnal Store videoStore; @Override public Observable get(Long playlistId) { return super.get(playlistId) .ﬂatMap(playlist -> from(playlist.videos()) .concatMap(video -> videoStore.get(video.id())) .toList() .map(videos -> builder().from(playlist) .videos(videos) .build())); }

Slide 86

Slide 86 text

Listening for changes

Slide 87

Slide 87 text

Step 1: Subscribe to Store, Filter what you need sectionFrontStore.stream() .observeOn(scheduler) .subscribeOn(Schedulers.io()) .filter(this::sectionIsInView) .subscribe(this::handleSectionChange); Step 2: Kick off a fresh request to that store public Observable refreshSections(final String sectionName) { return feedStore.get() .flatMapIterable(this::sectionsToUpdate) .map(section->sectionFrontStore.fetch(section))); }

Slide 88

Slide 88 text

Store3 features

Slide 89

Slide 89 text

store.getRefreshing(key) - will subscribe to get() which returns a single response, but unlike Get, Get Refreshing will stay subscribed. Anytime you call store.clear(key) anyone subscribed to getRefreshing(key) will resubscribe and force a new network response.

Slide 90

Slide 90 text

store.getWithResult(key) - a Result model which returns the parsed object accompanied by its source(network/cache). allows you to distinguish if the cache is from memory or disk

Slide 91

Slide 91 text

Yes!! We do kotlin as well! FluentStoreBuilder.barcode(myFetcher) { persister = myPersister memoryPolicy = myMemoryPolicy stalePolicy = myStalePolicy } FluentStoreBuilder.key().fetcher(myFetcher) { persister = myPersister memoryPolicy = myMemoryPolicy stalePolicy = myStalePolicy } FluentStoreBuilder.parsedWithKey(myFetcher) { persister = myPersister memoryPolicy = myMemoryPolicy stalePolicy = myStalePolicy parser = myParser parsers = myParsers }

Slide 92

Slide 92 text

Would love contributions & feedback! thanks for listening