Building a more Efficient Firestore Web App

Slide 1

Slide 1 text

Building a More Efficient Firestore Web App Google Developer Expert in Firebase Lim Shang Yi Firebase Dev Day 2023 GDG Bangkok Firebase Thailand Organized by

Slide 2

Slide 2 text

Quick Recap: Cloud Firestore

Slide 3

Slide 3 text

● Store and sync data in real-time ● NoSQL, document collections structure ● Offline handling and auto scaling

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

Concerns

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

A common consideration With Firestore you are billed by the number of document reads and writes. So customers tend to be mindful of how often their data gets read from the server.

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

Billed Reads Estimation 5,000,000 document reads 10,000 monthly visitors 20 items on page 5 page refresh x x 5 pages loaded x

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Choosing the right tech for future scale As an engineer, I want to choose a technology that will help accomplish my near term goals without too much effort. But that same technology must also support my future ambitions, in terms of scalability, cost, compliance, governance, etc.

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

I want to show data that doesn’t change often, such as the top 10 sales of the week. Which function should I use? getDocs onSnapshot

Slide 16

Slide 16 text

Reading Data from Firestore 1. getDocs() Call to get the data once. 2. onSnapshot() Set a listener to receive data-change events.

Slide 17

Slide 17 text

I want to show data that doesn’t change often, such as the top 10 sales of the week. Which function should I use? getDocs onSnapshot

Slide 18

Slide 18 text

Billed Reads Estimation (getDocs) 10,000 monthly visitors 10 items on page 2 navigation between pages 2 page refresh x x x 400,000 document reads

Slide 19

Slide 19 text

Billed Reads Estimation (onSnapshot) 10,000 monthly visitors 10 items on page 2 navigation between pages 2 page refresh x x x 200,000 document reads ( ↓200,000 reads)

Slide 20

Slide 20 text

Reading documents with getDocs App getDocs Firebase SDK Cloud Firestore

Slide 21

Slide 21 text

Reading documents with onSnapshot App onSnapshot Cloud Firestore Firebase SDK Cache onSnapshot Snapshot listener preserved Only fresh reads (such as a reload)

Slide 22

Slide 22 text

Flash Sale!! I want to display the 20 featured products that are on sale for today

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Billed Reads Estimation with onSnapshot() 10,000 daily visitors 20 items on page 20 page refresh x x 4,000,000 document reads

Slide 25

Slide 25 text

Solution: Query the catalogue once a day, and every other time load from cache

Slide 26

Slide 26 text

Reading documents with getFromCache App get Cloud Firestore Firebase SDK Cache getFromCache Only fresh reads (such as a reload)

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Billed Reads Estimation 10,000 daily visitors 20 items on page 20 1 page refresh x x 200,000 document reads (↓3,800,000 reads)

Slide 29

Slide 29 text

Problem: Still querying the server daily for each of the 10,000 visitors, even if content are the same

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

Solution: Query server once a day and share the results between all users using Bundles

Slide 32

Slide 32 text

What are Firestore Bundles? ● Packaged results of common queries ● Allow apps to load data quickly without querying backend ● Save billed reads by loading from cache

Slide 33

Slide 33 text

Products Bundle: Build it Cloud Firestore Firebase Server SDK await getDocs db.bundle() .add(‘products’, docs).build() Bundle

Slide 34

Slide 34 text

var bundleId = "latest-products"; var bundle = firestore.bundle(bundleId); var querySnapshot = await firestore.collection('product').get(); // Build the bundle // Note how querySnapshot is named "latest-stories-query" var bundleBuffer = bundle .add('latest-products-query', querySnapshot) // Add a named query. .build()

Slide 35

Slide 35 text

Products Bundle: Distribute it Distribute bundle Bundle

Slide 36

Slide 36 text

Reading documents with bundles App Firebase Server SDK Remote storage Bundle Create bundle Fetch & read bundle Store

Slide 37

Slide 37 text

// Fetch the bundle from Firebase Hosting, if the CDN cache is hit the 'X-Cache' // response header will be set to 'HIT' const resp = await fetch('/createBundle'); // Load the bundle contents into the Firestore SDK await db.loadBundle(resp.body); // Query the results from the cache // Note: omitting "source: cache" will query the Firestore backend. const query = await db.namedQuery('latest-products-query'); const storiesSnap = await query.get({ source: 'cache' });

Slide 38

Slide 38 text

Billed Reads Estimation 10,000 daily visitors 20 items on page 1 page refresh x x 20 document reads (↓199,980)

Slide 39

Slide 39 text

Problem: Loading a large bundle increases cache size, slowing down searching of data

Slide 40

Slide 40 text

Solution: Use client-side indexing to speed up queries against cached documents

Slide 41

Slide 41 text

What is Client-Side Indexing? ● Brings indexing to the Firestore Client SDK ● Indexing always on server, new to client ● Indexing enables very efficient queries

Slide 42

Slide 42 text

// You will normally read this from a file asset or cloud storage. let indexConfigJson = { indexes: [ ... ], fieldOverrides: [ ... ] } // Apply the configuration. Firestore.firestore().setIndexConfiguration(indexConfigJson)

Slide 43

Slide 43 text

To bundle or not to bundle?

Slide 44

Slide 44 text

When bundling makes sense: ● Config data read in upon startup. ● Data that doesn’t change frequently e.g top 10 stories every day. ● Show standing data when user is not logged in

Slide 45

Slide 45 text

When bundling does not makes sense: ● Data that changes very frequently ● Any data that contains private information

Slide 46

Slide 46 text

Alternate solution for Firestore Bundles?

Slide 47

Slide 47 text

Consider using Realtime Database to store common or rarely changed data

Slide 48

Slide 48 text

Reading common or less frequently changed data App onSnapshot(‘cart’) Cloud Firestore onValue(‘promotions’) Realtime Database

Slide 49

Slide 49 text

Problem: Determine size of a collection

Slide 50

Slide 50 text

No content

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

Getting the number of documents App getDocs(‘orders’) .size Firebase SDK Cloud Firestore

Slide 53

Slide 53 text

Billed Reads Estimation (Counting each order) 1,000,000 document reads 10,000 monthly visitors 1 listing x 100 orders x

Slide 54

Slide 54 text

Counting documents via Cloud Functions App Cloud Functions Firestore Trigger getDoc(‘orders’) Cloud Firestore updateDoc(‘orders’, { count : count +1 })

Slide 55

Slide 55 text

10,000 document reads 100 document writes (update counter) 10,000 monthly visitors 1 post x 100 orders x Billed Reads Estimation (Counter)

Slide 56

Slide 56 text

Solution: Aggregate Queries

Slide 57

Slide 57 text

Counting documents via Aggregate Queries App getCountFromServer(‘orders’) Cloud Firestore

Slide 58

Slide 58 text

const coll = collection(db, "orders"); const snapshot = await getCountFromServer(coll); console.log('count: ', snapshot.data().count);

Slide 59

Slide 59 text

Billed Reads Estimation (Aggregate Queries) 11,000 document reads (10,000 listing read + 1000 reads from count) ( 1 document read / 1,000 indexes) 10,000 monthly visitors 1 listing x 100 orders x

Slide 60

Slide 60 text

In summary ● Understand when to use getDocs vs onSnapshot ● Use bundles to save billing reads ● Take advantage of caches for different scenarios, client side indexing to speed it up ● Consider using Realtime Database as alternative for standing data ● Use aggregate queries instead of counting manually to reduce billing reads

Slide 61

Slide 61 text

Thank you Google Developer Expert in Firebase @shanggyilim Lim Shang Yi