Building a more Efficient Firestore Web App

Use Firestore in your Web App? Learn some tips and tricks to help you optimize Firestore billing.

Transcript

1. Building a More Efficient Firestore Web App Google Developer Expert in Firebase Lim Shang Yi Firebase Dev Day 2023 GDG Bangkok Firebase Thailand Organized by
2. Quick Recap: Cloud Firestore
3. • Store and sync data in real-time • NoSQL, document collections structure • Offline handling and auto scaling
6. Concerns
8. A common consideration With Firestore you are billed by the number of document reads and writes. So customers tend to be mindful of how often their data gets read from the server.
11. Billed Reads Estimation 5,000,000 document reads 10,000 monthly visitors 20 items on page 5 page refresh x x 5 pages loaded x
13. Choosing the right tech for future scale As an engineer, I want to choose a technology that will help accomplish my near term goals without too much effort. But that same technology must also support my future ambitions, in terms of scalability, cost, compliance, governance, etc.
15. I want to show data that doesn’t change often, such as the top 10 sales of the week. Which function should I use? getDocs onSnapshot
16. Reading Data from Firestore 1. getDocs() Call to get the data once. 2. onSnapshot() Set a listener to receive data-change events.
17. I want to show data that doesn’t change often, such as the top 10 sales of the week. Which function should I use? getDocs onSnapshot
18. Billed Reads Estimation (getDocs) 10,000 monthly visitors 10 items on page 2 navigation between pages 2 page refresh x x x 400,000 document reads
19. Billed Reads Estimation (onSnapshot) 10,000 monthly visitors 10 items on page 2 navigation between pages 2 page refresh x x x 200,000 document reads ( ↓200,000 reads)
20. Reading documents with getDocs App getDocs Firebase SDK Cloud Firestore
21. Reading documents with onSnapshot App onSnapshot Cloud Firestore Firebase SDK Cache onSnapshot Snapshot listener preserved Only fresh reads (such as a reload)
22. Flash Sale!! I want to display the 20 featured products that are on sale for today
24. Billed Reads Estimation with onSnapshot() 10,000 daily visitors 20 items on page 20 page refresh x x 4,000,000 document reads
25. Solution: Query the catalogue once a day, and every other time load from cache
26. Reading documents with getFromCache App get Cloud Firestore Firebase SDK Cache getFromCache Only fresh reads (such as a reload)
27. const db = getFirestore(); // remember to enable persistence enableIndexedDbPersistence(db); try { const querySnapshot = await getDocsFromCache(collection(db, "products")); let docs = []; if (querySnapshot.empty) { throw new Error("cache empty, refetch"); } querySnapshot.forEach((doc) => { docs.push(doc.data()); }); // do something } catch (err) { await getDocs(collection(db, "products")); // do something }
28. Billed Reads Estimation 10,000 daily visitors 20 items on page 20 1 page refresh x x 200,000 document reads (↓3,800,000 reads)
29. Problem: Still querying the server daily for each of the 10,000 visitors, even if content are the same
31. Solution: Query server once a day and share the results between all users using Bundles
32. What are Firestore Bundles? • Packaged results of common queries • Allow apps to load data quickly without querying backend • Save billed reads by loading from cache
33. Products Bundle: Build it Cloud Firestore Firebase Server SDK await getDocs db.bundle() .add(‘products’, docs).build() Bundle
34. var bundleId = "latest-products"; var bundle = firestore.bundle(bundleId); var querySnapshot = await firestore.collection('product').get(); // Build the bundle // Note how querySnapshot is named "latest-stories-query" var bundleBuffer = bundle .add('latest-products-query', querySnapshot) // Add a named query. .build()
35. Products Bundle: Distribute it Distribute bundle Bundle
36. Reading documents with bundles App Firebase Server SDK Remote storage Bundle Create bundle Fetch & read bundle Store
37. // Fetch the bundle from Firebase Hosting, if the CDN cache is hit the 'X-Cache' // response header will be set to 'HIT' const resp = await fetch('/createBundle'); // Load the bundle contents into the Firestore SDK await db.loadBundle(resp.body); // Query the results from the cache // Note: omitting "source: cache" will query the Firestore backend. const query = await db.namedQuery('latest-products-query'); const storiesSnap = await query.get({ source: 'cache' });
38. Billed Reads Estimation 10,000 daily visitors 20 items on page 1 page refresh x x 20 document reads (↓199,980)
39. Problem: Loading a large bundle increases cache size, slowing down searching of data
40. Solution: Use client-side indexing to speed up queries against cached documents
41. What is Client-Side Indexing? • Brings indexing to the Firestore Client SDK • Indexing always on server, new to client • Indexing enables very efficient queries
42. // You will normally read this from a file asset or cloud storage. let indexConfigJson = { indexes: [ ... ], fieldOverrides: [ ... ] } // Apply the configuration. Firestore.firestore().setIndexConfiguration(indexConfigJson)
43. To bundle or not to bundle?
44. When bundling makes sense: • Config data read in upon startup. • Data that doesn’t change frequently e.g top 10 stories every day. • Show standing data when user is not logged in
45. When bundling does not makes sense: • Data that changes very frequently • Any data that contains private information
46. Alternate solution for Firestore Bundles?
47. Consider using Realtime Database to store common or rarely changed data
48. Reading common or less frequently changed data App onSnapshot(‘cart’) Cloud Firestore onValue(‘promotions’) Realtime Database
49. Problem: Determine size of a collection
52. Getting the number of documents App getDocs(‘orders’) .size Firebase SDK Cloud Firestore
53. Billed Reads Estimation (Counting each order) 1,000,000 document reads 10,000 monthly visitors 1 listing x 100 orders x
54. Counting documents via Cloud Functions App Cloud Functions Firestore Trigger getDoc(‘orders’) Cloud Firestore updateDoc(‘orders’, { count : count +1 })
55. 10,000 document reads 100 document writes (update counter) 10,000 monthly visitors 1 post x 100 orders x Billed Reads Estimation (Counter)
56. Solution: Aggregate Queries
57. Counting documents via Aggregate Queries App getCountFromServer(‘orders’) Cloud Firestore
58. const coll = collection(db, "orders"); const snapshot = await getCountFromServer(coll); console.log('count: ', snapshot.data().count);
59. Billed Reads Estimation (Aggregate Queries) 11,000 document reads (10,000 listing read + 1000 reads from count) ( 1 document read / 1,000 indexes) 10,000 monthly visitors 1 listing x 100 orders x
60. In summary • Understand when to use getDocs vs onSnapshot • Use bundles to save billing reads • Take advantage of caches for different scenarios, client side indexing to speed it up • Consider using Realtime Database as alternative for standing data • Use aggregate queries instead of counting manually to reduce billing reads
61. Thank you Google Developer Expert in Firebase @shanggyilim Lim Shang Yi